Patent Application
20110202584
1. Field
The present system is related to the compression of integer data.
2. Description of the Related Art
As the Internet's popularity grows, more businesses are establishing a presence on the Internet. These businesses typically set up web sites that run one or more web applications. One disadvantage of doing business on the Internet is that if the web site goes down, becomes unresponsive or otherwise is not properly serving customers, the business is losing potential sales and/or customers. Similar issues exist with Intranets and Extranets. Thus, service oriented architecture (SOA) application management solutions have been developed which proactively detect and diagnose performance problems for web applications and web sites to ensure that they run properly. One such application management system is Introscope® by CA, Inc., Wily Technology Division, South San Francisco, Calif.
Applications such as Introscope® acquire and store a tremendous amount of data for their operation, and rely on data compression techniques to minimize storage space and transmission bandwidth. In general, data compression schemes involve an encoding engine which encodes data prior to storage or transmission, and a decoding engine which is capable of performing the reverse operation to decode the data upon receipt or retrieval from storage. One type of compression technique is a lossy data compression, where high compression rates are achieved, but at the sacrifice of some resolution of the original encoded data upon decoding. Lossy data compression techniques are often used to transmit multi-media data requiring high bandwidth transfer rates and where loss of some fidelity in the data will not be perceptible in the decoded media.
Other situations require lossless data compression where the decoded data is a full reconstruction of the encoded data. One well known technique for lossless compression of positive integers is called the Elias Delta coding algorithm, developed by M.I.T. professor Peter Elias in the 1970's. The conventional Elias Delta scheme for coding a natural number xε={1, 2, 3, . . . } will now be explained with reference to the prior art flowchart of
13base 10=1101base 2
In step 44, the number of digits, N, in the binary value of step 42 is determined.
1101 has N=4 digits
N may be determined mathematically by noting N−1 is the highest whole number power of 2 contained in the original integer value x. Stated another way, N−1=the largest integer contained in Log2 x. This is commonly written N−1=Log2 x, and is referred to with this notation herein.
N−1=Log2 13→N−1=3; N=4.
Another way of describing N is that N is the number of digits in the binary representation of x, and N−1 is the number of digits in the binary representation of x with the most significant bit (MSB) removed.
In step 46, N is converted to a binary number:
4base 10=100base 2
And in step 48, the number of digits, N′, in the binary conversion of N is determined.
100 has N′=3 digits; N′−1=2.
Again, N′ may be determined mathematically by noting N′−1 is the highest whole number power of 2 contained in N.
N′−1=Log2 4→N′−1=2, N′=3.
The Elias Delta code Eδ(x) for the original integer x is then obtained from three sub parts:
Eδ(x)1=N′−1 number of zeros (step 48);
Eδ(x)2=binary representation of N (step 42); and
Eδ(x)3=The remaining N−1 digits of the binary representation of x (i.e., the binary representation of x with the MSB removed).
Thus, Eδ(13)=
E
δ(13)1=00
E
δ(13)2=100
E
δ(13)3=1101=101
E
δ(13)=00100101.
The prior art in
Once the Elias Delta code value for an integer is obtained, it may be output in step 50. The output may be from a processor to a persistent storage media, or the output may be from the processor to a communications interface for transmission to another computing device. The Elias Delta code value for an integer may be stored on a persistent storage media or transmitted via a network connection as a single value. Alternatively, as shown in prior art
One problem with the Elias Delta coding scheme is that it must work for any input integer value and it must therefore reserve code space for any value. As can be seen for example in
The present invention, roughly described, pertains to a system where integer values to be transmitted and/or stored may be compressed with a higher compression ratio as compared to traditional compression techniques such as Elias Delta coding. In particular, by noting that a data system may store and/or transmit integer values over a predefined gamut having not only a minimum but a maximum limit, integer values at or near the maximum may be compressed to a greater degree than in conventional systems without any loss of data resolution. The present technology has parallels to Elias Delta coding. However, where it is determined that the number of digits in the binary integer to be stored or transmitted is equal to a predefined maximum number of digits for a given data system, the storage and/or transmission of the binary representation of log(2) of the integer value may be left out of the storage or transmission, thereby resulting in a greater compression ratio as compared to conventional Elias Delta coding.
The compression scheme of the present technology may have broad application in a wide variety of data systems. However, in one application, the present technology is used to compress data associated with application management systems which proactively detect and diagnose performance problems for web applications and web sites to ensure that they run properly.
The present technology can be accomplished using hardware, software, or a combination of both hardware and software. The software used for the present invention is stored on one or more processor readable storage media including hard disk drives, CD-ROMs, DVDs, optical disks, floppy disks, tape drives, RAM, ROM or other suitable storage devices. In alternative embodiments, some or all of the software can be replaced by dedicated hardware including custom integrated circuits, gate arrays, FPGAs, PLDs, and special purpose computers. In one embodiment, software implementing the present invention is used to program one or more processors. The processors can be in communication with one or more storage devices, peripherals and/or communication interfaces.
The present technology will now be described with respect to
Referring now to
With regard to the maximum integer value, many data systems and software applications work with a range of data values that are naturally limited by a system constraint. For example, a data system may use a register, address bus or data bus able to work with numbers of a set, predetermined size, such as for example 4 bits, so as to receive 16 possible integer values, e.g., between 0 and 15. As a further example, many application programs transmit and store data having a size and format that is set by industry standard. As one of many examples, data describing the color of a pixel in a bit mapped image or video frame buffer is often transmitted and stored as one of 256 values in an 8 bit format. Other telecommunications, audio and video data are stored in standard 8 bit, 16 bit, etc. data lengths.
Embodiments of the high order compression scheme assume that transmitted and stored integer values range over a gamut having a known maximum. The knowledge of the maximum is used by an encoding engine and a decoding engine implementing the high order compression scheme as explained below to improve the compression ratio of the compressed data. In embodiments, the present system operates with a maximum number of integer values equal to 2i, where i is a power of two (i=2, 4, 8, 16, 32, 64 . . . ). However, alternative embodiments of the present system may operate to compress and decode data having other integer value ranges in further embodiments.
An embodiment of the present system will now be described with reference to the flowchart of
The high order compression scheme described below may be performed by an encoding engine 230 (
x=14base(10)=1110base(2).
In step 104, the encoding engine 230 determines the number of bits, N, in the binary value of the integer. As described in the Background section, this may be determined by N−1=log2 x:
N−1=log2 14→N−1=3, N=4.
Those of skill in the art may understand alternative or additional methods of determining the number of digits in the binary representation of the integer value.
In step 106, the encoding engine 230 determines whether N is equal to the maximum log2 of the known gamut. In this example, the maximum value of N is 4. As N equals 4 for the integer x=14, the encoding engine would set a flag in step 108 indicating that N for the received integer value is at the predetermined maximum value. This is shown in
N′−1=log2 N
N′−1=log2 4→N′−1=2, N′=3.
Again, those of skill in the art may understand alternative or additional methods of determining the number of digits in the binary representation of N.
After step 114, the encoding engine may determine the high order compression value, denoted herein as M(x). If the integer value x has a number of bits, N, less than the predetermined maximum value (i.e., CHK not equal to max in step 118), then the high order compression value M(x) is determined in step 120 in the same way as the Elias Delta value. That is, the high order compression value M(x)=M(x)1 appended to M(x)2 appended to M(x)3, where:
M(x)1=N′−1 number of zeros (step 114);
M(x)2=the number of bits, N, in the binary representation of x (step 104); and
M(x)3=the remaining N−1 digits of the binary representation of x (i.e., the binary representation of x with the MSB removed).
However, where the integer value x takes up the maximum number of bits over the range of predetermined integer values in step 118, the present system allows a reduction in the size of the encoded value over conventional compression schemes such as the Elias Delta coding scheme. In step 124, if the flag is equal to the maximum value, then the high order compression value M(x)=M(x)1 appended to M(x)3, where M(x)1 and M(x)3 are as described above:
M(x)1=N′−1 number of zeros (step 114); and
M(x)3=The remaining N−1 digits of the binary representation of x (i.e., the binary representation of x with the MSB removed).
As it is known that the received integer value is at the maximum number of digits in its binary representation for the predetermined range, there is no need to reserve space in the encoded value representing the most significant bit (i.e., M(x)2).
In the current example where x=14, the number of bits in the binary representation (i.e., 4) is equal to the predetermined maximum. Thus:
M(14)=M(14)1 appended to M(14)3;
M(14)1=N′−1 number of zeros, or two zeros→M(14)1=00;
M(14)3=The remaining N−1 digits of the binary representation of x with the MSB removed. The binary representation of x=14 is 1110→M(14)3=110.
M(14)=M(14)1 appended to M(14)3;
M(14)=00110.
In step 130, the high order compression value M(x) is output. The output may be from a processor to a persistent storage media, or the output may be from the processor to a communications interface for transmission to another computing device. The high order compression value for an integer may be stored on a persistent storage media, or transmitted via a network connection, as a single value. Alternatively, as shown in
It is understood that the sub-components may alternatively be transmitted or stored in a LIFO data structure. In this case, the sub-components would be transmitted or stored in the reverse order from that described above.
Moreover, it is understood that the steps discussed above in
When an encoded value is either received upon transmission or retrieved from persistent storage, the encoded value must be decoded. The present system further includes a decoding engine 232 (
The decoding engine 232 may receive the entire high order compression value M(x) to work with, or the decoding engine may receive the first sub-component M(x)1 from a FIFO or LIFO data retrieval structure. In either case, in step 140, the decoding engine receives the first sub-component M(x)1, which is the number of leading zeros in the received or stored value M(x). As above, the number of leading zeros may be referred to here as N′−1.
For M(x)=00110, there are two leading zeros→N′−1=2, and N′=3.
In the high order compression values, the number of leading zeros, N′−1, will indicate the most significant bit, N, of the decoded binary value by the relationship:
N′−1=Log2 N
In a system where the integer values are constrained between 0 and 15, no leading zeros will indicate an integer value between 0 and 1, one leading zero will indicate an integer value between 2 and 7, and two (or more) leading zeros will indicate an integer value between 8 and 15.
In the above example, the integer values are constrained between 0 and 15 (N=a maximum of 4). However, as indicated above, the present invention may operate with any known gamut. In embodiments, the maximum number of integer values in a known gamut will be equal to 2i, where i is a power of two (i=2, 4, 8, 16, 32, 64 . . . ). Thus, in embodiments, the high order compression scheme may work with integer gamuts of 2 values, 4 values, 16 values (as in the above example), 256 values, 65,536 values, etc.
It can be seen that whenever the number of leading zeros in the high order compression value is equal to or greater than the log2 of the number of binary digits in the maximum integer value for the data system, the high order compression value represents an integer having the maximum number of binary digits for the data system. Thus:
for a gamut between 0 and 3, one leading zero in a given high order compression value indicates that the given high order compression value represents an integer having the maximum number of N=2 bits in the binary representation;
for a gamut between 0 and 15 (as above), two leading zeros in a given high order compression value indicates that the given high order compression value represents an integer having the maximum number of N=4 bits in the binary representation;
for a gamut between 0 and 255, three leading zeros in a given high order compression value indicates that the given high order compression value represents an integer having the maximum number of N=8 bits in the binary representation;
for a gamut between 0 and 65,535, four leading zeros in a given high order compression value indicates that the given high order compression value represents an integer having the maximum number of N=16 bits in the binary representation; etc.
Using this scheme and the known gamut, the decoding engine 232 is able to determine when a high order compression value represents a number at the maximum number of binary digits solely from the number of leading zeros in the high order compression value.
Continuing again with the example of
In step 152, a first intermediate decimal value J is defined as being equal to 2N-1. J represents the decimal representation of the most significant bit. In the example where M(x)=00110, N=4. Thus:
J=2N-1=23=8.
In step 158, a second intermediate decimal value K is defined as being equal to the base 10 conversion of the received/stored value M(x) with the most significant bit removed. This will be the last N−1 digits of M(x). In the above example where M(x)=00110, it was determined that N=4. Thus, N−1=3 and the last 3 digits of M(x) are 110. Thus, the base 10 conversion, K, of the N−1 digits of M(x) in this example is given by:
110base 2=Kbase 10→K=6.
Finally, in step 160, the original integer value, x, is determined by adding the intermediate values J and K together:
x=J+K
x=8+6=14.
As seen with respect to
In the above description, it was noted that where the leading zeros indicated a decoded binary value having the maximum number of bits, steps 148 and 150 may be skipped, thus resulting in a reduction of processing time. However, where it is determined that the number of digits in the binary representation is less than the maximum, steps 148 and 150 are performed. For example, assume the decoding engine 232 receives a high order compression value M(x)=01101 in a system where the maximum number of bits is still 4 (16 possible integer values in the gamut). In this example, there is one leading zero read in step 140 and N′−1 is not greater than or equal to 2. Accordingly, the decoding engine will perform step 148 and read the N′ number of bits in M(x) following the leading zeros. In the example of decoding 01101, there is one leading zero, so N′−1=1, and N′=2. Thus, the 2 bits following the leading zero in M(x) are read. In the example of decoding M(x)=01101, these digits are “11.” In step 150, N equals a base ten conversion of the read N′ number of bits. In the current example:
11base 2=3base 10→N=3.
Step 152 is then performed as described above. The first intermediate value J is determined where N=3
J=2N-1=22=4.
In step 158, the second intermediate value K is determined. As indicated above, K is the base 10 conversion of the remaining portion of the received/stored value M(x) with the most significant bit removed, i.e., the last N−1 digits of M(x). In the above example where M(x)=01101, it was determined that N=3. Thus, N−1=2 and the last 2 digits of M(x) are “01”. Thus, the base 10 conversion, K, of the N−1 digits of M(x) in this example is given by:
01base 2=Kbase 10→K=1.
Finally, in step 160, the original integer value, x, is determined by adding the intermediate values J and K together:
x=J+K
x=4+1=5.
It is understood that the steps discussed above in
As described above, in embodiments, any predefined gamut of integer values will have a maximum number of binary digits N that satisfy N=2i, where i is a power of 2. This constraint allows the decoder to automatically determine when an encoded high order compression value represents a high order integer value in the predefined gamut. Were it not a constraint on the system that i is a power of 2, the decoding of the high order compression values would be more complicated in that assumptions regarding N could not be made based on the leading number of zeros. For example, in a system having 16 possible integers where N is constrained to be 2i, it is known that, where there are 2 leading zeros in M(x), N′−1=2 and N′ is a 3 bit binary number equal to 100base 2, or N=4base 10.
However, in a system having 16 possible integers where N is not constrained to be 2i, it cannot be assumed that N=4 when there are 2 leading zeros. For example, where there are 2 leading zeros in M(x), N′−1=2 and thus N′ is a 3 bit binary number which could be 100 (N=4), 101 (N=5), 110 (N=6) or 111 (N=7).
However, although more complicated, embodiments of the high order compression scheme may work with values of N that are not constrained to be 2i, where i is a power of 2. This is so because the M(x) encoding of such values will still be unique for each possible value of N. In such embodiments, the leading zeros alone in the encoded value M(x) would not dictate the value of N, but additional steps may be performed in examining further digits of M(x) to allow M(x) to be decoded into its original integer value.
In embodiments, the high order compression scheme may result in efficiencies in coding the high order integer values, but not for integers below the high order integer values. In further embodiments, this issue may be addressed by employing a known regularization scheme to add and/or subtract some constant to integer values in a gamut prior to encoding by the high order compression scheme. The constant will depend on the size of the gamut. It is noted that in the example in
For example, the advantage of a regularized high order compression scheme is evident in
As indicated above, embodiments of the high order compression scheme may be used in a wide variety of situations where integer data is stored and/or transmitted. One embodiment where the high order compression scheme may be used is explained below with reference to the block diagram of
In one embodiment, a user specifies a threshold trace period and can initiate transaction tracing on one, some or all transactions running on a software system. Transactions with an execution time that exceeds the threshold trace period are reported to the user using a graphical user interface. The graphical user interface can include visualizations for reported transactions that enable the user to immediately understand where time was spent in the traced transaction.
The components 200, 220, 222, 224 and 226 shown in
With reference to
Computer 310 may include a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 310 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), EEPROM, flash memory or other memory technology, CD-ROMs, digital versatile discs (DVDs) or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disc storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 310. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.
The system memory 330 includes computer storage media in the form of volatile and/or nonvolatile memory such as ROM 331 and RAM 332. A basic input/output system (BIOS) 333, containing the basic routines that help to transfer information between elements within computer 310, such as during start-up, is typically stored in ROM 331. RAM 332 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 320. By way of example, and not limitation,
The computer 310 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, DVDs, digital video tapes, solid state RAM, solid state ROM, and the like. The hard disc drive 341 is typically connected to the system bus 321 through a non-removable memory interface such as interface 340, magnetic disc drive 351 and optical media reading device 355 are typically connected to the system bus 321 by a removable memory interface, such as interface 350.
The drives and their associated computer storage media discussed above and illustrated in
The computer 310 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 380. The remote computer 380 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 310, although only a memory storage device 381 has been illustrated in
When used in a LAN networking environment, the computer 310 is connected to the LAN 371 through a network interface or adapter 370. When used in a WAN networking environment, the computer 310 typically includes a modem 372 or other means for establishing communication over the WAN 373, such as the Internet. The modem 372, which may be internal or external, may be connected to the system bus 321 via the user input interface 360, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 310, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
In some embodiments, all or part of the transaction management system may be implemented in software that is stored on one or more processor readable storage devices and is used to program one or more processors. Referring again to
In general, the encoding engine 232 is capable of receiving integer values from a storage media on one of components 200, 220, 222, 224 and 226, encoding those values, and then storing the encoded values on a storage media of one or more of components 200, 220, 222, 224 and 226. In embodiments, the encoding engine 232 outputs encoded values, and new files and/or data structures are then generated to store those values. Similarly, the decoding engine is capable of receiving encoded values from a storage media on one of components 200, 220, 222, 224 and 226, decoding those values, and then outputting (including possibly displaying) the decoded values to one or more of components 200, 220, 222, 224 and 226. In embodiments, the decoding engine 232 outputs integer values, and new files and/or data structures are then generated to output and/or store those values on storage media.
In one embodiment, a user of the system in
The foregoing detailed description of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments were chosen in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto.
