Patent Application
20170048303
Field
The present disclosure relates generally to data systems, and more particularly, to techniques for accelerating the transfer of data across a network.
Background
Current file transfer technologies include the file transfer protocol (FTP). FTP is a protocol that enables a user to retrieve files from a remote location over a TCP/IP network. The user runs an FTP client application on a local computer, and an FTP server program resides on a remote computer. The user logs into the remote FTP server using a login name and password, which the server then authenticates. File transfers may also be conducted using hypertext transfer protocol (HTTP) in the form of a file download.
File transfers using FTP, HTTP and similar technologies are often inefficient. In the case where a large file is periodically modified or updated, a user will often repeatedly download over time the modified versions of the file. This is true even where the changes to the file are relatively small in comparison to the file size. The repetitive nature of downloads for such files changing over time often results in significant chunks of the same data being needlessly transferred across a network, producing latencies and clogging network bandwidth. The potentially deleterious effect of these repeated transfers often becomes more pronounced when the same embedded images or graphics persist in the file modified over time, or when large blocks of text in modified versions contain relatively insubstantial modifications.
Where many users are involved, this phenomenon can create a bottleneck. The result is that much of the same data ends up being repeatedly transferred over the network. Even where conventional compression techniques on individual files are used, the core problem of sending redundant chunks of data via the transmission of multiple versions of the same file is largely unaddressed. As the number of users, transfers and file modifications increase, the available bandwidth is taxed, resulting in network inefficiencies.
These and other limitations are addressed in the present disclosure.
In an aspect of the disclosure, a method, a computer program product, and an apparatus for transferring data are provided. The apparatus includes a memory having a first file stored therein, and a processor coupled to the memory and configured to create, in response to receiving a second file, a first fingerprint map (FM) corresponding to the first file, and generate, using the second file and the first FM, data representing a difference between the first file and the second file, wherein the data and the first file are sufficient to generate the second file.
In another aspect of the disclosure, the apparatus includes a memory having a first file stored therein, a transceiver, and a processor configured to transfer the first file via the transceiver over a network to a destination node, receive a second file, determine whether the second file comprises a modification to the first file, receive a fingerprint map (FM) corresponding to the first file, generate, using the second file and the FM, data representing a difference between the second file and the first file, and transfer the data via the transceiver to the destination node, wherein the data and the first file are sufficient to generate the second file.
In another aspect of the disclosure, the computer program product including a computer-readable medium has computer executable code for creating, in response to receiving a second file, a first fingerprint map (FM) corresponding to a first file, and generating, using the second file and the first FM, data representing a difference between the first file and the second file, wherein the data and the first file are sufficient to generate the second file.
In another aspect of the disclosure, the computer program product including a non-transitory computer-readable medium has computer executable code for transferring a first file via a transceiver over a network to a destination node, receive a second file, determine whether the second file comprises a modification to the first file, receive a fingerprint map (FM) corresponding to the first file, and generate, using the second file and the FM, data representing a difference between the second file and the first file, wherein the data and the first file are sufficient to generate the second file.
Additional advantages and novel features will be set forth in part in the description that follows, and in part will become more apparent to those skilled in the art upon examination of the following or upon learning by practice thereof.
The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations and is not intended to represent the configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well known structures and components are shown in block diagram form in order to avoid obscuring such concepts.
Several aspects of systems for data transfer will now be presented with reference to various apparatus and methods. These apparatus and methods will be described in the following detailed description and illustrated in the accompanying drawings by various blocks, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as “elements”). These elements may be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system.
By way of example, an element, or any portion of an element, or any combination of elements may be implemented with a “processing system” that includes one or more processors. Examples of processors include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. One or more processors in the processing system may execute software. Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
Accordingly, in one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, combinations of the aforementioned types of computer-readable media, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.
Block 102 represents a source node, such as the apparatus 800 for transferring files in
As illustrated in
In the above illustration, there is an original version of F1 (104) and a copy of F1 (114) where the copy was either transferred directly from source node 102, was reproduced from MAP1 and other data or was made available by some other means.
The technique of
Using an appropriate algorithm as illustrated below, large files may be represented as one or more maps of fingerprints where the map is comparatively smaller than the original file, where the fingerprint map may be used to identify similarities within files.
A file in general consists of any number of bytes that can be represented by values in the range {00h . . . FFh}. If a subset of byte patterns can be identified whose probability of random occurrence is very small and it is ensured that the patterns occur once, such patterns may be used as fingerprints of the file. A fingerprint is similar to a checksum in this regard, except that it is not computed but created from the file itself.
Four criteria for a statistically qualifying fingerprint include (i) minimal frequency of occurrence of the fingerprint within the data stream, (ii) minimal frequency of occurrence of the fingerprint with the fingerprint map, (iii) a high entropy (i.e., highly random bits), and (iv) low collision probability. In the context of the 128-bit fingerprinting algorithm discussed in greater detail below, the following general process may be used to gather the fingerprint map of a stream:
1. Divide the stream into fragments, which gives the sample window length.
2. Sample a 128 bit packet in window of data, which represents the 128 bit candidate fingerprint.
3. Check the 128 bit candidate fingerprint for minimum entropy.
4. Check the 128 bit candidate fingerprint for low frequency.
5. If the conditions are satisfied and the fingerprint meets a quality threshold, shift the window by 128 bits and repeat from step 2.
The bytes 906 and coordinates 908 are placed into a candidate 128-bit fingerprint 910. The candidate fingerprint 910 is then matched against any other corresponding fingerprints selected to date, as shown in pattern 912. In one configuration, a determination of whether a match is present is made at each byte boundary of the original pattern. If a match is found, the candidate fingerprint 910 is discarded and the next 256 byte section of the pattern is analyzed.
Generally, it can be shown that assuming a “normal” distribution of bits in a sample of any size, the probability of occurrence of any given fingerprint is the same, and that probability depends only on the size of the sample. For example, the probability of a 128 bit fingerprint occurring once in a one Terabyte random sample is about 1 in a five hundred million. While the presence of a “normal” distribution of bits is rare in practice with everyday files, matching fingerprints may be eliminated from the map by checking the file for fingerprint repetitions as described above. Once the distribution of a 256 byte sample is analyzed, the byte spectrum may be taken as the basis to select the fingerprint that would have the lowest probability of occurrence if the file had the same distribution as the block sample.
Thus, in the embodiment of
In sum, the 128-bit (16-byte) fingerprints are constructed by sampling 256-byte continuous blocks from the file. The coordinates 908 in
The offset at which the fingerprint found marks the beginning of the link, and the next fingerprint will mark the links end (1016). In this manner, each link is assigned a new fingerprint and each new candidate fingerprint is ensured to not match against any previous one, which in turn ensures that each fingerprint occurs only once in the file. In addition to the fingerprint, each link size and offset is also stored along with its sha2 message digest in the fingerprint map (1018). Continuing this technique, an array of fingerprints of the file is obtained (1020). The number of fingerprints may be determined by the number of divisions of the file and the byte entropy of the file. Generally, this number is no more than the granularity selected. The granularity, in turn, may be the predetermined average number of divisions, which in one embodiment is at a minimum ten times the number of expected file changes. The total fingerprint map FM in one embodiment is the total number of fingerprints, array hashes, message digests and link offsets of the original file. The map is sufficient to compute the A between the file and any other file, as described further below.
In an embodiment, the fingerprint map is computed using an entropy equation characterized by an alphabet of N letters, whereby for each data chunk a probability is determined that a K length word has less than L number of repeating letters. For a fingerprint of 16 bytes, this problem is solved for L=1 to 16, with K=16 and N=256. The data chunk length may vary depending on the boundary of each identified fingerprint.
In one embodiment, the minimum entropy is defined as the frequency of occurrence of any octet at an 8 bit boundary in the 128 bit fingerprint (128 bit sequence). This frequency of occurrence may be designated as a number range between 1-16 such that the higher the number the lower the entropy and the lower the worth of a fingerprint, and vice versa. If the entropy of a fingerprint is greater than a minimum threshold, it is not a good candidate and another candidate is selected. Based on real world data in the experience of the inventors, the best entropy minimum appears to be around 4 in the 1-16 number range. The given criteria can be mathematically verified based on the minimum entropy and assuming a normal random distribution of bits in a stream. The probabilities of occurrence for the different minimum entropies are set forth in the following tables:
In some situations, acceptable approximations may be made. Finding quality fingerprints cannot always be guaranteed as it is highly dependent upon the byte entropy of the file. For example, there may be large sections of a file consisting of only a few different bytes, and long sequences of repeating bytes. In many embodiments, the 128 bit fingerprint is strong down to a minimum of 4 different bytes in the maximum separation range of 256 bytes. At this or greater separation range, however, the data may become extremely compressible. Thus, while fingerprinting becomes progressively more difficult to find in the file, the file's compressibility may increase by magnitudes. In the event the fingerprinting algorithm identifies this situation, in one configuration, the algorithm may simply skip analysis of the entire link at issue because the link does not meet the minimum entropy requirement. Thereupon, the link may be marked as “compressible” and its byte statistics may be stored in the FM (MAP1 in
It is assumed for the purposes of this illustration that a number of changes or updates are made to F1, whether at source node 102 or otherwise, to produce file F2 as reflected at block 108. As is often the case, updated file F2 may contain large amount of redundancies. The redundancies may also consume a comparatively high amount of the overall file space. Examples of redundancies may include images, color graphics, and blocks of identical text. The greater the file size and the greater the number of redundancies, the greater the savings of bandwidth that can be achieved. Referring back to
In one embodiment, Δ1 is obtained by first checking F2 at each byte offset of the file for matches with the fingerprints of MAP1.
While the technique described in this embodiment does not guarantee this conclusion to a certainty (such a guarantee would generally require a byte to byte comparison), the conclusion is likely in view of the very small collision chance of the sha2 digest. Furthermore the chance that an sha2 collision occurs while the fingerprints themselves match is even smaller.
In the event that the sha2 digests do not match, the search for matching fingerprints continues, and all bytes that are processed are marked as new data (1110). Also, if an adjacent fingerprint does not match, all bytes processed continue to be marked as new data. After processing the remainder of F2 in this manner, the changes between F1 and F2 are identified and the unmodified links are discovered. The sum of the new data and the unmodified links is Δ, which can be applied to MAP1 to obtain F2.
In an embodiment, during the delta processing where certain portions of F2 are found to be compressible, they may be encoded with a run-length-encoded (RLE) compression formula, which can achieve an extremely high compression ratio. The degree of compressibility may be high in view of the fact that the byte entropy may be required to get very low before the fingerprinting fails for that portion. For example, a repeating pattern of 1 Kbyte may be compressed to 100:1 by RLE, and the compressed data is than added to Δ1. The described algorithms are easily tunable and scale well for multiple CPU cores. The fingerprint mapping may be computed simultaneously with the delta or from the backup copy of the file. How and when the delta is applied depends on the situation and may be implemented separately depending on the applications needs. The minimum link size and the fingerprint range are tunable to accommodate smaller files, but the algorithm is well suited for files greater than 1 Mb. In one configuration, the default and maximum fingerprint range is 256 bytes, which is the maximum distance between two fingerprint bytes. Compression is optional but for sparse files it may add significant additional benefits for accelerating file transfer operations.
Referring again back to
At this point in time, destination node 112 has the Δ1 and F1 (or MAP1). Thereupon, destination node 112 is able to rapidly compute F2 based on a standard application of Δ1 to F1 (block 118).
A number of alternative embodiments may be contemplated in view of
In
Whether any given file is a candidate for delta compression as described herein may be determined using any of a variety of methods at the source node. For example, certain fields or metadata corresponding to a document may be retrieved at the source node to determine whether the file has been identified as an updated or modified version of an existing file. In another configuration, the document title may be provided in an identifiable format. Alternatively, a basic comparison of the content of a candidate file may be made with one or more existing files to determine its suitability for delta compression.
In another embodiment where a file is subsequently updated a number of times, the source node may only maintain and save a FM of the original file, and may reuse the FM and recomputed respective deltas for each subsequent modified version of the file. Alternatively, the source node may create a new FM of each iteration of the file.
By using the techniques disclosed herein, substantial bandwidth savings can be achieved since only the changes to documents need to be transmitted over the network to a destination. The FM may be maintained in memory at the source node for use in computing deltas corresponding to future modifications.
Further, unlike the embodiment of
In alternative embodiments, MAP1 may be generated on the fly at the destination node and transferred to the source node, such as, for example, in response to determining that F2 at the source node and F2 at the destination node contain differences. Thereupon, Δ is calculated (block 310) using MAP1 and F2 (block 310). The A is then transmitted via network 320 to destination node 312, where it is used along with F1 to reproduce F2. Subsequently, where a third file is received that is a modified version of the second, the source node 302 or destination node 312 may create MAP2 on the fly corresponding to that file for subsequent delta operations (block 311).
In addition to the advantages associated with the embodiment of
In
Referring to
In alternative embodiments to those shown in
In
A2 represents one or more separate applications from which the data constituting the files are obtained. A2 may reside on the same machine or a different machine to that of source host 702. The “P/S” indication shows that the data may be sent via a pipe or socket connection to application A1, or another connection type depending on the physical configuration employed.
In another aspect of the disclosure, a technique for raw data differencing is disclosed, which can accelerate the transfer of raw data across a network. One use of the technique described in this implementation is an exemplary file transfer engine such as that described in connection with
In
As new files (e.g. D6, D7, D8) from stream D are generated and stored in the history cache, host 702 compares, via pipe 1208 or a similar interface, the new files to the fingerprint maps (e.g., FM1-FM5) stored in the history cache. If a match is detected, such as if packets in the file are detected to be identical to those represented in one of the stored fingerprint maps, the host 702 transmits an indication to the destination that the file (or particular sections thereof) is already present at the destination and also may send a pointer identifying the file or sections.
As an illustration involving a transfer of a data and a roughly 1 TB history cache 1204, as a first 100 MB of data arrives at the host 702, the history cache 1204 may be empty. After 1 TB of data is received, the host may create a 1 TB file constituting the data and a fingerprint map of the file. The file and fingerprint map may be stored in the history cache at the host. As additional data is received beyond the 1 TB, the data may be identified to correspond with one of the stored maps, whereupon the source 702 sends an appropriate indication over the network. It is understood that the specific order or hierarchy of blocks in the processes/flow charts disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes/flow charts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.
The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term “some” refers to one or more. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed as a means plus function unless the element is expressly recited using the phrase “means for.”
