Patent Grant
10579615
The present disclosure generally relates to methods for retrieving data from a data storage system including a plurality of storage nodes.
Data storage systems may include a large number of separate storage facilities. Such systems are becoming increasingly utilized for storing and sharing large amounts of data.
Often data flow between the large number of storage facilities included in in the data storage systems may be complex, and nodes may handle data acquisition requests from numerous nodes in the data storage system. Additionally, to decrease the risk of data corruption and to avoid losing data a redundant data storage system may be employed. For example, identical copies of the data may be stored at several storage facilities.
In addition, the data storage system may comprise various safety mechanisms, making the system more robust. However, the safety mechanism may add complexity to the system.
Although data may be read into and out of distributed storage systems, accessing data in the system may be burdensome or may require complex processing. Several storage nodes may be involved in both read and write operations.
A method for rendering the retrieval of data from a data storage system including a plurality of storage nodes is disclosed. The method may be faster and/or more efficient than previous methods.
According to an embodiment, there is provided a method for retrieving data from a data storage system including a plurality of storage nodes. The method may include sending a multicast (e.g., broadcast) message to at least a subset of the storage nodes. A multicast message may include a request for the identification of the subset of storage nodes to send the data. The multicast messages may further include a data identifier, which may indicate the data to be retrieved. The method may include receiving data from a first storage node of the subset of storage nodes. The data received from the first storage node may correspond to the requested data. Moreover, at least the act of sending a multicast message or the act of receiving data from the first storage node may be performed on a condition that an estimated size of the data is less than a predetermined value.
The method may be embodied as software/hardware combination which may be executed on a server or some other data retrieving device. In another example, the method may be embodied as software/hardware implementation which may be executed on one or several storage nodes included in the data storage system.
The data storage system may include a plurality of storage nodes. At least one of these storage nodes may store the data to be retrieved.
The multicast message may be directed to all or a subset of the storage nodes. One reason for directing the multicast message to a subset of the storage nodes may be that some of the storage nodes included in the data storage system may suffer from a temporary, or in another example persistent, malfunction. For example, the connection to some of the storage nodes may be down, temporarily or persistently.
As an example, rather than receiving information about which of the storage nodes may store the requested data, an instance of the requested data may be received directly, without additional communication with the storage nodes.
Thus, the number of communications between the storage nodes and a data retrieving device may be reduced. For example, the requested data may be received from a first storage node directly as a response to a multicast message, optionally along with additional information such as related storage location information. The additional information may include information regarding locations where the requested data is stored.
In an example, rather than receiving a reply message from each or some of the plurality of storage nodes as a response to the initial multicast message, wherein the reply message include storage location information about where the data to be retrieved is stored, the replies may directly include the data to be accessed, for example based on a determination that the size of the data is below a certain threshold value. In this example, rather than retrieving the data by means of an additional dedicated unicast communication to those storage nodes storing the data based on the storage location information received in the response to the multicast message, the data may be received directly. Thus, the method for retrieving data may involve fewer operation or communications between storage nodes.
Thus, in view of the above, the retrieval of data from a data storage system including a plurality of storage nodes may be rendered more effective.
According to an embodiment, the act of sending a multicast message and the act of receiving data from the first storage node may be performed on a condition that an estimated size of the data is less than a predetermined value. For example, when handling large amounts of small quantities of data, the method may be performed very efficiently, while at the same time avoiding the risk of undesired transmittals of large quantities of data which might overload or slow down the system.
An estimated size of the data may be established according to previous data retrievals, expectations of a typical size of data retrievals, and/or according to a method of weighing various factors to produce an estimated size of the data. For example, for data that may be frequently requested, information learned from previous requests (e.g., data size, location, file type, etc.) may be used to estimate the current size of the data. For example, these data quantities may be known to be modified in relatively small portions from one data retrieval to the next.
According to an embodiment, the data received from the first storage node may include reference data, which may correspond to the requested data, on a condition that the size of the data exceeds the predetermined value. For example, undesired transmittals of large quantities of data may be avoided. Instead of sending the requested data, reference data corresponding to the requested data may be received. The reference data may comprise information about an unsuccessful sending of the requested data and/or information about an actual size of the data (as opposed to an estimated size of the data), a current load condition, a free storage space available in the first storage node, a system age of the first storage node, position information about the first storage node, and/or the like.
According to an example, the method may further include analyzing whether the data received from the first storage node fulfills a set of criteria. The set of criteria may be predetermined or change with time according to an evaluation by a storage node and/or server. For example, the set of criteria may include a desired or accepted integrity state of the data, a required maximal or minimal size of the data, whether portions of the data have been received before, and/or the like. The reliability and robustness of the data storage system may be improved by considering such factors when determining what data may be transmitted and what data may be transmitted in the form of a reference.
In an example, portions of the received data may be analyzed by using a comparison to reference data; for example, reference data may be employed when a small portion of the received data differs from the reference data.
According to an embodiment, the method may further include discarding the data received from the first storage node based on the analysis. Improved reliability and robustness may be obtained. For example, the analysis may determine that some instances of the data may not be acceptable instances of the data. For example, acceptable instance may comply with a certain set of criteria, such as a desired integrity state of the data. The instances that comply with the criteria may be retrieved from the data storage system, while instance that do not comply may be ignored.
According to an example, the act of sending a multicast message and the act of receiving data from the first storage node may be repeated based on the analysis. In case the data does not fulfill a set of desired criteria, the process of retrieving the requested data may be repeated until an acceptable instance of the data is obtained. Optionally, a maximum, predetermined number of repetitions may be established, and the process may be halted once the maximum is reached.
According to an embodiment, the method may further include receiving data from a second storage node of the subset of storage nodes. The data received from the second storage node may correspond to the requested data. The data may be received from the second storage node in a manner similar to the method employed to receive data from the first storage node. In addition, in case the received data from the first storage node is corrupted, the data received from the second storage node may be utilized to ensure the data is accurate/and or to correct the corrupted data. Also, the first storage node may be suffering from a temporary or persistent malfunction, and the second storage node may serve as a backup storage node.
For example, the method may further include receiving data from more than two storage nodes of the subset of storage nodes.
According to an embodiment, the method may further include analyzing whether the data received from the first storage node and the data received from the second storage node fulfill a set of criteria. By accessing the first as well the second storage node, the two instances of data may be compared to each other, for example by means of a checksum or the like, and analyzed to establish whether the set of criteria are fulfilled.
As an example, if the data received from the first and second storage nodes are in compliance as determined by the analysis, a further process to receive data from additional storage nodes, such as additional instance of the data, may be stopped or interrupted.
According to an example embodiment, the method may further include discarding at least one of the data received from the first storage node and the data received from the second storage node based on the analysis. As explained above, the discarding may ensures that acceptable instances of data, according to a certain set of criteria, are retrieved from the data storage system, while corrupted or out-of-date data may be discarded.
If the data received from the first and second storage nodes both are considered to be acceptable as a result of the analysis, a further selection procedure may be initiated to select one of them and to discard the other. The selection procedure may be based on a current load condition, geographical positions, and/or other selection criteria of the first and second storage nodes, respectively. Note, however, that also other types of selection procedures may be possible.
In an example, rather than discarding the data that is not selected, the data may be saved. At least one of the data received from the first storage node and the data received from the second storage node may be stored as reference data for future analysis.
According to an example, the act of sending a multicast message and at least one of the act of receiving data from the first storage node and the act of receiving data from the second storage node may be repeated based on the analysis. For example, the instances of the requested data may have unacceptable qualities, a repetition of the acts of sending and receiving may be performed.
The act of sending a multicast message and at least one of the act of receiving data from the first storage node and the act of receiving data from the second storage node may be repeated until an acceptable instance of the requested data is obtained. Optionally, only a maximum, predetermined number of repetitions may be permitted.
According to an embodiment, the data received from the first storage node may be sent by means of unicast communication.
According to an embodiment, the method may further include determining whether a capability of the first storage node fulfills a set of characteristics. The capability of the first storage node may be at least one of a current load condition of the first storage node, an expected or estimated load condition of the first storage node, information about a free storage space available in the first storage node, information about a system age of the first storage node, position information about the first storage node, and/or the like.
The sending of the requested data from the first storage node may be selected according to a queue scheme in relation to other storage nodes included in the data storage system. Information about such capabilities may be stored at the first storage node or, optionally, at some other storage nodes included in the data storage system. For example, this information may be stored at other devices outside of the data storage system, such as at a server in communication with the data storage system.
According to another example, the method may further include determining whether a capability of a communications network connecting the plurality of storage nodes fulfills a set of characteristics. The plurality of storage nodes and a data retrieving device, such as a server, may be connected to each other via a communications network, such as a Local Area Network, a Wide Area Network and/or the Internet. The capability of the communications network may, for example, be a current connection speed to each of the storage nodes or the data retrieving device, a type of connection associated to each of the storage nodes or the data retrieving device, and or the like.
According to an embodiment, the data received from the first or second storage node may include a collection object including a reference to data of the data storage system. The data may include at least one of a set of another collection object or a data file other than a collection object.
According to an embodiment, a server may be adapted to retrieve data from a data storage system that includes the plurality of storage nodes. The server may include a transmitter adapted to send a multicast message to at least a subset of the storage nodes. The multicast message may include a request for the subset of storage nodes to send the data. The sever may also include a receiver adapted to receive data from a first storage node of the subset of storage nodes. The data received from the first storage node may correspond to the requested data.
Thus the sever may access data stored on the plurality of storage nodes. In an example, the server may act as a storage node in the system.
Other features of the disclosed embodiments will become apparent to those skilled in the art upon review of the following drawings, the detailed description, and the appended claims.
The above, as well as additional objects and features will be better understood through the following illustrative and non-limiting detailed description of embodiments disclosed herein, with reference to the appended drawings, where like reference numerals may be used for like elements.
The structure of an example embodiment of a distributed data storage system 100 is outlined schematically in
Each API 136, 138 may communicate with storage nodes 140 in the data storage system 100. The storage nodes 140 may communicate with each other. Each storage node 140 may include a network interface card (NIC) so that a storage node may receive and send data to/from other storage nodes 140 and the server 130. These communications may be based on TCP (Transmission Control Protocol) and UDP (User Datagram Protocol).
Different APIs 136, 138 on the same server 130 may access different sets of storage nodes 140. There may exist more than one server 130 which accesses each storage node 140. Additional servers, however, may not significantly affect the way in which the storage nodes operate.
The components of the distributed data storage system may be the storage nodes 140 and the APIs 136, 138 in the server 130 which access the storage nodes 140. The present embodiment as illustrated in
The storage node 140 may typically be embodied by a file server which is provided with a number of functional blocks. The storage node may thus comprise a tangible storage medium 142, which typically comprises of a number of internal (e.g. connected via Integrated Device Electronics (IDE) or Serial Advanced Technology Attachment (SATA) or external hard drives (e.g. connected via USB or Firewire), optionally configured as a RAID (Redundant Array of Independent Disk) system. Other types of storage media are however conceivable as well.
The storage node 140 may further include a directory 144, which may include lists of data entity/storage node relations as a host list.
In addition to the host list, each storage node 140 may further contain a node list including the IP addresses of storage nodes in its set or group of storage nodes. The number of storage nodes in a group may vary from a few to hundreds of storage nodes. The node list may further have a version number.
Additionally, the storage node 140 may further include a cluster monitor block 146. The cluster monitor block 146 may be configured to carry out monitoring of other storage nodes 140 in the data storage system 100.
The storage nodes 140 of the distributed data storage system 100 may be considered to exist in the same hierarchical level. In an example, there may be no permanent master storage node that is responsible for maintaining a directory of stored data entities and monitoring data consistency, etc. Instead, all storage nodes 140 may be considered equal, and may, at times, carry out data management operations vis-à-vis other storage nodes in the system. This equality may ensure that the system is robust. In case of a storage node malfunction, other nodes in the system may cover up the malfunctioning node and ensure reliable data storage.
According to the embodiment as illustrated in
In the following, the operation of the data storage system may be described with reference to
The data, or data set, to be retrieved from the data storage system may be one or several data files, a folder comprising data files or additional folders, a computer-readable collection object including a set of computer-readable references, and/or the like.
In more detail, the collection object may include a reference to data of the data storage system, the data including at least one of a set of one or more collection objects and data files, other than a collection object. A reference may be a reference to another collection object stored in the data storage system. A reference may also be a reference to one or more computer-readable data files, other than the collection objects, stored in the data storage system. A reference may include a pointer (e.g. a memory address) to a storage location at a storage node. A reference may include an identifier of the collection object or data file referred to.
A collection object may be embodied as a directory in a file system of a storage node. Data files referenced in the collection object may be data files stored in the directory. Collection objects referenced in the collection object may represent subdirectories stored in the directory.
A collection object may also be embodied as a file having a predetermined format. The file may be a special file in the file system of the storage medium. Alternatively, the file may be a data file in the file system of the storage medium. Such a file may be readable using the same routines of the file system as the data files. For example the file may be a plain text file including references to collection objects and/or data files.
The reading or retrieval of data, as well as other functions in the system, may utilize multicast communication to communicate simultaneously with a plurality of storage nodes. By a multicast or IP multicast is here meant a point-to-multipoint communication which may be accomplished by sending a message to an IP address which is reserved for multicast applications.
For example, a message, typically a request, may be sent to such an IP address (e.g. 244.0.0.1), and a number of recipient servers may be registered as subscribers to that IP address. Each of the recipient servers may have its own IP address. When a switch in the network receives the message directed to 244.0.0.1, the switch may forward the message to the IP addresses of each server registered as a subscriber.
In principle, a single server may be registered as a subscriber to a multicast address, in which case a point-to-point, communication may be achieved. However, in the context of this disclosure, such a communication may nevertheless considered a multicast communication since a multicast scheme is employed.
Unicast communication may also be employed. Unicast communication may be a communication with a single recipient.
With reference to
The storage nodes 220, 222, 224, 226 and 228 may scan themselves for data corresponding to the identifier. If such data is found at some of the storage nodes, a response may be sent which may be received (box 320) by the server 210, cf.
As illustrated, the response may optionally contain further information in addition to an indication that the storage node has an instance, or copy, of the requested data. For example, the response may contain information from the storage node directory about other storage nodes containing the data, storage location information, information regarding which version of the data set is contained in the storage node, information about which API, and/or which server, that created the version of the data set that is contained in the storage node, information about a free storage space available at the storage node, information about a system age of the storage node, and/or information regarding which load the storage node at present is exposed to.
If, on the other hand, the data set is not stored at a particular storage node, this storage node may be silent or may send reference data to the server. This reference data may include information that the requested data set is not stored at the storage node in question and/or may include information about the storage node, such as a current load condition, information about a current data storage availability, storage location information, information about a system age of the storage node, and/or the like.
The received data sets from the storage nodes 222 and 228 may then be analyzed to check whether at least one instance of the requested data set has a maintained integrity (box 330). A maintained data integrity of a data set may correspond to the data set being similar as compared to an original data set, up to certain acceptable data discrepancies. The integrity check may ensure the data is accurate, for example the integrity check may ensure the data set is an accurate copy of the original.
The act of analyzing may be implemented by analyzing received data from one storage node, from two storage nodes, or alternatively from several, storage nodes. The analysis of the data integrity may, for example, be performed in the server 210. For example, the analysis may be implemented by means of a checksum algorithm, a cryptographic hash function such as MD5 (Message-Digest Algorithm), a comparison of portions of some or all of the different instances of the requested data set, a comparison with a reference data set, and/or the like.
As a consequence of the analysis, it may be established whether the integrity of some of the received data sets from the storage nodes 222, 228 meets a desired quality threshold (box 340). If at least one instance of the requested data set meets the desire quality threshold, one of the instances that meets the threshold may be selected (box 350). If several instances of the requested data set have met the quality threshold, one of these may be selected depending on a set of additional requirements. These requirements may for instance include a current load condition of the storage nodes, geographical positions of the storage nodes, a system age of the storage nodes, etc. Alternatively, the storage node having the latest version of the requested data set and/or the lowest load may be selected.
In one example, one instance of the requested data set that meets the quality threshold, for example the received data from the storage node 222, may be selected. The received data set from the other storage node 228 may be discarded irrespective of whether this data set also meets the desired quality threshold. In another example, the received data set from the other storage node 228 may be stored as reference data, for example at the server 210. Upon selecting a data set that meets the quality threshold, the server may stop analyzing integrity of additional data sets.
Optionally, if the integrity of the at least one instance of the requested data set meets the quality threshold, a further analysis of the data sets received from additional storage nodes may be stopped or interrupted.
In one example, a number of instances of the requested data set, say two instances, may be compared. If these two instances agree as to a desired accuracy (e.g. they are the same or substantially similar) or comply with a desired integrity state of the data, and the checking of additional received data set may be interrupted or stopped.
In another example, the comparison between different instances may be interrupted when a certain fraction of the total number of instances agrees to a desired accuracy. Example fractions may be 50%, 60% or 70%, but also other fractions may be equally conceivable.
A desired accuracy may be that the compared data sets fully agree, certain portions of the compared data sets fully agree, etc. In some instances, however, a less strict desired accuracy may be sufficient.
The data integrity may be lost for example when the connection between the server 210 and some of the storage nodes 220, 222, 224, 226 or 228 is corrupted so that the transfer of the requested data set fails, or when the data storage system suffers from some other malfunction. In
If no instance of the requested data set from the storage nodes 222, 228 meets the desired quality threshold, a new multicast message may sent to the storage nodes (box 310), and the procedures described with reference to boxes 320, 330 and 340 may be repeated. The loop thus formed by the boxes 310, 320, 330 and 340 may be reiterated until an instance of the requested data set that meets the desired quality threshold has been received from some of the storage nodes 220, 222, 224, 226 or 228, and/or until a predetermined number of iterations have been performed.
Optionally, the server 130 may send a unicast message to a storage node associated with a data set having a lost integrity (e.g., the data set from this node may not meet the desired quality threshold). The message may include information about the nature of the failed transfer, information about the received data set, the correct data, and/or the like.
An embodiment may be implemented in a data storage system where the majority of the files are small and a minority of the data sets to be handled are large. The definitions of small or large data set depends on the system capabilities, such as connection speeds between the various entities comprised in the system, storage capabilities, etc. The definitions may vary from system to system (e.g. “small” may imply data blocks with sizes less than or equal to 512 kB or less than a predetermined fraction of the data bandwidth available in the system).
If the majority of the data sets to be handled by the system are considered to be small, it may be assume that the size of all the data sets to be retrieved are small. Moreover, if the size of the data to be retrieved exceeds a predetermined value, the transmittal of the data may be stopped, interrupted or may not occur. For example, the transfer from the storage node storing the data, as well as from the additional storage nodes storing instances of the data, to the server may be halted or interrupted. This decision to stop the transfer of large files may be made locally at the storage nodes storing the data to be retrieved. Optionally, the decision may be made by the server.
In another example, the storage nodes storing the data to be retrieved may send reference data back to the server including information about the sizes and the storage locations of the data sets which were supposed to be sent, file handles, and/or additional relevant information about the storage nodes. The additional information may include a current load condition, a free storage space available, a system age, position information, and/or the like. In addition, the reference data may include instructions for the server to retrieve the data set from a particular set of the storage nodes storing the requested data. In one example, the data set may be retrieved from one particular storage node by means of a unicast communication.
It may occur that a data set stored at two or more storage nodes 140 may be updated or created substantially simultaneously by two different APIs 136, 138. Such a situation may give rise to different versions of the data set at different storage nodes and thereby giving rise to a conflict when retrieving the data according to the present inventive concept. These APIs 136, 138 may be included in the same, or alternatively, at different servers. The contents of the respective creation or updating processes may differ, thereby giving rise to versions of the data set having different contents. In this case, a selection procedure may be initiated in which one of the versions are selected and thereafter stored at the different storage nodes. Optionally, all or some of the different versions may be stored at the storage nodes.
Two APIs (e.g. API1 and API2) may be in the process of updating a data set which is stored at three different storage nodes (e.g. storage nodes SN1, SN2 and SN3). The data sets stored at the respective storage nodes may each be associated with a specific version number. Each of the data sets may also be associated with information about which API updated the particular version of the data set. This version number and API information may be stored in the data sets at the storage nodes as well as at the APIs. In the following example, the version number V and API information N are collectively denoted by the version data (V|N). For example, the version data (34|2) represents the version number 34 of the data set has been updated by the second API (i.e. API2).
The storage nodes may be arranged to send acknowledgements to at least one of the APIs. The acknowledgement may include information about whether the updating process was successful or not. The acknowledgements may include the version data, i.e. (V|N), associated to the updated data set.
For example, at a particular time instant, all the data sets may have the same version number, for example, version number V=25. The data sets stored at the first two storage nodes (e.g. the storage nodes SN1 and SN2) may have both previously been updated by the first API (e.g. API1). However, before the first API had the time to do so, the second API (e.g. API2) may have managed to update the data set stored at the third storage node (e.g. storage node SN3). Put differently, the first API may have updated the data sets stored at the first and second storage nodes with version 25 before the second API had the time to do so. The acknowledgements sent back to the APIs, as well as the information stored at the storage nodes, thereby may include the version data (25|1), (25|1) and (25|2), for the data sets stored at the first, second and third storage node, respectively.
Thus, when the first API at a subsequent time instant attempts to update the third storage node with its version 25 of the data set, the updating may be halted or may not occur since there is already a version 25 stored at the third storage node as sent from the second API. Moreover, when the second API at a subsequent time instant attempts to update the first and second storage nodes, the updating of these may not occur, since there are already versions 25 at the first and second storage nodes as sent from the first API.
The APIs receiving the acknowledgements including the version data may be arranged to instruct the storage nodes included in the data storage system 100 to utilize and store the data set having the version data (V|N) of the data set which is most frequently occurring in the received acknowledgements. For example, if the version data (V|Nmax) of the data set occurs in more than 50% of the acknowledgements, this data set may be maintained and distributed to other storage nodes storing other versions of the data set. In the following, the data set associated to a version data (V|N) may be referred to as a correct and a wrong version of the data set, depending on whether the data set is the most frequently occurring or not, respectively.
The APIs may be arranged to send instructions to the storage nodes keeping the wrong versions to delete the wrong versions of the data set. Moreover, wrong versions of the data set may be replaced or amended by the correct versions of the data set at these storage nodes by means of the APIs instructing the storage nodes to send the correct version. For example, the other storage nodes included in the data storage system 100 may send the correct version e.g. according to the method as disclosed in U.S. patent application Ser. No. 13/125,524. It should be noted that the wrong versions of the data set may be replaced or overwritten. In another example, the wrong versions may be amended, in which case the parts of the correct version which differ from the wrong version in question may be added to the wrong version of the data set. Thereby, the data from the wrong versions may be kept in the data storage system.
Thus, in the example involving the three storage nodes and the two APIs, version 25 as sent from the first API, e.g. having version data (25|1), may be kept as the correct version 25, since this version had been successfully stored at two storage nodes (i.e. SN1 and SN2).
In an example, the data set sent from the second API (i.e. API2) to the third storage node (i.e. SN3), which was deduced to be the wrong version according to the above, may be resent to all of the storage nodes with a new version number, e.g. version number 26. This data set may then be associated with the version data (26|2). After the selection procedure has been performed, the process of retrieving data according to the methods described above may be repeated.
The above description illustrated several example, and the methods and system disclosed were described with reference to a limited number of examples. However, as is readily appreciated by a person skilled in the art, other examples than the ones disclosed above are equally possible within the scope of the disclosure, and the breadth of the protection should be judged as defined by the appended claims.
This application is a continuation of U.S. patent application Ser. No. 13/224,433, filed Sep. 2, 2011, the contents of which are hereby incorporated by reference herein. This application includes subject matter that may be related to subject matter included in the following applications: U.S. patent application Ser. No. 13/224,393 entitled “A Method And Device For Maintaining Data In A Data Storage System Comprising A Plurality Of Data Storage Nodes,” filed Sep. 2, 2011; U.S. patent application Ser. No. 13/224,404 entitled “Method And Device For Writing Data To A Data Storage System Comprising A Plurality Of Data Storage Nodes,” filed Sep. 2, 2011; U.S. patent application Ser. No. 13/224,415 entitled “Method For Data Maintenance,” filed Sep. 2, 2011; U.S. patent application Ser. No. 13/224,424 entitled “Method For Handling Requests In A Storage System And Storage Node For A Storage System,” filed Sep. 2, 2011; and U.S. patent application Ser. No. 13/224,446 entitled “Method For Updating Data In A Distributed Data Storage System,” filed Sep. 2, 2011.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3707707 | Spencer et al. | Dec 1972 | A |
| 5713017 | Lin et al. | Jan 1998 | A |
| 5787247 | Norin et al. | Jul 1998 | A |
| 6003065 | Yan et al. | Dec 1999 | A |
| 6021118 | Houck et al. | Feb 2000 | A |
| 6055543 | Christensen et al. | Apr 2000 | A |
| 6259701 | Shur et al. | Jul 2001 | B1 |
| 6345308 | Abe | Feb 2002 | B1 |
| 6389432 | Pothapragada et al. | May 2002 | B1 |
| 6401120 | Gamache et al. | Jun 2002 | B1 |
| 6470420 | Hospodor | Oct 2002 | B1 |
| 6510162 | Fijolek et al. | Jan 2003 | B1 |
| 6732237 | Jacobs | May 2004 | B1 |
| 6775688 | Kakimoto | Aug 2004 | B1 |
| 6782389 | Chrin et al. | Aug 2004 | B1 |
| 6839815 | Kagami et al. | Jan 2005 | B2 |
| 6925737 | Bolduan et al. | Aug 2005 | B2 |
| 6985956 | Luke et al. | Jan 2006 | B2 |
| 7032168 | Gerace | Apr 2006 | B1 |
| 7039661 | Ranade | May 2006 | B1 |
| 7200664 | Hayden | Apr 2007 | B2 |
| 7206836 | Dinker et al. | Apr 2007 | B2 |
| 7266556 | Coates | Sep 2007 | B1 |
| 7320088 | Gawali | Jan 2008 | B1 |
| 7340510 | Liskov et al. | Mar 2008 | B1 |
| 7352765 | Dai et al. | Apr 2008 | B2 |
| 7376945 | Kakumani | May 2008 | B1 |
| 7385978 | Yang et al. | Jun 2008 | B1 |
| 7406484 | Srinivasan et al. | Jul 2008 | B1 |
| 7487305 | Hill et al. | Feb 2009 | B2 |
| 7503052 | Castro et al. | Mar 2009 | B2 |
| 7546486 | Slik et al. | Jun 2009 | B2 |
| 7565419 | Kwiatkowski | Jul 2009 | B1 |
| 7568069 | Jantz et al. | Jul 2009 | B2 |
| 7574488 | Matsubara | Aug 2009 | B2 |
| 7590672 | Slik et al. | Sep 2009 | B2 |
| 7593966 | Therrien et al. | Sep 2009 | B2 |
| 7624155 | Nordin et al. | Nov 2009 | B1 |
| 7624158 | Slik et al. | Nov 2009 | B2 |
| 7631023 | Kaiser et al. | Dec 2009 | B1 |
| 7631045 | Boerries et al. | Dec 2009 | B2 |
| 7631313 | Mayhew et al. | Dec 2009 | B2 |
| 7634453 | Bakke et al. | Dec 2009 | B1 |
| 7647329 | Fischman et al. | Jan 2010 | B1 |
| 7694086 | Bezbaruah et al. | Apr 2010 | B1 |
| 7769711 | Srinivasan et al. | Aug 2010 | B2 |
| 7778972 | Cormie et al. | Aug 2010 | B1 |
| 7822711 | Ranade | Oct 2010 | B1 |
| 7822766 | Arndt et al. | Oct 2010 | B2 |
| 7840992 | Dufrene et al. | Nov 2010 | B1 |
| 7873650 | Chapman et al. | Jan 2011 | B1 |
| 7885982 | Wight et al. | Feb 2011 | B2 |
| 8060598 | Cook et al. | Nov 2011 | B1 |
| 8073881 | Georgiev | Dec 2011 | B1 |
| 8190561 | Poole et al. | May 2012 | B1 |
| 8255430 | Dutton et al. | Aug 2012 | B2 |
| 8296398 | Lacapra et al. | Oct 2012 | B2 |
| 8401997 | Tawri et al. | Mar 2013 | B1 |
| 8417828 | Ma et al. | Apr 2013 | B2 |
| 8443062 | Voutilainen et al. | May 2013 | B2 |
| 8468214 | Bourlas | Jun 2013 | B1 |
| 8504758 | McHugh | Aug 2013 | B1 |
| 8561115 | Hattori et al. | Oct 2013 | B2 |
| 8577957 | Behar et al. | Nov 2013 | B2 |
| 8707091 | Gladwin et al. | Apr 2014 | B2 |
| 8745342 | Amano et al. | Jun 2014 | B2 |
| 8769042 | Sedeffow | Jul 2014 | B2 |
| 8898236 | Kanakadandi | Nov 2014 | B2 |
| 9098683 | Rose | Aug 2015 | B2 |
| 9225780 | Braam | Dec 2015 | B2 |
| 9367448 | Botelho | Jun 2016 | B1 |
| 9471593 | Anderson | Oct 2016 | B1 |
| 9928174 | Connell | Mar 2018 | B1 |
| 20010034812 | Ignatius et al. | Oct 2001 | A1 |
| 20010047400 | Coates et al. | Nov 2001 | A1 |
| 20010048728 | Peng | Dec 2001 | A1 |
| 20020042693 | Kampe et al. | Apr 2002 | A1 |
| 20020073086 | Thompson et al. | Jun 2002 | A1 |
| 20020103888 | Janz et al. | Aug 2002 | A1 |
| 20020114341 | Sutherland et al. | Aug 2002 | A1 |
| 20020145786 | Chang et al. | Oct 2002 | A1 |
| 20020156912 | Hurst | Oct 2002 | A1 |
| 20030026254 | Sim | Feb 2003 | A1 |
| 20030065663 | Chu | Apr 2003 | A1 |
| 20030097468 | Hamadi | May 2003 | A1 |
| 20030120654 | Edlund et al. | Jun 2003 | A1 |
| 20030126122 | Bosley et al. | Jul 2003 | A1 |
| 20030145127 | Unice | Jul 2003 | A1 |
| 20030154238 | Murphy et al. | Aug 2003 | A1 |
| 20030172089 | Douceur et al. | Sep 2003 | A1 |
| 20030177261 | Sekiguchi et al. | Sep 2003 | A1 |
| 20030233455 | Leber | Dec 2003 | A1 |
| 20040010654 | Yasuda | Jan 2004 | A1 |
| 20040030787 | Jandel et al. | Feb 2004 | A1 |
| 20040059805 | Dinker et al. | Mar 2004 | A1 |
| 20040064729 | Yellepeddy | Apr 2004 | A1 |
| 20040078466 | Coates et al. | Apr 2004 | A1 |
| 20040088297 | Coates et al. | May 2004 | A1 |
| 20040088646 | Yeager | May 2004 | A1 |
| 20040111730 | Apte | Jun 2004 | A1 |
| 20040243675 | Taoyama et al. | Dec 2004 | A1 |
| 20040260775 | Fedele | Dec 2004 | A1 |
| 20050010618 | Hayden | Jan 2005 | A1 |
| 20050015431 | Cherkasova | Jan 2005 | A1 |
| 20050015461 | Richard et al. | Jan 2005 | A1 |
| 20050038990 | Sasakura et al. | Feb 2005 | A1 |
| 20050044092 | Adya et al. | Feb 2005 | A1 |
| 20050055418 | Blanc et al. | Mar 2005 | A1 |
| 20050114711 | Hesselink | May 2005 | A1 |
| 20050177550 | Jacobs et al. | Aug 2005 | A1 |
| 20050193245 | Hayden et al. | Sep 2005 | A1 |
| 20050204042 | Banerjee et al. | Sep 2005 | A1 |
| 20050246393 | Coates et al. | Nov 2005 | A1 |
| 20050256894 | Talanis et al. | Nov 2005 | A1 |
| 20050278552 | Delisle et al. | Dec 2005 | A1 |
| 20050283649 | Turner et al. | Dec 2005 | A1 |
| 20060023969 | Lara | Feb 2006 | A1 |
| 20060031230 | Kumar | Feb 2006 | A1 |
| 20060031439 | Saffre | Feb 2006 | A1 |
| 20060047776 | Chieng et al. | Mar 2006 | A1 |
| 20060080574 | Saito et al. | Apr 2006 | A1 |
| 20060090045 | Bartlett et al. | Apr 2006 | A1 |
| 20060090095 | Massa et al. | Apr 2006 | A1 |
| 20060095397 | Torres | May 2006 | A1 |
| 20060112154 | Douceur et al. | May 2006 | A1 |
| 20060218203 | Yamato et al. | Sep 2006 | A1 |
| 20060253504 | Lee | Nov 2006 | A1 |
| 20060288053 | Holt | Dec 2006 | A1 |
| 20070022087 | Bahar et al. | Jan 2007 | A1 |
| 20070022121 | Bahar et al. | Jan 2007 | A1 |
| 20070022122 | Bahar et al. | Jan 2007 | A1 |
| 20070022129 | Bahar et al. | Jan 2007 | A1 |
| 20070028291 | Brennan et al. | Feb 2007 | A1 |
| 20070055703 | Zimran et al. | Mar 2007 | A1 |
| 20070088703 | Kasiolas et al. | Apr 2007 | A1 |
| 20070094269 | Mikesell et al. | Apr 2007 | A1 |
| 20070094354 | Soltis | Apr 2007 | A1 |
| 20070189153 | Mason | Aug 2007 | A1 |
| 20070192451 | Tran et al. | Aug 2007 | A1 |
| 20070198467 | Wiser et al. | Aug 2007 | A1 |
| 20070220320 | Sen et al. | Sep 2007 | A1 |
| 20070220328 | Liu et al. | Sep 2007 | A1 |
| 20070255763 | Beyerle | Nov 2007 | A1 |
| 20070276838 | Abushanab | Nov 2007 | A1 |
| 20070288494 | Chrin et al. | Dec 2007 | A1 |
| 20070288533 | Srivastava et al. | Dec 2007 | A1 |
| 20070288638 | Vuong et al. | Dec 2007 | A1 |
| 20080005195 | Li | Jan 2008 | A1 |
| 20080005199 | Chen et al. | Jan 2008 | A1 |
| 20080010372 | Khedouri | Jan 2008 | A1 |
| 20080043634 | Wang et al. | Feb 2008 | A1 |
| 20080077635 | Sporny et al. | Mar 2008 | A1 |
| 20080084889 | Archer et al. | Apr 2008 | A1 |
| 20080104218 | Liang et al. | May 2008 | A1 |
| 20080104666 | Dillaway | May 2008 | A1 |
| 20080109830 | Giotzbach et al. | May 2008 | A1 |
| 20080126357 | Casanova et al. | May 2008 | A1 |
| 20080168157 | Marchand | Jul 2008 | A1 |
| 20080171556 | Carter | Jul 2008 | A1 |
| 20080172478 | Kiyohara et al. | Jul 2008 | A1 |
| 20080177767 | Lin et al. | Jul 2008 | A1 |
| 20080198752 | Fan et al. | Aug 2008 | A1 |
| 20080216174 | Vogel | Sep 2008 | A1 |
| 20080235321 | Matsuo | Sep 2008 | A1 |
| 20080243950 | Webman et al. | Oct 2008 | A1 |
| 20080244674 | Hayashi | Oct 2008 | A1 |
| 20080270822 | Fan et al. | Oct 2008 | A1 |
| 20090043922 | Crowther | Feb 2009 | A1 |
| 20090083810 | Hattori et al. | Mar 2009 | A1 |
| 20090132543 | Chatley et al. | May 2009 | A1 |
| 20090172211 | Perry et al. | Jul 2009 | A1 |
| 20090172307 | Perry et al. | Jul 2009 | A1 |
| 20090228669 | Slesarev et al. | Sep 2009 | A1 |
| 20090249222 | Schmidt | Oct 2009 | A1 |
| 20090271412 | Lacapra et al. | Oct 2009 | A1 |
| 20090287842 | Plamondon | Nov 2009 | A1 |
| 20100057785 | Khosravy | Mar 2010 | A1 |
| 20100070515 | Dutton et al. | Mar 2010 | A1 |
| 20100115078 | Ishikawa et al. | May 2010 | A1 |
| 20100146497 | Kogan | Jun 2010 | A1 |
| 20100161138 | Lange et al. | Jun 2010 | A1 |
| 20100169391 | Baptist et al. | Jul 2010 | A1 |
| 20100169415 | Leggette et al. | Jul 2010 | A1 |
| 20100185693 | Murty et al. | Jul 2010 | A1 |
| 20100198888 | Blomstedt et al. | Aug 2010 | A1 |
| 20100198889 | Byers et al. | Aug 2010 | A1 |
| 20100223262 | Krylov et al. | Sep 2010 | A1 |
| 20100303071 | Kotalwar et al. | Dec 2010 | A1 |
| 20100306270 | McDiarmid | Dec 2010 | A1 |
| 20110047413 | McGill | Feb 2011 | A1 |
| 20110047620 | Mahaffey | Feb 2011 | A1 |
| 20110055353 | Tucker et al. | Mar 2011 | A1 |
| 20110072206 | Ross et al. | Mar 2011 | A1 |
| 20110087690 | Cairns | Apr 2011 | A1 |
| 20110125814 | Slik et al. | May 2011 | A1 |
| 20110153557 | Bernstein | Jun 2011 | A1 |
| 20110161335 | Dash | Jun 2011 | A1 |
| 20110208825 | Lee et al. | Aug 2011 | A1 |
| 20110231374 | Jain | Sep 2011 | A1 |
| 20110238909 | Kumar et al. | Sep 2011 | A1 |
| 20110252204 | Coon et al. | Oct 2011 | A1 |
| 20120066498 | Engert | Mar 2012 | A1 |
| 20120109883 | Iordanov | May 2012 | A1 |
| 20120180070 | Pafumi et al. | Jul 2012 | A1 |
| 20120191771 | Preston | Jul 2012 | A1 |
| 20120290870 | Shah | Nov 2012 | A1 |
| 20120331021 | Lord | Dec 2012 | A1 |
| 20130031423 | Barrow | Jan 2013 | A1 |
| 20130060884 | Bernbo et al. | Mar 2013 | A1 |
| 20130103851 | Umeki et al. | Apr 2013 | A1 |
| 20130133030 | Xue | May 2013 | A1 |
| 20130254314 | Chow et al. | Sep 2013 | A1 |
| 20130268632 | Baron | Oct 2013 | A1 |
| 20130324259 | McCaffrey | Dec 2013 | A1 |
| 20140006796 | Smith | Jan 2014 | A1 |
| 20140173112 | Seago et al. | Jun 2014 | A1 |
| 20140289702 | McMahon | Sep 2014 | A1 |
| 20140298135 | Dhuse | Oct 2014 | A1 |
| 20150156174 | Fahey | Jun 2015 | A1 |
| 20150288659 | Lukacs | Oct 2015 | A1 |
| 20160091948 | Mitchell | Mar 2016 | A1 |
| 20160099963 | Mahaffey | Apr 2016 | A1 |
| 20160259795 | Brandys | Sep 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 1726454 | Jan 2006 | CN |
| 101710339 | May 2010 | CN |
| 0774723 | Jul 1998 | EP |
| 0934568 | Jun 2003 | EP |
| 1521189 | Apr 2005 | EP |
| 1578088 | Sep 2005 | EP |
| 1669850 | Jun 2006 | EP |
| 1798934 | Jun 2007 | EP |
| 2031513 | Mar 2009 | EP |
| 6-348527 | Dec 1994 | JP |
| 11-249874 | Sep 1999 | JP |
| 2000-322292 | Nov 2000 | JP |
| 2003-030012 | Jan 2003 | JP |
| 2003-223286 | Aug 2003 | JP |
| 2003-248607 | Sep 2003 | JP |
| 2003-271316 | Sep 2003 | JP |
| 2004-005491 | Jan 2004 | JP |
| 2007-058275 | Mar 2007 | JP |
| 2008-250767 | Oct 2008 | JP |
| 2009-259007 | Nov 2009 | JP |
| WO 1999038093 | Jul 1999 | WO |
| WO 2001018633 | Mar 2001 | WO |
| WO 2002035359 | May 2002 | WO |
| WO 2002044835 | Jun 2002 | WO |
| WO 2004053677 | Jun 2004 | WO |
| WO 2006124911 | Nov 2006 | WO |
| WO 2007014296 | Feb 2007 | WO |
| WO 2007115317 | Oct 2007 | WO |
| WO 2007134918 | Nov 2007 | WO |
| WO 2008069811 | Jun 2008 | WO |
| WO 2008102195 | Aug 2008 | WO |
| WO 2009048726 | Apr 2009 | WO |
| WO 2010046393 | Apr 2010 | WO |
| WO 2010046393 | Apr 2010 | WO |
| WO 2010080533 | Jul 2010 | WO |
| WO 2011131717 | Oct 2011 | WO |
| Entry |
|---|
| Deering et al., “Multicast Routing in Datagram Internetworks and Extended LANs”, ACM Transactions on Computer Systems, vol. 8, No. 2, May 1990, pp. 85-110. |
| Hewlett-Packard Development Company L. P., “HP Volume Shadowing for OpenVMS”, OpenVMS Alpha 7.3-2, Sep. 2003, 162 pages. |
| Katsurashima et al., “NAS Switch: a Novel CIFS Server Virtualization”, Proceedings. 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, Apr. 7-10, 2003, pp. 82-86. |
| Kronenberg et al., “VAXclusters: A Closely-Coupled Distributed System”, ACM Transactions on Computer Systems, vol. 4, No. 2, May 1986, pp. 130-146. |
| Parris, Keith, “Using OpenVMS Clusters for Disaster Tolerance”, System/Software Engineer, HP Services—Systems Engineering, 2003, 27 pages. |
| SAP Library, “Queues for Prioritized Message Processing”, SAP Exchange Infrastructure, Available online at http://help.sap.com/saphelp_nw04/helpdata/en/04/827440c36ed562e10000000a155106/content.htm, Feb. 6, 2009, pp. 1-2. |
| Squid-Cache.org, “Squid Configuration Directive Reply_Body_Max_Size”, Available online at < http://www.squid-cache.org/Doc/config/reply_body_max_size/>, Dec. 21, 2008, pp. 1-2. |
| Suryanarayanan et al., “Performance Evaluation of New Methods of Automatic Redirection for Load Balancing of Apache Servers Distributed in the Internet”, Proceedings. 25th Annual IEEE Conference on Local Computer Networks, Nov. 8-10, 2000, pp. 644-651. |
| Tang et al., “An Efficient Data Location Protocol for Self-Organizing Storage Clusters”, Supercomputing, ACM/IEEE Conference, Phoenix, AZ, USA, Nov. 15-21, 2003, 13 pages. |
| Trustwave, “How Do I Block Large Files by Content Size Before Download?”, Available online at < https://www.trustwave.com/support/kb/article,aspx?id=13166>, Mar. 23, 2009, pp. 1-2. |
| Weatherspoon et al., “Antiquity: Exploiting a Secure Log for Wide-Area Distributed Storage”, Proceedings of the EuroSys Conference, ACM 2007, Lisbon, Portugal, Mar. 21-23, 2007, pp. 371-384. |
| Wikipedia, “FastTrack”, Available online at: http://de.wikipedia.org/w/index.php?title=FastTrack&01did=83614953, Jan. 8, 2011, pp. 1-2. |
| Wikipedia, “Load Balancing (Computing)”, Available online at http://en.wikipedia.org/w/index.php?title=Load_balancing_%28computing%29&01did=446655159, Aug. 25, 2011, pp. 1-7. |
| Zhang et al., “Brushwood: Distributed Trees in Peer-to-Peer Systems”, Peer-to-Peer Systems IV Lecture Notes in Computer Science vol. 3640, 2005, pp. 47-57. |
| Gavrilov, Eugene. Distributed Database and Storage, Lections, *. Replication. Last Modified on Nov. 20, 2006. Retrieved on Jun. 28, 2017 from http://mai.pmoproject.ru/display/MSP/8.+Replication. |
| Handley, Mark, “Session Directories and Scalable Internet Multicast Address Allocation”, Proceedings of the Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, Aug. 31-Sep. 4, 1998, 12 pages. |
| Jiang Lan et al: “Consistency Maintenance in Peer-to-Peer File Sharing Networks”, Aug. 24, 2007, 11 pages. |
| International Telecommunication Union, “Series X: Data Networks and Open System Communications: OSI networking and system aspects—Naming, Addressing and Registration” Sep. 2004. ITU-T Recommendation X.667. |
| Wikipedia, “Universally Unique Identifier”, Available at http://en.wikipedia.org/wiki/Universally_unique_identifier, Aug. 24, 2011, 9 pages. |
| Extended European Search Report received in EP Application No. 19169627.7 dated Sep. 6, 2019. |
| Lohmann, Niels et al., Petri Net Transformations for Business Processes—A Survey, In: 11 Serious Games 11 2009. Springer International Publishing, Cham 032682. XP55534162, ISSN: 0302-9743, ISBN: 978-3-540-37274-5 vol. 5460. pp. 46-63. DOI: 10.1007/978-3-642-00899-3_3 section 2.3; figure 2. |
| Wikipedia, et al., URL:https://en.wikipedia.org/w/index.php?title=Query flooding&oldid=369549021 as accessed on Aug. 28, 2019. |
| Number | Date | Country | |
|---|---|---|---|
| 20140279936 A1 | Sep 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13224433 | Sep 2011 | US |
| Child | 14291223 | US |
