Patent Application
20130275697
The present invention generally relates to distributed data storage systems. In particular, the present invention relates to a method of data placement in a distributed data storage system that combines high data availability with a low impact on network and data storage resources, in terms of bandwidth needed for exchange of data between network nodes and in terms of number of network nodes needed to store an item of data.
With the rapidly spreading deployment of mass data handling devices, such as video and image handling devices, reliable storage of huge amounts of data is required, for direct storage or as part of backup storage. As more and more devices are provided with network connectivity, distributed storage of data in network connected devices is considered as a cost effective solution. In such distributed data storage systems that can be deployed over non-managed networks such as on the Internet, methods have been developed that copy a same item of data to multiple network connected devices to ensure data availability and resilience to data loss. This is called data replication. Data replication has to be taken in a broad sense, and covers mere data duplication as well as usage of coding techniques such as erasure or regenerating codes (where encoded data is placed on storage devices for resilience). To cope with a risk of permanent data loss due to device failure or temporary data loss due to temporary device unavailability, a high replication factor (i.e. a high number of copies) is wished. However, to reduce costs in terms of communication and storage size needed (so-called replication costs) it is rather wished to have a low replication factor.
What is needed is a solution that achieves high level of data availability of distributed data storage that jointly considers availability requirements and replication costs.
The present invention aims at alleviating some of the inconveniences of prior art.
In a large-scale distributed data networks such as peer-to-peer networks, devices join and leave the network continuously. Each device has its own disconnection and connection behavior. Certain devices are for example always connected, while others are only connected during the day, and disconnected during the night, while others have a more random connection behavior.
For peer-to-peer networks that are based on Internet connection gateways, the gateways provided by telecommunication operators to subscribers of their services are considered as peers, which subscribers are free to switch off during nighttime or during absence.
As a peer or device in a distributed data storage system is switched off or fails, the data that it stores is no longer available. If all devices that store a particular item of data are switched off or fail, the data item is no longer available, and the data item is considered being lost for at least the time that the switched off devices are switched on again, or are respectively repaired. During this time, the availability of the stored data is thus not ensured. Increasing the data replication factor is a possible solution to this problem, but highly impacts data storage costs in terms of data communication and data storage size needed.
The objective of the current invention is to optimize the tradeoff between replication factor and storage cost for a given (desired) availability while taking into account a certain equity in distribution of the data over the devices that are part of the distributed data storage system (load balancing).
For this, the invention proposes a selection of a device set for data replication that is based on knowledge of the availability of devices. According to a particular embodiment of the invention, this availability knowledge is acquired by monitoring of the network by the network devices or by only some network devices themselves. According to a variant embodiment, this knowledge is acquired by measurement on connections to the distributed data storage network acquired from one of the network's servers, such as an Internet server. The method of acquiring the availability knowledge is either active or passive, an example of an active method is through usage of ‘ping’ messages, and an example of a passive method is using connection logs stored by a centralized server of the distributed data storage system. A device is considered being unavailable when it is disconnected, either temporarily or permanently due to a breakdown.
In order to optimize data storing in a distributed data storage system, the invention proposes a method of data storing in a distributed data storage system that comprises network devices that are at least used as storage devices. That the devices are at least used as storage devices means that the devices are used as storage devices for the distributed storage network, but can also be used at a same time for other purposes, such as reception of audio and video programs. As an example, the devices may be Set Top Boxes for reception of audio and video programs, or gateways giving access to an external network, personal computers, or handheld mobile devices. The method comprises a step of reception of a request to store a data item in the distributed data storage system, a step of a first selection of a first network device as a reference device and determination of an availability and of an unavailability over time of the reference device, a step of second selection of at least one second network device as a function of a correspondence of an availability over time of the at least one second network device to the availability over time of the reference device third selection of at least one third network device as a function of a correspondence of an availability over time of the at least one third network device to an unavailability over time of the reference device and storage of the data item on the at least one second and at least one third network devices selected in the second and the third selection steps.
According to a variant embodiment, k is the number of network devices over which the data item is to be replicated, k being specified in the request to store the data item, and the first, the second and the third selection steps are being repeated until the data item is replicated over at least k network devices, the first selection step of a reference device excluding a selection of a network device that has already been selected as a reference device in a previous iteration of the method.
According to a variant embodiment, k is the number of network devices over which the data item is to be replicated, k being specified in the request to store the data item, and the second and the third selection steps are being repeated until the data item is replicated over at least k network devices.
According to a variant embodiment, the unavailability is stored as a vector (24) of which each element is representative of a unit of time, and where a number of elements of the vector (24) is representative of a time span, and where each element value is representative of an availability during the unit of time of a network device to which the vector (24) is associated, a first predetermined element value being representative of availability, and a second predetermined element value being representative of unavailability.
According to a variant embodiment, the first predetermined element value is a positive predetermined value, and the second predetermined value is a negative predetermined value, and the selection of the second network device as a function of the unavailability of the reference device is determined according to level of anti-correlation determined through a calculation of an angle between two of the vectors (24), two vectors (24) being highly correlated when the angle approaches zero, and two vectors (24) being highly anti-correlated when the angle approaches π, the value of the angle being representative of the level of anti-correlation.
According to a variant embodiment, the angle is calculated according to:
where θ is the angle, and x and y are the two vectors.
According to a variant embodiment, the first predetermined element value is a binary one, and the second predetermined value is a binary zero, and a level of anti-correlation is determined through a calculation of a logical Boolean XOR operation between two of the vectors (24), two vectors (24) being highly anti-correlated when a result of the logical Boolean XOR operation comprises a high number of 1's that approaches the number of the elements of each of the two vectors (24), and two vectors (24) being highly correlated when the result of the logical Boolean XOR operation comprises a number of 1's that approaches 0, the number of 1's in the result of the Boolean XOR operation being representative of the level of anti-correlation.
According to a variant embodiment, the reference network device is selected randomly from the network devices.
According to a variant embodiment, the reference network device is selected deterministically from the network devices. As an example, a round-robin selection is a deterministic selection.
In order to optimize data storing in a distributed data storage system, the invention also comprises a device for data storing in a distributed data storage system comprising network devices that are at least used as storage devices, the device comprising the following means: a receiver for receiving of a request to store a data item in the distributed data storage system; a selector for a first selection of a first network device as a reference network device and means for determination of an availability and of an unavailability over time of the reference device; a selector for a second selection, of at least one second network device as a function of a correspondence of an availability over time of the second network device to an availability over time of the reference network device; a selector for a third selection of at least one third network device as a function of a correspondence of an availability over time of the at least one third network device to an unavailability over time of the reference network device: and storage means for storing of the data item on the network devices selected by the selector for a second selection and by the selector for the third selection.
More advantages of the invention will appear through the description of particular, non-restricting embodiments of the invention.
The embodiments will be described with reference to the following figures:
Other types of networks than illustrated in
The invention can be implemented in any type of network device, such as a home gateway connecting a home network to the internet, a web server, a video server, or a wireless device, handheld or not.
According to a variant embodiment, the sample time and/or sample period are predetermined. According to a more advantageous embodiment, the sample time and period are determined after measurements on connectivity behavior of the distributed data storage devices Then, the best sample time and sample period can be determined from these measurements that captures best recurrent periods of unavailability of devices. For example, a recurrency may be observed when the sample period is set to one week, so as to comprise recurrent day/night and weekend availability/unavailability alternations. According to a variant embodiment, the distributed storage devices can execute the task of determining a good sample time and period in a collaborative manner, thereby relieving a centralized server from this task.
Thus, each device has its own availability vector, according to a common format, such as the previous described format. According to a variant embodiment, each device stores its own availability vector. This variant has the advantage to avoid centralized monitoring of device availability and reduces the need for communication between devices and the centralized availability storage.
The objective is now to build a devices set that, when data is replicated over the replication set, ensures availability of the data through availability of at least one device of the set over the whole sample period. An example is illustrated in
This formula gives a value for the angle θ between a vector x and a vector y. When the angle approaches 0, the vectors are correlated. When the angle approaches π, the vectors are anti-correlated. Two anti-correlated vectors represent two devices that have opposite behavior, for example a device that is uniquely available during the day, whereas the other is uniquely available during the night. Then, for a given device with a given availability vector, if we select for a replication set at least one device having an availability vector that is most correlated with the given device and at least one device having an availability vector that is most anti-correlated with the given device, the sample period is covered with better performance than prior-art methods of random selection of devices to comprise a replication set. According to prior art, an equivalent or even lower level of availability is obtained by increasing the number of devices in the replication set.
According to a variant embodiment of the invention, correlated and anti-correlated devices are not detected as previously described by an angle between vectors, but rather through logical binary operations. Like for the previous described variant embodiment, a device's availability data is translated in a vector where each index represents a sample time, for example 1 h, and the vector length represents a sample period for example 24 h. But where the vector is filled with +1 or −1 according to the previous variant embodiment, in the current described variant embodiment the vector is filled with zeros and ones, a 0 indicating unavailability during the sample time and a 1 indicating availability during the sample time. For example:
Device i: 010110010010010100100100
Perfect anti-correlated device: 101001101101101011011011
According to this variant embodiment, a perfectly anti-correlated device of a device i is a device that has logical 1's where device i has 0's and 0's where device i has 1's. According to this variant embodiment, a selection between anti-correlated devices is done through comparing the results of a logical XOR operation between availability vector of a device i and the availability vectors of candidates for anti-correlation. The more 1's the results of the XOR operation comprises, the more the devices are anti-correlated. This method results in filling up the gaps in the availability of the reference device i with devices that minimize their overlap in availability with device i. As the previously described variant embodiment based on measurement on angle between vectors, this variant embodiment distributes the data well over the distributed data storage system rather than resulting in poor distribution of data over a limited selection of devices that are available over the sample period without interruption, which has an advantage of a better spreading of risks of grouped data loss through the differentiation of the devices that are included in replication sets associated to different data items. Like with the previously described ‘angle’ variant embodiment where a measured angle is representative of the level of anti-correlation, the current described variant the number of 1's in the result of the XOR operation between the availability vectors of two devices is a measure for the level of anti-correlation. For example, if the vectors comprise 24 elements, twenty-four 1's represents the highest possible (perfect) level of anti-correlation, and zero 1's represents the lowest possible level of anti-correlation. Intermediate numbers are representative of intermediate levels of anti-correlation. Of course, the levels of anti-correlation can be scaled to any given scale, for example on a scale from 1 to 10 through applying simple arithmetic operations.
Of course, the above variant embodiments of the invention supposes a certain predictability of the availability behavior of the devices, since the embodiments use availability data measured in the past. However, tests have proven that in practice, the availability behavior of devices is rather cyclic; for example over a 24 h period or over a 7 day period. When the availability behavior of the devices is rather unpredictable, the method of the invention still obtains availability that is as good as pure random selection methods of storage devices, because each reference device i, as has been mentioned, is selected randomly.
According to another variant embodiment, further referred to as the ‘patching’ variant, instead of selecting a certain number of correlated devices and a certain number of anti-correlated devices, a single anti-correlated device with i is chosen. The data item is copied on these devices; if k>2, the process is repeated with the selection of a ‘new’ reference device i and search for a new device that is anti-correlated with i, and copying of the data item on these devices. The process is repeated until k is attained, this selection of devices boils down to repeating the selection of devices including the selection of the reference device i.
Both variants are an improvement to prior art, and the ‘group’ variant is particularly suited for having always t encoded blocks among n (for reasonable values of n), which is very well adapted to erasure codes for example, which require the availability of t erasure code blocks among n in order to cope with a failure. The ‘patching’ variant is more efficient to keep continuously at least one copy of a data item available, because the ‘group’ variant can at times result in some temporal gaps, i.e. gives a lesser guarantee than the patching variant that there is always at least one device available that stores a given item of data.
As mentioned above, according to a variant embodiment of the invention, when a device “j” enters the distributed data storage system to store data, the system determines a reference device “i” that will serve as a basis for determination of the replication set comprising correlated and anti-correlated devices. According to a variant embodiment discussed above, the reference device “i” is chosen randomly by a central server in the network. According to another variant embodiment, the reference device “i” is determined in a deterministic manner, so as to be sure not to more than once a same reference device, and thus resulting in a better distribution of data over the network. According to yet another particular embodiment of the invention, both variants are combined so as to choose randomly a reference device i from a set of devices in which previously selected reference devices are excluded, resulting in an even better distribution of data over the distributed data storage network.
According to a variant embodiment of the invention, the number of replicas is dependent on the capacity of the replication set to “cover” a whole sample period. According to this variant, the steps of finding anti-correlated devices with the resulting availability of the already chosen device set are repeated until there is no more gap in availability.
The device 60 comprises a Read-Only Memory (ROM) 600, a Random Access Memory (RAM) 610, a Central Processing Unit (CPU) 620, a clock 630, a network interface 640, a graphics interface 650, and a user interface 660. All of these components are interconnected via a data- and communication bus 670. CPU 620 controls the device 60 according to a program that is stored in ROM 600 in a memory zone 601. Clock device 630 provides the components of device 60 with a common timing in order to sequence and synchronize their operation. Network interface 640 receives and transmits data with external devices via connection 6000. Graphics interface 650 is connected to an external rendering device via connection 6001. User interface 660 receives input commands from a user via connection 6002. ROM memory zone 601 comprises instructions that implement the method of the invention. Upon powering on of device 60, CPU 620 copies the program ‘Prog’ from ROM memory zone 601 into RAM memory zone 611, and executes the copied program. Upon execution of the copied program, the program allocates memory space for variables needed for its execution in RAM memory zone 615, and copies list of device addresses 602, device availability vectors 603, and device sets 604 into respective RAM memories 612-614. The device 60 is now operational and upon reception of a request for storing of a data item, the method of the invention is activated, which method is part of the program stored in ROM 600 memory zone 601. The network interface (640) serves as a receiver for reception of requests to store a data item. The CPU (620) serves among others as a selector for making a first selection of one of the network devices as a reference device. CPU (620) also serves as a selector for a second selection of at least one network device that has an availability over the predetermined time period that is anti-correlated to an availability determined for the reference network device, and CPU (620) serves as a means for adding of the at least one selected network device to the replication set for storing of the data item. ROM memory 600 and RAM memory 610 are storage means for storing of the data item on the network devices in the replication set.
According to a variant embodiment, the invention is entirely implemented in hardware, for example as a dedicated component (for example as an ASIC, FPGA or VLSI) (respectively <<Application Specific Integrated Circuit>>, <<Field-Programmable Gate Array>> and <<Very Large Scale Integration>>) or as distinct electronic components integrated in a device or in a form of a mix of hardware and software.
Then, in a step 704, a network device is selected. This selected device is then added to the replication set. The notion of correlation has already been explained previously. In a next step 705, the data item is replicated (copied) to the devices of the replication set, and the algorithm continues with repetition of step 701. In each step 704, k/2 devices are selected for the replication set, while again k/2 correlated devices will be chosen later on (this is the previously described ‘group’ variant). According to a variant embodiment (the previously described ‘patching’ variant), one device is selected in each of the step 704, and steps 702-704 are repeated until k devices are selected, so that in both variants, but in a different manner, the desired replication factor is obtained. Each of the ‘group’ and the ‘patching’ variant has its own advantages as has been previously discussed.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10305999.4 | Sep 2010 | EP | regional |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/EP2011/066256 | 9/20/2011 | WO | 00 | 5/28/2013 |
