Patent Application
20140279946
The present invention relates to storage technology, and, in particular embodiments, to a system and method for automatic integrity checks in a key/value store.
In some storage systems, such as distributed hash table (DHT), key/value store, and noSQL database systems, data is stored in the form of data objects, where each object includes a key and value. The key is used to identify the data object, and the value represents the data content. A data object may correspond to a single data structure or a set of data (e.g., a file or a set of files). Alternatively, the data object may correspond to a block or chunk of data, such as a portion of a file or a file from a set of files. The stored data integrity can be compromised when the data is changed or corrupted due to hardware failure, a malicious attack (e.g., by a hacker), or other reasons.
For some storage or database systems, a mechanism can be included to check the integrity of stored data. The mechanism includes associating each individual data block in a database with a corresponding message digest. The message digest is a signature that describes the uniqueness of the corresponding data block. A message digest algorithm can be used for examining the message digest of a data block to check the block integrity. The algorithm ensures that the data has not been modified or corrupted. The data that fails the block integrity check is identified to prevent the delivery of such data to a user, e.g., by disregarding such data. However, there is a need for a mechanism in a key/value store, or similar systems that store data objects with a key and value, to deliver the correct data to the user when the integrity check fails.
In accordance with an embodiment, a method for data integrity check and recovery in a key-value store system includes receiving a command to retrieve a data block stored in the key-value store system, the command indicating a key associated with the data block, retrieving one or more copies of the stored data block including the key, a value, a data integrity check algorithm, and an integrity value, and performing data integrity check on the retrieved one or more copies of the data block using the data integrity check algorithm, the integrity value, and at least one of the key and the value. The method also includes, upon one or more copies of the data block failing the data integrity check, repairing the one or more copies of the data block and delivering the data block.
In accordance with another embodiment, a method for data storage supporting data integrity check and recovery in a key-value store system includes receiving a command to store a data block in the key-value store system, the data block including a key identifying the data block and a value representing data content, calculating an integrity value for the data block using a data integrity check algorithm, adding the calculated integrity value and the data integrity check algorithm to the data block, and storing one or more copies of the data block including the key, the value, the data integrity check algorithm, and the integrity value in one or more storage nodes of the key-value store system.
In accordance with yet another embodiment, in a storage system, an apparatus for data storage supporting data integrity check and recovery in a key-value store system includes one or more storage nodes configured for storing one or more data blocks including key and value pairs, a processor, and a computer readable storage medium storing programming for execution by the processor. The programming includes instructions to receive a command to retrieve a data block stored at the one or more storage nodes, the command indicating a key associated with the data block, retrieve one or more copies of the stored data block including the key, a value, a data integrity check algorithm, and an integrity value, and perform data integrity check on the retrieved one or more copies of the data block using the data integrity check algorithm, the integrity value, and at least one of the key and the value. Upon one or more copies of the data block failing the data integrity check, the one or more copies of the data block are repaired and the data block is delivered.
For a more complete understanding of the present invention, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawing, in which:
The making and using of the presently preferred embodiments are discussed in detail below. It should be appreciated, however, that the present invention provides many applicable inventive concepts that can be embodied in a wide variety of specific contexts. The specific embodiments discussed are merely illustrative of specific ways to make and use the invention, and do not limit the scope of the invention.
System and method embodiments are provided for integrity check and recovery in a key/value store. The system and method embodiments can also be implemented for any storage system that stores data objects including key and value pairs, such as DHT and noSQL database systems. The embodiments include calculating a block integrity value for a key and value pair and storing the integrity value with the key and value pair in one or more storage nodes. Multiple copies can be stored at different locations or nodes. Upon retrieving the stored data for a user from one or more storage nodes, the integrity value is recalculated for each retrieved key and value pair and compared with the previously stored integrity value of the retrieved key and value pair to check data integrity. The integrity value can be calculated using the message digest algorithm.
A copy passes the integrity check when the two integrity values match and hence the copy can be forwarded to the user. A copy that fails the integrity check (upon detecting a mismatch between the two integrity values) is repaired and then stored and can be forwarded to the user if needed. The corrupted or changed data can be repaired using any suitable data recovery or repair mechanism, for example using one or more other stored copies that are not corrupted (e.g., that pass the integrity check). The schemes herein allow the storage system to check each storage node for data integrity and repair failures for any storage node that fails the integrity check to guarantee delivering a correct or intact version of the requested data to the user. The schemes also guarantee delivering the correct value associated with the key provided by the user.
The key/value store system is configured to store (in the storage nodes 130) data objects or blocks 110, each including a key 102 and a value 104. A block integrity value 108 and a block integrity algorithm 107 used for calculating the integrity value 108 can also be included in a field 106 in the data block 110. To store the data block 110, a user (or the system) can initiate the put operation 100, for example using a put command or function that includes the data block 110 with the key 102 and the value 104 (without the algorithm 107 and the integrity value 108).
Upon receiving the command and the data block 110, the system uses a message digest algorithm to calculate the integrity value 108, e.g., a message digest. The integrity value 108 can be calculated based on, for instance, the key 102, a key length 103 (of the key 102), a value 104, and a value length 105 (of the value 104). The resulting data block 120 that includes the block integrity algorithm 107 and the integrity value 108 is then stored in one or more storage nodes 130, where each storage node 130 stores a version or copy of the same resulting data block 110. Multiple copies can be stored in multiple storage nodes 130 to provide redundancy and resilience to errors, system failures, or data losses.
The system then performs an integrity check for each retrieved copy of the data block 110. To check the data integrity, the system uses the algorithm 107 (e.g., a message digest algorithm) to calculate an integrity value (e.g., a digest message) using the information in the data blocks 110, such as the key 102 and the value 104. The calculated integrity value is then compared to the integrity value 108 in the retrieved data block 110. The data block 110 passes the integrity check if the two integrity values match. Hence, a copy of the data block 110 is forwarded to the user. If the two values do not check, the integrity check fails. The mismatch may be cause due to change or corruption in the data (e.g., in the value 104), for example due to hardware failure or other reasons. In this case, the system does not disregard or remove the corrupted data block 110. Instead, the corrupted data block 110 is repaired using a suitable recovery mechanism, for instance using a Read-Repair operation. The mechanism may use one or more other uncorrupted copies of the data block 110 that passed the integrity check to repair the corrupted copy. If the integrity check reveals only corrupted copies, the system sends a copy after repair to the user. The repaired copy can also be stored in a storage node 110 replacing the corrupted copy.
The CPU 410 may comprise any type of electronic data processor. The memory 420 may comprise any type of system memory such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, the memory 420 may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs. In embodiments, the memory 420 is non-transitory. The mass storage device 430 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus. The mass storage device 430 may comprise, for example, one or more of a solid state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like.
The processing unit 401 also includes one or more network interfaces 450, which may comprise wired links, such as an Ethernet cable or the like, and/or wireless links to access nodes or one or more networks 480. The network interface 450 allows the processing unit 401 to communicate with remote units via the networks 480. For example, the network interface 450 may provide wireless communication via one or more transmitters/transmit antennas and one or more receivers/receive antennas. In an embodiment, the processing unit 401 is coupled to a local-area network or a wide-area network for data processing and communications with remote devices, such as other processing units, the Internet, remote storage facilities, or the like.
While this invention has been described with reference to illustrative embodiments, this description is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the invention, will be apparent to persons skilled in the art upon reference to the description. It is therefore intended that the appended claims encompass any such modifications or embodiments.
