Patent Grant
9684598
A portion of the disclosure of this patent document may contain command formats and other computer language listings, all of which are subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
This Application is related to U.S. patent application Ser. No. 14/319,343 entitled “CACHE CONSISTENCY” filed on Jun. 30, 2014 the teachings of which application are hereby incorporated herein by reference in their entirety.
This application relates to data storage.
Computer systems are constantly improving in terms of speed, reliability, and processing capability. As is known in the art, computer systems which process and store large amounts of data typically include a one or more processors in communication with a shared data storage system in which the data is stored. The data storage system may include one or more storage devices, usually of a fairly robust nature and useful for storage spanning various temporal requirements, e.g., disk drives. The one or more processors perform their respective operations using the storage system. Mass storage systems (MSS) typically include an array of a plurality of disks with on-board intelligent and communications electronics and software for making the data on the disks available.
Companies that sell data storage systems and the like are very concerned with providing customers with an efficient data storage solution that minimizes cost while meeting customer data storage needs. It would be beneficial for such companies to have a way for reducing the complexity of implementing data storage.
Example embodiments of the present invention relate to a method, a system, and a computer program product for re-syncing a distributed cache after node disconnection. The method includes stopping cache coherency among a plurality of nodes at times a first node among the plurality of nodes is in a disconnected state and determining the first node is no longer in the disconnected state. A second cache at a second node then may dump portions of the second cache at the second node inconsistent with changes made to a first cache at the first node at times the first node was in the disconnected state.
Objects, features, and advantages of embodiments disclosed herein may be better understood by referring to the following description in conjunction with the accompanying drawings. The drawings are not meant to limit the scope of the claims included herewith. For clarity, not every element may be labeled in every Figure. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments, principles, and concepts. Thus, features and advantages of the present disclosure will become more apparent from the following detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings in which:
Traditionally, data storage systems employ distributed cache in active-active cluster implementations. Generally, a distributed cache requires synchronization of content across member nodes of an active-active cluster implementation. Conventionally, a common solution for synchronizing distributed cache is using clock synchronization between member nodes of an active-active cluster implementation. Typically, clock synchronization enables time-stamping for correct ordering of data written to distributed cache. Generally, clock synchronization enables a data storage system to pause caching and use timestamps to order the I/O requests in a correct order to know what the right data is, however, the clock synchronization degrades performance of the data storage system. Typically, clock synchronization increases the complexity of a data storage system. Conventionally, enabling a data storage system to maintain a consistent distributed cache without clock synchronization would be beneficial to the performance of the data storage system.
As understood in the art, a typical problem with active-active clusters is that if the first node reads data from the storage system and then the second node writes new data to the same location, the cache on the first node should be aware that the data was updated. Therefore, as described in U.S. patent application Ser. No. 14/319,343 entitled “CACHE CONSISTENCY” filed on Jun. 30, 2014 the teachings of which application are hereby incorporated herein by reference in their entirety, example embodiments of the present invention provide cache coherency across each node of a distributed cache. Cache coherency is a method of synchronizing the cache between nodes in an active-active cluster. It should be noted, however, that example embodiments of the present invention relate to a write through cache in which a “master” version of the data is always on the storage system (i.e., an application writes data and the data is written to the storage system and then acknowledged by to the application) which is different than a write back cache in which a “master” version of the data may be in the cache (i.e., where data is written to cache and acknowledged to application, with the data later being written to the storage system).
However, the use of a write through cache poses a problem when one of the nodes or communication links between nodes is down and requires complex synchronization algorithms. For example, in a distributed cache with two nodes synchronizing I/O operations (e.g., reads and writes), when the first node receives a write operation, it has to notify the second node that is it updating the affected chunk of data. However, until the second node acknowledges that it updated (or, in certain embodiments, does not cache the data), the first cannot acknowledge the write operation to issuing application. Here, the first node may hang is the second node fails or the communications link between the first node and the second node fails. Traditionally, the entire cache was invalidated after disconnection.
Example embodiments of the present invention provide fast synchronization after disconnection of a node (e.g., by node failure or communications link failure). In a preferred embodiment, the algorithm keeps the cache as warm (i.e., synchronized) as possible by identifying a disconnection and, at each node, keeping a list of write I/O requests. In certain embodiments, as the list keeps growing, the granularity is increased, as will be described in greater detail below (ultimately, the entire logical unit (LU) may be marked as changed). Once the disconnected node reconnects, or the failed communications link is restored, the bitmap of write I/O request changes is requested from each other node with which the connection is reestablished. Upon receiving the bitmap (i.e., list of changes), the node may invalidate and evict the data from its cache for the marked locations. Thus, after reconnection, caches do not include data that was written on other nodes. Caching then may continue.
Further, physical and logical storage provided by the storage system 100 may be shared across each node 110, thereby enabling the nodes 110 in the cluster to access such storage resources on the same storage system 100. In certain embodiments, the storage system 100 may be one or more external storage arrays, such as VMAX® and VNX® by EMC Corporation of Hopkinton, Mass. Further, in certain embodiments, the cache 115 may be a Flash card, such as XtremCache™ by EMC Corporation of Hopkinton, Mass., or any other kinds of cache.
As illustrated in
As a result of this update, the first node 1101, such as via the data management module 1201, sends an update message 145 to the second node 1102, such as to its data management module 1202, to update the cache location 1152-2 of the second node's local cache 1152 corresponding to the changed cache location 1151-2 of the first node's local cache 1151. The local caches 115 at the nodes 110 are now synchronized for cache coherency.
As described above, in traditional active-active distributed caches, upon a node disconnection, the nodes may stop caching and dump the cache as there is no unique data in the caches (i.e., the “master” data is at the storage array). As understood in the art, stopping caching until all nodes begin synchronizing causes the storage system to take a significant performance hit. Another option is to stop all I/O operations until the failed node (e.g., the first node 2101) responds because read operations on the second node 2102 may not be accurate as there is a possibility that the first node 2101 updated the data that the second node 2102 is trying to read before the change was synchronized between the nodes 210 (i.e., the second node 2102 has stale data in its cache 2152). However, this option is not desirable as node failure can be frequent and taking the entire storage system 200 offline is unnecessarily disruptive.
Rather, example embodiments of the present invention take a more balanced approach and allow the nodes 210 to continue to work while a node (e.g., the first node 2101) is in a disconnected state.
As illustrated in
As illustrated in
Then, unlike in normal cache coherency operation, the data management module 2201 may pause sending an update message (e.g., update message 145 of
The data management module 2201 then may set a bit in a bitmap 2251 for a respective chunk of the local cache 2151 at the first node 2101 changed by processing the write I/O 235 (520). Therefore, as illustrated in
As illustrated in
The data management module 3202 at the second node 3102 then may compare the changed portions of the local cache 3151 at the first node 3101 to respective portions of the local cache 3152 at the second node 3102 to determine portions of the local cache 3152 at the second node 3102 that are inconsistent with the changes made to the local cache 3151 at the first node 3101 (645). Finally, the data management module 3202 at the second node 3102 may remove from the local cache 3152 at the second node 3102 the portions of the local cache 3152 at the second node 3102 that are inconsistent with the changes made to the local cache 3151 at the first node 3101 (650).
For example, as illustrated in
One of ordinary skill in the art will appreciate that extended downtime of a node (e.g., the first node 2101 of
The data management module 2201 at the first node 2101 then may continually monitor for whether the first node 2101 has reconnected (745). If the first node 2101 has not reconnected (747), the data management module 2201 at the first node 2101 then may determine whether a number of changes made to the cache 2151 exceeds a threshold (750) requiring a change in the granularity of the chunks. If the threshold has not been met (752), the data management module 2201 at the first node 2101 may continue to set bits in the bitmap 2251 as I/Os 235 are processed to reflect chunks of the cache 2151 changed by processing the I/O 235 (740).
However, if the threshold has been met (753), the data management module 2201 at the first node 2101 may decrease the granularity of the chunks (e.g., 1 MB chunks to 1 GB chunks) (755) and reset the bits in the bitmap 2251 according to the decreased granularity (760). Losing granularity was result in the local cache 2152 at the second node 2102 dumping more data than under higher granularity after reconnection; however, the tradeoff is that some cache data can be retained after reconnection rather than having to dump the whole cache 2152. As described above, the data management module 2201 at the first node 2101 may process I/Os 235 and set a bit in the bitmap 2251 (according to the decreased granularity) corresponding to the one or more chunks of the cache 2151 changed by processing the I/O 235 (740).
If the data management module 3201 at the first node 3101 determines that the first node 3101 has reconnected to the cluster (748), the data management module 3201 at the first node 3101 may send the bitmap 3251 as a records of changes 345 made at times the first node 3101 was disconnected to the data management module 3202 at the second node 3102 (765). The data management module 3202 at the second node 3102 then may determine whether a number of inconsistencies between the local cache 3151 at the first node 3101 and the local cache 3152 at the second node 3102 exceeds a threshold such that dumping the entire local cache 3152 at the second node 3102 should be performed regardless of the record of changes 345 (770). If the threshold has not been met (772), the data management module 3202 at the second node 3102 may dump the chunks of the local cache 3152 at the second node 3102 marked in the bitmap 3251 as indicated by the records of changes 345 (775). Otherwise, if the threshold has been met (773), the data management module 3202 at the second node 3102 may dump the entirety of the local cache 3152 at the second node 3102 regardless of the record of changes 345 (785).
It should be understood that any data dumped from the local cache 3152 at the second node 3102 may be recached from the storage array 305 at a different time or, if a host tries to access the data, the second node 3102 may bring it in from the storage array 305 and put it in the local cache 3152 at the second node 3102. Although dumping data from the local cache 3152 at the second node 3102 may decrease short term performance, the long term benefits are greater.
Further, it should be understood that example embodiments of the present invention may operate under one or more of the following assumptions. First, it is likely that a node operating in a disconnected state is a transient or temporary occurrence. Therefore, the number of writes and/or updates performed during this finite period is likely low compared to an entire cache. Therefore, storage systems according to example embodiments of the present invention may pay a performance penalty at times the disconnected node is in the disconnected state. However, it is likely that most of the cache stays intact and that only a small portion of the cache will need to be synchronized and/or recached at a different time.
The methods and apparatus of this invention may take the form, at least partially, of program code (i.e., instructions) embodied in tangible non-transitory media, such as floppy diskettes, CD-ROMs, hard drives, random access or read only-memory, or any other machine-readable storage medium. When the program code is loaded into and executed by a machine, such as the computer of
The logic for carrying out the method may be embodied as part of the aforementioned system, which is useful for carrying out a method described with reference to embodiments shown. For purposes of illustrating the present invention, the invention is described as embodied in a specific configuration and using special logical arrangements, but one skilled in the art will appreciate that the device is not limited to the specific configuration but rather only by the claims included with this specification.
Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present implementations are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7120824 | Burton | Oct 2006 | B2 |
| 20040044850 | George | Mar 2004 | A1 |
| 20050108191 | Iyengar | May 2005 | A1 |
| Entry |
|---|
| Jim Handy. 1998. The Cache Memory Book. Academic Press Prof., Inc., San Diego, CA, USA. |
