Patent Application
20250165177
Contained herein is material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction of the patent disclosure by any person as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all rights to the copyright whatsoever. Copyright @2021, NetApp, Inc.
Various embodiments of the present disclosure generally relate to multi-site distributed data storage systems. In particular, some embodiments relate to usage of operation (OP) logs for fast recovery of bidirectional synchronous replication in a dual copy data storage system with simultaneous read-write ability on each copy.
Multiple storage nodes organized as a cluster may provide a distributed storage architecture configured to service storage requests issued by one or more clients of the cluster. The storage requests are directed to data stored on storage devices coupled to one or more of the storage nodes of the cluster. The data served by the storage nodes may be distributed across multiple storage units embodied as persistent storage devices, such as hard disk drives (HDDs), solid state drives (SSDs), flash memory systems, or other storage devices. The storage nodes may logically organize the data stored on the devices as volumes accessible as logical units. Each volume may be implemented as a set of data structures, such as data blocks that store data for the volume and metadata blocks that describe the data of the volume.
Business enterprises rely on multiple clusters for storing and retrieving data. Each cluster may be a separate data center with the clusters able to communicate over an unreliable network. The network can be prone to failures leading to connectivity issues such as transient or persistent connectivity issues that disrupt operations of a business enterprise.
Systems and methods are described for performing persistent inflight tracking of operations (Ops) and resynchronization between storage objects within a cross-site storage solution. According to one embodiment, a method performed by one or more processing resources of a distributed storage system comprises maintaining state information regarding a data synchronous replication status for a storage object of a primary storage cluster with the storage object being replicated to a replicated storage object of a secondary storage cluster. The method includes temporarily disallowing input/output (I/O) operations for the storage object when the storage object of the primary storage cluster has a failure, which causes an internal state as out of sync for the storage object of the primary storage cluster while maintaining an external state as in sync for external entities in order to provide time for reestablishing synchronous replication within duration of an operation (Op) timeout period. The method further includes performing persistent inflight tracking and reconciliation of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster and performing a resynchronization between the storage object and the replicated storage object based on the persistent inflight tracking and reconciliation of I/O operations with the first Op log of the primary storage cluster and the second Op log of the secondary storage cluster.
According to another embodiment, a method performed by one or more processing resources of a distributed storage system comprises establishing bi-directional synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiating a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 due to a loss of the bi-directional synchronous replication between the one or more storage objects of the CG1 and the one or more storage objects of the CG2, and performing the resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.
Other features of embodiments of the present disclosure will be apparent from accompanying drawings and detailed description that follows.
In the Figures, similar components and/or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label with a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.
Systems and methods are described for efficiently tracking and reconciling inflight operations and fast resynchronization within a cross-site distributed storage system to provide zero recovery point objective (RPO) protection. In the context of cross-site distributed storage system (including cross-site HA storage solutions that perform synchronous data replication to support zero RPO protection), a certain degree of consistency over time is maintained between a mirror copy and a primary dataset depending upon the particular implementation. Certain operations on a storage object (e.g., data container/volume) of a consistency group (CG) having numerous data containers/volumes hosting the data at issue should be managed independently for an out of sync storage object in order to avoid transitioning all storage objects of a CG out of sync and avoid delays needed for transitioning all storage objects with resynchronization from out of sync back in sync.
In one example, a primary and a secondary storage cluster are diverged due to inflight I/O operations (ops) that are not yet acknowledged to a client device. An inflight op is an op that is in progress on either primary or secondary storage cluster and its response is held by a synchronous replication circuitry (SR circuitry), which includes a splitter component (or replicating component). An inflight Op can be a data Op (e.g., write, punch hole, etc.) or a metadata op (e.g., create, unlink, set attribute, etc.). An inflight Op can have the following states:
A splitter component can include a queue to store incoming operations and a splitter object that is configured to split (replicate) operations targeting a storage object. The splitter object replicates the operations to a replicated storage object of the second storage cluster. Operations that been acknowledged to the client device have been executed by a storage cluster and hence committed on both primary and secondary endpoints for the primary and secondary storage clusters. However, at a given instance of time, one or more Ops could be in flight i.e., executed on neither of endpoints (e.g., first storage object hosted by primary storage cluster, replicated second storage object hosted by secondary storage cluster), both of the endpoints, or executed on one of the endpoints. As a consequence, the primary and second storage clusters at a given point in time could be divergent with respect to inflight Ops. A common snapshot may be performed periodically to serve as resynchronization points.
Data operations are designed with an idempotent property while metadata operations are designed with a non-idempotent property. To address this divergence, the present design when in a state of synchronous replication (in sync state) will persistently track inflight operations. Also, before opening up a storage object for I/O operations of a client device, the present design will replace the inflight operations prior to resuming synchronous replication.
In another example, the present design in order to maintain a secondary storage cluster as being failover capable will cause an out of sync (OOS) event for a data container/volume to be an internal event and will also avoid write order inconsistency. The present design disallows I/O operations on the OOS data container/volume until resumption of synchronous replication and this avoids dependent write order inconsistencies between the OOS data container/volume and a replicated data container/volume. The present design performs a fast establishment of a transfer engine for resynchronization and keeps the CG in sync while one or more data containers/volume in the CG having an internal event to indicate OOS.
Embodiments described herein seek to improve various technological processes associated with cross-site storage solutions and ensure the process of quickly establishing resynchronization between a storage object (e.g., a first storage object) of a primary storage cluster and a replicated storage object (e.g., a second storage object) of a secondary storage cluster. Various embodiments of the present technology provide for a wide range of technical effects, advantages, and/or improvements to stretched storage systems and participating distributed storage systems. For example, various embodiments may include one or more of the following technical effects, advantages, and/or improvements: (i) provide an Op log on a primary storage cluster and another Op log on the secondary copy. Both Op logs will specify which operations are committed on each of the storage clusters. These two Op logs can be used to find how the two filesystems are differing and carry out resynchronization; (ii) maintain the benefit of parallel splitting of Ops to primary and secondary storage clusters in synchronous replication while adding support for Op logging on both primary and secondary storage clusters; and (iii) start Op logging early and marking the Op log usable once the synchronous replication relationship between a storage object and a replicated storage object enters In Sync state. Early engagement of Op logging avoids I/O latency for a client device. One or more of which may include various additional optimizations described further below.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present disclosure. It will be apparent, however, to one skilled in the art that embodiments of the present disclosure may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form.
Brief definitions of terms used throughout this application are given below.
A “computer” or “computer system” may be one or more physical computers, virtual computers, or computing devices. As an example, a computer may be one or more server computers, cloud-based computers, cloud-based cluster of computers, virtual machine instances or virtual machine computing elements such as virtual processors, storage and memory, data centers, storage devices, desktop computers, laptop computers, mobile devices, or any other special-purpose computing devices. Any reference to “a computer” or “a computer system” herein may mean one or more computers, unless expressly stated otherwise.
The terms “connected” or “coupled” and related terms are used in an operational sense and are not necessarily limited to a direct connection or coupling. Thus, for example, two devices may be coupled directly, or via one or more intermediary media or devices. As another example, devices may be coupled in such a way that information can be passed there between, while not sharing any physical connection with one another. Based on the disclosure provided herein, one of ordinary skill in the art will appreciate a variety of ways in which connection or coupling exists in accordance with the aforementioned definition.
If the specification states a component or feature “may”, “can”, “could”, or “might” be included or have a characteristic, that particular component or feature is not required to be included or have the characteristic.
As used in the description herein and throughout the claims that follow, the meaning of “a,” “an,” and “the” includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.
The phrases “in an embodiment,” “according to one embodiment,” and the like generally mean the particular feature, structure, or characteristic following the phrase is included in at least one embodiment of the present disclosure, and may be included in more than one embodiment of the present disclosure. Importantly, such phrases do not necessarily refer to the same embodiment.
In the context of the present example, the multi-site distributed storage system 102 includes a data center 130, a data center 140, and optionally a mediator 120. The data centers 130 and 140, the mediator 120, and the computer system 110 are coupled in communication via a network 105, which, depending upon the particular implementation, may be a Local Area Network (LAN), a Wide Area Network (WAN), or the Internet.
The data centers 130 and 140 may represent an enterprise data center (e.g., an on-premises customer data center) that is owned and operated by a company or the data center 130 may be managed by a third party (or a managed service provider) on behalf of the company, which may lease the equipment and infrastructure. Alternatively, the data centers 130 and 140 may represent a colocation data center in which a company rents space of a facility owned by others and located off the company premises. The data centers are shown with a cluster (e.g., cluster 135, cluster 145). Those of ordinary skill in the art will appreciate additional IT infrastructure may be included within the data centers 130 and 140. In one example, the data center 140 is a mirrored copy of the data center 130 to provide non-disruptive operations at all times even in the presence of failures including, but not limited to, network disconnection between the data centers 130 and 140 and the mediator 120, which can also be located at a data center.
Turning now to the cluster 135, it includes multiple storage nodes 136a-n and an Application Programming Interface (API) 137. In the context of the present example, the multiple storage nodes 136a-n are organized as a cluster and provide a distributed storage architecture to service storage requests issued by one or more clients (not shown) of the cluster. The data served by the storage nodes 136a-n may be distributed across multiple storage units embodied as persistent storage devices, including but not limited to HDDs, SSDs, flash memory systems, or other storage devices. In a similar manner, cluster 145 includes multiple storage nodes 146a-n and an Application Programming Interface (API) 147. In the context of the present example, the multiple storage nodes 146a-n are organized as a cluster and provide a distributed storage architecture to service storage requests issued by one or more clients of the cluster.
The API 137 may provide an interface through which the cluster 135 is configured and/or queried by external actors (e.g., the computer system 110, data center 140, the mediator 120, clients). Depending upon the particular implementation, the API 137 may represent a Representational State Transfer (REST)ful API that uses Hypertext Transfer Protocol (HTTP) methods (e.g., GET, POST, PATCH, DELETE, and OPTIONS) to indicate its actions. Depending upon the particular embodiment, the API 137 may provide access to various telemetry data (e.g., performance, configuration, storage efficiency metrics, and other system data) relating to the cluster 135 or components thereof. As those skilled in the art will appreciate various other types of telemetry data may be made available via the API 137, including, but not limited to measures of latency, utilization, and/or performance at various levels (e.g., the cluster level, the storage node level, or the storage node component level).
In the context of the present example, the mediator 120, which may represent a private or public cloud accessible (e.g., via a web portal) to an administrator associated with a managed service provider and/or administrators of one or more customers of the managed service provider, includes a cloud-based, monitoring system.
While for sake of brevity, only two data centers are shown in the context of the present example, it is to be appreciated that additional clusters owned by or leased by the same or different companies (data storage subscribers/customers) may be monitored and one or more metrics may be estimated based on data stored within a given level of a data store in accordance with the methodologies described herein and such clusters may reside in multiple data centers of different types (e.g., enterprise data centers, managed services data centers, or colocation data centers).
In the context of the present example, the system 202 includes data center 230, data center 240, and optionally a mediator 220. The data centers 230 and 240, the mediator 220, and the computer system 210 are coupled in communication via a network 205, which, depending upon the particular implementation, may be a Local Area Network (LAN), a Wide Area Network (WAN), or the Internet.
The data centers 230 and 240 may represent an enterprise data center (e.g., an on-premises customer data center) that is owned and operated by a company or the data center 230 may be managed by a third party (or a managed service provider) on behalf of the company, which may lease the equipment and infrastructure. Alternatively, the data centers 230 and 240 may represent a colocation data center in which a company rents space of a facility owned by others and located off the company premises. The data centers are shown with a cluster (e.g., cluster 235, cluster 245). Those of ordinary skill in the art will appreciate additional IT infrastructure may be included within the data centers 230 and 240. In one example, the data center 240 is a mirrored copy of the data center 230 to provide non-disruptive operations at all times even in the presence of failures including, but not limited to, network disconnection between the data centers 230 and 240 and the mediator 220, which can also be a data center.
The system 202 can utilize communications 290 and 291 to synchronize a mirrored copy of data of the data center 240 with a primary copy of the data of the data center 230. Either of the communications 290 and 291 between the data centers 230 and 240 may have a failure 295. In a similar manner, a communication 292 between data center 230 and mediator 220 may have a failure 296 while a communication 293 between the data center 240 and the mediator 220 may have a failure 297. If not responded to appropriately, these failures whether transient or permanent have the potential to disrupt operations for users of the distributed storage system 202. In one example, communications between the data centers 230 and 240 have approximately a 5-20 millisecond round trip time.
Turning now to the cluster 235, it includes at least two storage nodes 236a-b, optionally includes additional storage nodes (e.g., 236n) and an Application Programming Interface (API) 237. In the context of the present example, the multiple storage nodes are organized as a cluster and provide a distributed storage architecture to service storage requests issued by one or more clients of the cluster. The data served by the storage nodes may be distributed across multiple storage units embodied as persistent storage devices, including but not limited to HDDs, SSDs, flash memory systems, or other storage devices.
Turning now to the cluster 245, it includes at least two storage nodes 246a-b, optionally includes additional storage nodes (e.g., 246n) and includes an Application Programming Interface (API) 247. In the context of the present example, the multiple storage nodes are organized as a cluster and provide a distributed storage architecture to service storage requests issued by one or more clients of the cluster. The data served by the storage nodes may be distributed across multiple storage units embodied as persistent storage devices, including but not limited to HDDs, SSDs, flash memory systems, or other storage devices.
In one example, each cluster can have up to 5 CGs with each CG having up to 12 volumes. The system 202 provides a planned failover feature at a CG granularity. The planned failover feature allows switching storage access from a primary copy of the data center 230 to a mirror copy of the data center 240 or vice versa.
The cluster 310 includes nodes 311 and 312 while the cluster 320 includes nodes 321 and 322. In one example, the cluster 320 has a data copy 330 in node 321 that is a mirrored copy of data copy 330 in node 311. A data copy 331 in node 322 is a mirrored copy of the data copy 331 in node 312 to provide non-disruptive operations at all times even in the presence of failures including, but not limited to, network disconnection between the data centers 302 and 304 and the mediator 360.
The multi-site distributed storage system 300 provides correctness of data, availability, and redundancy of data. In one example, the nodes 311 and 312 are designated as a master and the nodes 321 and 322 are designated as a slave. The master is given preference to serve I/O commands to requesting clients and this allows the master to obtain a consensus in a case of a race between the clusters 310 and 320. The mediator 360 enables an automated unplanned failover (AUFO) in the event of a failure. The data copy 330 (master), data copy 331 (slave), and the mediator 360 form a three way quorum. If two of the three entities reach an agreement for whether the master or slave should serve I/O commands to requesting clients, then this forms a strong consensus.
In one embodiment, node 311 has a failure and the data copy 331 for a storage object of node 312 remains in sync. The node 312 handles a takeover operation for data copy 330 (master). Upon a volume mount time, the node 311 temporarily disallows input/output operations (e.g., both read and write) with a retriable error. The I/O operations from a computer system 308 are not allowed at node 311 until resynchronization occurs or a timeout occurs.
Next, the cluster 320 performs an automatic Fast Resynchronization (Fast Resync) to maintain zero recovery point objective (RPO) protection. The Fast Resync is based on reestablishing a Sync Data Path between data copy 330 (master) of node 311 and data copy 330 (slave) of mirrored node 321, and reconciling inflight regions based on persistent inflight tracking of I/O operations (IFT-P). The secondary storage cluster 320 can be provided with necessary information about a high availability partner to avoid cross-cluster calls between the primary and secondary storage cluster. Note, no asynchronous transfers and transition are allowed during the Fast Resync, which will establish a transfer engine session and start persistent inflight op tracking replay. A Fast Resync can be triggered as soon a storage object on the secondary storage cluster is mounted.
Subsequently, node 311 waits for resumption of synchronous replication and allows I/O upon completion of the synchronous replication.
If Fast Resync experiences an error or failure resulting in the Fast Resync not being possible within a certain time period (e.g., 30-90 seconds, 60 seconds), then the following phases occur:
The master and slave roles for the clusters 310 and 320 help to avoid a split-brain situation with both of the clusters simultaneously attempting to serve I/O commands. There are scenarios where both master and slave copies can claim to be a master copy. For example, a recovery post failover or failure during planned failover workflow can results in both clusters 310 and 320 attempting to serve I/O commands. In one example, a slave cannot serve I/O until an AUFO happens. A master doesn't serve I/O commands until the master obtains a consensus.
The multi-site distributed storage system 300 presents a single virtual logical unit number (LUN) to a host computer or client using a synchronized-replicated distributed copies of a LUN. A LUN is a unique identifier for designating an individual or collection of physical or virtual storage devices that execute input/output (I/O) commands with a host computer, as defined by the Small System Computer Interface (SCSI) standard. In one example, active or passive access to this virtual LUN causes read and write commands to be serviced only by node 311 (master) while operations received by the node 321 (slave) are proxied to node 311.
Each slice service 420 may include one or more volumes (e.g., volumes 421a-x, volumes 421c-y, and volumes 421e-z). Client systems (not shown) associated with an enterprise may store data to one or more volumes, retrieve data from one or more volumes, and/or modify data stored on one or more volumes.
The slice services 420a-n and/or the client system may break data into data blocks. Block services 415a-q and slice services 420a-n may maintain mappings between an address of the client system and the eventual physical location of the data block in respective storage media of the storage node 400. In one embodiment, volumes 421 include unique and uniformly random identifiers to facilitate even distribution of a volume's data throughout a cluster (e.g., cluster 135). The slice services 420a-n may store metadata that maps between client systems and block services 415. For example, slice services 420 may map between the client addressing used by the client systems (e.g., file names, object names, block numbers, etc. such as Logical Block Addresses (LBAs)) and block layer addressing (e.g., block IDs) used in block services 415. Further, block services 415 may map between the block layer addressing (e.g., block identifiers) and the physical location of the data block on one or more storage devices. The blocks may be organized within bins maintained by the block services 415 for storage on physical storage devices (e.g., SSDs).
As noted above, a bin may be derived from the block ID for storage of a corresponding data block by extracting a predefined number of bits from the block identifiers. In some embodiments, the bin may be divided into buckets or “sublists” by extending the predefined number of bits extracted from the block identifier. A bin identifier may be used to identify a bin within the system. The bin identifier may also be used to identify a particular block service 415a-q and associated storage device (e.g., SSD). A sublist identifier may identify a sublist with the bin, which may be used to facilitate network transfer (or syncing) of data among block services in the event of a failure or crash of the storage node 400. Accordingly, a client can access data using a client address, which is eventually translated into the corresponding unique identifiers that reference the client's data at the storage node 400.
For each volume 421 hosted by a slice service 420, a list of block IDs may be stored with one block ID for each logical block on the volume. Each volume may be replicated between one or more slice services 420 and/or storage nodes 400, and the slice services for each volume may be synchronized between each of the slice services hosting that volume. Accordingly, failover protection may be provided in case a slice service 420 fails, such that access to each volume may continue during the failure condition.
According to some embodiments, various operations (e.g., data replication, data migration, data protection, failover, and the like) may be performed at the level of granularity of a CG (e.g., CG 115a or CG 115b). A CG is a collection of storage objects or data containers (e.g., volumes) within a cluster that are managed by a Storage Virtual Machine (e.g., SVM 111a or SVM 111b) as a single unit. In various embodiments, the use of a CG as a unit of data replication guarantees a dependent write-order consistent view of the dataset and the mirror copy to support zero RPO and zero RTO. CGs may also be configured for use in connection with taking simultaneous snapshot images of multiple volumes, for example, to provide crash-consistent copies of a dataset associated with the volumes at a particular point in time. The level of granularity of operations supported by a CG is useful for various types of applications. As a non-limiting example, consider an application, such as a database application, that makes use of multiple volumes, including maintaining logs on one volume and the database on another volume.
The volumes of a CG may span multiple disks (e.g., electromechanical disks and/or SSDs) of one or more storage nodes of the cluster. A CG may include a subset or all volumes of one or more storage nodes. In one example, a CG includes a subset of volumes of a first storage node and a subset of volumes of a second storage node. In another example, a CG includes a subset of volumes of a first storage node, a subset of volumes of a second storage node, and a subset of volumes of a third storage node. A CG may be referred to as a local CG or a remote CG depending upon the perspective of a particular cluster. For example, CG 115a may be referred to as a local CG from the perspective of cluster 110a and as a remote CG from the perspective of cluster 110b. Similarly, CG 115a may be referred to as a remote CG from the perspective of cluster 110b and as a local CG from the perspective of cluster 110b. At times, the volumes of a CG may be collectively referred to herein as members of the CG and may be individually referred to as a member of the CG. In one embodiment, members may be added or removed from a CG after it has been created.
A cluster may include one or more SVMs, each of which may contain data volumes and one or more logical interfaces (LIFs) (not shown) through which they serve data to clients. SVMs may be used to securely isolate the shared virtualized data storage of the storage nodes in the cluster, for example, to create isolated partitions within the cluster. In one embodiment, an LIF includes an Internet Protocol (IP) address and its associated characteristics. Each SVM may have a separate administrator authentication domain and can be managed independently via a management LIF to allow, among other things, definition and configuration of the associated CGs.
In the context of the present example, the SVMs make use of a configuration database (e.g., replicated database (RDB) 112a and 112b), which may store configuration information for their respective clusters. A configuration database provides cluster wide storage for storage nodes within a cluster. The configuration information may include relationship information specifying the status, direction of data replication, relationships, and/or roles of individual CGs, a set of CGs, members of the CGs, and/or the mediator. A pair of CGs may be said to be “peered” when one is protecting the other. For example, a CG (e.g., CG 115b) to which data is configured to be synchronously replicated may be referred to as being in the role of a destination CG, whereas the CG (e.g., CG 115a) being protected by the destination CG may be referred to as the source CG. Various events (e.g., transient or persistent network connectivity issues, availability/unavailability of the mediator, site failure, and the like) impacting the stretch cluster may result in the relationship information being updated at the cluster and/or the CG level to reflect changed status, relationships, and/or roles.
While in the context of various embodiments described herein, a volume of a CG may be described as performing certain actions (e.g., taking other members of a CG out of synchronization, disallowing/allowing access to the dataset or the mirror copy, issuing consensus protocol requests, etc.), it is to be understood such references are shorthand for an SVM or other controlling entity, managing or containing the volume at issue, performing such actions on behalf of the volume.
While in the context of various examples described herein, data replication may be described as being performed in a synchronous manner between a paired set of CGs associated with different clusters (e.g., from a primary or master cluster to a secondary or slave cluster), data replication may also be performed asynchronously and/or within the same cluster. Similarly, a single remote CG may protect multiple local CGs and/or multiple remote CGs may protect a single local CG. For example, a local CG can be setup for double protection by two remote CGs via fan-out or cascade topologies. In addition, those skilled in the art will appreciate a cross-site high-availability (HA) solution may include more than two clusters, in which a mirrored copy of a dataset of a primary (master) cluster is stored on more than one secondary (slave) cluster.
While a given CG is in the InSync state, the mirror copy of the primary dataset associated with the member volumes of the given CG may be said to be in-synchronization with the primary dataset and asynchronous data replication or synchronous data replication, as the case may be, are operating as expected. When a given CG is in the OOS state, the mirror copy of the primary dataset associated with the member volumes of the given CG may be said to be out-of-synchronization with the primary dataset and asynchronous data replication or synchronous data replication, as the case may be, are unable to operate as expected. Information regarding the current state of the data replication status of a CG may be maintained in a configuration database (e.g., RDB 512a or 512b).
As noted above, in various embodiments described herein, the members (e.g., volumes) of a CG are managed as a single unit. In the context of the present example, the data replication status of a given CG is dependent upon the data replication status of the individual member volumes of the CG. A given CG may transition 611 from the InSync state to the not ready for resync state 621 of the OOS state responsive to any member volume of the CG becoming OOS with respect to a peer volume with which the member volume is peered. A given CG may transition 622 from the not ready for resync state 621 to the ready for resync state 623 responsive to all member volumes being available. In order to support recovery from, among other potential disruptive events, manual planned disruptive events (e.g., balancing of CG members across a cluster) a resynchronization process is provided to promptly bring the CG back into the InSync state from the OOS state. Responsive to a successful CG resync, a given CG may transition 624 from the ready for resync state 623 to the InSync state.
Although outside the scope of the present disclosure, for completeness it is noted that additional state transitions may exist. For example, in some embodiments, a given CG may transition from the ready for resync state 623 to the not ready for resync state 621 responsive to unavailability of a mediator (e.g., mediator 120) configured for the given CG. In such an embodiment, the transition 622 from the not ready for resync state 621 to the ready for resync state 623 should additionally be based on the communication status of the mediator being available.
A given volume may transition 631 from the InSync state to the OOS state responsive to a peer volume being unavailable. A given volume may transition 64 from the OOS state to the InSync state responsive to a successful resynchronization with the peer volume. As described below in further detail, in one embodiment, dependent write-order consistency of the mirror copy is preserved by responsive to any member volume of a given CG detecting it has gone OOS for any reason (e.g., a network failure), driving all member volumes OOS.
Each storage cluster may include a configuration database (e.g., persistent replicated database (RDB) 717, 719, RDB 512a, RDB 512b,), which is available on all storage nodes of a storage cluster. Each storage cluster includes synchronization replication circuitry (SR circuitry) 713 and 714 for synchronous replication between the storage clusters.
The operation logs or journals synchronize across a filesystem from a primary storage cluster 710 having a primary copy of data and secondary storage cluster 720 having a mirror copy of the data. In the event of an Out of Sync state for a volume due to a network glitch or a node crash, etc., a mechanism is designed to protect data for the volume and its mirror copy, avoid a coordinated OOS state for other volumes within the same CG as the OOS volume, and also avoid an OOS notification from nodes of the secondary storage cluster acting as a slave to an external mediator.
Embodiments of the present disclosure provide an Op log 741 on primary copy of node 712 and another Op log 742 on the secondary copy of node 722. Both of the copies will specify which operations are committed on each of the sides of the storage clusters. These two copies can be used to find how the filesystems for each storage cluster are differing and carry out resynchronzation if necessary. Embodiments of the present disclosure eliminate design options that involve the synchronization replication circuitry (SR) components 713 and 714 directly accessing non-volatile memory contents.
In one example, persistent inflight tracking uses only In-Volume metafiles and has a minimal impact on the Op path length. A Write to metafile isn't logged in memory (e.g., non-volatile memory). Instead, a non-volatile log replay of the Op regenerates the entry in the metafile.
The SR circuitry 713 includes an active Ops log file 740 and can be implemented with a circular array. For each Op, the SR circuitry 713 specifies its view of an Inflight Op range <Head, Tail> in a message payload. Even though responses come out of order, the SR circuitry 713 waits for a head Op to be completed and frees up all consecutive Ops which are responded to next.
The various nodes (e.g., storage nodes 136a-n and storage node 400) of the distributed storage systems described herein, and the processing described below with reference to the flow diagrams of
State information regarding a data replication status of a mirror copy of a dataset associated with a local CG may be maintained, for example, to facilitate automatic triggering of resynchronization. For example, the state information may include information relating to the current availability or unavailability of a peer volume of a local CG.
At operation 802, computer-implemented method 800 may initiate a resynchronization process due to a failure of a storage object of a first node of a primary storage cluster with the storage object becoming out of sync. A second node of the primary cluster can remain in sync state and handle operations for the first node. At operation 804, the computer-implemented method establishes or activates a transfer engine session for resynchronization from the storage object to a mirrored storage object of a secondary storage cluster where a data copy of the storage object with a failure will be moved to a node of this mirrored storage object. The storage object is not allowed to process I/O operations during this failure. A connectivity loss or failure for the storage object of the first node causes an internal state of the first node and the primary storage cluster to be out of sync (OOS) while maintaining an external state for any external entity as in sync in order to provide time for the transfer engine session to be established for reestablishing synchronous replication within duration of an Op timeout. In one example, the internal state for OOS does not cause the storage object to generate an out of sync (OOS) event for processing of a mediator or external entity. The internal state of OOS is with respect to the first node and the primary storage cluster while the external state is with respect to any external entities outside of the primary storage cluster. During this internal state with OOS, no user I/O operations are allowed on the storage object that is OOS. If the mediator or a controlling external entity views the storage object as being in sync, then the secondary storage cluster is capable of handling I/O operations for an application (e.g., database application) if the primary storage cluster fails during a resynchronization.
In a different solution, for asynchronous replication between the storage object and the mirrored storage object, the storage object will generate an external OOS event that is sent to a mediator or external entity for processing. For the OOS state, an automatic failover is disallowed.
At operation 806, the computer-implemented method waits for a filesystem log replay to finish for an endpoint of the resynchronization. The endpoint can be a storage object of the primary storage cluster. At operation 808, the primary storage cluster obtains content from an active Op log of a node of the secondary storage cluster, starts a persistent inflight tracker (IFT-P) replay of Ops, and waits for this replay to issue all Ops needed for resynchronization between the primary and secondary storage clusters. At operation 810, the primary storage cluster waits for completion of the IFT-P replay and allows new Ops onto the storage object of the primary storage cluster.
At operation 902, a computer-implemented method 900 establishes an Op range based on both of the Op log files from the primary and secondary storage clusters. A comparison between Op ranges of both of the Op log files from the primary and secondary storage clusters is performed to establish the Op range as discussed in conjunction with
At operation 904, the computer-implemented method obtains a next Op in the established Op range to be considered. At operation 906, the primary storage cluster determines whether the Op being considered is present or absent in both of the Op log files from the primary and second storage clusters. If the Op is present or absent in both of the Op log files from the primary and second storage clusters, then this Op is skipped at operation 908 and the method returns to operation 904 to consider a next Op.
If the Op is not present or absent in both of the Op log files, then the primary storage cluster determines if the Op is present (e.g., executed) on the primary storage cluster but not present on the secondary storage cluster at operation 910. If so, then the primary storage cluster replicates the Op to the secondary storage cluster at operation 912 and the method returns to operation 904.
If not, then the Op is undone from the secondary storage cluster by reading from the primary storage cluster and updating the Op log file of the secondary storage cluster and the method returns to operation 904 to obtain a next Op.
Operations 916, 918, and 920 may optionally occur in connection with operations 912 or 914. At operation 916, a dependent graph module will queue or dispatch an Op. At operation 918, the method includes performing a check (e.g., allocation check for allocation or allocation failure) on the Op with an overlapping write manager. At operation 920, the Op is sent to the secondary storage cluster.
When operations are sequentially split such that operations are first completed on a primary storage cluster and then completed on a secondary storage cluster, the active Op Log entries at a destination will be either a subset of Op log entries at a primary storage cluster or at best be the same as the Op log entries on the primary storage cluster. In one example, establishing a range of operations to be considered for persistent inflight tracking (IFT-P) replay can be a range of Ops on the Op log of the primary storage cluster.
In case of a parallel split of operations, an incoming Op (e.g., incoming WRITE operation) is sent to a synchronization replication circuitry (SR circuitry) in a primary storage cluster and a secondary storage cluster in parallel. However, this poses challenges from an Op logging perspective:
Various embodiments of the present technology provide for a wide range of technical effects, advantages, and/or improvements to stretched storage systems and participating distributed storage systems. For example, various embodiments may include one or more of the following technical effects, advantages, and/or improvements in case of a parallel split of operations:
State information regarding a data replication status of a mirror copy of a dataset associated with a local CG may be maintained, for example, to facilitate automatic triggering of resynchronization. For example, the state information may include information relating to the current availability or unavailability of a peer volume of a local CG.
At operation 1002, a computer-implemented method 1000 establishes Op ranges for Op logs of the primary and second storage clusters.
Every Op or message that is split, will carry <HeadOpSeqNum, OpSeqNum> which basically is the SR circuitry's view of what is inflight. Both the Active Op Log of the primary storage cluster and Active Op Log of the secondary storage cluster know the <HeadOpSeqNum, TailOpSeqNum> which indicates the range of Ops that are potentially in the Log file. The Head Op Sequence Numbers indicates a first sequence number for a first Op and Tail Op Sequence Number indicates a last sequence number for a last Op within the range of Ops. Using the primary and secondary Op logs for the parallel split, it is possible to determine which Ops have to be replayed.
At operation 1004, the computer-implemented method compares the Op logs of the primary and secondary storage clusters. The Op-ranges from both of the primary and secondary Op logs can have the following relation between them:
The following are salient features of logging:
In one example, an Op log file is a circular log file where an Op is inserted into Active Op Log at the index=(OpSeqNum % MaxEntries). The following examples T0 to T12 of Table 1 will clarify how range information in the Active Op Log file header will be put to use during replay: Consider Active Op Log of max size of 5 elements. Parallel split Ops don't have a suffix, sequentially split Ops will have a ‘S’ suffix, Lun metadata Ops have a ‘L’ suffix, while successful QR will have a ‘Q’ suffix. Each snapshot shows the contents of the Op Log files and how the entries can be used for IFTP-replay.
Quick reconcile is an undo of the write at the secondary storage cluster because a write Op failed on the primary storage cluster. The undo occurs via a read Op of the old content at the secondary storage cluster. Since the undo is completed at end of QR, its completion needs to be recorded. The record at the secondary storage cluster can be deleted or mark the earlier recorded Op as QR.
Consider a case where a successful QR is followed by a delete of the LUN or truncate of the LUN. In such a case, QR induced read Op is taught to deal with this failure. Instead, logging QR avoids IFT-P replay from repeating a QR, which intuitive and maintenance friendly.
During an IFT-P replay, write Ops which are committed only onto the secondary storage cluster will have to be undone by reading from primary and applying the same to secondary storage cluster. This is very much like a QR which is encountered during the steady state. Like in the steady state, the undo will read the old data from primary storage cluster and apply the same at the secondary storage cluster. The Op will be logged in Op log of the secondary storage cluster as a successful completed QR like in the steady state.
Op logging is relevant when the primary and secondary storage clusters are in sync state of replication. If Op logging is implemented by pausing all I/O just before entering in sync state, start Op logging and resume user I/O, then this will cause a spike in user I/O latency. The present design maintains the benefit of a transition state while adding Op logging support.
State information regarding a data replication status of a mirror copy of a dataset associated with a local CG may be maintained, for example, to facilitate automatic triggering of resynchronization. For example, the state information may include information relating to the current availability or unavailability of a peer storage object of a local CG.
A splitter component of SR circuitry goes through various sub-phases to allow a smooth transition from async replication to sync replication. Depending on the sub-phase, a decision for Op logging or no Op logging will occur. The following table 2 explains the various sub-phases of the splitter component for transitioning from async replication to sync replication.
At operation 1102, a computer-implemented method 1100 determines whether a sub-phase (e.g., dirty region logging (DRL) sub-phase, MDL drain sub-phase to drain metadata log, DRL drain phase sub-phase (also referred to as cut over split phase), splitting sub-phase) of a splitter component causes Op logging. If so, then at operation 1104 the method starts Op logging early and engages the splitter component of SR circuitry before a synchronous replication relationship between a storage object of the primary storage cluster and a replicated storage object of the secondary storage cluster transitions from OOS state to in sync state. After the splitter component is engaged and before the synchronous replication relationship enters in sync state, the following can happen to incoming Ops depending on the sub-phase of the splitter component.
At operation 1106, for DRL sub-phase, metadata Ops (e.g., Create, Link, Unlink, Rename, Set Attribute, Open, Close) are executed and op logged on the primary storage cluster and entered into a metadata log (MDL) journal of the primary storage cluster. At operation 1107, for a MDL drain sub-phase to drain metadata log, these metadata Ops from the MDL journal will be replicated later to secondary storage cluster by metadata scan process and enter Op log at the secondary storage cluster.
At operation 1108, for a DRL drain sub-phase, metadata Ops are executed on primary storage cluster and replicated to secondary storage cluster as well. The metadata Ops are logged into Op log of primary storage cluster and replicated Ops enter Op log of secondary storage cluster as well. For DRL drain, the data ops that are independent of the regions marked in DRL and independent of DRL scan will be Op logged on the primary storage cluster. The data ops are replicated to secondary storage cluster when replication occurs and then Op log these data Ops on secondary storage cluster.
At operation 1110, for a splitting sub-phase, metadata Ops are executed at primary and secondary storage clusters and Op logged at primary and secondary storage clusters. Data ops are executed at primary and secondary storage clusters and Op logged at primary and secondary storage clusters.
At operation 1112, given a splitter sub-phase that does not cause Op logging of data Ops, for DRL or MDL drain sub-phases, data Ops (e.g., write, punch holes) are executed on the primary storage cluster with changes being tracked in a DRL bitmap. The data Ops are not logged into the Op log of the primary storage cluster.
A dirty region log is used to track regions within the storage object that are modified by data operations, such as write operations executed during a last incremental transfer. The dirty region log may comprise bits that can be set to either a dirty indicator or a clean indicator. A bit may be mapped to a region within the storage object. Thus, the bit can be set to the dirty indicator to indicate that a data operation has modified the region (e.g., the region now comprises data not yet replicated to the replicated storage object). The bit can be set to the clean indicator to indicate that the region is now clean (e.g., the region has not been modified with data not yet replicated to the replicated storage object, and thus the region comprises the same data as a corresponding region within the replicated storage object).
The computer-implemented method marks the Op log usable once the synchronous replication relationship between a storage object of a primary storage cluster and a replicated storage object of a secondary storage cluster enters in sync state.
Although the operations in the computer-implemented methods disclosed herein are shown in a particular order, the order of the actions can be modified. Thus, the illustrated embodiments can be performed in a different order, and some operations may be performed in parallel. Some of the operations listed in the methods disclosed herein are optional in accordance with certain embodiments. The numbering of the operations presented is for the sake of clarity and is not intended to prescribe an order of operations in which the various operations must occur. Additionally, operations from the various flows may be utilized in a variety of combinations.
In one example, the operations 1106, 1107, 1108, and 1108 are illustrated as separate parallel operations. In another example, these operations can occur sequentially such as operation 1107 being performed after operation 1106.
It would be trivial to start Op Logging while only in the In Sync state for a synchronous replication relationship between a storage object (e.g., a first storage object) of the primary storage cluster and a replicated storage object (e.g., a second storage object) of the secondary storage cluster. Conventional techniques for transitioning from asynchronous replication to synchronous replication must pause client I/O (e.g., stop, block, fail, or queue the client I/O for later execution), which increases latency (e.g., increased latency while the client I/O is queued). This also affects the operation of client devices accessing data within the existing volume (e.g., an application may timeout or experience errors when data access operations, attempting to access data, are blocked or failed).
Accordingly, methods and/or systems are provided herein that can transition a storage object from an asynchronous replication state or other non-synchronous state (e.g., an out of sync state) to a synchronous replication state in a manner that mitigates client disruption and latency. However, this will need an explicit differentiation between transition state and the In Sync state and this can cause a latency spike in the user I/O into a transition state. Thus, the method 1100 starts Op logging early and marks the Op Log as usable once the synchronous replication relationship enters into the In Sync state.
A metadata log replay can play Ops out of order at destination from an Op Log seq number point of view.
Subsequently, at operation 4, the splitter component 1220 using module 1218 will update the record state in splitter active Ops as responded for the Op.
At operation 3′ the replicated Op is sent to writer 1240 to be executed on a replicated storage object. The replicated Op is provided into a record of file system 1250, which also computes a location for a record of the replicated Op, inserts the replicated Op, updates a log marker if permitted, and deletes any older entries in the Op log if these entries are not needed. At operation 4′, the splitter component 1220 will update the record state in splitter active Ops as responded for the replicated Op. Responses for operations 5 and 6 complete the transition or resynchronization process.
It is important to assign sequence numbers after Dependent Graph and Overlapping Write Manager checks have been cleared. The following example highlights the reason by showing what happens if the above rule wasn't followed:
The operations of computer-implemented method 1300 may be executed by a storage controller, a storage virtual machine (e.g., SVM 511a, SVM 511b), a mediator (e.g., mediator 120, mediator 220, mediator 360), a mediator agent, a multi-site distributed storage system, a computer system, a machine, a server, a web appliance, a centralized system, a distributed node, or any system, which includes processing logic (e.g., one or more processors such as processors 1504, a processing resource). The processing logic may include hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine or a device), or a combination of both.
At block 1310, an operation is received by a node (e.g., one of storage nodes 136a-n or storage nodes 146-n). The operation may represent a data operation (e.g., WRITE or PUNCH HOLE) or a metadata operation (e.g., CREATE to create a file, OPEN to open a file, RENAME to rename a file, LINK, UNLINK, set attribute (SETATTR) to set a volume size or assign permissions, etc.) to be executed on a file associated with a volume hosted by the node. In one embodiment, by convention, operations to be executed in relation to a particular member volume of a CG may be directed to the monarch node and performed by an SVM (e.g., SVM 511a or 511b).
At decision block 1320, processing logic (e.g., processing logic of the node, processing logic of a node of the primary storage cluster, processing logic of a node of the secondary storage cluster) determines whether the received Op is of a type that is to be logged. If so, processing continues with decision block 1340; otherwise, processing branches to block 1330. In various examples described herein, the purpose of keeping track of inflight Ops is to facilitate re-establishment of symmetry between the primary storage cluster and the secondary storage cluster. This re-establishment of symmetry may refer to ensuring any changes that are made to the primary storage cluster are also made to the secondary storage cluster. In one embodiment, only operations that perform some kind of modification to a file system, a file, and/or a volume (collectively, referred to herein as “Modify-Ops”) are tracked within the Op Logs. Non-Modify-Ops (e.g., READ and GETATTR) need not be tracked as there is reconciliation (e.g., execution of an operation that failed on the secondary cluster, but completed successfully on the primary cluster or undoing of an operation that failed on the primary cluster, but completed successfully on the secondary cluster) to be performed for such operations. According to one embodiment, when the received Op is a Modify-Op, processing continues with decision block 1340 and when the received Op is a non-Modify-Op, processing continues with block 1330.
At block 1330, the processing logic determines that the received Op is not of a type to be logged. As such, no Op Log entry need be created and Op logging is complete for the received Op.
The processing logic determines in decision block 1320 that the received Op is of a type to be logged. As such, at block 1340 it is further determined whether the node that is responsible for executing the received Op is associated with the primary storage cluster. If so, processing continues with block 1342; otherwise processing branches to block 1350. While, in the context of the present example, the amount and/or type of data logged for an Op differs depending upon whether the Op logging is being performed by the primary cluster or the secondary cluster, in alternative embodiments, Op logging may be consistent regardless of where the Op logging is being performed.
As described further below, in some examples, Op Log entries may have a different structure (e.g., more or fewer fields) depending upon the version number (the file version number) of the Op Log file at the time the Op Log entry was created. Depending upon the particular implementation the file version number may be included within a header of the Op Log file or reference may be made to the current version of the storage operating system, which may be used as the file version number of the Op Log. According to one embodiment, creation of the Op Log entry may involve determining the file version number (e.g., from the Op Log or based on the version of the storage operating system), determining a corresponding Op Log entry format based on the file version number, and then creating the Op Log entry having a structure that is based upon the determined format. In some examples, the Op Log entry includes a header and the Op payload, the structure of which may depend further upon the type of operation. When the node's file version number is greater than that of the peer node, the additional fields in the newer version Op Log entries may represent placeholders; otherwise, all fields in the newer version Op Log entries are valid
At block 1342, the processing logic determines whether the Op is a write Op or not. If so, then at block 1344, the processing logic creates an Op log entry in the primary storage cluster for minimal details (e.g., a sequence number, a file identifier, and an Op range including offset and length for the write Op) to support replay and reconciliation. At block 1346, the processing logic populates the Op log entry in the primary storage cluster with minimal details (e.g., a sequence number, a file identifier, and an Op range including offset and length for the write Op). However, pay load data of the write Op is not stored.
If the Op is not a write Op at block 1342, then at block 1346 the processing logic creates an Op log entry and populates the Op log entry at block 1348 with complete full details for the Op log entry including a sequence number, a file identifier, an Op range including offset and length, and pay load data of the Op.
Depending upon the particular implementation, an Op Log entry may be populated at the same or a different stage of the Op execution pipeline than creation of the Op Log entry. In one embodiment, the Op Log entry is populated during the modify phase of an Op handler. As noted above, it is desirable to minimize the amount of data logged, for example, to reduce the amount of time for performance of periodic consistency point (CP) generation by the filesystem (which may be performed asynchronously of the write path). According to one embodiment, different data may be stored in the header of the Op Log entry for (i) metadata Ops, (ii) PUNCH-HOLE and partial file SIS-CLONE, and (iii) WRITE. Similarly, the Op payload may or may not be logged depending upon the type of Op. According to one embodiment, the following data is stored as part of the Op Log entry for metadata Ops:
In one embodiment, the following data is stored in the Op Log entry for PUNCH-HOLE and partial file SIS-CLONE:
In one embodiment, the following data is stored in the Op Log entry for WRITEs:
In one embodiment, during replay/reconciliation, as the Op Log entries for WRITEs does not include the data payload, the data to be replicated to the secondary cluster or the old data to replace a WRITE that is to be undone on the secondary cluster may be read from the file at issue on the primary cluster. Further details regarding undo of WRITEs during IFT-P replay are provided below.
In one implementation, the ability to exclude the data payload from the Op Log entries for WRITEs is enabled by avoiding conflicting operations during synchronous replication. For example, while a volume is in an InSync state (e.g., InSync state 630) of replication, an overlap write manager (OWN) (e.g., OWM 1214) may serialize conflicting operations, for example, to preclude concurrent performance of WRITEs having a range overlap on the same file. When the volume is in an Out of Sync state (e.g., OOS state 640) of replication, and until the resynchronization process has finished (e.g., the volume has returned to the InSync state of replication), execution of Ops on the primary cluster may be disallowed temporarily.
At block 1350 (for node associated with the secondary storage cluster), the processing logic determines whether the Op is a write Op or not. If so, then at block 1352, the processing logic creates an Op log entry and at block 1356 populates the Op log entry in the primary storage cluster with minimal details (e.g., a sequence number, a file identifier, and an Op range including offset and length for the write Op) to support replay and reconciliation. However, pay load data of the write Op is not stored.
If the Op is not a write Op at block 1350, then the processing logic creates an Op log entry for only a sequence number of the Op in the Op log at block 1354 and at block 1358 populates the Op log entry with the sequence number. A file identifier, an Op range including offset and length, and pay load data of the Op are not created at block 1354.
At block 1358, as the existence of an Op Log entry on the secondary storage cluster containing a sequence number of an Op is sufficient in some embodiments to confirm its successful execution, an Op Log entry may be stored containing only the sequence number of the Op. Further data may, but need not be logged. As noted above, while, in the context of the present example, the amount and/or type of data logged for an Op differs depending upon whether the Op logging is being performed by the primary cluster or the secondary cluster, in alternative embodiments, Op logging may be performed consistently on the primary and secondary storage clusters.
In one embodiment, the complete header may be logged for parallel split Ops, like WRITE. In case of sequentially split Ops, it is sufficient to store the sequence number. Performing minimal logging at the secondary cluster, will provide for more efficient operations, for example, by reducing the CP overhead at the destination. Also, it helps save effort in the cases where the Ops are not the same between primary cluster and the secondary cluster.
Following any of blocks 1344, 1346, 1352, and 1356, the <head, tail>pointers of the Op Log file may be updated as noted above.
For completeness, the following is a non-limiting example of a generic Op Log header for the WAFL filesystem:
In one embodiment, during an IFT-P replay, WRITEs that are committed only onto the secondary cluster are undone by reading from the primary cluster and applying the same to the secondary cluster. This is very much like a QR which is encountered during the steady state. Like in the steady state, the undo may read the old data from primary cluster and apply the same at the secondary cluster and the Op may be logged on the secondary cluster as a successfully completed QR like in the steady state.
Of particular interest here is the handling of the case where there is WRITE which that is not block aligned (offset/length or both) and the original file block numbers is a hole. For example, the WAFL filesystem has a default disk block size of 4 KB (or 4k). Assuming for purposes of the present example a 4k block size, during splitting, an OWM would have been taken at the byte range for which the WRITE was issued. A punch-hole (due to undo) should only be issued at 4k boundary and for the whole block. The following are examples of various options for addressing the same:
Initially, operations are received and handled by a protocol service 1412 (e.g., network attached storage (NAS) protocol service, storage area network (SAN) protocol service, such as Small Computer System Interface (SCSI) or Fiber Channel Protocol (FCP), etc.) of the primary cluster 1410 and then the operations can be queued in a queue 1422 of splitter component 1420. The operations may represent a data operation (e.g., WRITE or PUNCH HOLE) or a metadata operation (e.g., CREATE, OPEN, RENAME, LINK, UNLINK, set attribute (SETATTR), etc.) to be executed on a file associated with a volume hosted by a node of a storage cluster (e.g., primary storage cluster 1410, secondary storage cluster 1450). In one embodiment, by convention, operations to be executed in relation to a particular member volume of a CG may be directed to the monarch node and performed by an SVM (e.g., SVM 511a or 511b).
An entry for the Op is added to the queue. The protocol service 1412 or splitter component 1420 will determine whether the Op is a data operation or a metadata operation to be executed on a file. If a data operation is being processed, then a parallel split process 1421 causes the data operation to be sent along paths 1425 and 1426 to the file systems 1430 and 1470, respectively, in parallel simultaneously. Next, the file system 1430 computes a location for a record of the data operation, inserts the data operation, updates a log marker if permitted, and deletes any older entries in the Op log of the primary storage cluster if these entries are not needed.
The splitter component 1420 will update the record state in splitter active Ops as responded for the data operation. The replicated data operation is sent to writer 1460 to be executed on a replicated storage object. The replicated data Op is provided into a record of file system 1470, which also computes a location for a record of the replicated Op, inserts the replicated Op, updates a log marker if permitted, and deletes any older entries in the Op log if these entries are not needed. Subsequently, the result processor 1424 will receive updates for the data operation along paths 1431 and 1432. The record state in splitter active Ops are updated as responded for the data operation. The result processor 1424 sends updates to the protocol service 1412.
If a metadata operation is being processed, then a sequential split process 1445 causes the metadata operation to be sent initially to the file system 1430 and then sequentially to the file system 1470. The splitter component 1420 will update the record state in splitter active Ops as responded for the data operation. The replicated data operation is sent to writer 1460 to be executed on a replicated storage object. Subsequently, the result processor 1424 will receive an update for the data operation along sequential path 1435. The record state in splitter active Ops are updated as responded for the data operation. The result processor 1424 sends updates to the protocol service 1412.
Embodiments of the present disclosure include various steps, which have been described above. The steps may be performed by hardware components or may be embodied in machine-executable instructions, which may be used to cause a processing resource (e.g., a general-purpose or special-purpose processor) programmed with the instructions to perform the steps. Alternatively, depending upon the particular implementation, various steps may be performed by a combination of hardware, software, firmware and/or by human operators.
Embodiments of the present disclosure may be provided as a computer program product, which may include a non-transitory machine-readable storage medium embodying thereon instructions, which may be used to program a computer (or other electronic devices) to perform a process. The machine-readable medium may include, but is not limited to, fixed (hard) drives, magnetic tape, floppy diskettes, optical disks, compact disc read-only memories (CD-ROMs), and magneto-optical disks, semiconductor memories, such as ROMs, PROMs, random access memories (RAMs), programmable read-only memories (PROMs), erasable PROMs (EPROMs), electrically erasable PROMs (EEPROMs), flash memory, magnetic or optical cards, or other type of media/machine-readable medium suitable for storing electronic instructions (e.g., computer programming code, such as software or firmware).
Various methods described herein may be practiced by combining one or more non-transitory machine-readable storage media containing the code according to embodiments of the present disclosure with appropriate special purpose or standard computer hardware to execute the code contained therein. An apparatus for practicing various embodiments of the present disclosure may involve one or more computers (e.g., physical and/or virtual servers) (or one or more processors within a single computer) and storage systems containing or having network access to computer program(s) coded in accordance with various methods described herein, and the method steps associated with embodiments of the present disclosure may be accomplished by modules, routines, subroutines, or subparts of a computer program product.
Computer system 1500 also includes a main memory 1506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1502 for storing information and instructions to be executed by processor 1504. Main memory 1506 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1504. Such instructions, when stored in non-transitory storage media accessible to processor 1504, render computer system 1500 into a special-purpose machine that is customized to perform the operations specified in the instructions.
Computer system 1500 further includes a read only memory (ROM) 1508 or other static storage device coupled to bus 1502 for storing static information and instructions for processor 1504. A storage device 1510, e.g., a magnetic disk, optical disk or flash disk (made of flash memory chips), is provided and coupled to bus 1502 for storing information and instructions.
Computer system 1500 may be coupled via bus 1502 to a display 1512, e.g., a cathode ray tube (CRT), Liquid Crystal Display (LCD), Organic Light-Emitting Diode Display (OLED), Digital Light Processing Display (DLP) or the like, for displaying information to a computer user. An input device 1514, including alphanumeric and other keys, is coupled to bus 1502 for communicating information and command selections to processor 1504. Another type of user input device is cursor control 1516, such as a mouse, a trackball, a trackpad, or cursor direction keys for communicating direction information and command selections to processor 1504 and for controlling cursor movement on display 1512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
Removable storage media 1540 can be any kind of external storage media, including, but not limited to, hard-drives, floppy drives, IOMEGA® Zip Drives, Compact Disc-Read Only Memory (CD-ROM), Compact Disc-Re-Writable (CD-RW), Digital Video Disk-Read Only Memory (DVD-ROM), USB flash drives and the like.
Computer system 1500 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware or program logic which in combination with the computer system causes or programs computer system 1500 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 1500 in response to processor 1504 executing one or more sequences of one or more instructions contained in main memory 1506. Such instructions may be read into main memory 1506 from another storage medium, such as storage device 1510. Execution of the sequences of instructions contained in main memory 1506 causes processor 1504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
The term “storage media” as used herein refers to any non-transitory media that store data or instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media or volatile media. Non-volatile media includes, for example, optical, magnetic or flash disks, such as storage device 1510. Volatile media includes dynamic memory, such as main memory 1506. Common forms of storage media include, for example, a flexible disk, a hard disk, a solid state drive, a magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 1502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 1504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 1500 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 1502. Bus 1502 carries the data to main memory 1506, from which processor 1504 retrieves and executes the instructions. The instructions received by main memory 1506 may optionally be stored on storage device 1510 either before or after execution by processor 1504.
Computer system 1500 also includes a communication interface 1518 coupled to bus 1502. Communication interface 1518 provides a two-way data communication coupling to a network link 1520 that is connected to a local network 1522. For example, communication interface 1518 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 1518 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 1518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 1520 typically provides data communication through one or more networks to other data devices. For example, network link 1520 may provide a connection through local network 1522 to a host computer 1524 or to data equipment operated by an Internet Service Provider (ISP) 1526. ISP 1526 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 1528. Local network 1522 and Internet 1528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 1520 and through communication interface 1518, which carry the digital data to and from computer system 1500, are example forms of transmission media.
Computer system 1500 can send messages and receive data, including program code, through the network(s), network link 1520 and communication interface 1118. In the Internet example, a server 1530 might transmit a requested code for an application program through Internet 1528, ISP 1526, local network 1522 and communication interface 1518. The received code may be executed by processor 1504 as it is received, or stored in storage device 1510, or other non-volatile storage for later execution.
A symmetric active active architecture is a fully symmetric storage solution that allows simultaneous read-write access to both the primary and secondary copies of the data. This symmetric storage solution provides application-granular zero recovery point objective (ZRPO) data protection that prevents any data loss and zero recovery time objective (ZRTO) transparent failover that provides instant recovery in the event of various faults. The concurrent read/write access to both copies in the Active/Active system is facilitated by bi-directional synchronous replication. This means that any write operation (e.g., WRITE op) initiated on a primary copy of data on a primary storage site is synchronously replicated to a secondary copy of data on a secondary storage site before a client device receives an acknowledgment (ACK). Similarly, a WRITE op initiated on the secondary storage site is synchronously replicated to the primary storage site before the client device receives an ACK. This bi-directional sync replication ensures that both copies of the data on the primary and secondary storage site are always up-to-date and consistent with each other.
The dual copies of data remain in a synchronous state (InSync) as far as a synchronous replication session between the dual copies is healthy and locally received Ops on a local storage site (e.g., primary storage site, secondary storage site) are replicated to a remote copy of a remote storage site (e.g., secondary storage site, primary storage site). However, a synchronous replication session can be torn down due to planned or unplanned events. For example, a storage aggregate hosting a storage volume can be taken over by its high availability (HA) partner second node when a first node encounters a failure condition. This is one of the Non-Disruptive-Operations (NDO) that the symmetric storage solution supports.
In another case, a customer may move a storage volume to a different node to balance a storage headroom or CPU utilization. A traditional way to re-establish the synchronous replication after such events is to perform a catch-up on the secondary storage site using snapshot based incremental asynchronous transfer followed by bootstrapping and resuming the synchronous replication. However, this traditional way can take several minutes or hours depending on the time of last common snapshot. The ZRPO and ZRTO protection is compromised during this time window.
An Op log journaling based fast-resync was previously introduced by a business continuity asymmetric solution to handle such faults during unidirectional replication from a primary storage site to a secondary storage site.
In a steady state of bidirectional synchronous replication, each Op received from a client device is logged in an In-Flight-Tracker-Persistent (IFTP) metafile on both primary and secondary storage site. Once synchronous replication relationship goes Out-Of-Sync (OOS) due to an NDO event or a network glitch, the Ops on primary and secondary are reconciled with the help of IFTP journal to bring primary and secondary copies of data back to the InSync state. However, more sophisticated algorithms are needed to handle the bidirectional replication. The major challenges are described below.
For bidirectional synchronous replication for a symmetric storage solution, an Op received by a primary or secondary storage site needs to be assigned with a unique number (e.g., sequence number, or seq_num in short) before it is logged into a metafile of a primary or secondary storage site. In an asymmetric configuration, Ops are received and processed on primary storage site, hence primary storage site generates the sequence number. In a symmetric configuration, an Op can be received and processed by any storage site. The technical problem is how to guarantee two different Ops received on different storage sites do not have a same sequence number.
A fast resync feature is useful as long as added overhead in terms of additional latency associated with the fast resync feature in a steady state is low. Overhead of fast resync is governed by an additional cost of maintaining a metafile. More entries in metafile means more overhead. Hence, how fast a storage solution can purge a metafile plays an important role. In an asymmetric configuration, a fast resync feature of the primary storage site responds to the client devices and purges the metafile. In a symmetric configuration, both storage sites will respond to client devices and hence purging of a metafile on both storage sites will be dependent on each other. Keeping both storage sites updated so that purging is fast is a challenge. A metafile is a generic term for a file format containing multiple types of data streams along with descriptive metadata.
For terminology, primary storage site/Site A will be used interchangeably and will convey a same meaning. Secondary storage site/Site B will be used interchangeably and will convey a same meaning.
For an asymmetric configuration, the purging of the metafile is contingent on which operation (Op) has been acknowledged to the client device. Each Op is assigned a unique sequence number, and a counter, referred to as the Cumulative Response Sequence Number (CSN), is maintained. For instance, when an Op is received on Site A, it is assigned a sequence number (seq_num) of 1 and logged in the metafile. Once this Op is acknowledged to the client, the CSN of Site A is updated to 1. When another Op is received on Site A and assigned a seq_num of 2, the previous entry with seq_num of 1 is deleted from the metafile because the CSN of Site A is 1.
However, in a symmetric active/active configuration, complications arise. For example, when an Op is received on Site B, the Op is assigned a seq_num of 1, logged in the metafiles of both Site A and Site B, and then acknowledged to the client device by Site B. Consequently, the CSN of Site B is updated to 1. The next Op is received on Site A and assigned a seq_num of 2. At this juncture, Site A is unaware of the status of the Op with seq_num 1 since it was acknowledged by Site B, thus the CSN of Site A remains at 0. The Op with seq_num 2 is then logged in the metafile. This process repeats when the next Op is received on Site A and assigned a seq num of 3. As a result, the metafile continues to grow, leading to increased overhead.
To address the above problem,
In one embodiment, the multi-site distributed storage system includes a primary storage site have a primary storage cluster that includes a primary copy of data in a consistency group (CG1). The consistency group of the primary storage cluster is initially assigned a leader role. A secondary storage site includes a secondary storage cluster that has a mirror copy of the data of the primary copy in the consistency group. The consistency group of the secondary storage cluster (CG2) is initially assigned a follower role.
At operation 1602, the primary storage site (Site A) is designated as a global sequence number generator, which resolves the first issue of ensuring unique sequence numbers for operations received on different storage sites. At operation 1604, counters for Operation Sequence Number (OpSeqNum) and Cumulative Response Sequence Number (CSN) are maintained on the primary storage site. At operation 1606, a counter for CSN is also maintained on the secondary storage site.
At operation 1608, when operations for one or more storage objects of CG1 are received on the primary storage site, the received operations are assigned an Op Sequence Number and CSN. At operation 1610, the operations having the Op Sequence Number and CSN are sent from the primary storage site to the secondary storage site.
At operation 1612, a response from the secondary storage site carries its CSN back to the primary storage site. At operation 1614, the CSN of the primary storage site is compared to the received CSN from the secondary storage site. At operation 1616, the method determines if the CSN of the primary storage site is less than the received CSN from the secondary storage site. If so, at operation 1617, the CSN is updated to be equal to the received CSN. At operation 1618, if the CSN of the primary storage site is equal to or greater than the received CSN from the secondary storage site, then no update is needed for the CSN of the primary storage site.
For
In case of NDO event, due to workflow mentioned above, primary storage site will be ahead of secondary storage site. Hence to bring the secondary storage site updated with the primary storage site, a replay (IFT-P replay) of Ops missing from the Op log of the secondary storage but present on the Op log the primary storage site will be performed. After replay is complete, both copies of data on the primary and secondary storage sites have been updated and consistent with each other and will be ready to service read/write request from any site.
At operation 1702, a first sync engine of the primary storage site will be set in forward direction (from primary to secondary storage site). At operation 1704, an In-Flight-Tracker-Persistent (IFTP) metafile on the primary storage site will be read, to know which Ops has been committed on the primary storage site. At operation 1706, a request will be sent from the primary storage site to the secondary storage site. At operation 1708, the secondary storage site will read In-Flight-Tracker-Persistent (IFTP) metafile on the secondary storage site to know which Ops have been committed to the secondary storage site. At operation 1710, the primary storage site will perform a comparison of these ops in the metafile of the primary site and in the metafile of the secondary site. After the comparison, the computer-implemented method will know which ops have not been committed to the secondary storage site. At operation 1712, these non-committed Ops will then be read from the primary storage site and will be replicated to the primary storage site.
At operation 1714, once all Ops are replicated from the primary storage site to the secondary storage site, both of the dual copies of data will be updated and consistent with each other. At operation 1716, at this point, the primary storage site will start accepting IO Ops from client devices. At operation 1718, due to a reverse (secondary to primary) sync engine not yet being established, any IO request landing on the secondary storage site will be proxied to the primary storage site. At operation 1720, a reverse (secondary to primary) sync engine will be established and after this point the secondary storage site will also start accepting IO Ops from client devices.
Initially, the primary storage site (Site A) is designated as a global sequence number generator, which resolves the first issue of ensuring unique sequence numbers for operations received on different storage sites. Counters for Operation Sequence Number (OpSeqNum) and Cumulative Response Sequence Number (CSN) are maintained on the primary storage site. A counter for CSN is also maintained on the secondary storage site.
At operation 1802, operations are received on the primary storage site from one or more client devices. At operation 1804, the method determines whether received operations are from client devices or from a secondary storage site. If operations are received from the secondary storage site, then the method extracts a cumulative response sequence number (CSN) from a response of the secondary storage site at operation 1806. At operation 1808, the method compares the CSN of the primary storage site to the received CSN from the secondary storage site. If the CSN of the primary storage site is less than the received CSN from the secondary storage site, the CSN is updated to be equal to the received CSN at operation 1810. Otherwise, if the CSN of the primary storage site is equal to or greater than the received CSN from the secondary storage site, then no update is needed for the CSN of the primary storage site. The method proceeds to operation 1812 with the received operations from the primary storage site being assigned an Op Sequence Number. At operation 1814, the Op Sequence Number and CSN of the operations are added to a payload of an Op message. At operation 1816, the method updates a filesystem of the primary storage site with the operations and initiates an inflight tracking and replay reconciliation between Op log metafiles of the primary and secondary storage sites. The operations are logged into the Op log metafile of the primary storage site.
At operation 1818 of
At operation 1820, if the secondary storage site's CSN is lower than the received CSN from the primary storage site, it is updated to be equal to the received CSN at operation 1822. If the secondary storage site's CSN is equal to or greater than the received CSN, no update is needed and the method proceeds to operation 1824 to determine if a received Op is from the secondary storage site. If so, at operation 1826, the method performs an active Op log metafile update. If the received Op is not from the secondary storage site, then the method prepares a response that increments the CSN at operation 1828 and sends the response to the primary storage site. At operation 1830, primary storage site extracts the CSN from the response of the secondary storage site. At operation 1832, the method performs an active Op log metafile update.
At operation 1834, the method determines if operations are received on the secondary storage site. If so, the method prepares a message that increments the CSN and sends the message to the primary storage site at operation 1836.
The virtual storage system 2900 may present storage over a network to clients 2905 using various protocols (e.g., object storage protocol (OSP), small computer system interface (SCSI), Internet small computer system interface (ISCSI), fibre channel (FC), common Internet file system (CIFS), network file system (NFS), hypertext transfer protocol (HTTP), web-based distributed authoring and versioning (WebDAV), or a custom protocol. Clients 2905 may request services of the virtual storage system 2900 by issuing Input/Output requests 2906, 2907 (e.g., file system protocol messages (in the form of packets) over the network). A representative client of clients 2905 may comprise an application, such as a database application, executing on a computer that “connects” to the virtual storage system over a computer network, such as a point-to-point channel, a shared local area network (LAN), a wide area network (WAN), or a virtual private network (VPN) implemented over a public network, such as the Internet.
In the context of the present example, the virtual storage system 2900 includes virtual storage nodes 2910 and 2920 with each virtual storage node being shown includes an operating system. The virtual storage node 2910 includes an operating system 2911 having layers 2913 and 2914 of a protocol stack for processing of object storage protocol operations or requests.
The virtual storage node 2920 includes an operating system 2921, layers 2923 and 2924 of a protocol stack for processing of object storage protocol operations or requests.
The storage nodes can include storage device drivers for transmission of messages and data via the one or more links 2960. The storage device drivers interact with the various types of hyperscale disks 2915, 2925 supported by the hyperscalers.
The data served by the virtual storage nodes may be distributed across multiple storage units embodied as persistent storage devices (e.g., non-volatile memory 2940, 2942), including but not limited to HDDs, SSDs, flash memory systems, or other storage devices (e.g., 2915, 2925).
To create a virtualized high availability host clustering 2020 across two sites A and B, hosts are used and managed by a server appliance 2010. The virtual machine (VM-1) can be migrated 2021 from host 01 to host 02. The server appliance 2010 is a centralized management system that enables administrators to effectively operate hosts in host clusters. The server appliance 2010 facilitates key functions such as VM provisioning, High Availability (HA), Distributed Resource Scheduler (DRS), Kubernetes Grid, and more. It is an important component in cloud environments.
The virtual storage system 2000 provides advanced business continuity if one or more failure domains suffer a total outage. The virtual storage system 2000 may present storage over a network to clients using various protocols (e.g., object storage protocol (OSP), small computer system interface (SCSI), Internet small computer system interface (ISCSI), fibre channel (FC), common Internet file system (CIFS), network file system (NFS), hypertext transfer protocol (HTTP), web-based distributed authoring and versioning (WebDAV), or a custom protocol. Clients may request services of the virtual storage system 2000 by issuing Input/Output requests (e.g., file system protocol messages (in the form of packets) over the network). A representative client may comprise an application, such as a database application, executing on a computer that “connects” to the virtual storage system over a computer network, such as a point-to-point channel, a shared local area network (LAN), a wide area network (WAN), or a virtual private network (VPN) implemented over a public network, such as the Internet.
In the context of the present example, the clusters 01 and 02 each include virtual storage nodes with each virtual storage node including an operating system. The storage nodes can include storage device drivers for transmission of messages and data via the one or more links 2041 and 2042.
The data served by the virtual storage nodes may be distributed across multiple storage units embodied as persistent storage devices (e.g., non-volatile memory), including but not limited to HDDs, SSDs, flash memory systems, or other storage devices.
The clusters 01 and 02 enable business services to continue operating even through a complete site failure, supporting applications to fail over transparently using a secondary copy. Neither manual intervention nor custom scripting are required to trigger a failover with active sync. The active sync supports a symmetric active active capability, enabling read and write I/O operations from both copies of a protected LUN (e.g., L1, L2, L3) with bidirectional synchronous replication, enabling both LUN copies to serve I/O operations locally.
A data protection relationship to protect for business continuity is created between the source storage system (e.g., cluster 01) and destination storage system (e.g., cluster 02), by adding the application specific LUNs from different volumes within a storage virtual machine (SVM) to the consistency group. Under normal operations, the enterprise application writes to the primary consistency group (e.g., CG 2040), which synchronously replicates this I/O to the mirror consistency group (e.g., CG 2050). Even though two separate copies of the data exist in the data protection relationship, because active sync maintains the same LUN identity, the application host sees this as a shared virtual device with multiple paths (e.g., active/optimized paths 2022, 2023; active/non-optimized path 2025, 2026) while only one LUN copy is being written to at a time. Active Optimized paths are a path state in ALUA (Asymmetric Logical Unit Access) where the target storage system responds to I/O requests using the most efficient path. In this case, the active/optimized path 2022 is between host 01 and cluster 01 at site A while the active/optimized path 2023 is between host 02 and cluster 02 at site B. The active non-optimized paths 2025 and 1226 are between different sites. This results in higher performance and reduced latency.
When a failure renders the primary storage system offline, the operating system detects this failure and uses the Mediator 2090 for reconfirmation. If neither the operating system nor the Mediator 2090 are able to ping the primary site with cluster 01, the operating system performs the automatic failover operation. This process results in failing over only a specific application without the need for the manual intervention or scripting which was previously required for the purpose of failover.
The external Mediator 2090 is external from sites A and B and installed in a third failure domain, distinct from the two distinct failure domains of the clusters 01 and 02. The Mediator 2090 acts as a passive witness to active sync copies. In the event of a network partition or unavailability of one copy, active sync uses Mediator 2090 to determine which copy continues to serve I/O, while discontinuing I/O on the other copy. The Mediator 2090 plays a crucial role in active sync configurations as a passive quorum witness, ensuring quorum maintenance and facilitating data access during failures. It acts as a ping proxy for controllers to determine liveliness of peer controllers. Although the Mediator does not actively trigger switchover operations, it provides a vital function by allowing the surviving node to check its partner's status during network communication issues. In its role as a quorum witness, the Mediator provides an alternate path (effectively serving as a proxy) to the peer cluster.
Furthermore, the Mediator allows clusters to get this information as part of the quorum process. The Mediator 2090 utilizes the node management LIF and cluster management LIF for communication purposes. The Mediator 2090 establishes redundant connections through multiple paths to differentiate between site failure and InterSwitch Link (ISL) failure. When a cluster loses connection with the Mediator software and all its nodes due to an event, it is considered not reachable. This triggers an alert and enables automated failover to the mirror Consistency Group (CG) in the secondary site, ensuring uninterrupted I/O for the client. The replication data path relies on a heartbeat mechanism, and if a network glitch or event persists beyond a certain period, it can result in heartbeat failures, causing the relationship to go out-of-sync. However, the presence of redundant paths, such as LIF failover to another port, can sustain the heartbeat and prevent such disruptions.
Some embodiments relates to Example 1 that includes a computer-implemented method performed by one or more processing resources of a distributed storage system, the computer-implemented method comprises establishing bi-directional synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiating a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 due to a loss of the bi-directional synchronous replication between the one or more storage objects of the CG1 and the one or more storage objects of the CG2, and performing the resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.
Example 2 includes the subject matter of Example 1, the computer-implemented method further comprises designating the primary storage site as a global sequence number generator to ensure unique sequence numbers for operations received on each of the primary and secondary storage sites, and maintaining counters for Operation Sequence Number and Cumulative Response Sequence Number (CSN) on the primary storage site.
Example 3 includes the subject matter of any of any of Examples 1-2, the computer-implemented method further comprises maintaining a counter for CSN on the secondary storage site.
Example 4 includes the subject matter of any of Examples 1-3, the computer-implemented method further comprises upon operations for one or more storage objects of CG1 being received on the primary storage site, assigning the received operations an Op Sequence Number and CSN, sending the operations having the Op Sequence Number and CSN from the primary storage site to the secondary storage site, and receiving, with the primary storage site, a response from the secondary storage site having a CSN for the secondary storage site.
Example 5 includes the subject matter of any of Examples 1-4, the computer-implemented method further comprises comparing the CSN of the primary storage site to the received CSN from the secondary storage site, and updating the CSN to be equal to the received CSN when the CSN of the primary storage site is less than the received CSN from the secondary storage site.
Example 6 includes the subject matter of any of Examples 1-5, the computer-implemented method further comprises receiving operations on the secondary storage site for one or more storage objects of CG2, sending the received operations with a CSN of the secondary storage site to the primary storage site, sending a response from the primary storage site having an Op Sequence Number that is generated on the primary storage site and a CSN of the primary storage site to the secondary storage site, and updating the CSN of the secondary storage site to be equal to the CSN of the primary storage site when the CSN of secondary storage site is lower than the received CSN.
Example 7 includes the subject matter of any of Examples 1-6, the computer-implemented method further comprises initiating the inflight tracking and reconciliation of I/O operations based on the first Op log metafile of the primary storage site and the second Op log metafile of the secondary storage site.
Example 8 includes the subject matter of any of Examples 1-8, the computer-implemented method further comprises replaying the I/O operations from one of the first and second Op log metafiles and reconciling the I/O operations in the first Op log metafile with the I/O operations in the second Op log metafile by determining whether an Op being considered is missing from one of the first and second Op log metafiles and sending the missing Op from the primary storage site to the secondary storage site when the Op is missing from the second Op log metafile.
Some embodiments relates to Example 9 that includes a non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a multi-site distributed storage system cause the one or more processing resources to establish bi-directional synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiate a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 due to a loss of the bi-directional synchronous replication between the one or more storage objects of the CG1 and the one or more storage objects of the CG2, and perform the resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.
Example 10 includes the subject matter of Example 9, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to designate the primary storage site as a global sequence number generator to ensure unique sequence numbers for operations received on each of the primary and secondary storage sites, maintain counters for Operation Sequence Number and Cumulative Response Sequence Number (CSN) on the primary storage site, and maintain a counter for CSN on the secondary storage site.
Example 11 includes the subject matter of any of Examples 9-10, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to upon operations for one or more storage objects of CG1 being received on the primary storage site, assign the received operations an Op Sequence Number and CSN, send the operations having the Op Sequence Number and CSN from the primary storage site to the secondary storage site, and receive, with the primary storage site, a response from the secondary storage site having a CSN for the secondary storage site.
Example 12 includes the subject matter of any of Examples 9-11, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to compare the CSN of the primary storage site to the received CSN from the secondary storage site and update the CSN to be equal to the received CSN when the CSN of the primary storage site being less than the received CSN from the secondary storage site.
Example 13 includes the subject matter of any of Examples 9-12, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to receive operations on the secondary storage site for one or more storage objects of CG2, send the received operations with a CSN of the secondary storage site to the primary storage site, send a response from the primary storage site having an Op Sequence Number that is generated on the primary storage site and a CSN of the primary storage site to the secondary storage site, and update the CSN of the secondary storage site to be equal to the CSN of the primary storage site when the CSN of secondary storage site CSN is lower than the received CSN.
Example 14 includes the subject matter of any of Examples 9-13, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to initiate the inflight tracking and reconciliation of I/O operations based on the first Op log metafile of the primary storage site and the second Op log metafile of the secondary storage site, replay the I/O operations from one of the first and second Op log metafiles, and reconcile the I/O operations in the first Op log metafile with the I/O operations in the second Op log metafile by determining whether an Op being considered is missing from one of the first and second Op logs and sending the missing Op from the primary storage site to the secondary storage site when the Op is missing from the second Op log metafile.
Some embodiments relate to Example 15 that includes a multi-site storage system having a primary storage site with a primary storage cluster and a secondary storage site with a secondary storage cluster, comprising one or more processing resources and a non-transitory computer-readable medium coupled to the one or more processing resources, having stored therein instructions, which when executed by the one or more processing resources cause the one or more processing resources to establish bi-directional synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiate a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 due to a loss of the bi-directional synchronous replication between the one or more storage objects of the CG1 and the one or more storage objects of the CG2, and perform the resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.
Example 16 includes the subject matter of Example 15, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to designate the primary storage site as a global sequence number generator to ensure unique sequence numbers for operations received on each of the primary and secondary storage sites, maintain counters for Operation Sequence Number and Cumulative Response Sequence Number (CSN) on the primary storage site, and maintain a counter for CSN on the secondary storage site.
Example 17 includes the subject matter of any of Examples 15-16, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to upon operations for one or more storage objects of CG1 being received on the primary storage site, assign the received operations an Op Sequence Number and CSN, send the operations having the Op Sequence Number and CSN from the primary storage site to the secondary storage site, and receive, with the primary storage site, a response from the secondary storage site having a CSN for the secondary storage site.
Example 18 includes the subject matter of any of Examples 15-17, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to receive operations on the secondary storage site for one or more storage objects of CG2, send the received operations with a CSN of the secondary storage site to the primary storage site, send a response from the primary storage site having an Op Sequence Number that is generated on the primary storage site and a CSN of the primary storage site to the secondary storage site, and update the CSN of the secondary storage site to be equal to the CSN of the primary storage site when the CSN of secondary storage site CSN is lower than the received CSN.
Example 19 includes the subject matter of any of Examples 15-18, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to initiate the inflight tracking and reconciliation of I/O operations based on the first Op log metafile of the primary storage site and the second Op log metafile of the secondary storage site.
Example 20 includes the subject matter of any of Examples 15-19, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to replay I/O operations from one of the first and second Op log metafiles; and reconcile the I/O operations in the first Op log metafile with the I/O operations in the second Op log metafile by determining whether an Op being considered is missing from one of the first and second Op logs and sending the missing Op from the primary storage site to the secondary storage site when the Op is missing from the second Op log metafile.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202141020578 | May 2021 | IN | national |
| 202141020579 | May 2021 | IN | national |
This application is a continuation-in-part of U.S. patent application Ser. No. 18/421,649, filed Jan. 24, 2024, which is a continuation of U.S. patent application Ser. No. 17/510,788, filed Oct. 26, 2021, which claims the benefit of Indian Provisional Application No. 20/214,1020578, filed on May 5, 2021, and Indian Provisional Application No. 20/214,1020579, filed on May 5, 2021, which are hereby incorporated by reference in its entirety for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17510788 | Oct 2021 | US |
| Child | 18421649 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18421649 | Jan 2024 | US |
| Child | 19034412 | US |
