Computer data is increasingly vital to modern organizations; therefore, protecting against data loss in the event of a system failure is an increasingly important organization objective. Data protection systems for storing organizational source (e.g., production) site data on a periodic basis suffer from several drawbacks. First, they require a system shutdown during backup, since the data being backed up cannot be used during the backup operation. Second, they limit the points in time to which the production site can recover. For example, if data is backed up on a daily basis there may be several hours of lost data in the event of a disaster. Third, the data recovery process itself is time consuming as well as system resource consuming.
Some data protection systems use data replication by creating a copy of the organization's production site data on a secondary backup storage system and updating the backup with changes. In some instances, multiple backup storage systems may be employed in which some backup storage locations are local to the production storage system (i.e., the source site), while others are remote (i.e., target sites). Data replication processes may take longer to complete for backup storage locations that are remote from the production storage system as compared to backup storage locations that are local to the production site.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described herein in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In an asynchronous cascading replication process, replication is performed whereby a storage cluster A creates cycles of consistency via snapshots, sends commands to a storage cluster B to create matching (empty) containers, and sends the differences between generations of snapshots from A to B. For example, a cycle counter N on storage cluster A may advance from time to time. When the cycle counter advances, a new set of snapshots is created in A, called A(N). Storage cluster A then compares A(N) with the previously generated snapshot set A(N−1) and sends the difference to storage cluster B. When the containers in B receive the full difference, they become snapshots. Once storage cluster B receives a command to create container B(N), it immediately sends a similar command to a storage cluster C, creating snapshot containers C(N). Data transmission from A to B begins. Storage cluster B starts a scan process that captures any data sent from A to B and sends it to C in containers C(N). Storage cluster B does not need to wait for cycle N to finish—it can start sending data as it receives it, or in the background. When all of the cycle data is transmitted to B, storage cluster B marks the cycles as receive complete, When all of the cycle data is sent from B to C, storage cluster B marks the cycles as transmit complete. When both marks are on, B announces that the cycle is complete by sending a message to storage cluster C. Storage cluster C can then mark the container as a complete snapshot set and a new consistent point-in-time in the asynchronous replication process.
One aspect may provide a method for performing a data replication process in a storage system. The method includes creating, at a first target site, an empty container in a storage system. The empty container matches a container at a source site in response to initiation of an asynchronous data replication process. The method also includes transmitting, by the first target site, a command to a second target site to create a container at the second target site. The first target site performs the asynchronous data replication process, which includes scanning the data upon receipt from the source site for a first target replication cycle and transmitting the scanned data to the container at the second target site for a second target replication cycle.
Another aspect may provide a system performing a data replication process in a storage system. The system includes a memory having computer-executable instructions. The system also includes a processor operated by a first target site. The process executes the computer-executable instructions. When executed by the processor, the computer-executable instructions cause the processor to perform operations. The operations include creating, at the first target site, an empty container in a storage system. The empty container matches a container at a source site in response to initiation of an asynchronous data replication process. The operations also include transmitting a command to a second target site to create a container at the second target site. The first target site performs the asynchronous data replication process, which includes scanning the data upon receipt from the source site for a first target replication process and transmitting the scanned data to the container at the second target site for a second target replication process.
Another aspect may provide a computer program product embodied on a non-transitory computer readable medium. The computer program product includes instructions that, when executed by a computer at a first target site, causes the computer to perform operations. The operations include creating, at the first target site, an empty container in a storage system. The empty container matches a container at a source site in response to initiation of an asynchronous data replication process. The operations also include transmitting a command to a second target site to create a container at the second target site. The first target site performs the asynchronous data replication process, which includes scanning the data upon receipt from the source site for a first target replication process and transmitting the scanned data to the container at the second target site for a second target replication process.
Objects, aspects, features, and advantages of embodiments disclosed herein will become more fully apparent from the following detailed description, the appended claims, and the accompanying drawings in which like reference numerals identify similar or identical elements. Reference numerals that are introduced in the specification in association with a drawing figure may be repeated in one or more subsequent figures without additional description in the specification in order to provide context for other features. For clarity, not every element may be labeled in every figure. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments, principles, and concepts. The drawings are not meant to limit the scope of the claims included herewith.
In an asynchronous cascading replication setup for three storage clusters A, B, and C, a consistency group in the first storage cluster A is replicated to the second storage cluster B, and from the second storage cluster B to the third storage cluster C. Embodiments described herein enable the creation of consistent delayed copies of replicated data on multiple backup targets while using minimal overhead.
Turning now to
Source site 102 may include a host 104, storage application 106, and data storage 108. In some embodiments, storage 108 may include one or more storage volumes (not shown), that operate as active or production volumes.
Host 104 may perform input/output (I/O) operations on storage 108 (e.g., read data from and write data to storage 108). In some embodiments, the I/O operations may be intercepted by and controlled by the storage application 106. As changes are made to data stored on storage 108 via the I/O operations from host 104, or over time as storage system 100 operates, storage application 106 may perform data replication from the source site 102 to the target site 112 over a communication network 110. In some embodiments, the communication network 110 may include internal (e.g., short distance) communication links (not shown) to transfer data between storage volumes for storing replicas 118 (also referred to herein as snapshots), such as an InfiniBand (IB) link or Fibre Channel (FC) link.
In illustrative embodiments, storage system 100 may employ a snapshot (or replication) mechanism to replicate data between source site 102 and target site 112. A snapshot (or replica) may be created from data within storage 108 and transferred to the target site 112 during a data replication cycle by data replication.
Data replication may be performed based on data replication policies that may define various settings for data recovery operations, shown as policy 114 in target site 112. For example, policy 114 may define a plurality of attributes, such as a frequency with which replicas are generated and how long each replica 118 is kept at target site 112. Policy 114 may also define a remote replica lag (e.g., the length of time during which updates may be lost in case of a source site failure), a recovery point objective (RPO) (e.g., a maximum acceptable lag time between the time data is committed to the source site 102 and the time the data is committed to the target site 112 or an acceptable amount of data loss measured in time), a recovery time objective (RTO) (e.g., the time taken to perform the recovery), the mode of replication (e.g., synchronous, asynchronous, continuous data protection (CDP), point in time (PIT), and so forth), and/or other attributes.
As described herein, in example embodiments, data replication may be asynchronous data replication performed at time-based intervals during operation of storage system 100. The timing of asynchronous replication cycles and the retention of the replicas 118 may be managed by replica manager 116 of target site 112. Data replication may alternatively be synchronous data replication performed when data is changed on source site 102.
For example, asynchronous data replication may periodically generate snapshots (or replicas), scan and compare the snapshots to determine changes in data between the snapshots, and transfer the data difference to target site 112. In the scanning process, the entire range of a snapshot is traversed, from address 0 to the last address in the snapshot. In this context, the scan traverses the address-to-hash (A2H) meta-data structure and locates each entry in this structure. Each entry in the A2H structure is then compared with the same entry in the snapshot of the previous cycle to determine any changes. In embodiments, the storage application 106 may employ a cycle counter (not shown), which upon advancing, generates a new set or generation of snapshots on the storage 108 and is referred to herein as A(N). The storage application 106 compares A(N) with a previously generated snapshot set A(N−1), sends a command to the target site 112 to create a new container (e.g., container 119), and sends the difference between the snapshots to target site 112. Once target site 112 receives the difference, it stores the data differences in container 119. Once the full difference among snapshots is received at target site 112, the difference is stored as the new set of snapshots.
The frequency with which to perform replication cycles may be determined by a recovery point objective (RPO) and/or retention policy settings of policy 114. For example, policy 114 may define an RPO setting of x seconds, where x may be a positive integer, to ensure that the most recent available replica 118 stored on target site 112 reflects the state of data stored on source site 102 no longer than x seconds ago. Policy 114 may also determine how many replicas 118 should be maintained, at what time (e.g., what time of day, week, month, etc.), and for what duration.
In illustrative embodiments, policy 114 may set the RPO to a relatively short duration (e.g., on the order of 1 or 2 seconds) between asynchronous data replication cycles.
In addition to managing replicas 118 according to a policy 114 (e.g., a replication and/or retention policy), the replica manager 116 may also include a cycle counter 117 to track generations of snapshot sets over time, as will be described further herein.
In an embodiment, target site 112 is communicatively coupled to a second target site 112′ over a communications link 110′. The communication link 110′ may be a long-distance communication network of a storage area network (SAN), e.g., over an Ethernet or Internet (e.g., TCP/IP) link that may employ, for example, the iSCSI protocol. The target site 112 receives data transmissions from the source site 102 and forwards the data onto the target site 112′. In an embodiment, the target site 112 tracks the progress of the incoming data as it is received from the source site 102 and further tracks the progress of the data transmissions delivered to the second target site 112′. Replicas 118 and 118′, or snapshots, of the replicated data are stored both in storage containers 119 (on target site 112) and 119′ (on target site 112′), respectively.
Referring to
Turning now to
The process 300 of
At block 302, storage cluster B receives the command and creates an empty container 119 to store the replicated data at block 304. In turn, storage cluster B initiates a command to storage cluster C over a network (e.g., network 110′ of
Blocks 308-312 and 316 in
In block 312, it is determined whether the transmission of data is complete between storage cluster A and storage cluster B (e.g., when all of the data is received by the storage cluster B). If not, the storage cluster B continues to receive replication data from storage cluster A, scanning the data at block 308 and sending the scanned data to storage cluster C in block 310. If, however, the transmission of data has completed in block 312, which is referred to herein as a first target replication cycle, storage cluster B marks the cycle as “receive” complete in block 314.
If the transmission of data from storage cluster A to storage cluster B has completed in block 312, it is determined whether the transmission of data from storage cluster B to storage cluster C has completed in block 316 (e.g., when all of the data has been transmitted from storage cluster B to storage cluster C). If not, the process returns to block 310 and storage cluster B continues to send replication data to storage cluster C. If, however, the transmission of data from storage cluster B to storage cluster C has completed in block 316, which is referred to herein as a second target replication cycle, the transmission is marked as cycle “transmit” complete in block 318 and an acknowledgement message is sent from storage cluster B to storage cluster C to indicate the end of the data transmission in block 320.
Storage cluster C can mark the container 119′ as a complete snapshot set and as a new consistent point in time for the asynchronous replication process.
Referring to
Process 300 shown in
The processes described herein are not limited to the specific embodiments described. For example, process 300 is not limited to the specific processing order shown in
Processor 402 may be implemented by one or more programmable processors executing one or more computer programs to perform the functions of the system. As used herein, the term “processor” is used to describe an electronic circuit that performs a function, an operation, or a sequence of operations. The function, operation, or sequence of operations can be hard coded into the electronic circuit or soft coded by way of instructions held in a memory device. A “processor” can perform the function, operation, or sequence of operations using digital values or using analog signals. In some embodiments, the “processor” can be embodied in an application specific integrated circuit (ASIC). In some embodiments, the “processor” can be embodied in a microprocessor with associated program memory. In some embodiments, the “processor” can be embodied in a discrete electronic circuit. The “processor” can be analog, digital or mixed-signal.
While illustrative embodiments have been described with respect to processes of circuits, described embodiments may be implemented as a single integrated circuit, a multi-chip module, a single card, or a multi-card circuit pack. Further, as would be apparent to one skilled in the art, various functions of circuit elements may also be implemented as processing blocks in a software program. Such software may be employed in, for example, a digital signal processor, micro-controller, or general purpose computer. Thus, described embodiments may be implemented in hardware, a combination of hardware and software, software, or software in execution by one or more processors.
Some embodiments may be implemented in the form of methods and apparatuses for practicing those methods. Described embodiments may also be implemented in the form of program code, for example, stored in a storage medium, loaded into and/or executed by a machine, or transmitted over some transmission medium or carrier, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation. A non-transitory machine-readable medium may include but is not limited to tangible media, such as magnetic recording media including hard drives, floppy diskettes, and magnetic tape media, optical recording media including compact discs (CDs) and digital versatile discs (DVDs), solid state memory such as flash memory, hybrid magnetic and solid state memory, non-volatile memory, volatile memory, and so forth, but does not include a transitory signal per se. When embodied in a non-transitory machine-readable medium, and the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the method.
When implemented on a processing device, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits. Such processing devices may include, for example, a general purpose microprocessor, a digital signal processor (DSP), a reduced instruction set computer (RISC), a complex instruction set computer (CISC), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array (PLA), a microcontroller, an embedded controller, a multi-core processor, and/or others, including combinations of the above. Described embodiments may also be implemented in the form of a bitstream or other sequence of signal values electrically or optically transmitted through a medium, stored magnetic-field variations in a magnetic recording medium, etc., generated using a method and/or an apparatus as recited in the claims.
Various elements, which are described in the context of a single embodiment, may also be provided separately or in any suitable subcombination. It will be further understood that various changes in the details, materials, and arrangements of the parts that have been described and illustrated herein may be made by those skilled in the art without departing from the scope of the following claims
In the above-described flow chart of
Some embodiments may be implemented in the form of methods and apparatuses for practicing those methods. Described embodiments may also be implemented in the form of program code, for example, stored in a storage medium, loaded into and/or executed by a machine, or transmitted over some transmission medium or carrier, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation. A non-transitory machine-readable medium may include but is not limited to tangible media, such as magnetic recording media including hard drives, floppy diskettes, and magnetic tape media, optical recording media including compact discs (CDs) and digital versatile discs (DVDs), solid state memory such as flash memory, hybrid magnetic and solid state memory, non-volatile memory, volatile memory, and so forth, but does not include a transitory signal per se. When embodied in a non-transitory machine-readable medium and the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the method.
When implemented on one or more processing devices, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits. Such processing devices may include, for example, a general purpose microprocessor, a digital signal processor (DSP), a reduced instruction set computer (RISC), a complex instruction set computer (CISC), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array (PLA), a microcontroller, an embedded controller, a multi-core processor, and/or others, including combinations of one or more of the above. Described embodiments may also be implemented in the form of a bitstream or other sequence of signal values electrically or optically transmitted through a medium, stored magnetic-field variations in a magnetic recording medium, etc., generated using a method and/or an apparatus as recited in the claims.
For example, when the program code is loaded into and executed by a machine, such as the computer of
In some embodiments, a storage medium may be a physical or logical device. In some embodiments, a storage medium may consist of physical or logical devices. In some embodiments, a storage medium may be mapped across multiple physical and/or logical devices. In some embodiments, storage medium may exist in a virtualized environment. In some embodiments, a processor may be a virtual or physical embodiment. In some embodiments, a logic may be executed across one or more physical or virtual processors.
For purposes of illustrating the present embodiment, the disclosed embodiments are described as embodied in a specific configuration and using special logical arrangements, but one skilled in the art will appreciate that the device is not limited to the specific configuration but rather only by the claims included with this specification. In addition, it is expected that during the life of a patent maturing from this application, many relevant technologies will be developed, and the scopes of the corresponding terms are intended to include all such new technologies a priori.
The terms “comprises,” “comprising”, “includes”, “including”, “having” and their conjugates at least mean “including but not limited to”. As used herein, the singular form “a,” “an” and “the” includes plural references unless the context clearly dictates otherwise. Various elements, which are described in the context of a single embodiment, may also be provided separately or in any suitable subcombination. It will be further understood that various changes in the details, materials, and arrangements of the parts that have been described and illustrated herein may be made by those skilled in the art without departing from the scope of the following claims.
Number | Name | Date | Kind |
---|---|---|---|
6496908 | Kamvysselis et al. | Dec 2002 | B1 |
6553464 | Kamvysselis et al. | Apr 2003 | B1 |
6640280 | Kamvysselis et al. | Oct 2003 | B1 |
6862632 | Halstead et al. | Mar 2005 | B1 |
6883018 | Meiri et al. | Apr 2005 | B1 |
6886164 | Meiri | Apr 2005 | B2 |
6898685 | Meiri et al. | May 2005 | B2 |
6910075 | Marshak et al. | Jun 2005 | B2 |
6938122 | Meiri et al. | Aug 2005 | B2 |
6944726 | Yoder et al. | Sep 2005 | B2 |
6968369 | Veprinsky et al. | Nov 2005 | B2 |
6976139 | Halstead et al. | Dec 2005 | B2 |
7000086 | Meiri et al. | Feb 2006 | B2 |
7024525 | Yoder et al. | Apr 2006 | B2 |
7032228 | McGillis et al. | Apr 2006 | B1 |
7051176 | Meiri et al. | May 2006 | B2 |
7054883 | Meiri et al. | May 2006 | B2 |
7113945 | Moreshet et al. | Sep 2006 | B1 |
7114033 | Longinov et al. | Sep 2006 | B2 |
7165141 | Cochran | Jan 2007 | B2 |
7174423 | Meiri et al. | Feb 2007 | B2 |
7191299 | Kekre | Mar 2007 | B1 |
7197616 | Meiri et al. | Mar 2007 | B2 |
7228456 | Lecrone et al. | Jun 2007 | B2 |
7240116 | Marshak et al. | Jul 2007 | B2 |
7292969 | Aharoni et al. | Nov 2007 | B1 |
7376651 | Moreshet et al. | May 2008 | B2 |
7380082 | Meiri et al. | May 2008 | B2 |
7383385 | Meiri et al. | Jun 2008 | B2 |
7383408 | Meiri et al. | Jun 2008 | B2 |
7386668 | Longinov et al. | Jun 2008 | B2 |
7392360 | Aharoni et al. | Jun 2008 | B1 |
7409470 | Halstead et al. | Aug 2008 | B2 |
7430589 | Veprinsky et al. | Sep 2008 | B2 |
7577957 | Kamvysselis et al. | Aug 2009 | B1 |
7613890 | Meiri | Nov 2009 | B1 |
7617372 | Bjornsson et al. | Nov 2009 | B1 |
7702871 | Arnon et al. | Apr 2010 | B1 |
7702953 | Spear | Apr 2010 | B2 |
7870195 | Meiri | Jan 2011 | B1 |
8046545 | Meiri et al. | Oct 2011 | B2 |
8078813 | LeCrone et al. | Dec 2011 | B2 |
8332687 | Natanzon et al. | Dec 2012 | B1 |
8335771 | Natanzon et al. | Dec 2012 | B1 |
8335899 | Meiri et al. | Dec 2012 | B1 |
8468180 | Meiri et al. | Jun 2013 | B1 |
8578204 | Ortenberg et al. | Nov 2013 | B1 |
8600943 | Fitzgerald et al. | Dec 2013 | B1 |
8677087 | Meiri et al. | Mar 2014 | B2 |
8694700 | Natanzon et al. | Apr 2014 | B1 |
8706959 | Arnon et al. | Apr 2014 | B1 |
8719497 | Don et al. | May 2014 | B1 |
8732124 | Arnon et al. | May 2014 | B1 |
8782357 | Halstead et al. | Jul 2014 | B2 |
8799211 | Bezbaruah | Aug 2014 | B1 |
8812595 | Meiri et al. | Aug 2014 | B2 |
8825964 | Sopka et al. | Sep 2014 | B1 |
8838849 | Meiri et al. | Sep 2014 | B1 |
8862546 | Natanzon et al. | Oct 2014 | B1 |
8914596 | Lecrone et al. | Dec 2014 | B2 |
8966211 | Arnon et al. | Feb 2015 | B1 |
8977826 | Meiri et al. | Mar 2015 | B1 |
9002904 | Meiri et al. | Apr 2015 | B1 |
9009437 | Bjornsson et al. | Apr 2015 | B1 |
9026492 | Shorey et al. | May 2015 | B1 |
9026696 | Natanzon et al. | May 2015 | B1 |
9037816 | Halstead et al. | May 2015 | B1 |
9037822 | Meiri et al. | May 2015 | B1 |
9100343 | Riordan et al. | Aug 2015 | B1 |
9110693 | Meiri et al. | Aug 2015 | B1 |
9304889 | Chen et al. | Apr 2016 | B1 |
9323750 | Natanzon et al. | Apr 2016 | B2 |
9342465 | Meiri | May 2016 | B1 |
9378106 | Ben-Moshe et al. | Jun 2016 | B1 |
9396243 | Halevi et al. | Jul 2016 | B1 |
9418131 | Halevi et al. | Aug 2016 | B1 |
9483355 | Meiri et al. | Nov 2016 | B1 |
9524220 | Veprinsky et al. | Dec 2016 | B1 |
9558083 | LeCrone et al. | Jan 2017 | B2 |
9606739 | LeCrone et al. | Mar 2017 | B1 |
9606870 | Meiri et al. | Mar 2017 | B1 |
9753663 | LeCrone et al. | Sep 2017 | B1 |
9959063 | Meiri et al. | May 2018 | B1 |
9959073 | Meiri | May 2018 | B1 |
10007466 | Meiri et al. | Jun 2018 | B1 |
10025843 | Meiri et al. | Jul 2018 | B1 |
10055161 | Meiri et al. | Aug 2018 | B1 |
10095428 | Meiri et al. | Oct 2018 | B1 |
10152527 | Meiri et al. | Dec 2018 | B1 |
20040260902 | Stanley | Dec 2004 | A1 |
20050050115 | Kekre | Mar 2005 | A1 |
20050182910 | Stager | Aug 2005 | A1 |
20070100909 | Padovano | May 2007 | A1 |
20170295230 | Hatfield | Oct 2017 | A1 |
20180150229 | Brown | May 2018 | A1 |
20180349042 | Brown | Dec 2018 | A1 |
20190018727 | Huh | Jan 2019 | A1 |
Entry |
---|
U.S. Appl. No. 16/050,247, filed Jul. 31, 2018, Schneider et al. |
U.S. Appl. No. 16/177,782, filed Nov. 1, 2018, Hu et al. |
U.S. Appl. No. 16/264,825, filed Feb. 1, 2019, Chen et al. |
U.S. Appl. No. 16/263,414, filed Jan. 31, 2019, Meiri et al. |
U.S. Appl. No. 15/001,789, filed Jan. 20, 2016, Meiri et al. |
U.S. Appl. No. 15/076,775, filed Mar. 22, 2016, Chen et al. |
U.S. Appl. No. 15/076,946, filed Mar. 22, 2016, Meiri. |
U.S. Appl. No. 15/085,188, filed Mar. 30, 2016, Meiri et al. |
U.S. Appl. No. 15/499,297, filed Apr. 27, 2017, Kucherov et al. |
U.S. Appl. No. 15/499,303, filed Apr. 27, 2017, Kucherov et al. |
U.S. Appl. No. 15/499,226, filed Apr. 27, 2017, Meiri et al. |
U.S. Appl. No. 15/499,199, filed Apr. 27, 2017, Stronge et al. |
U.S. Appl. No. 15/797,329, filed Oct. 30, 2017, Parasnis et al. |
U.S. Appl. No. 15/971,310, filed May 4, 2018, Kucherov et al. |
U.S. Appl. No. 15/971,325, filed May 4, 2018, Kucherov et al. |
U.S. Appl. No. 15/971,445, filed May 4, 2018, Kucherov et al. |
U.S. Appl. No. 15/970,243, filed May 3, 2018, Schneider et al. |
Number | Date | Country | |
---|---|---|---|
20190339870 A1 | Nov 2019 | US |