Patent Grant
7702667
The invention is directed to methods and systems for determining and validating accessibility and currency, i.e., the actual status, of data replicated in a networked environment.
Data replication is a technique commonly used for achieving high data availability. When multiple replicas of a data set are created and stored at different locations, a replica of the data will more likely be available to a client application even when some components fail or some data sets are corrupted.
In computing systems many techniques exist for copying data and for managing multiple replicas. Replication techniques can be classified to two main categories: synchronous and asynchronous replication. Synchronous replication processes enforce continuous full synchronization between the source data set and the replicas. This involves strong transactional guarantees and ensures that any update to a source data set is consistently and immediately reflected in all the synchronous replicas. However, achieving synchronous replication can in some environments be prohibitively expensive in terms of the overhead it imposes on computing resources, and in some cases not be possible at all (for example due to temporary failure of some component in the environment).
Asynchronous replication, on the other hand, requires a much less stringent time-consistency between replicas by creating copies only periodically. Thus a replica may represent some past state of the data source rather than the current state of the data source. Depending on how far back in the past that reference point is, such discrepancy may still be acceptable for some client applications under some exceptional circumstances (e.g., when recovering from a catastrophic failure). Asynchronous replication imposes a much lower overhead on the computing resources and is commonly used in many environments, such as maintaining geographically remote copies of application data for Disaster-Recovery (DR) purposes.
However, ensuring continuous conformance of the data sets and their replicas with the applications requirements is a difficult challenge for a number of reasons: different applications may have different minimal currency requirements for replicated-data (that is, there are typically differences in their cost/currency trade-off considerations); there may be multiple data-copiers in a typical environment that may be executing concurrently; copy activities may be based on replicas (which may themselves not be fully current) rather than on the original data set, thus creating chains of dependencies; individual copy activities may fail entirely, and a replica at a remote site may be inaccessible to a host due for example to a network or component configuration problem.
Consequently an application may not have a replica of sufficient currency accessible to it at a remote site, if required. Currently, such a deficiency may not be detected until an application actually requires that replica. Present replication technologies focus on the actual correctness of individual copy mechanism, but not on continuous end-to-end validation of the currency and accessibility of multiple replicas of data.
It would therefore be desirable to provide systems and processes for continuously validating replicated data sets in networks as being in conformance with defined application requirements for currency and accessibility, and for identifying and notifying a user of any discrepancies so that corrective actions can be taken before any undesirable consequences.
The invention is directed to systems and processes for continuously validating replicated data sets in networks as being in conformance with a replicated-data policy.
According to one aspect of the invention, a process for validating replicated data residing on network devices in a network includes the steps of defining a replicated-data policy for replicating data in the network, monitoring access paths between network devices or between applications running on the network devices, monitoring data replication activities in the network, and comparing currency and accessibility of a replica with the requirements in the replicated-data policy to identify discrepancies with the replicated-data policy.
According to another aspect of the invention, a replication validation manager for validating replicated data residing on network devices in a network includes a policy engine that stores replicated-data policy for replicating data in the network and a validation engine that monitors access paths between network devices or between applications running on the network devices. The validation engine further monitors data replication activities in the network and compares currency and accessibility of a replica with the requirements in the replicated-data policy to identify discrepancies with the replicated-data policy. The replication validation manager also includes a notification engine that provides a violation notification if a replica cannot be validated.
Embodiments of the invention may include one or more of the following features. The replicated-data policy may specify an access path between a host or application and a replica, and/or a number of replicas in a network, locations of the replicas, one or more access paths between the replicas, and/or a maximum age of a corresponding replica at each location. The data replication activities may include monitoring at least one of synch, split, copy start, copy complete, source data volume, target data volume, and a time of the replication.
Each replica may be associated with a tag which may include at least one of a source data set name, the copy status, and a time stamp. The tag of a replica may be updated in response to a copier event.
Access path attributes may include redundancy and/or number of intermediate components and/or performance and/or security and/or interoperability, sharability, or capacity.
Discrepancies may be identified from a currency violation and/or missing access path and/or unauthorized access path, and/or path attribute violation.
In another embodiment, replication reports may be generated which may include properties of applications, replicas and replication events, and their impact on replicas; replication violations and time to correct them; replication resource utilization over time, or a combination thereof.
The following figures depict certain illustrative embodiments of the invention in which like reference numerals refer to like elements. These depicted embodiments are to be understood as illustrative of the invention and not as limiting in any way.
In the context of the invention, the following terminology will be used for classifying components of replicated data environments:
Definitions
Host Components are platforms on which application software programs can execute to achieve some useful purpose. At any point in time each host can have one or more applications executing on it. Each application can be executing on one or more host components. In addition each host may contain control programs which control the access to host resources and to external components.
Storage Components are platforms on which data can be stored and retrieved. Each storage device contains a memory sub-component containing multiple addresses in each of which one or more bits can be stored. Data is read from and written to storage devices in units which are referred to as volumes. A volume may contain any amount of data represented by any number of bits. Volumes are stored on storage devices, each starting with a particular start address on a particular storage component. In addition, each storage component may also contain a controller sub-component which controls access to the data in the memory sub-component.
Network components are platforms via which data can be transferred and routed from any source component to any target component. Each network component can control the flow of data depending on the source, destination and status circumstances.
Each of the aforementioned components have a unique identifier (name) associated with it. Moreover, each of the components has one or more ports which enable input and output data to flow to and from that component. Each component can furthermore have a local state which represents a current “Control Configuration” of that component defining certain information flow characteristics such as which data flow can be enabled based on the source and the target components.
Different components can be connected to each other by “Communication Links” via which data can flow from one component to another. Such communication links can connect components located in very close proximity at single site, or at remote geographical locations. Example communication channels can include a cable a point-to-point connection, a local area network, a wide area network and others.
An Access Path exists between two end points (components, data sets, etc) if there is communication links connectivity, and if each intermediate component as well as the end point themselves are configured to enable data flow between these end points.
Environment Configuration Events that can occur in an environment can be classified to different classes including among others: components configuration changes, components addition and deletion, components failed and recovered, data send and receive, data volumes reads and writes, and others.
Applications running on hosts can generate new “Data Volumes” and submit them for storage on storage devices, as well as update or read existing data volumes stored on storage devices.
A Data Copier is a program which at certain points in time reads a volume from a storage device and writes an identical copy of that volume to another location (address) on the same storage device or a different storage device. A copier can execute on any component (host component, storage component, or network component). An initial source volume updated by an application is referred to a source volume, and any volume which was generated by a copier is referred to as a replica volume.
A Replicated-Data Validation Manager 101 is connected to the networks, as indicated by arrows 172, 174, 176, 178, and performs the process described below in order to validate the currency and accessibility of all data replicas in accordance with a specified application data-replication policy.
At step 204, a replicated-data application policy is defined. For each data volume the policy defines: application access path (and attributes) to that volume, access path between replicas of that volume, and attributes, access paths from replicas to other hosts, required minimal currency (or maximally tolerable age) requirements of replicas at different locations (also referred to as Recovery Point Objectives (RPO))
At step 206, information is obtained about events in the environment that affect the currency of replicas or their accessibility. For example, information may be received indicating that a copy action was successfully completed from replica 130 to replica 132 (
At step 208, an analysis is performed to derive from the event information about the status of currency and accessibility of the volumes and replicas.
At step 210, the conclusions of the analysis performed at step 208 are compared with the currency and accessibility requirements that are specified in the policy and if violations are detected, at step 212, notifications are provided regarding these violations. Such violations can be associated with the status or level of currency, for example all replicas of source volume 128 located at a remote location are at the moment less current than the minimum currency specified in the currency and accessibility requirements. Such violations can also be associated with accessibility, for example, if no remote replicas of source volume 128 are currently accessible by remote host-components as required.
In general, an applications replicated-data policy can specify for each volume in the environment: (1) which hosts and application should have an access path to that volume; (2) how many replicas should it have, where should they reside, and what should be the access path between these replicas; (3) what should be the maximum age of the replica at each location (what is the RPO); and (4) should any remote hosts have an access path to any of the replicas.
Thus the policy can represent many types of application requirements including, for example, the following exemplary requirements:
The next step in the replicated-data validation process (represented by step 206 in
Data copier replication events information include for example:
A conventional asynchronous copy of one volume to another volume may also be represented by a Synch event followed by a simultaneous Copy-Completed and Split events. Any volume generated or updated by a Synch event of a copier is referred to as a Replica Volume.
At point in time 401, a synch copy event occurs between volume 1 and 2, at point in time 402 the copy is completed, and at point in time 403 a split event occurs. If a copy is initiated between volume 2 and 3 at point in time 404 and the copy is completed and a split is performed at point in time 405. In order to determine the currency of volume 3 after point 405 the history of previous copy events need to be considered. Note also that between the time a copy is started (for example, point in time 401) to the point in time it is successfully completed (for example, point in time 402), the state of the target replica is inconsistent.
Environment configuration events may also affect access paths. Event information collected include:
The next step in the replication validation process (represented by step 208 in
The currency conclusions are derived from the replication event information in the following way. An association is maintained between each replica volume and a Replica Header Tag, which may include among others the following attributes:
The Replica Header Tag of a replica is updated in response to different copier events, as described above. These Tag updates can, for example, adhere to the following principles:
Accessibility conclusions are derived from the environment configuration events information by determining all the access paths in the environments. That is, deriving all the volumes and replicas at all storage devices that each initiating component can obtain data from or provide data to. The analysis is performed repeatedly after each event using the following underlying principles:
An access path exists between an initiator component and a target component if there exists a communication connectivity (at least one communication link and possible additional intermediate components interconnected with communication link) between the initiator component and the target, and the access control configuration on each component in the sequence (starting with the host and ending with the storage device) is set to enable data flow between the sources in the prefix of the sequence to the destinations in the postfix of that sequence.
Furthermore, the current step in the process derives for each identified access paths various end-to-end attributes of that access-path as determined by cumulative analysis of all the state information and events information associated with components in that access path. These derived access path attributes represent various service dimensions such as level of redundancy in the path, level of access performance within the path, number of intermediate components between host and target volume, storage capacity, among others.
This analysis may also establish in a similar way the existence of appropriate access paths between data copiers and data volumes (source and target volumes).
An extended type of an access path is a Disaster-Recovery Access-Path (DR-Path) which represents an access relationship between host-components and a sequence of a number of replica volumes (local and remote).
Such a notion of a DR-Path is a useful way to specify and enforce both the host to volume required access attributes as well as the volume to replica and replica to replica required access attributes.
Such DR-Paths can be formally defined in a number of ways, for example, as follows:
Let νl . . . νn be data sets. Each data set νi is of a certain type tk that belongs to a set of types {tl, . . . , tm}.
A path segment exists between νi and νi+l if there is physical connectivity between νi and νi+l and also logical connectivity between them. νi and νi+l have logical connectivity if the physical path/s between them complies with certain set of rules that are a function of νi and νi+l's types.
A path exists from νk . . . νl if for every k≦i<l there exists a path segment from νi to νi+l.
Another exemplary definition:
Let νl . . . νn be computing nodes. Each node νi is of a certain type tk that belongs to a set of types {tl, . . . , tm}.
A Stored-Data Access-Path νk . . . νl exists if there is physical connectivity from νk to νl, and if the logical state of each node νi k≦i<l in the sequence is set to enable data flow along that sequence to or from a longer-term storage type node (in response to an initiation action by one of the other nodes in the sequence).
Another exemplary definition:
Let Hi denote hosts, Let Dj denote Storage Area network (SAN) devices (switches, routers, HBAs, etc.—local area or wide area networks), and let Vk denote stored data sets (volumes on storage devices)
A DR path is a sequence Ha-Vm- . . . -Vn[-Hb], such that:
Multiple types of DR-Paths can be predefined, each representing a particular permutation of replica types, and potentially a remote host associated with a specific sequence at the other end.
For example, EMC Corporation, Hopkington, Mass. (USA) supplies infrastructure storage components. EMC's technology refers to specific names for replica types (e.g. BCVs, R1s, and R2s representing local and remote synchronous and asynchronous replicas respectively) and imposes certain constraints on the replica type copy sequence relationship.
In the context of EMC infrastructures, the following are examples of possible predefined DR-Paths types which can be used and enforced:
Each represents an access-path relationship between a host and a volume, between a volume and replica, possibly between a replica and a replica, and between a replica and a host.
The next step in the replicated-data validation process (represented by step 210 in
These requirements and others can be established with the replication analysis and access-path analysis mechanisms outlined above as follows:
Any discrepancy between the derived conclusions and the policy requirements is considered a violation. Thus a replicated data violation can represent a case in which a volume is not accessible to a host or to another replica in a manner consistent with its DR-Path specification in the policy, that one of the attributes of the existing access path or DR-path is different than the one specified in the policy, or that the most current replica of that volume beyond a specified distance is older than the Recovery Point Objectives (RPO) specified by the policy.
The next step of the replication-validation process process (represented by step 212 in
The violation and the context information are listed and stored, and appropriate notification messages may be generated and sent to appropriate parties based on filtering rules that are applied to the violation information. The correction of a violation is also automatically detected by the mechanisms described, and the stored repositories are updated and appropriate notification messages can be sent.
The current process also stores received event information (from components and from data copiers) in a history structure in a database, and associates with each event the derived access path conclusion results. That history database represents the complete evolution of the access paths status in the environment is available for further analysis and summary reporting.
Another possible embodiment of the invention enables planning and simulation of future changes, events, and failures to determine their expected impact on the replication requirements, and prevent problems before they occur.
The events that can be specified are possible future component change actions, possible data copiers action, or possible component failure or site failure.
The analysis performed is analogous to the one described above using the current state of the environment components and replicas, and considering the simulated event input as if this is the event information that was collected from the environment (represented by step 204 of
The context information of each such future violation is generated and provided in the same way provided for regular violations (see above). Once validated, the innovation can actually track the execution of any corresponding change events, track implementation progress, and reports upon successful completion (or notify of any resulting violations).
Yet another embodiment of the invention enables summary reporting of the information accumulated and processed during the replication validation process. The information collected and analyzed by the current innovation enables the generation of a wide variety of useful data replication validation summary reports and trending reports.
The types of reports that can be generated include for example the following, among many others:
While the invention has been disclosed in connection with the preferred embodiments shown and described in detail, various modifications and improvements thereon will become readily apparent to those skilled in the art. Various aspects of access path, their validation and management, are described, for example, in commonly assigned U.S. patent applications Ser. No. 10/693,632, filed 23 Oct. 2003; and Ser. Nos. 11/112,942 and 11/112,624, both filed 22 Apr. 2005, the contents of which is incorporated herein by reference in their entirety. Accordingly, the spirit and scope of the present invention is to be limited only by the following claims.
This application claims the benefit of U.S. Provisional Patent Application No. 60/720,977, filed Sep. 27, 2005, the entire content of which is incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5043866 | Myre et al. | Aug 1991 | A |
| 5280611 | Mohan et al. | Jan 1994 | A |
| 5327556 | Mohan et al. | Jul 1994 | A |
| 5381545 | Baker et al. | Jan 1995 | A |
| 5452448 | Sakuraba et al. | Sep 1995 | A |
| 5774377 | Eidson et al. | Jun 1998 | A |
| 5825772 | Dobbins et al. | Oct 1998 | A |
| 5940819 | Beavin et al. | Aug 1999 | A |
| 6014673 | Davis et al. | Jan 2000 | A |
| 6223176 | Ricard et al. | Apr 2001 | B1 |
| 6233240 | Barbas et al. | May 2001 | B1 |
| 6240463 | Benmohamed et al. | May 2001 | B1 |
| 6327598 | Kelley et al. | Dec 2001 | B1 |
| 6347335 | Shagam et al. | Feb 2002 | B1 |
| 6434626 | Prakash et al. | Aug 2002 | B1 |
| 6636981 | Barnett et al. | Oct 2003 | B1 |
| 6691169 | D'Souza | Feb 2004 | B1 |
| 6751228 | Okamura | Jun 2004 | B1 |
| 6792503 | Yagi et al. | Sep 2004 | B2 |
| 6795399 | Benmohamed et al. | Sep 2004 | B1 |
| 6816927 | Bouchet | Nov 2004 | B2 |
| 6904143 | Peterson et al. | Jun 2005 | B1 |
| 6909700 | Benmohamed et al. | Jun 2005 | B1 |
| 7051029 | Fayyad et al. | May 2006 | B1 |
| 7058702 | Hogan | Jun 2006 | B2 |
| 7062559 | Yoshimura et al. | Jun 2006 | B2 |
| 7069480 | Lovy et al. | Jun 2006 | B1 |
| 7103653 | Iwatani | Sep 2006 | B2 |
| 7103712 | Mizuno | Sep 2006 | B2 |
| 7120654 | Bromley | Oct 2006 | B2 |
| 7127633 | Olson et al. | Oct 2006 | B1 |
| 7149886 | Fujibayashi et al. | Dec 2006 | B2 |
| 7194538 | Rabe et al. | Mar 2007 | B1 |
| 7260628 | Yamamoto et al. | Aug 2007 | B2 |
| 7376937 | Srivastava et al. | May 2008 | B1 |
| 7380239 | Srivastava et al. | May 2008 | B1 |
| 7512954 | Srivastava et al. | Mar 2009 | B2 |
| 20020145981 | Klinker et al. | Oct 2002 | A1 |
| 20030005119 | Mercier et al. | Jan 2003 | A1 |
| 20030018619 | Bae et al. | Jan 2003 | A1 |
| 20030055932 | Brisse | Mar 2003 | A1 |
| 20030101258 | Parham | May 2003 | A1 |
| 20030131077 | Hogan | Jul 2003 | A1 |
| 20030191992 | Kaminsky et al. | Oct 2003 | A1 |
| 20030208589 | Yamamoto | Nov 2003 | A1 |
| 20030237017 | Jibbe | Dec 2003 | A1 |
| 20040019833 | Riedl | Jan 2004 | A1 |
| 20040030739 | Yousefi'zadeh | Feb 2004 | A1 |
| 20040030768 | Krishnamoorthy et al. | Feb 2004 | A1 |
| 20040075680 | Grace et al. | Apr 2004 | A1 |
| 20040205089 | Alon et al. | Oct 2004 | A1 |
| 20040215749 | Tsao | Oct 2004 | A1 |
| 20050010682 | Amir et al. | Jan 2005 | A1 |
| 20050044088 | Lindsay et al. | Feb 2005 | A1 |
| 20050097471 | Faraday et al. | May 2005 | A1 |
| 20050114403 | Atchison | May 2005 | A1 |
| 20050160431 | Srivastava et al. | Jul 2005 | A1 |
| 20060004830 | Lora et al. | Jan 2006 | A1 |
| 20060143492 | LeDuc et al. | Jun 2006 | A1 |
| 20060161883 | Lubrecht et al. | Jul 2006 | A1 |
| 20060161884 | Lubrecht et al. | Jul 2006 | A1 |
| 20060218366 | Fukuda et al. | Sep 2006 | A1 |
| 20070094378 | Baldwin et al. | Apr 2007 | A1 |
| 20070112883 | Asano et al. | May 2007 | A1 |
| 20070198722 | Kottomtharayil et al. | Aug 2007 | A1 |
| Number | Date | Country |
|---|---|---|
| WO-0182077 | Nov 2001 | WO |
| WO-02088947 | Nov 2002 | WO |
| WO-03054711 | Jul 2003 | WO |
| WO-2004111765 | Dec 2004 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20070088763 A1 | Apr 2007 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60720977 | Sep 2005 | US |
