This invention relates to storage systems, and in particular to techniques of assuring appropriate performance of remote copy systems in such storage systems.
Large organizations throughout the world now are involved in millions of transactions which include enormous amounts of text, video, graphical and audio information. This information is being categorized, stored, accessed, and transferred every day. The volume of such information continues to grow. One technique for managing such massive amounts of information is to use storage systems. Conventional storage systems include large numbers of hard disk drives operating under various control mechanisms to record, mirror, remotely backup, and reproduce this data. The rapidly growing amount of data requires most companies to manage the data carefully with their information technology systems, and to assure appropriate performance within such systems.
One common occurrence in the management of such data is the need to assure its preservation by making remote copies of the information in a location away from a primary or production site. Maintaining such records in a remote site helps assure the owner of the data that the data will be available even if there are natural disasters or other unexpected events which occur at the primary site and destroy the data there. By having stored the data in a remote location, protection is also provided in the event of failures in the primary storage system, as well as other events. Should an event occur at the primary site, the data from the remote copy operation can be retrieved and replicated for use by the organization, thereby preventing data loss or the need to recreate the data at considerable cost and delay.
Typically the data at the remote site (the “remote copy”) is remotely copied to be stored at that site via a communications network which is either dedicated to the transmission of data between the primary site and the remote site, via the Internet, or by other means. Of course, because the remote site is, by definition, located at a distance from the primary site to provide enhanced data protection, there is a delay between the time the data is stored at the primary site and the time the data is transmitted to and stored at the remote site. Depending upon the bandwidth of the connection and the particular equipment at the remote site, this delay can be significant. Examples of storage-based remote copy technology as provided by leading vendors are Hitachi TrueCopy™, EMC SRDF™, and IBM PPRC™.
To date, however, there has been no satisfactory means available for auditing the remote copy and the remote site to determine its currency with respect to the fresh data being provided to the primary site. Users of storage systems must assure that the remote copy is always properly operating, because the timing of and nature of a particular disaster will be unknown. In addition, the Securities and Exchange Commission is now issuing standards with respect to practices for maintaining business records and the like, some of which require certain users to execute various disaster recovery tests periodically. As a result, there is a growing need for users to have storage systems and software which assure the remote copy systems are operating correctly, and which determine the consistency time, i.e., the time that describes how far behind the remote copy is from the primary data in terms of WRITE operation time.
Some of the existing solutions, for example, IBM's PPRC CQUERY command enable assuring the remote site is operational. This solution may be satisfactory in simple implementations; however, the remote copy configuration environment has become more complex in which various applications cause an inconsistent WRITE workload. Furthermore, because of limited budgets, users want to utilize storage systems as efficiently as possible, and with minimum network costs. These network costs can rise when high network bandwidth is required to provide remote copy functionality. Accordingly, a need has arisen for monitoring remote copy operations and reporting to users so that they can understand their storage systems, and particular the remote copy operations therein, better than before.
A further issue with respect to remote copy operations is the need to satisfy various recovery objectives, and in particular to assure that the remote copy satisfies particular recovery objectives. The Storage Networking Industry Association has defined two recovery objectives. The recovery point objective (RPO) is the maximum desired time period prior to a failure or a disaster during which changes to the data may be lost as a consequence. Data changes preceding the failure or disaster by at least this time period are preserved by a recovery. In other words, changes to the data made later than this time will not be reflected in the remote copy. A zero value is equivalent to no loss of data. In general, the consistency time indicates RPO, which is how much time of WRITE data potentially could be lost if the production storage system goes down.
A second established definition for a recovery objective is the recovery time objective (RTO). This is the maximum desired time period required to bring one or more applications and its associated data back to a correct operational state. Because a major factor of RTO is the time to restart a server and its applications, it is not necessarily a desirable measure of the quality of remote copy operations.
The method and apparatus of this invention provides a technique for auditing a remote copy system to determine if the system is satisfying particular recovery objectives. This invention provides a technique by which host input/output (“I/O”) (mainly WRITE) operations are monitored and compared with corresponding operations to a remote copy system. It thereby allows a determination of the relationship between the data presently being stored at a remote copy location, and the current data being operated upon by the processor. As a result, the user obtains a measure of the time difference between the two, and therefore a measure of the extent to which data would be lost if the primary storage system failed and the user were forced to rely entirely upon the remote system. It further allows calculation of a parameter we refer to as consistency time, which is an estimate of how far behind the remote copy is from the primary data, as measured in terms of WRITE operation time.
In a preferred embodiment, a method of implementing this invention includes providing a storage system coupled to a host, with primary storage media at a primary site and remote storage media at a remote site. WRITE operations of the host are monitored at first times, and at corresponding times, WRITE, COPY or APPLY operations of the remote site are monitored. From a comparison of the monitored information, a consistency time, reflecting the delay between storage of data at the primary site and storage of data at the remote site is obtained, thereby providing a measure of the performance of the remote system.
In a preferred implementation of the system, a first probe is used to monitor host WRITE operations, either at the host or at the storage system, at first times, a remote copy probe is used to monitor remote copy operations at the first times, and a remote copy auditor coupled to receive the data from the first probe and the remote copy probe. The auditor compares the information from the two to thereby provide information about the time delay between operations to the primary storage system and the remote storage system.
The system described can be implemented with particular facility in a journal remote copy system in which all changes being made to a primary volume are written to a journal volume before being stored to the primary volume. The journal data is also sent to the remote site and used to update a remote volume there.
The consistency time can be calculated by measuring the number of WRITE operations to be copied in the primary system and dividing that by the average number of WRITE operations per second. Alternatively, the size of the WRITE data to the primary journal can be measured. Dividing that information by the average throughput provides an approximation of the consistency time.
Although other techniques may be used, the system depicted in
Storage system 120 also includes a probe 123 to measure the remote copy workload within the storage system. The particular measurements taken by the probe are discussed in more detail later in conjunction with
In an alternative embodiment, rather than measuring the host WRITE performance, probe 112 is placed in the storage system. In this implementation, it measures storage I/O operations from the point of view of the storage system 120. In other words, probe 112 is placed in the storage system and monitors the WRITE operations received by the storage system, rather than those sent by the host. In such implementations, there is no need for timer 113 or probe 112 to be situated in the host. This embodiment also eliminates the need for synchronization of timers, because timer 125 can be used for both the probe of received host I/O operations and the remote copy operations.
As also shown, the primary site 100 includes a remote copy auditor 130. The remote copy auditor 130 provides the capability of reporting and analyzing remote copy operation status and performance. The auditor 130 enables users to determine causes or errors, or impediments in remote copy operations for enabling greater efficiency. Generally the auditor 130 is implemented as software on a computer system, for example, a server or a maintenance computer coupled to the storage system 120. The remote copy auditor 130 collects the host workload 132 from the host probe 112. It also collects remote copy workload information from probe 123 in storage system 120. The collected information is stored in a history database 133, which may itself comprise a volume in the storage system 120. The particular information collected and stored is discussed below. The host and the remote copy auditor are preferably connected using a network, for example a TCP/IP protocol-based local area network 103. The auditor may communicate with the storage system through the storage area network (102) itself, or a TCP/IP protocol-based local area network (102). There may be an implementation in which the auditor resides on the same host as the application.
Remote site 150 also typically contains a remote host 160 and a remote storage system 170. More than one storage system may be employed, but no hosts need be employed to reduce cost. The remote storage system 150 is similar to the primary storage system 120, except that it receives the journal information from the primary storage 120 and stores that journal information in journal volume 172. The changes written into the journal volume are then applied to the remote volume 171. The remote host 160 itself is not an essential element of the remote site. It operates typically only when the workload of host 110 fails over to site 150. In this case, the host I/O probe 162 and host application 161, as well as timer 163, will operate, otherwise, they may be quiescent. As described with respect to the primary site, the host and the storage system in the remote site are coupled via the communications channel 151. The remote host may also execute the same application 161 as the application 111, for example, a database program.
The embodiment depicted in
Remote copy probe 123 also collects information from the remote storage controller about its throughput 305 and from the remote journal volume 172. An example of the throughput 305 is, as a mark 174 indicates, the amount of data applied to the secondary volume per second (apply throughput) in Mbps. As allow indicates, it also measures around the remote journal volume 172. The remote journal volume information collected includes the amount of data pending write operations 306, typically in megabytes, the I/O time stamp 303 of the oldest entry in the remote journal volume 172, and the time stamp of the newest entry 304 in that journal volume. The remote copy probe 123 may access to the remote storage system 170 through the wide area storage network 105 or TCP/IP based wide area network.
As mentioned above, to compare all of the measurements taken from the host probe 112 and the remote copy probe 123, it is necessary to synchronize the timers. To achieve this, timer 113 in the host and timer 125 in the storage system are regularly synchronized. Such synchronization can be performed using a variety of techniques. For example, in a master and slave operation host 110 becomes a master of the timer, and storage 120 receives commands from the host 110. The host regularly issues timer synchronization commands with the exact time to the storage 120. The storage collects the timer information and adjusts the timer 125 as necessary.
An alternative synchronization technique is arbitration. In this case, the host and the storage arbitrate their timers with each other and decide what time should be followed. In yet another implementation, the host and the storage regularly check for the exact time with a trusted third party such as a time stamp authority. The host and the remote then synchronize their timers in that manner. This technology is well known, and commonly referred to as Network Time Protocol (NTP).
Assuming the timers to be synchronized, the remote copy auditor 130 collects the host workload information described in
The history database 133 stores the workload information regularly collected from each of the probes.
At step 501 the I/O time stamp of the oldest entry 303 at a certain measuring time is retrieved. Then at step 502 the I/O time stamp of the newest entry 304 at the same time is also retrieved. By subtracting the newest from the oldest, the consistency time at that measurement time is determined. The manner in which this is presented to the user will be discussed in conjunction with
In another technique for calculating the consistency time, the number of WRITE 1,0 operations to the primary (production) journal is measured, and the average number of WRITE 1,0 operations per second is determined. By dividing the later quantity into the former, an approximate consistency time in seconds is determined. Alternatively, the size of the WRITE I/O in the production volume may be measured, then that quantity divided by the average throughput of the WRITE I/O per second to obtain an approximate consistency time in seconds.
As shown in
Although these two parameters are discussed in conjunction with
/O operations impact storage resources.
Some remote copy systems may be configured to have multiple storage systems at multiple sites. In some cases such systems configure more than three storage systems at different locations or sites. Such systems also can be measured using the techniques described herein. In such systems typically one storage system receives I/O from at least one production host, and performance of those may be measured in the same manner as described above (i.e. Host I/O Probe resides in the storage system). Of course, the remote copy probe would need to be capable of accessing each such storage system, and the auditor would need to be aware of the multiplicity of storage systems. In these situations, the host and the storage system may have the same configuration as the host 110 and the storage system 120 discussed above. The other storage systems may have the same configuration as the storage system 170. The difference from the first embodiment is that the probe in the production storage system should have the capability to access multiple storage systems at different locations. Remote copy auditor 130 should be aware of the total number of storage systems at different sites, for example, by having them included in the history database 133.
There is another remote copy system in which the remote copy is operated by the hosts. In this case, a probe of remote copy may exist within the host, and its configuration and function are the same as the probe of remote copy 123 discussed above. Also, there is another remote copy system in which the remote copy is operated by combination of a storage system and a host. An example is IBM XRC(r). In this case, probes of remote copy may be provided at both the storage system and the host. The measurements that those probes collect are the same as the first embodiment. In addition, the timers to which the probes refer should be synchronized in the same way described above.
In a further embodiment, multiple production storage systems and multiple remote storage systems can be employed and their performance measured. In this case, there may be multiple hosts in which probes 112 are provided, and multiple production storage systems in which probe 123 is provided. In this implementation, the remote copy auditor 130 stores measurements from the probes, but maintains the relationship among the multiple storage systems and the multiple hosts. Then the calculations and analysis described in conjunction with
Although the foregoing has been a description of the preferred embodiments of the invention, it will be understood that these are provided for illustrative purposes only. The scope of the invention may be determined from the appended claims.