Patent Application
20090240744
1. Field
The present invention relates generally to database backup systems, and more specifically to an apparatus and method of replicating and storing backup databases in real-time or near real-time.
2. Background
PourOver is a method and system that replicates continually generated transaction data to a backup location in order to achieve real-time or near real-time database consistency between the active and backup databases. It can be used to construct a consistent backup database that is up-to-date (real-time) or within a few minutes' worth of transactions (near real-time). The backup location can be situated at a remote site.
High-availability systems that handle millions of transactions per day require that the data be reliably and continuously replicated to backup locations. This is so in the situation of a disaster where the running production site has failed or even potentially become inaccessible, the backup site can be brought up with no loss (real-time) or almost no loss of transaction data (near real-time).
An ideal system is a real-time system that operates constantly with extremely low downtime requirements, where inter-related data is stored across multiple files and distributed across multiple machines and disk drives. Further, due to high availability requirements a remote backup site needs to be used that replicates the primary site. When failing over from one site to another, either as a planned and/or unplanned action, the files need to be replicated in a consistent manner on the backup site in real or near real time with minimal or no data loss. Once, the files are restored on the backup site, that site can preferably take over operation from the site that was previously being backed up.
Presently, there are some commercial databases that implement database replication across distributed locations. However, they do not handle the general case of replicating arbitrary inter-related files, across multiple file-systems/machines/disks. In addition, some operating systems (and disk subsystems) can mirror data across distributed sites. This requires that all the files be on the same mirror, that the mirror gets all the writes in order from the operating system and that the mirror replicates in the order of the writes to ensure that all the files are consistent. Typically, when a transaction is lost the mirror needs to be rebuilt and while in the rebuilding process the mirrored site may be in an inconsistent state and may not be available. Some journaled file systems implement the required functionality, but not on a distributed basis.
Some of these prior art systems include fault tolerant clustered file systems, such as Calypso as described at http://www.research.ibm.com/caly, Sprite as described at http://portal.acm.org/citation.cfm?id=42183 and AFS as described at http://portal.acm.org/citation.cfm?doid=35037.35059. These systems implement the required functionality within a cluster but not across clusters. Veritas Cluster Server, as described at http://www.symantec.com/enterprise/products/overview.jsp?pcid=1019&pvid=20—1, implements the required functionality by combining a Veritas file system and volume replicator. The Veritas Cluster Server uses a Veritas file system to track all file modifications and uses a volume replicator to replicate any modified data to the remote cluster. The replicated data is not journaled, but is applied immediately to the remote cluster. Pratima, as described at http://www.linuxjournal.com/article/7265, and DRDB, as described at http://www.linbit.com/en/drbd, implements the required functionality at the block level using a customized Linux device driver. The Pratima and DRDB systems use custom Linux device drivers to replicate block updates across remote clusters. The replicated data is not journaled, but is applied immediately to the remote cluster.
There are several disadvantages and shortcomings of these prior art systems. The Veritas Cluster Server, Pratima and DRDB systems are volume based (not file based). In addition, these systems do not maintain journals, so during a rebuild, the remote cluster is not in a consistent state and cannot be used as a backup. This creates a potentially large window in time in which no backup cluster is available for fail over and potentially cannot be made consistent in the event of a network failure during this window (this can be overcome by replicating to a third location, but requires a third copy of the data and the network resources to make the extra copy). The Veritas Cluster Server requires a Veritas file system and is not currently available for virtual memory system (VMS) and the Pratima and DRDB systems require a Linux O/S and a customized device driver and is also is not available for VMS. These systems require the use of proprietary databases and specialized hardware. Further, these systems are not universal. They cannot back up multiple inter-related files across file systems or machines, they cannot solve the problem of retaining the ability to use the previous backup while rebuilding (unless two backup sites are used and only one site is rebuilt at a time) and they do not ensure file consistency to a specific point in time
Aspects disclosed herein address the above stated needs by providing a system, method and computer program for replicating a system's data. Periodically, the databases on the production site are copied over to the backup site. This process is called a snapshot. PourOver replicates the system's data by applying a snapshot/journaling mechanism. In essence the “snapshot” is taken periodically of all of the read/write databases (across all file-systems/machines on the production site), and changes to the databases (across all file-systems/machines on the production site) are saved in journal files, which contain the delta between the database snapshot and the current state of the actual databases. All transactions are recorded or journaled in real or near real-time and are sent over the network to the backup site immediately. Each recorded transaction “belongs” to the last recorded snapshot. The process replicates the databases across clusters, in order to achieve real and/or near real-time database consistency between clusters (e.g., continually provide the ability to construct a consistent database on the alternate cluster that is up to date within a few minutes worth of transactions). In order to replicate across clusters the latest snapshots are copied, while the journals are streamed in real or near real-time to the destination cluster. Due to the desire to limit the impact on the existing network management computer (NMC) architecture, one aspect uses near real-time replication. Real-time replication requires the NMC processes be changed to use an asynchronous database update model, which is a major change to the current process architecture of the NMC. Once the copying of all the databases that compose a snapshot is complete, the destination cluster can rebuild the databases by replaying the journal files (i.e., applying the changes described in the journal files to the snapshot database files). Thus, the timeliness of the destination database is dependent upon the delay between the streaming of the journal records and the time of their insertion into the journal file.
The backup site processes collect the transactions, and stores them in configurable locations in the order that they were generated. Each transaction is then replayed onto a snapshot database in order to rebuild the database that is up-to-date with the production database.
The entire process beginning with recording of the transactions up to replaying them is streamlined and enables replication of data in near real-time. The backup site itself can be operated in a high-availability mode where standby backup processes take over in the event that the “active” backup processes fail.
The aspects of the present invention provide continuously backing up of a set of potentially inter-related files, such that they can be restored to a specific moment in time, thus ensuring their inter-consistency. The backup location may be to a local and/or remote site/cluster. The time delay between sending data between sites may exceed the maximum acceptable transaction delay (i.e., only near-real time replication). The files can be distributed across one or more machines/computers and/or disk drives. The backup is made in real and/or near-real time, even when performed to a remote site. The backed up files can be used in real/near-real time at the backup site. When a transaction is lost and a new backup is being started, the previous backup is available until the new backup is in a consistent state. Lost transactions automatically start a new consistent backup set. Once a backup is in a consistent state it remains in the backup until a transaction is lost. No transactions are lost in the normal replication process.
In a second implementation, normal NMC I/O operation doesn't wait for the journal record to be made persistent, so as to not introduce extra latency into normal NMC operations.
Given that NMC operations are mostly I/O bound, and that I/O resources on the active cluster should be spared for normal NMC operations, this implementation transfers the journal records to a backup node where they are made persistent. In this implementation, journaling modules have to deal with a stream (of journal records) that is not flow controlled, because of which, journaling will keep up only to the extent to which “communication resources to backup node” and “I/O resources on backup node” support it. Hence, resource planning is critical to ensuring successful journaling.
This implementation uses Transmission Control Protocol, TCP, connections to transfer records to cluster-wide journaling process, JRNLC, which can be run on the backup node. When there is a backup node, which can be used, this design slightly increases the CPU usage on the active node, but have no additional impact on I/O usage. This implementation allows running journaling on the production node, provided that the CPU & I/O resources are adequate. Therefore, in this implementation, NMC application transfers DB updates to a system journaling process, JRNLS which transfers the records to JRNLC over TCP/IP connections, JRNLS are configured for memory-buffer or file-buffer mode, multiple JRNLSs are run to overcome throughput and ramp-up limits of a single TCP/IP connection, JRNLC writes to multiple JRNL files to overcome maximum IO rate limits of a single file (i.e., single spindle or group), JINTG reads from multiple JRNL files, a helper process, JSTRM copies any database files that need to copied to the backup cluster. JRNLC is set-up for either local or remote mode. A system snapshotting process, JSNAP is responsible for making a copy of databases and performing consistency checks.
The disclosed aspects support arbitrary file replication like Disk/Volume replication and journal files like structured query language (SQL) databases. It also maintains prior replicated files in a consistent state, even during a rebuild (snapshot), limits disk space by automatically taking snapshots based on time/records replicated and removing old backup sets that are no longer required. The system can replay journals in order from older snapshots, is capable of sustaining high transaction rates with small (˜1 k byte) objects and works on VMS.
The disclosed implementations provide a real and/or near real-time backup of a distributed set of files between clusters and ensure that a distributed set of backed up files are consistent with respect to a point in time within a backup set.
An advantage of the aspects of the present invention is that it provides continuous backup with no windows in which the backup site cannot be restored to the last consistent state.
Another advantage of the disclosed aspects of the present inventions is that it allows the backup site to be continuously updated to the latest consistent state available.
Yet another advantage of the aspects of the present invention is that it allows the backup site to be rebuilt to any point in time at which a consistent state is available for a snapshot to the point in time that a transaction was lost, up to the latest available record.
Another advantage of the aspects of the present inventions is that it allows journals to be replayed in order from older snapshots.
Another advantage of the aspects of the present inventions is that it replicates only the changed records of a file in the journals.
Another advantage of the aspects of the present inventions is that it copies only the journaled files (not a whole volume) on a snapshot.
Another advantage of the aspects of the present inventions is that it provides fast replication even for large numbers of small records.
Another advantage of the aspects of the present inventions is that it does not add any significant disk I/O load to the local cluster.
Another advantage of the aspects of the present inventions is that it does not require any special hardware.
Another advantage of the aspects of the present inventions is that it is not an operating system or file system dependent
As database updates 124 are made, a description of the changes are appended 126 to journal files 118 in the active snapshot/journal set 104, which will grow until the point in time that a new snapshot is taken. Once copying of databases 106 for a snapshot is complete, databases 112 have been replicated across clusters and associated journal files 118 have been streamed to the point in time that the database replication is completed, previous snapshot journal sets 100, 102 can be deleted. Therefore, in order to reduce disk space consumption, periodic snapshots should be made by the journaling components based upon the size of the journal files and/or the time interval between snapshots. It is also desirable to make snapshots in order to reduce the time to reconstruct the databases from journal files 118, since the time to reconstruct a database from a journal is in direct proportion to the number of records in journal files 118 and the size of databases 106.
All updates to a database 124 are journaled 126.
PourOver is configured using a configuration file, which specifies the databases to be journaled 106 (for both clusters), journal files to use 118 and the other configurable information. The journal configuration file itself is part of each snapshot 100, 102, 104. For PourOver there is the concept of the root configuration file. This file is only loaded if no snapshot has been previously taken or an explicit load command is performed. Under normal conditions the PourOver processes will load the configuration file from the currently active snapshot 104, unless specifically commanded to do otherwise.
In order to ensure that databases 106 can be reconstructed in a consistent manner (i.e., the reconstructed databases are identical to the original databases at some exact instance in time), it is necessary to be able to replay database updates 124 in the exact order in which they were originally made (across all processes in the cluster).
When a process wishes to update a database 106 it replaces the usual system I/O request with the equivalent journal I/O request 200 (e.g., sys$update is replaced by journal_update, . . . ). Journaling operation 202 then performs the following steps:
Get Record Lock 300. A record lock 318 is acquired for the record being modified (assuming this is a record operation). The name of record lock 318 is based upon the file ID and the key of the record being modified.
Get Journal File Lock 302. The journal file to be used for the journaling operation is locked 320.
Get PourOver lock 304. Pour over lock 322 is used to synchronize all journaled requests.
Increment the cluster wide unique ID 316 and release/update pour over lock 306. Update Database 308. Database 120 is updated (as would have been done originally).
Append Journal Entry 310. Locked journal file 128 is appended to, with just enough information to reconstruct the database update.
Release Journal File Lock 312. Journal lock 320 is released.
Release Record Lock 314. Record lock 318 is released.
The systems starts by getting record lock 300 (assuming a record oriented operation) to ensure that no other process can update the same record between the time of the database update and the journal update. This is important, since otherwise it would be possible for the entries in the journal files to be in a different order than the actual updates to the database. Next journal file lock is obtained 302. This maintains increasing record identifications (ID's) within a journal file. Once the journal file is locked the process proceeds to get pour over lock 304 to update cluster wide unique ID 316. Once the pour over lock is obtained, unique ID 316 is updated and pour over lock is released 306, journaling the transaction (updating the database and journaling the change) is performed 308 and 310. Incrementing unique ID 316 is an integral part of the transaction, since aborting anytime between incrementing the ID and journaling the operation will cause a missing ID in the journaled files and will be interpreted as a missing record on replay. Thus, making the journal/snapshot set no longer re-playable beyond that point. Finally, all remaining locks 312 and 314 are released (note, the unlocking order is not necessary).
There is a potential race condition between the point in time journal lock is acquired 302 and when pour over lock is acquired 304. It is possible for another process to request a journal snapshot 324 between these operations. If a snapshot is taken after journal file lock is acquired 302 and before pour over lock is granted 304, unique ID 316 in pour over lock 322 will refer to new journal/snapshot set 104, while journal file lock 320 will refer to the pre snapshot journal file. Thus, if one were to proceed, it would journal the operation into the old journal file with an out of sequence snapshot ID. In order to resolve this race condition, journal file lock 320 must be released and reacquired whenever the snapshot number changes between the two operations.
Note, causing a missing journal ID in the case that a process crashes between updating pour over lock 322 and updating the journal file is a means to detect a failure in the critical region between the time that a database update is made and the time that the update is journaled. If a process crashes between the time that a journaled database is updated and the transaction is journaled, on replay it is not possible to reconstruct the lost transaction. Therefore, a new snapshot must be made in order to recover the missing operation. It should be noted, that by treating the moment that unique ID 316 gets updated as the start of the critical region, crashes will be detected between the pour over lock update and the database update as a critical error, even though it is not. However all crashes between the database update and journal update will be appropriately detected.
The process of journaling an I/O operation has been designed to make minimal assumptions about the processes that are using the journaling calls and the databases being journaled. The basic goal is to provide a drop in replacement for the standard I/O routines without any other modifications to the original code. Initialization of the journaling code is performed on the fly when the first journaling call is performed and further state synchronization is done either as further I/O calls are made or by asynchronous timer interrupts.
The journal entries consist of:
Once the lock block is updated, the application applies the change across the cluster by releasing write lock 306 on pour over lock 322. The very next process to obtain a write lock on pout over lock 322 will get the modified values and in turn proceed to make any journaling updates into the journal files associated with the updated snapshot ID (regardless of whether or not the snapshot process has begun to make a snapshot of the underlying databases).
Between snapshot requests the snapshot process may choose to stream active snapshot journal set's 104 journal files 118 looking for a missing record (i.e., a process crashed in a critical region while journaling). If a missing record is detected, snapshot process 512 will itself request that a new snapshot be taken. Snapshot process 512 will also request a new snapshot based upon the time interval between snapshots exceeding a maximum interval and/or number of records.
Since the journaling stream is not flow controlled at the source, it is preferred to use a smaller receive buffer so that the ramp-up is faster. This will limit the peak rate on a single TCP connection 806. Higher peak rates can be achieved by streaming over multiple TCP connections. Because of the nature of TCP connections 806, JRNLS 802 will have a buffer to store journal records to accommodate temporary slowness in TCP connections 806 due to ramp-ups. To get better peak rates than a single connection can support, multiple JRNLSs should be run.
JRNLS 802 receives journal records from NMC applications via mailbox (Mbx) 808.
A TCP write can sometimes take up to 500 ms, so ‘blocking TCP writes’ cannot be used because it could result in dropped mail packets as the mail stream from NMC applications is not flow-controlled. JRNLS uses asynchronous/non-blocking TCP writes so as to return immediately to the event loop to process the mailbox, while TCP write is going in parallel.
In most cases where the network link is congestion free and reliable, a memory buffer 814 as shown in
If the necessary throughput cannot be attained, then multiple JRNLSs can be started. JRNLS will be able to handle quick and temporary network glitches by re-transmitting journal records from the buffer. JRNLC will be able to handle duplicate journal records. A re-transmit index will be used in JRNLS to point to the journal record in the buffer from where the re-transmit begins.
Multiple journal files should be used only if they can leverage parallel use of multiple spindles or disc groups. In order to keep the journal record stream processing lean (to prevent any latency) JRNLS 802 and JRNLC 818 will avoid performing lengthy operations. So, JRNLS 802 will delegate any lengthy processing to JSTRM 834 on the same gateway. Because of this, JSTRM 834 will become a necessary process on every gateway. To avoid latency problems JRNLC 818 does not load the journal configuration file. This moves some of the functionality, which is performed in the first implementation from the JRNLC 818 into JINTG 836.
The processes running on the remote node will also support local mode. In this mode end-to-end journaling can be performed on the same cluster. This will be supported for sites which don't have a backup node and have resources to support journaling on the local nodes.
Those of skill in the art would understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
The steps of a method or algorithm described in connection with the aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), flash memory, Read Only Memory (ROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these implementations will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
