Data stored on computers is lost or damaged every day. Accidents, human error, virus attacks, hardware failures and power problems are just some of the thousands of possible reasons for loss or damage of information stored on a computer. To protect against the unexpected loss of data, smart people (and businesses) commonly backup their files. A backup can be made by simply making a copy of a file or set of files on some kind of removable medium for use in the event of failure or loss of the original, or the data can be compressed as it is copied, using a backup utility. When a data loss or data corruption occurs, the damaged or lost file or files are typically restored from the backup. “Restoring” in this sense, means copying from the removable medium back to the computer or copying and decompressing the data, if a data utility were used. When the files are small, and when a backup is available, restoring files from a backup is a convenient and efficient means to regain information.
As the size, importance and/or the degree to which the files change over time increase, simple copies of files taken periodically are no longer so appealing. For example, suppose a business depends on the reliable availability of a set of very large files that change frequently, as would occur in database files maintained by an airline, for instance. Periodic snapshots of the data (a set of files and directories taken at a particular point in time) may no longer be sufficient. Mirroring may be a better choice. A mirror in computing is a direct copy of a data set such that there are exact duplicate copies of the data on separate machines. The copies are created and then are continually updated so that the copies stay synchronized with the principal database. The mirror can be maintained as a physical copy at the hardware level or through database mechanisms (sometimes called “replication”). A mirror is differentiated from a snapshot in that a snapshot represents the state of the file or database at a particular point in time. A mirror, in contrast, is an active, dynamic copy which is kept up to date with a dynamically changing source.
When a small portion of a database becomes corrupt, the option of restoring the entire database from backups is not optimal because most of the work performed is unnecessary (most of the database is fine). The restoration process is slow, requires the handling of external media (backup tapes or backup disks) and requires human intervention (a database administrator to select which backups to use, etc., a computer operator to find and load the tapes, maybe others). Furthermore, while the restore process is occurring, the database is typically not available to users. Another way to handle the corruption of a page is to try to repair the page. Repairing a page is fast but almost always results in partial or complete loss of the page data, causing logical inconsistencies within the database.
It would be helpful if there were a way to regain the data stored on a corrupted page (a page is a fixed number of bytes of data recognized as a unit by the DBMS, usually 8K bytes) that would be fast and would result in no lost data or data inconsistencies. It may be useful to have this process initiate automatically upon detection of the data corruption and occur without human intervention, without requiring the management and handling of tapes or other removable media.
A set of servers takes advantage of the existing data redundancy of a mirrored database to restore page corruptions. The page restore may occur with none of the time and/or administration costs of a restore from backup media and without the data loss associated with repair. Furthermore, online page restore from a database mirror can be initiated and performed by the computer system automatically upon corruption detection. The concept can be extended to allow the restore of an entire file or database instead of an individual page or set(s) of pages. The mechanism can be used both to restore corrupt pages on the principal server from a mirror, or corrupt pages on a mirror from the principal server. Online page restore from a database mirror enables nearly instantaneous fixing of page corruptions without data loss. It also allows page data recovery without the need to find/load/scan through and apply data and log backups, allowing efficient and potentially automatic data recovery.
Hence a restore can be performed without requiring backups to be provided or even to be existent. One or more pages may be requested from one or more mirrors, and verification may be performed to ensure that the returned pages are caught up in time to those of the principal server at the time of corruption detection (a “redo” operation on the mirror may not be caught up to the principal's “do” operation when the page request is received by the mirror). Page corruptions may be automatically fixed during a crash recovery scenario or during normal operation when a corruption is detected. During crash recovery, corrupted pages deterring transaction rollback may be automatically restored with no human intervention, enabling rollback of deferred transactions to proceed without human intervention. When multiple mirrors are available, the mirror selected to return the requested page may be selected based on which mirror has the fastest response time historically or based on which mirror is furthest along in replaying the log from the principal server (i.e., which mirror is most up-to-date). A multi-page restore across multiple mirrors may be load-balanced. One or more page restores may be performed automatically upon corruption detection or may be user-driven. Pages may be served for read only queries from the mirror until the corruption on the principal server is fixed. Pages may be served for read/write queries from the mirror until the corruption on the principal server is fixed. Alternatively, the mirror may become the principal server. A corruption on the principal server may be restored from a mirror and conversely, a corruption on the mirror may be restored from the principal server.
In the drawings:
Overview
In accordance with embodiments of the invention, a disaster recovery solution that is entirely software based is described more fully below. Briefly, a simple scenario would be:
1. a corrupt page is detected
2. the database administrator executes a restore function (e.g., “RESTORE DATABASE foo PAGE x FROM MIRROR”) or the restore function is initiated automatically (without human intervention) by the computer
3. the server locks the corrupt page in the database
4. a request is sent from the principal server to at least one mirror asking for the database page. The request includes the page identifier(s) of the corrupted page or pages and a log sequence number (LSN) for the current point in time on the principal. The LSN for the current point in time is provided because the LSN on the corrupted page cannot be trusted.
LSNs are important because SQL Server writes changes made to the database to a transaction log, so that if a transaction starts but fails to complete, the changes from the log can be retrieved and re-applied (“rolled back”) or can be undone. When a transaction commits, SQL Server writes all the log records pertaining to that transaction to permanent storage on disk. Thus, even if the system fails before SQL Server writes the changed data pages to disk, the log records are on disk. When SQL Server starts again, the log provides enough information to recover, or roll forward, any transactions that completed but whose corresponding data pages were not written to disk. Each record written to the transaction log is assigned a (generally increasing) sequential log sequence number, providing an easy way to track the order in which transactions were applied.
5. the mirror waits for its “redo” operations to pass the LSN provided in the request to ensure that all changes to the requested page have been replayed from the log and applied to the page.
6. the mirror fetches the page from either its buffer pool or its disk. The page now held by the mirror would be guaranteed to be consistent with the database on the principal server, because no updates could have been made in the interim to the page (because it was locked at step 3) and the log has been replayed past the point of locking (at 5).
7. using the database mirroring communication infrastructure, a new message type is used to send the page from the mirror to the principal.
8. upon receiving the page, the principal writes the page to disk to persist the restore and releases the lock, making the fixed page available for queries again.
Error handling processing associated with what happens when mirrors are unavailable, what happens when mirroring is suspended, what happens if a database mirroring failover is triggered during a restore operation, etc. may be provided.
In automatic mode, the process is initiated automatically by the computer without human intervention upon detection of a corrupt page either during crash recovery or during regular operation. When a corrupt page is detected, locks are held automatically for the transaction doing the update to that page. A deferred transaction is a transaction that cannot be resolved (aborted or committed) until some external event occurs. Within the present context, the indicated event is recovery of a consistent page, which can be automatically generated. (Traditionally, the resulting ‘deferred transactions’ require administrator intervention to resolve the underlying issue.) When the automatic mode feature of the online page restore from a database mirror mechanism is invoked, the corrupt pages (identified by page ids) are locked, the page restores are completed from the mirror, and then code for rolling back the deferred transactions can be invoked, resulting in seamless repair of database page corruptions.
The described technique can be extended to restore an entire file (for instance in the case of a disk crash) from the mirror. Alternatively, the database may fail over to the mirror, the mirror becoming the active (principal) database and the failed copy becoming a mirror. The mirror, in this case, may become the target of the automatic repair, by shipping data from the new principal. If manual mode (human intervention required) is invoked, a file location is optionally specified for cases where the original location is not usable. For automatic mode, a default location may be attempted, otherwise the server may wait for a manual operation to be executed.
If automatic mode is turned off or is not implemented, the underlying page restore mechanism may be used to serve pages from the mirror for read-only queries until corrupt pages are fixed. In this option, the method described above is followed but the page(s) received from the mirror is not written back to disk on the principal. This enables greater data availability while still allowing the administrator to maintain manual control of the restore.
Optimizations include:
1) With multiple mirrors, the principal keeps track of which mirror responds fastest (faster response time may be due to a number of factors, including network differences, physical location, and so on) and request the restore pages from the fastest-responding mirror.
2) If multiple mirrors are at different stages of being caught up with the redo from the log received from the principal, the principal may ask the most up-to-date (current) mirror for the restore pages.
3) When a number of pages are being restored, the mirror may load balance by asking for blocks of pages from the different mirrors.
Exemplary Computing Environment
Although not required, the invention can be implemented via an application programming interface (API), for use by a developer, and/or included within the network browsing software which will be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers, such as client workstations, servers, or other devices. Generally, program modules include routines, programs, objects, components, data structures and the like that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations. Other well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers (PCs), automated teller machines, server computers, hand-held or laptop devices, multi-processor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network or other data transmission medium. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
With reference to
Computer 110 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 110. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. A graphics interface 182, such as Northbridge, may also be connected to the system bus 121. Northbridge is a chipset that communicates with the CPU, or host processing unit 120, and assumes responsibility for accelerated graphics port (AGP) communications. One or more graphics processing units (GPUs) 184 may communicate with graphics interface 182. In this regard, GPUs 184 generally include on-chip memory storage, such as register storage and GPUs 184 communicate with a video memory 186. GPUs 184, however, are but one example of a coprocessor and thus a variety of coprocessing devices may be included in computer 110. A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190, which may in turn communicate with video memory 186. In addition to monitor 191, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 195.
The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in
When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
One of ordinary skill in the art can appreciate that a computer 110 or other client device can be deployed as part of a computer network. In this regard, the present invention pertains to any computer system having any number of memory or storage units, and any number of applications and processes occurring across any number of storage units or volumes. The present invention may apply to an environment with server computers and client computers deployed in a network environment, having remote or local storage. The present invention may also apply to a standalone computing device, having programming language functionality, interpretation and execution capabilities.
Online Page Restore from a Database Mirror
In some embodiments of the invention, online page restore from a database modules 306, 326, 336, etc. comprise a disaster recovery solution that is entirely software based, as described more fully below. In some embodiments of the invention, the online page restore module detects a corrupt page or pages on the principal server. In manual mode, the module 306 may receive an instruction executing a restore function. An exemplary, non-limiting instruction may be, for example, “RESTORE DATABASE foo PAGE x FROM MIRROR”. Alternatively, in automatic mode, upon detection of the corrupt page, the restore software 306 may be invoked automatically by the computer without human intervention. The principal server 302 may then lock the corrupt page or pages (represented by page 309 in
Error handling processing is performed if a mirror or mirrors are unavailable, mirroring is suspended, or if a database mirroring failover is triggered during a restore operation, etc.
In automatic mode, the process is initiated automatically by the computer without human intervention upon detection of a corrupt page either during crash recovery or during regular operation. When a corrupt page is detected, locks are held automatically for the transaction doing the update to that page. (Traditionally, the resulting ‘deferred transactions’ require administrator intervention to resolve the underlying issue.) When the automatic mode feature of the online page restore from a database mirror mechanism is invoked, the corrupt pages (identified by page ids) are locked, the page restores are completed from the mirror, and then code for rolling back the deferred transactions can be invoked, resulting in seamless repair of database page corruptions.
The described method can be extended to restore an entire file (for instance in the case of a disk crash) from the mirror. If manual mode (human intervention required) is invoked, a file location is optionally specified for cases where the original location is not usable. For automatic mode, a default location may be attempted, otherwise the server may wait for a manual operation to be executed.
If automatic mode is turned off or is not implemented, the underlying page restore mechanism may be used to serve pages from the mirror for read-only queries until corrupt pages are fixed. In this option, the method described above is followed but the page received from the mirror is not written back to disk on the principal. This enables greater data availability while still allowing the administrator to maintain manual control of the restore.
Optimizations include:
1) With multiple mirrors, the principal keeps track of which mirror responds fastest (faster response time may be due to a number of factors, including network differences, physical location, and so on) and request the restore pages from the fastest-responding mirror.
2) If multiple mirrors are at different stages of being caught up with the redo from the log received from the principal, the principal may ask the most up-to-date mirror for the restore pages.
3) When a number of pages are being restored, the mirror may load balance by asking for blocks of pages from the different mirrors.
If the online page restore is operating in automatic mode, at 504, upon detection of the corruption, the restore is initiated by software running on the computer and without requiring human intervention. In automatic mode, the parameters described in the preceding paragraph are set by the online page restore software running on the computer. At 508, whether in manual or in automatic mode, the corrupted page or pages in the principal database is/are locked. At the point of locking, the corrupted page or pages become unavailable but the rest of the database is still accessible (e.g., available for user queries and updates, etc). The corrupted page or set of pages is requested from at least one mirror. The mirror or mirrors that receive the page request may be selected based on the mirror which has historically had the fastest response time, based on the most up-to-date (current) mirror or based on other suitable criteria. If a large number of pages or the entire file have to be restored, load balancing may be performed by sending an (optionally non-overlapping) request for a subset of the required pages to a number of mirrors. Suppose, for ease of understanding that a single page is corrupted and an appropriate mirror, say, mirror 1 has been selected to receive the page request. It will be appreciated that the invention as contemplated is not so limited however. At 508 the principal server may send a request for page id X to mirror 1 and a LSN as described above. At 510, the mirror may receive the request, and may wait until its log updates have been applied to the LSN received to ensure that the page is up-to-date (all changes to the page made before corruption detection) have been applied. Once the log has been applied to at least the point indicated by the received LSN, the mirror page or pages corresponding to the corrupted page may be fetched either from a buffer pool on the mirror or from a mirror disk. A message of a specified type identifying the message as an online page restore message may be generated and sent to the principal server. At 512, the page may be received at the principal server and applied to the database. The page may be written to disk to persist the restored page. The lock may be released (514), making the restored page available for queries and updates as determined by the characteristics of the database.
In some embodiments of the invention, while the corrupt page(s) are being restored, queries directed to the corrupt page or pages may be served from the mirror, allowing for greater availability of the data.
The various techniques described herein may be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention. In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. One or more programs that may utilize the creation and/or implementation of domain-specific programming models aspects of the present invention, e.g., through the use of a data processing API or the like, are preferably implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations.
While the present invention has been described in connection with the preferred embodiments of the various figures, it is to be understood that other similar embodiments may be used or modifications and additions may be made to the described embodiments for performing the same function of the present invention without deviating therefrom. Therefore, the present invention should not be limited to any single embodiment, but rather should be construed in breadth and scope in accordance with the appended claims.
Number | Name | Date | Kind |
---|---|---|---|
5307481 | Shimazaki et al. | Apr 1994 | A |
5966706 | Biliris et al. | Oct 1999 | A |
6205449 | Rastogi et al. | Mar 2001 | B1 |
6772309 | Ohr et al. | Aug 2004 | B1 |
6799189 | Huxoll | Sep 2004 | B2 |
6877016 | Hart et al. | Apr 2005 | B1 |
6883112 | Maeda et al. | Apr 2005 | B2 |
7177993 | Kiselev et al. | Feb 2007 | B1 |
7254683 | Burton et al. | Aug 2007 | B2 |
20030097533 | Maeda et al. | May 2003 | A1 |
20040002975 | Scardo et al. | Jan 2004 | A1 |
20040030721 | Kruger et al. | Feb 2004 | A1 |
20040098425 | Wiss et al. | May 2004 | A1 |
20040215998 | Buxton et al. | Oct 2004 | A1 |
20050055559 | Bucher | Mar 2005 | A1 |
20050071391 | Fuerderer et al. | Mar 2005 | A1 |
20070174583 | Shinkai et al. | Jul 2007 | A1 |
Number | Date | Country |
---|---|---|
2 273 180 | Jun 1994 | GB |
Number | Date | Country | |
---|---|---|---|
20070038682 A1 | Feb 2007 | US |