The present invention generally relates to the efficient recovery of transactional data stores.
With the explosive growth in the number and complexity of Web 2.0 applications, software-as-a-service (SaaS), cloud computing, and other enterprise applications, datacenter workloads have increased dramatically. The business opportunities created by these new applications are substantial, but the demands they place on the datacenter are daunting.
The success of modern web sites and other enterprise applications depends heavily on the ability to effectively scale both the data tier and the caching tier on which these applications depend. Unfortunately, ordinary server, database, data store, and caching infrastructures are loosely integrated and minimally optimized. As a result, existing datacenter solutions do not adequately address the performance, capacity, scaling, reliability, and power challenges of supporting dynamic online data and services effectively.
Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
Approaches for the efficient recovery of transactional data stores are described. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
Embodiments of the invention are directed towards the replication of write sets in a distributed transactional system. Embodiments may be employed in a wide variety of distributed transactional systems. For example, embodiments of the invention may involve the replication of write sets in many different types of distributed object stores, such as a memcached caching system, a MySQL database, or a key-value data store. Further, in certain embodiments, nodes of the distributed transactional system may chiefly or wholly employ the use of solid state devices to persistently store data. Advantageously, the architecture of embodiments is specifically tailored for using solid state devices in a fast, efficient, and scalable manner to obtain better performance than prior approaches.
Device 100 may be used in a variety of contexts to efficiently manage large amounts of data. To illustrate the capabilities of device 100, consider
In contrast, in the approach of embodiment 240, device 100 may perform the work of all of the one or more slave DBMSs 214. Thus, in the example of
Returning to
Operating environment 120 refers to software that is designed to support the operation of object store 130 on hardware platform 110. Operating environment 120 may be specifically tailored to operate efficiently on one or more solid state devices comprised within hardware platform 110. The embodiment of
Caching component 122 refers to software components, within operating environment 120, which are responsible for performing caching services in a manner that is optimized or specifically tailored for solid state devices. Caching component 122 may support write-through and/or write-back caching.
SSD access component 124 refers to software components, within operating environment 120, which are responsible for enabling highly parallel read and write access to solid state devices. SSD access component 124 may be configured to minimize the wear of solid state devices and provide data durability with high performance. SSD access component 124 may provide redundant array of integrated disks (RAID) support.
Scalability component 126 refers to software components, within operating environment 120, which are responsible for ensuring that object store 130 may scale to support a large number of users. In certain embodiments, scalability component 126 may provide fine-grain locking, scalable and concurrent data structures, optimized thread-to-core allocation, and efficient handling of network interrupts.
HA/DR component 128 refers to software components, within operating environment 120, which are responsible for ensuring that object store 130 is highly available as well as for recovering object store 130. In an embodiment, HA/DR component 128 may perform synchronous and/or asynchronous replication of data within object store 130, perform failure detection of object store 130, automated virtual IP address (VIP) failover, perform incremental data recovery, and perform an incremental or full online backup and restore process.
As broadly used herein, object store 130 refers to software designed to store, either persistently or non-persistently, objects within an organized data store. Typically, object store 130 receives and processes requests from one or more of clients 50(1) to (N). In processing such requests, object store may store objects on or read objects from storage mediums within hardware platform 110, such as a solid state device.
Object store 130 may correspond to a variety of different types of mechanisms for storing data, such as a MySQL DBMS, a memcached object caching system, or any type of key-value data store for example. In certain embodiments, object store 130 may implement a NoSQL database while in other embodiments object store 130 may implement a traditional relational database.
In
MySQL object store 132 refers to a MySQL DBMS, memcached object store 134 refers to the memcached caching system, and key-value object store 136 refers to any type of key-value data store. Object store 130 may support a wide variety of different types of object stores, and so, object stores 132-136 are merely illustrative of several examples data stores of embodiments and are not intended to be a comprehensive list of all the types of data stores which may be implemented by object store 130.
Hardware platform 110 may comprise one or more solid state devices (SSDs) 310 and one or more parallel SSD controller(s) 312. As broadly used herein, SSD(s) 310 may be implemented using any type of solid state device, although examples discussed herein shall be explained in the context of SSD(s) 310 being implemented using flash memory. Each SSD in SSD(s) 310 contains a write cache 328.
In an embodiment, hardware platform 110 may include one or more hard-disk drive(s) 314 and one or more HDD controller(s) 316. In an embodiment, each HDD controller in HDD controller(s) 316 may include a non-volatile (NV) DRAM 326. In an embodiment, NV DRAM 326 may store one or more of transaction log(s) 330 and one or more double-write buffer(s) 332 for object store 130.
NV DRAM 326 may be constructed using a DRAM which includes a battery so that if the power to the DRAM is disrupted, the battery will supply power to the DRAM, thereby ensuring that the data stored thereon may be persistently stored (at least until the battery runs out).
In an embodiment, hardware platform 110 also comprises network controller 318, PCIe HUB 320, one or more processors 322, and dynamic random access memory (DRAM) 324.
Embodiments involve the synchronous, semi-synchronous, and/or asynchronous replication of transactions performed in a distributed object store. Embodiments of the invention may comprise a plurality of nodes that may collectively be referred to as a cluster. Each node of the cluster may correspond to a machine that executes one or more instances of a transactional system. For example, each instance executing on a node of the cluster may correspond to a database management system (DBMS), a memcached application, a MySQL database, or any type of object store. Each node of a cluster may, but need not, correspond to device 100 of
In an embodiment, a number of transactions may be performed by each instance executing on a node of the cluster. When a transaction is committed by an instance, the instance assigns to the transaction a global commit number (other names may be used to refer to a global commit number by embodiments). The global commit number is an identifier that may be used by any instance of the cluster to determine when, relative to other transactions committed by an instance of the cluster, the transaction associated with the global commit number was committed. As certain transactions may need to be applied in a particular order (since a transaction may assume or require that a prior transaction has already been committed), global commit numbers are used to ensure that replicated transactions are applied in the proper order.
Write operations may be replicated serially from one node to another. To replicate a write operation serially, initially a write operation is sent from Node A to Node B. Node B then reads the received write operation, and determines to which object the write operation is writing. Node B next determines in which data block the object to be written is stored and then loads that data block in a buffer cache in memory. After the data block is loaded in the buffer cache, Node B then performs the write operation to object B stored in the buffer pool. Node B may then commit the transaction after the changes to object B are, in some fashion, persistently stored.
It is observed that write operations may be replicated in less time if portions of the process are performed in parallel. Thus, in an embodiment, Node B may maintain a list of write operations which have been replicated to Node B. Each of the write operations on the list is to be performed by Node B, although there is no requirement that each write operation on the list was sent to Node B from Node A, as any node of the cluster may send replicated write operations to Node B for Node B to perform. Node B may scan the list of write operations to determine which objects are written to by the write operations in the scanned portion of the list. After Node B determines which objects write operations in the scanned portion of the list reference, Node B may thereafter determine which data blocks contain objects referenced by the write operations in the scanned portion of the list. Node B may then load those data blocks into the buffer cache memory concurrently.
By loading all or most of the blocks concurrently, Node B can load in the buffer cache all or most of the data blocks which will be required to process write operations in the scanned portion of the list in roughly the same amount of time as Node B would require to load a single data block, since this process may be performed in parallel. Moreover, while Node B still needs to observe data dependencies, data blocks loaded into the buffer pool may be written to in parallel. For example, a write operation to object B and a write operation to object C may be performed in parallel, since the performance of one operation does not affect the performance of another. However, two write operations to object B should be performed in the order in which the write operations were issued—which can be determined using the global commit number.
This approach may be used in synchronous, semi-synchronous, or asynchronous replication and may be used in a variety of different types of object stores. For example, it is observed this approach has particular utility in a MySQL relational database management system (RDMS) involving asynchronous replication.
Pre-fetching data blocks in parallel is especially important in embodiments that use flash memory as the block storage system. Flash memory supports much more concurrency and IO throughput than traditional hard disk storage subsystems. To fully exploit this high throughput, the reading and writing of data blocks must be parallelized as much as possible.
In a MySQL RDMS, change information that identifies changes to the object store of the master is stored in a bin log. This change information currently takes one of three forms. Change information may be (a) statement-based information that identifies the processed SQL statements, (b) row-based information that identifies the particular changes made to rows of the object store, or (c) mixed mode where statement-based information coexists with row-based information and each mode is utilized as deemed necessary. After the change information is stored in the bin log, one or more slave instances are able to read the change information from the master's bin log and save it locally to a relay log. A new slave can join the asynchronous replicated cluster by providing the binary log file name and an offset (offset represents a point in time) in the file where it can begin or continue applying updates that are read from the master. Thereafter, the relay log is read and the change information applied to the object store of the slave. After the change information is applied to the object store of the slave, the object store of the slave should reflect the state of the object store of the master at the time the change information was copied to the bin log.
It is observed that the current approach for performing asynchronous replication in a MySQL RDMS yields a variety of subtle problems that makes it difficult to ensure the states of the master's object store and the slave's object store are the same due to the nature of the change information. Embodiments of the invention overcome these limitations through the use of change information that identifies per-transaction write operations. Thus, in an embodiment, a node of the cluster logs a per-transaction write set, which is the collection of all write operations of each individual transaction that have been performed to the object store maintained at that node. Advantageously, the write operations in the log may be ordered in the logical order in which they were committed. In other words, the per-transaction write sets in the log may be ordered by the global commit number associated with each transaction. Write operations in the log may then be replayed against the object store of the slave in the same order in which the write operations were committed. As a result, asynchronous replication may be performed faster and with greater assurances of data coherency between object stores in the cluster than prior approaches.
In a MySQL RDMS, a slave communicates with the master to read from the tuple {filename, location in file referred as offset}. However, when any slave is promoted as master, the filename and location with the file may be different from the earlier master. Due to this issue, rejoining a failed-over master may include manual intervention and is error prone. The ability to use global commit number effectively solves the above stated issues by employing global commit numbers to search for a point in time in any replicated data stream, irrespective of the master or slave(s).
In one embodiment the replication stream may take the form of a network packet based stream or a file.
Grouping write operations into per-transaction write sets enables the write sets to be applied on slave nodes in parallel. Currently, databases typically require that the same thread process an entire transaction from start to finish, which makes is difficult or impossible to parallelize individual write operations within transactions. However, if write operations are grouped into per-transaction write sets, then the write sets can be processed in parallel with a dedicated thread processing each write set from start to finish.
Processing write sets in parallel may require dependency checking to ensure that data dependencies are maintained. A non-limiting, illustrative example of such dependency checking include ensuring that write operations to the same data object are performed according to global transaction commit order. Another example of dependency checking is ensuring that write operations to columns in a particular table that are used as foreign key in another “child” table are ordered with respect to writes to the “child” table. For example, table A may have a column C_A that references column C_B in table B as a foreign key. This means that any row written to table A must use a value in column C_A that already exists in column C_B in table B. Whenever an application requires a new value for column C_A that does not already exist in column C_B, it must first write the new value into table B before doing a write to table A. If a foreign key ordering constraint like this is satisfied in the global commit order on a master, this order must also be maintained when write sets are applied on slaves. Thus, when applying write operations in parallel at a node, foreign key ordering constraints should be observed and followed.
In an embodiment, enforcing the global write order on individual objects can be accomplished by comparing the object names that are touched by each write set. Only write sets that write to disjoint objects can be applied in parallel. Foreign key constraints are enforced by serializing the processing of any write sets that operate on tables with foreign key dependencies (either the parent or child table for the foreign key dependency).
As explained above, embodiments of the invention may proactively pre-fetch blocks which are referenced by write operations in a write set prior to processing the write operations in the write set.
Embodiments of the invention may employ any replication technique discussed with reference to an asynchronous replication environment in a semi-synchronous replication environment. In semi-synchronous replication, the master waits for an acknowledgement that data was received by at least one of its slaves before allowing a transaction to commit.
An advantage of write-set based replication is that slave nodes need only apply the write-sets of transactions that are committed on another node. Thus, the situation where a slave applies a portion of the write operations comprised within a write-set before the slave receives notice that the write-set has been cancelled (and thus, the slave needlessly applied the portion of write operations within that write-set) may be avoided, thereby conserving time and resources.
Another advantage is that it is more efficient to dispatch groups of updates from the master node to parallel applier threads executing on the slave node as opposed to dispatching each update individually.
The assignment of the global commit numbers may be aided or accomplished with the use of a distributed system. For example, in
Currently, GCN daemons assume that a master/master replication scheme is used. In other words, GCN daemons assume that any node of a cluster could replicate transactions to other nodes of the cluster and thus leverage an ordering or consensus protocol to monotonically assign GCN through the cluster. However, it is observed that in a master/slave replication environment (that is, a unidirectional replication environment where transactions are only replicated in one direction), the algorithm used to generate the global commit numbers may be optimized using a simpler algorithm than prior approaches.
In one embodiment, the process of agreement on a global commit number between a plurality of nodes in master/slave replication arrangement is simplified and optimized by assigning a master the unique capability of generating and maintaining the GCN. In master/slave replication, the slave(s) do not explicitly start update transactions that require a commit. By not employing the multi-node GCN protocol, the network path involved and the latency of GCN assignment is eliminated, thereby making the process efficient.
In one embodiment, the process of agreement on a global commit number between a plurality of nodes in a master/master replication arrangement is optimized by assigning a node a batch of commit numbers. In the cases where a particular node has queued transactions ready for commit, the node utilizes a batch of commit numbers to make forward progress without involving a network based protocol on getting each commit number. This reduces the network patch involved in the GCN assignment and reduces the average latency of GCN assignment, thereby making the process efficient.
Reducing the complexity of assigning the global commit order can significantly increase the number of transactions that can be completed per second. It is of particular benefit in asynchronous replication applications in which there is a long communication between nodes in the cluster. Such situations arise when data centers are thousands of miles apart.
When a node of a cluster initially becomes operational, the node (hereby denoted the “recovering node”) needs to update its object store to reflect the same state as the other object stores of nodes of the cluster. The object store of the recovering node may not contain any data (such as when the recovering node is powered on for the first time) or it may have a partial or incomplete set of data (for example, the recovering node may have been powered down for a period of time, thereby becoming unsynchronized with the remainder of the cluster).
Currently, to recover a node, a copy of an existing node's object store is made. For example, as depicted in
However, while the object store of recovering Node B is being updated in this fashion, the cluster ceases to accept any write requests from clients. This means that existing node A, or any other node in the cluster, cannot perform any write transactions, otherwise the object store on recovering node B will not reflect the current state of the object stores maintained in the cluster.
Embodiments of the invention address and overcome this limitation. In an embodiment, while recovering Node B is synchronizing its object store with other object stores in the cluster, existing Node A retains its ability to process transactions (including write operations as well as read operations) performed locally. In this approach, a copy of the object store of existing Node A is made. A copy of the object store of existing Node A may be stored on recovering Node B. A copy of the object store of existing Node A may be made using a third party utility, such as “Extra backup” or using other methods like LVM snapshot, etc. Before, during, and after the copying of the object store of existing Node A, existing Node A continues to perform transactions as well as send, to recovering Node B, information (denoted “committed transaction information”) about transactions committed by existing Node A to enable recovering Node B to replay those transactions against the object store at recovering Node B. Module B at recovering Node B may maintain the received committed transaction information in a buffer. The buffer may reside in system DRAM or a hard-disk drive (HDD), in a solid state device (SSD), etc. Recovering Node B may initially start the buffering at DRAM and later move the buffer to alternate local storage, such as a SSD, a hard-disk drive (HDD), or the like, if the buffering requires more space than is available on the DRAM.
The copy of the object store of existing Node A is used to synchronize the object store at recovering node B. After synchronizing the object store at recovering Node B, the object store at recovering Node B will reflect the same state as the copy of the object store of existing Node A (at the earlier point in time, and earlier point in global commit order, when the copy was made). At this point, recovering Node B notes the most recent global commit number for transactions performed on the copy of the object store of existing Node A. Recovering Node B uses the most recent global commit number as a “high water mark” by (a) replaying transactions using the committed transaction information stored in the buffer maintained by module B while (b) disregarding any committed transaction information for transactions having a global commit number that is below this high water mark. Recovering Node B may safely disregard committed transaction information that fall below this “high water mark” as these transactions will already have been made to the copied object store of existing Node A. Once recovering Node B has replayed transactions using the committed transaction information maintained by module B, the object store at recovering Node B will reflect the same state as the object store in existing Node A. Note that during the time that recovering Node B is recovering (i.e., synchronizing its object store to reflect the state of other object stores in the cluster), the buffer maintained by module B will still receive committed transaction information from other nodes of the cluster.
While the description of
Embodiments of the invention support incremental recovery. If recovering Node B stores partial data (i.e., the object store at Node B stores some data, but the object store is not current), then the incremental recovery functionality allows recovering Node B to copy the difference instead of copying the entire object store from another node. To accomplish this, when recovering Node B in the cluster goes down or otherwise becomes inoperable, one of the nodes in the cluster that is operational (for example, existing Node A) starts logging replication transaction information. Recovering node B checks with existing Node A (which may involve checking with each node of the cluster sequentially or in parallel) to determine if such replication log exists, and if so, recovering Node B copies the difference rather than copying the whole object store from existing Node A.
In an embodiment, existing Node A (and naturally any and all other nodes of the cluster) may also throttle down the rate at which transactions are committed, thereby reducing the amount of transactions (and by extension the amount of time) which recovering Node B needs to process to synchronize its object store with other object stores of the cluster.
While embodiments have been described by module B reviewing committing transaction information and discarding any committed transaction information for transactions that are lower than the “high water mark,” other embodiments may perform this type of review on the node replicating the committed transaction information. For example, module A on existing Node A may be configured to not replicate any committed transaction information to recovering Node B for transactions that are beneath the “high water mark.”
In an embodiment, the in-memory buffer pool on the recovering node may be “pre-warmed,” that is to say, may be updated to store database pages that are anticipated to be used in the near future. Most databases use an in-memory buffer pool to cache database pages that are frequently used. Since accessing main memory is much faster than accessing disk or flash storage, this dramatically improves performance. When a node is recovering, the recovering node typically starts with an empty buffer pool since no database pages have been accessed yet. Certain known recovery algorithms may be employed to load some pages into the buffer pool of the recovering node as it replays buffered transactions. However, such known recovery algorithms are not usually sufficient to fully prime the buffer pool of the recovering node.
The recovery mechanism of an embodiment of the invention can more fully prime the recovering buffer pool as compared to prior approaches by capturing the list of pages in the “donor” buffer pool at the time of recovery and pre-loading the recovering buffer pool using this list. Priming the recovering buffer pool in this way should increase the likelihood that referenced data will be found in memory in the recovering buffer pool and therefore need not be retrieved from disk. The recovering node does not have to wait for the buffer pool to be fully preloaded—the recovering node can accept client queries while the preloading is carried out concurrently in the background.
The process of transferring backup data from the donor node to the recovering node can consume most or all of the available network bandwidth between the two nodes. This can cause problems in the database cluster by “starving” other cluster management processes that share the network. For example, the cluster manager may send heartbeat messages between nodes to maintain cluster status. If the network is consumed by data transfer for recovery, then such heartbeat messages may be delayed and not be acknowledged within their timeout interval. The cluster manager may then erroneously think that a node has crashed. As another example, the cluster management console may communicate with the nodes of the cluster using the same network that is used for data transfer during recovery. If network response is severely degraded by the recovery process, cluster management operations will become unacceptably slow.
To prevent these forms of degradation from occurring, the data transfer stage of recovery can be “throttled” by limiting the fraction of network bandwidth that it can use. This ensures that there is always a certain portion of network bandwidth that is available for other cluster functions.
There are many ways to perform this throttling. One method is to pipe the network stream from the data transfer process through a bandwidth limiting utility, such as cstream, trickle, or throttle for example. Bandwidth available to the recovery process can also be limited by setting bandwidth limits in common Linux networking utilities such as iptables, the squid proxy, or tc. The bandwidth limit can be set manually or by auto-detecting the bandwidth limit of the network between the donor node and the recovering node and setting the data transfer limit to some reasonable fraction of capacity, such as 80%.
During the recovery process, while the backup is being taken and applied to the recovering node, all new write operations that are performed at any node other than the recovering node are replicated to the recovering node. For a large database, there may be many gigabytes worth of replicated write operations that must be buffered by the recovering node. Since replicate write operations are written in a serial stream, hard disks are the most cost effective medium for storing the replicated write operations. If the hard disks are configured with an intelligent disk controller that can combine many small serial writes into fewer, larger writes to the disk, very high bandwidth can be sustained. When the backup of the data store on the recovering node is complete, the writes that were buffered on disk must be retrieved and applied to the data store on the recovering node to bring the data store on the recovering node up to date with the data store maintained by the donor node. A naive implementation of this process would read each buffered write one-at-a-time and apply it to the data store on the recovering node. With hard disks as the buffering medium, however, this would be very slow because of the limited I/O rate of hard disks (typically <200 reads/sec per disk).
It is much more efficient to reduce the number of read operations to disk by reading the buffered writes in large chunks. For maximum efficiency, the read operations should be pipelined with respect to the application of the writes to the database. In other words, the next chunk(s) of buffered write operations should be pre-fetched in parallel with the application of the buffered write operations that were just retrieved. This prefetching optimization can significantly accelerate this portion of the recovery process.
While on-line recovery is taking place it is desirable to provide a measure of progress to the database administrator. This can be done as follows for each of the three phases of online recovery: In the first phase (corresponding to the backup of the data store maintained on the donor node to the recovering node), to obtain a measure of progress, initially the size of all the files to be transferred from the data store on the donor node to the recovering node is determined. Thereafter, the size of the files received on the recovering node is monitored. A percentage of size of all the files received to the size of all the files to be transferred may be determined and updated in real time. This percentage may be used to determine the progress of the first phase.
In the second phase (corresponding to applying the backup on the recovering node), the utility xtrabackup, available from Percona Inc. of Pleasanton Calif., may be used to determine how much of the apply logs has completed. This information may be used to determine the progress of the second phase.
In the third and final phase (corresponding to applying the buffered client updates to catch up the data store maintained on the recovering node to the current state of the data store maintained on the donor node), to obtain a measure of progress, initially the amount of data waiting to be applied to the data store at the recovering node is determined. This amount is roughly the size of the buffer files. The size of the data waiting to be applied may be maintained below a certain size by using flow control. As the size of the data waiting to be applied decreases, a percentage or measure of how close the process is to completion may be computed.
The measure of progress of the on-line recovery may be displayed to the administrator on a user interface to inform the administrator of how far the recovery progress has progressed. Such information is helpful to show that the recovery progress has not stalled or reached an impasse, as often the on-line recovery process may take many hours to perform.
The term cluster typically refers to a plurality of nodes that host applications that work in concert with each other.
Logical clusters provide great flexibility in managing resources of the cluster. For example, logical clusters may be used in replication and fault tolerance. A logical cluster can be completely transparent to a user of the logical cluster.
A logical cluster may support automatic instance migration, which can be used to support manual or automatic load balancing and “autosharding.” Instance migration is the process of moving an instance on one node to another different node. This is used to reduce the load on a busy server by transferring load to another less busy server. “Autosharding” is the ability to automatically split an instance on one server and migrate one of the resulting instances to another machine. Thus, if at some point an instance is running out of space and resources on a particular node, autosharding would enable the instance to grow or move to another node.
Instance migration may used to entirely move an instance from a first node to a second node, e.g., this may be desirable to facilitate maintenance on the first node. Such maintenance may include, for example, installing a new version of the database code or changing the node configuration (operating system, hardware changes, etc.). Once maintenance is complete, instance migration can be used to move the original instance back to its original home node. In an embodiment, if the instance receives a single instruction to migrate to a new location, then the instance may then copy its object store to the new location and establish a new instance at the new location.
Autosharding may be used to transfer a portion of an instance on a first node to a second node, e.g., to increase the capacity of the instance. In an embodiment, if the instance receives a single instruction to grow an instance on a first node to also be implemented on a second node, then the instance uses an algorithm to determine how to divide its object store (for example, split the keys of the object store) between the first node and the second node. The algorithm may consider the capacity and speed of the nodes, e.g., if the first node and the second node have similar speed and capacity, then the keys of the object store may be split evenly. On the other hand, if the second node has twice as much capacity and speed as the first node, then the algorithm may assign the second node to support 66% of the keys of the object store. Any approach for dividing the object store between nodes may be used by embodiments of the invention.
Autosharding may also be used to coalesce two instances to a single instance, e.g., to migrate an instance to a new node or to simplify management of the object stores. In an embodiment, if a first instance receives a single instruction to coalesce two instances to a single instance, then the first instance communicates with the second instance to determine how to merge their object stores into a single object store.
A node of a cluster may be configured to perform a scheduled backup of an object store to a different node (the “backup node”). However, if the backup node is offline or otherwise unavailable, then the scheduled backup cannot proceed.
In an embodiment embodied as a synchronous replication cluster, a node of the cluster may maintain a list of alternate backup nodes or alternate locations. If a backup node is offline or otherwise unavailable when a scheduled backup is to occur (or an instance crashes during the backup process), then the scheduled backup may be attempted at a different location in the list of alternate backup nodes or alternate locations within the cluster. The list of alternate backup nodes or locations may be prioritized so that the selection of an alternate backup node or location is made based on priority. A scheduled backup may either be a full backup or a partial (or incremental) backup of an object store.
Currently, if an administrator makes a mistake in interacting with a object store (for example, the administrator accidently deletes an object, table, or object store) of one node of a cluster supporting an asynchronous replication environment, the regrettable action will be replicated, at some point in time, even if the administrator detects the mistake prior to the mistake being replicated to other nodes of the cluster.
To address this concern, embodiments may support a configurable wait period before asynchronously replicating any transaction to another node. In this way, if a user identifies the mistake during the configurable wait period, the user may cancel the replication of the mistake and correct the mistake. The configurable wait period may be any length of time, such as one hour or one minute, for example.
It is anticipated that a configurable wait period may be used with a cluster of MySQL instances that each asynchronously replicate transaction to each other.
The embodiments of inventions described above for improving synchronous and asynchronous replication can be combined in a hierarchical fashion so that one synchronous cluster can be configured to replicate asynchronously to one or more synchronous clusters. Each synchronous cluster uses synchronous replication to support highly consistent, highly available storage, typically within a data center or within a metropolitan area. Asynchronous replication is used to ensure data availability across distant sites with less consistency. This is commonly used to facilitate disaster recovery. The embodiments of the inventions described above can be used to improve the performance and consistency of both the synchronous and asynchronous replication operations in such a hierarchical system.
In an embodiment, device 100 may be implemented on or using a computer system.
Computer system 800 may be coupled to a display 812, such as a cathode ray tube (CRT), a LCD monitor, and a television set, for displaying information to a user. An input device 814, including alphanumeric and other keys, is coupled to computer system 800 for communicating information and command selections to processor 804. Other non-limiting, illustrative examples of input device 814 include a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 804 and for controlling cursor movement on display 812. While only one input device 814 is depicted in
Embodiments of the invention are related to the use of computer system 800 for implementing the techniques described herein. According to one embodiment of the invention, those techniques are performed by computer system 800 in response to processor 804 executing one or more sequences of one or more instructions contained in main memory 806. Such instructions may be read into main memory 806 from another machine-readable medium, such as storage device 810. Execution of the sequences of instructions contained in main memory 806 causes processor 804 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement embodiments of the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.
The term “machine-readable storage medium” as used herein refers to any medium that participates in storing instructions which may be provided to processor 804 for execution. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 810. Volatile media includes dynamic memory, such as main memory 806.
Non-limiting, illustrative examples of machine-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read.
Various forms of machine readable media may be involved in carrying one or more sequences of one or more instructions to processor 804 for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a network link 820 to computer system 800.
Communication interface 818 provides a two-way data communication coupling to a network link 820 that is connected to a local network. For example, communication interface 818 may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 818 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 818 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 820 typically provides data communication through one or more networks to other data devices. For example, network link 820 may provide a connection through a local network to a host computer or to data equipment operated by an Internet Service Provider (ISP).
Computer system 800 can send messages and receive data, including program code, through the network(s), network link 820 and communication interface 818. For example, a server might transmit a requested code for an application program through the Internet, a local ISP, a local network, subsequently to communication interface 818. The received code may be executed by processor 804 as it is received, and/or stored in storage device 810, or other non-volatile storage for later execution.
In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
This application claims priority to U.S. provisional patent application No. 61/359,237, entitled “Approaches for Replication in a Distributed Transactional System Employing Solid State Devices,” filed Jun. 28, 2010, invented by John Busch et al., the entire contents of which are incorporated by reference for all purposes as if fully set forth herein. This application is related to U.S. provisional patent application No. 61/323,351, entitled “Distributed Data Access Using Solid State Storage,” filed Apr. 12, 2010, invented by John Richard Busch et al., the entire contents of which are incorporated by reference for all purposes as if fully set forth herein. This application is related to U.S. non-provisional patent application Ser. No. 12/983,754, entitled “Efficient Flash Memory-Based Object Store,” filed on Jan. 3, 2011, invented by John Busch et al., the entire contents of which are incorporated by reference for all purposes as if fully set forth herein. This application is related to U.S. non-provisional patent application Ser. No. 12/983,758, entitled “Flexible Way of Specifying Storage Attributes in a Flash Memory-Based Object Store,” filed on Jan. 3, 2011, invented by Darryl Ouye et al., the entire contents of which are incorporated by reference for all purposes as if fully set forth herein. This application is related to U.S. Non-provisional patent application Ser. No. 12/983,762, entitled “Minimizing Write Operations to a Flash Memory-Based Object Store,” filed on Jan. 3, 2011, invented by Darpan Dinker, the entire contents of which are incorporated by reference for all purposes as if fully set forth herein. This application is related to U.S. non-provisional patent application Ser. No. 13/084,368, entitled “Event Processing in a Flash Memory-Based Object Store,” filed on Apr. 11, 2011, invented by Mana Krishnan, the entire contents of which are incorporated by reference for all purposes as if fully set forth herein. This application is related to U.S. non-provisional patent application Ser. No. 13/084,432, entitled “Write Operations in a Flash Memory-Based Object Store,” filed on Apr. 11, 2011, invented by Xiaonan Ma, the entire contents of which are incorporated by reference for all purposes as if fully set forth herein. This application is related to U.S. non-provisional patent application Ser. No. 13/084,511, entitled “Recovery and Replication of a Flash Memory-Based Object Store,” filed on Apr. 11, 2011, invented by Johann George, the entire contents of which are incorporated by reference for all purposes as if fully set forth herein.
Number | Name | Date | Kind |
---|---|---|---|
4916605 | Beardsley et al. | Apr 1990 | A |
5046002 | Takashi et al. | Sep 1991 | A |
5057996 | Cutler et al. | Oct 1991 | A |
5117350 | Parrish et al. | May 1992 | A |
5212789 | Rago | May 1993 | A |
5287496 | Chen et al. | Feb 1994 | A |
5297258 | Hale et al. | Mar 1994 | A |
5394555 | Hunter et al. | Feb 1995 | A |
5403639 | Belsan et al. | Apr 1995 | A |
5423037 | Hvasshovd | Jun 1995 | A |
5509134 | Fandrich et al. | Apr 1996 | A |
5537534 | Voigt et al. | Jul 1996 | A |
5603001 | Sukegawa et al. | Feb 1997 | A |
5611057 | Pecone et al. | Mar 1997 | A |
5613071 | Rankin et al. | Mar 1997 | A |
5680579 | Young et al. | Oct 1997 | A |
5692149 | Lee | Nov 1997 | A |
5701480 | Raz | Dec 1997 | A |
5742787 | Talreja | Apr 1998 | A |
5887138 | Hagersten et al. | Mar 1999 | A |
5897661 | Baranovsky et al. | Apr 1999 | A |
5897664 | Nesheim et al. | Apr 1999 | A |
5960436 | Chang et al. | Sep 1999 | A |
5963983 | Sakakura et al. | Oct 1999 | A |
6000006 | Bruce et al. | Dec 1999 | A |
6023745 | Lu | Feb 2000 | A |
6052815 | Zook | Apr 2000 | A |
6130759 | Blair | Oct 2000 | A |
6141692 | Loewenstein et al. | Oct 2000 | A |
6151688 | Wipfel et al. | Nov 2000 | A |
6216126 | Ronstrom | Apr 2001 | B1 |
6298390 | Matena et al. | Oct 2001 | B1 |
6308169 | Ronstrom et al. | Oct 2001 | B1 |
6434144 | Romanov | Aug 2002 | B1 |
6467060 | Malakapalli et al. | Oct 2002 | B1 |
6615313 | Kato et al. | Sep 2003 | B2 |
6658526 | Nguyen et al. | Dec 2003 | B2 |
6728826 | Kaki et al. | Apr 2004 | B2 |
6745209 | Holenstein et al. | Jun 2004 | B2 |
6804766 | Noel et al. | Oct 2004 | B1 |
6874044 | Chou et al. | Mar 2005 | B1 |
6938084 | Gamache et al. | Aug 2005 | B2 |
6944699 | Bugnion et al. | Sep 2005 | B1 |
6981070 | Luk et al. | Dec 2005 | B1 |
7003586 | Bailey et al. | Feb 2006 | B1 |
7010521 | Hinshaw et al. | Mar 2006 | B2 |
7043621 | Merchant et al. | May 2006 | B2 |
7082481 | Lambrache et al. | Jul 2006 | B2 |
7162467 | Eshleman et al. | Jan 2007 | B2 |
7200718 | Duzett | Apr 2007 | B2 |
7203890 | Normoyle | Apr 2007 | B1 |
7249280 | Lamport et al. | Jul 2007 | B2 |
7251749 | Fong et al. | Jul 2007 | B1 |
7269708 | Ware | Sep 2007 | B2 |
7269755 | Moshayedi et al. | Sep 2007 | B2 |
7272605 | Hinshaw et al. | Sep 2007 | B1 |
7272654 | Brendel | Sep 2007 | B1 |
7281160 | Stewart | Oct 2007 | B2 |
7305386 | Hinshaw et al. | Dec 2007 | B2 |
7334154 | Lorch et al. | Feb 2008 | B2 |
7359927 | Cardente | Apr 2008 | B1 |
7383290 | Mehra et al. | Jun 2008 | B2 |
7406487 | Gupta et al. | Jul 2008 | B1 |
7415488 | Muth et al. | Aug 2008 | B1 |
7417992 | Krishnan | Aug 2008 | B2 |
7467265 | Tawri et al. | Dec 2008 | B1 |
7529882 | Wong | May 2009 | B2 |
7542968 | Yokomizo et al. | Jun 2009 | B2 |
7562162 | Kreiner et al. | Jul 2009 | B2 |
7584222 | Georgiev | Sep 2009 | B1 |
7610445 | Manus et al. | Oct 2009 | B1 |
7623494 | Zhu et al. | Nov 2009 | B2 |
7627618 | Honigfort | Dec 2009 | B2 |
7647449 | Roy et al. | Jan 2010 | B1 |
7657710 | Loewenstein | Feb 2010 | B2 |
7809691 | Karmarkar et al. | Oct 2010 | B1 |
7822711 | Ranade | Oct 2010 | B1 |
7885923 | Tawri et al. | Feb 2011 | B1 |
7917472 | Persson | Mar 2011 | B2 |
8015352 | Zhang et al. | Sep 2011 | B2 |
8018729 | Skinner | Sep 2011 | B2 |
8024515 | Auerbach et al. | Sep 2011 | B2 |
8037349 | Mandagere et al. | Oct 2011 | B2 |
8069328 | Pyeon | Nov 2011 | B2 |
8099391 | Monckton | Jan 2012 | B1 |
8103643 | Danilov et al. | Jan 2012 | B2 |
8161248 | Blumrich et al. | Apr 2012 | B2 |
8205206 | Özer et al. | Jun 2012 | B2 |
8225053 | McCorkendale et al. | Jul 2012 | B1 |
8239617 | Linnell | Aug 2012 | B1 |
8261266 | Pike et al. | Sep 2012 | B2 |
8261289 | Kasravi et al. | Sep 2012 | B2 |
8321450 | Thatte et al. | Nov 2012 | B2 |
8335776 | Gokhale | Dec 2012 | B2 |
8370853 | Giampaolo et al. | Feb 2013 | B2 |
8401994 | Hoang et al. | Mar 2013 | B2 |
8504526 | Gokhale et al. | Aug 2013 | B2 |
8666939 | O'Krafka et al. | Mar 2014 | B2 |
8671074 | Wang et al. | Mar 2014 | B2 |
20010032253 | Duxbury | Oct 2001 | A1 |
20020089933 | Giroux et al. | Jul 2002 | A1 |
20020129192 | Spiegel et al. | Sep 2002 | A1 |
20020166031 | Chen et al. | Nov 2002 | A1 |
20020184239 | Mosher, Jr. et al. | Dec 2002 | A1 |
20030016596 | Chiquoine et al. | Jan 2003 | A1 |
20030097610 | Hofner | May 2003 | A1 |
20030177408 | Fields et al. | Sep 2003 | A1 |
20030220985 | Kawamoto et al. | Nov 2003 | A1 |
20040010502 | Bomfim et al. | Jan 2004 | A1 |
20040078379 | Hinshaw et al. | Apr 2004 | A1 |
20040143562 | Chen et al. | Jul 2004 | A1 |
20040148283 | Harris et al. | Jul 2004 | A1 |
20040172494 | Pettey et al. | Sep 2004 | A1 |
20040172577 | Tan et al. | Sep 2004 | A1 |
20040205151 | Sprigg et al. | Oct 2004 | A1 |
20040230862 | Merchant et al. | Nov 2004 | A1 |
20040267835 | Zwilling et al. | Dec 2004 | A1 |
20050005074 | Landin et al. | Jan 2005 | A1 |
20050021565 | Kapoor et al. | Jan 2005 | A1 |
20050027701 | Zane et al. | Feb 2005 | A1 |
20050028134 | Zane et al. | Feb 2005 | A1 |
20050034048 | Nemawarkar et al. | Feb 2005 | A1 |
20050081091 | Bartfai et al. | Apr 2005 | A1 |
20050086413 | Lee et al. | Apr 2005 | A1 |
20050120133 | Slack-Smith | Jun 2005 | A1 |
20050131964 | Saxena | Jun 2005 | A1 |
20050240635 | Kapoor et al. | Oct 2005 | A1 |
20050246487 | Ergan et al. | Nov 2005 | A1 |
20060059428 | Humphries et al. | Mar 2006 | A1 |
20060064549 | Wintergerst | Mar 2006 | A1 |
20060085594 | Roberson et al. | Apr 2006 | A1 |
20060123200 | Ito et al. | Jun 2006 | A1 |
20060130063 | Kilian et al. | Jun 2006 | A1 |
20060161530 | Biswal et al. | Jul 2006 | A1 |
20060174063 | Soules et al. | Aug 2006 | A1 |
20060174069 | Shaw et al. | Aug 2006 | A1 |
20060179083 | Kulkarni et al. | Aug 2006 | A1 |
20060195648 | Chandrasekaran et al. | Aug 2006 | A1 |
20060212795 | Cottrille et al. | Sep 2006 | A1 |
20060218210 | Sarma et al. | Sep 2006 | A1 |
20060242163 | Miller et al. | Oct 2006 | A1 |
20060253724 | Zhang | Nov 2006 | A1 |
20070038794 | Purcell et al. | Feb 2007 | A1 |
20070043790 | Kryger | Feb 2007 | A1 |
20070043860 | Pabari | Feb 2007 | A1 |
20070073896 | Rothman et al. | Mar 2007 | A1 |
20070143368 | Lundsgaard et al. | Jun 2007 | A1 |
20070156842 | Vermeulen et al. | Jul 2007 | A1 |
20070174541 | Chandrasekaran et al. | Jul 2007 | A1 |
20070234182 | Wickeraad et al. | Oct 2007 | A1 |
20070276784 | Piedmonte | Nov 2007 | A1 |
20070283079 | Iwamura et al. | Dec 2007 | A1 |
20070288692 | Bruce et al. | Dec 2007 | A1 |
20070288792 | Thorpe et al. | Dec 2007 | A1 |
20070294564 | Reddin et al. | Dec 2007 | A1 |
20070299816 | Arora et al. | Dec 2007 | A1 |
20080016300 | Yim et al. | Jan 2008 | A1 |
20080034076 | Ishikawa et al. | Feb 2008 | A1 |
20080034174 | Traister et al. | Feb 2008 | A1 |
20080034249 | Husain et al. | Feb 2008 | A1 |
20080046538 | Susarla et al. | Feb 2008 | A1 |
20080046638 | Maheshwari et al. | Feb 2008 | A1 |
20080126706 | Newport et al. | May 2008 | A1 |
20080172402 | Birdwell et al. | Jul 2008 | A1 |
20080256103 | Fachan et al. | Oct 2008 | A1 |
20080288713 | Lee et al. | Nov 2008 | A1 |
20080295105 | Ozer et al. | Nov 2008 | A1 |
20080301256 | McWilliams | Dec 2008 | A1 |
20090006500 | Shiozawa et al. | Jan 2009 | A1 |
20090006681 | Hubert et al. | Jan 2009 | A1 |
20090006888 | Bernhard et al. | Jan 2009 | A1 |
20090019456 | Saxena et al. | Jan 2009 | A1 |
20090024871 | Emaru et al. | Jan 2009 | A1 |
20090030943 | Kall | Jan 2009 | A1 |
20090059539 | Ryu et al. | Mar 2009 | A1 |
20090070530 | Satoyama et al. | Mar 2009 | A1 |
20090150599 | Bennett | Jun 2009 | A1 |
20090177666 | Kaneda | Jul 2009 | A1 |
20090198791 | Menghnani | Aug 2009 | A1 |
20090240664 | Dinker et al. | Sep 2009 | A1 |
20090240869 | O'Krafka et al. | Sep 2009 | A1 |
20090327751 | Koifman et al. | Dec 2009 | A1 |
20100058021 | Kawamura | Mar 2010 | A1 |
20100125695 | Wu et al. | May 2010 | A1 |
20100241895 | Li et al. | Sep 2010 | A1 |
20100262762 | Borchers et al. | Oct 2010 | A1 |
20100299490 | Attarde et al. | Nov 2010 | A1 |
20100306448 | Chen et al. | Dec 2010 | A1 |
20100318821 | Kwan et al. | Dec 2010 | A1 |
20100325498 | Nagadomi | Dec 2010 | A1 |
20110022566 | Beaverson et al. | Jan 2011 | A1 |
20110072206 | Ross et al. | Mar 2011 | A1 |
20110082985 | Haines et al. | Apr 2011 | A1 |
20110099420 | MacDonald McAlister | Apr 2011 | A1 |
20110167038 | Wang et al. | Jul 2011 | A1 |
20110179279 | Greevenbosch et al. | Jul 2011 | A1 |
20110185147 | Hatfield et al. | Jul 2011 | A1 |
20110191299 | Huynh Huu et al. | Aug 2011 | A1 |
20110225214 | Guo | Sep 2011 | A1 |
20120005154 | George et al. | Jan 2012 | A1 |
20120072449 | Patch et al. | Mar 2012 | A1 |
20130066948 | Colrain et al. | Mar 2013 | A1 |
Number | Date | Country |
---|---|---|
1548600 | Jan 2007 | EP |
1746510 | Jan 2007 | EP |
Entry |
---|
Amza et al., “Data Replication Strategies for Fault Tolerance and Availability on Commodity Clusters”, Proceedings International Conference on Dependable Systems and Networks, pp. 459-467, IEEE, 2000. |
Thomas et al., “Calvin: fast distributed transactions for partitioned database systems”, SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data , pp. 1-12, 2012, ACM. |
Feeley et al., “Integrating coherency and recoverability in distributed systems”, OSDI '94 Proceedings of the 1st USENIX conference on Operating Systems Design and Implementation, Article No. 16, 1994, ACM. |
Chandramohan et al., “Frangipani: a scalable distributed file system”, SOSP '97 Proceedings of the sixteenth ACM symposium on Operating systems principles, pp. 224-237, 1997, ACM. |
Wang et al., “Log-based middleware server recovery with transaction support”, The VLDB Journal Jun. 2011, vol. 20, Issue 3, pp. 347-370, 2011, Springer-Verlag. |
bsn-modulestore, Versioning Concept, Oct. 13, 2010, 2 pgs. |
Btrfs, http://en.wikipedia.org, Oct. 3, 2011, 9 pgs. |
Buchholz, The Structure of the Reiser File System, Jan. 26, 2006, 21 pgs. |
Chacon, Git, The Fast Version Control System, Oct. 3, 2011, 3 pgs. |
Email Communication from James Bodwin to Christopher Brokaw re prior art, Sep. 13, 2011, 4 pgs. |
Git (Software), http://en.wikipedia.org, Oct. 3, 2011, 10 pgs. |
Hitz, File System Design for an NFS File Server Appliance, Jan. 19, 1994, 23 pgs. |
McDonald, Architectural Semantics for Practical Transactional Memory, Jun. 2006, 12 pgs. |
McGonigle, A Short History of btrfs, Aug. 14, 2009, 11 pgs. |
Mellor, ZFS—the future of file systems? Aug. 14, 2006, 5 pgs. |
Mercurial, http://en.wikipedia.org, Oct. 2, 2011, 6 pages. |
Module: Mongoid: Versioning, http://rdoc.info, Documentation by Yard 0.7.2, 6 pages Oct. 3, 2011. |
Noach, Database Schema under Version Control, code.openarck.org, Apr. 22, 2010, 6 pages. |
Reiser FS, http://enwikipedia.org, Sep. 17, 2011, 5 pgs. |
Rice, Extension Versioning, Update and Compatibility, Aug. 9, 2011, 11 pgs. |
Rice, Toolkit Version Format, Aug. 19, 2011, 4 pgs. |
Russell, Track and Record Database Schema Versions, Jun. 28, 2005, 8 pgs. |
Schooner Information Technology, IPAF, PCT/US2008/065167, Oct. 23, 2008, 7 pgs. |
Schooner Information Technology, ISR/WO, PCT/US2008/065167, Jan. 28, 2009, 16 pgs. |
SQL Server Database Schema Versioning and Update, Dec. 2, 2009, 2 pgs. |
Sufficiently Advanced Bug, File Versioning, Caching and Hashing, Oct. 3, 2011, 3 pgs. |
The Z File System (ZFS), FreeBSD Handbook, Oct. 3, 2011, 8 pgs (Author not provided). |
Tux3 Linux Filesystem Project, 2008, 1 pg. |
Tux3 Versioning Filesystem, Jul. 2008, 67 pgs. |
Tux3, http://en.wikipedia.org, Jun. 2, 2010, 3 pgs. |
Vijaykumar, Speculative Versioning Cache, Dec. 1, 2001, 13 pgs. |
WAFL—Write Anywhere File Layout, 1999, 1 pg. |
Write Anywhere File Layout, Sep. 9, 2011, 2 pgs. |
ZFS, http://en.wikipedia.org Sep. 30, 2011, 18 pgs. |
Ajmani, Automatic Software Upgrades for Distributed Systems, MIT, Sep. 2004, 164 pgs. |
Chockler, Active Disk Paxos with infinitely many processes, Springer-Verlag, Apr. 2005, 12 pgs. |
Dwork, Concensus in the presence of partial synchrony, MIT, 1988, 6 pgs. |
Guerraoui, A Leader Election Protocol for Eventually Synchronous Shared Memory Systems, IEEE, 2006, 6 pgs. |
Lamport, Cheap Paxos, Microsoft, 2004, 9 pgs. |
Lamport, Fast Paxos, Microsoft, Jul. 2005, 43 pgs. |
Lamport, Generalized Consensus and Paxos, Microsoft, Mar. 2004, 25 pgs. |
Lamport, Paxos Made Simple, Nov. 2001, 14 pgs. |
Malkhi, Lecture notes in computer science [Section: Omega Meets Paxos, Leader election and stability without eventual timely links], 2005, pp. 199-213. |
Pease, Reaching Agreement in the Presence of Faults, ACM, 1980, pp. 228-234. |
Schneider, Implementing fault tolerant services using the state machine, Cornell Univ., 1990, 21 pgs. |
Mukherjee et al., Verification of an Industrial CC-NUMA server, Proceedings of ASP-DAC 2002, 7th Asia and South Pacifric and the 15th International Conference on VLSI Design, Jan. 7-11, 2002, 6 pages. |
Shacham et al., Verificaiton of chip multiprocessor memory systems using a relaxed scoreboard, Microarchitecture, 2008, MICRO-41, 2008, 41st IEEE/ACM International Symposium, Nov. 8-12, 2008, 12 pages. |
Walker, Hash Table Tutorial, Oct. 13, 2007, 14 pgs. |
Unknown Author, Supermicro, “Intel Itanium Processor 9300 Series Based Server Systems,” Jul. 8, 2010, http://www.supermicro.com/products/info/itanium.cfm, 3 pages. |
Number | Date | Country | |
---|---|---|---|
20120005154 A1 | Jan 2012 | US |
Number | Date | Country | |
---|---|---|---|
61359237 | Jun 2010 | US |