Replication of data between mirrored data sites

Abstract

Computer-implemented methods and systems for replication of data between mirrored data sites are provided. An exemplary method may comprise receiving a data object at a mirrored data site, the mirrored data site including one or more data nodes. The data objects are replicated to the one or more data nodes internally. Then, a data object reference associated with the data object is generated. The reference referring to the data object is queued for transmission to all other mirrored data sites. The data object reference associated with the data object is transmitted to the other mirrored data sites. When the data object is received at each mirrored data site, it is replicated to one or more data nodes within the site. After transmitting the data object, the data object reference is discarded.

Description

TECHNICAL FIELD

This disclosure relates generally to systems and methods for data replication, and more specifically to replication infrastructures including mirrored data sites to provide a single consistent view of the file system available from any site.

DESCRIPTION OF RELATED ART

The approaches described in this section could be pursued but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

In computing systems, remote replication is a form of data protection that involves copying of data between multiple sites to improve data protection, fault tolerance and to provide disaster recovery. As used herein, a term “site” may refer to physically distinct geographic locations, or it may refer to distinct groupings that require failure handling. For example, protection from earthquakes could mean placing replicas in sites that are not affected by the same fault lines. If the protection is directed against power-related failures, the two sites may be in the same building or perhaps even in the same rack, but each site would have different power sources.

Procedures used for data protection with a single site and procedures used for replication between different sites may differ substantially. Therefore, in conventional systems, two entirely different methodologies may be used.

Furthermore, replication of data objects between sites may suffer from various network and node failures. Such failures need to be detected and recovered from.

SUMMARY

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

In accordance with various embodiments of the disclosure, a method for replication between mirroring sites is provided. In some embodiments, the method may include replicating a content addressable object store between multiple sites, where each object is addressable by a signature that is derived from the object data. In some embodiments, the method may include replicating a file system that is constructed on top of a content addressable object store.

The mirroring may be bi-directional such that changes to the data at any site are copied to all other sites, and all sites may simultaneously access the data according to the procedures available for the file system.

Additionally, the method may include guaranteeing, across the set of replicated sites, the read and write ordering and locking guarantees that the file system is required to deliver to its clients. In some embodiments, these ordering rules may be defined by various standards or protocols. Examples of such standards and protocols include POSIX, NFS, CIFS, SMB, RESTful, WebDav, and so forth.

Each mirrored site may include one or more nodes, one of which may be elected as a gateway. In some embodiments, gateway nodes may cooperate to elect one site as an arbitrator. Alternatively, the gateways may cooperatively share this responsibility. The arbitrator guarantees that all file system ordering rules are adhered to.

Sites may be added to a mirror or removed from it. An added site may already contain data objects. In some embodiments, these data objects may be replicated using an “initial synchronization” method. The initial synchronization method may be also used whenever sites are reconnected after a disconnection (for example, due to a network failure).

Using the method described herein, data objects of a mirrored site may be accessed, created, modified, or changed by other sites while the mirrored site is disconnected. For this purpose, a data object may be received at one site. Then, the data object may be stored on one or more nodes at that site. Furthermore, the data object may be forwarded to the gateway. For some data objects, the gateway may synchronously replicate the data object and metadata to the mirrored sites. For other data objects the gateway may synchronously send the data object signature and object metadata to the mirrored sites. The data object may be queued for asynchronous transmission. In some embodiments, the metadata may include a site identifier.

When a client at a mirrored site requires access to a data object that has not yet been replicated, that mirrored site sends a request for the data object to the site identified by the object metadata previously transmitted. The requested site may then send the object data to the requesting site and remove it from the queue of objects pending transmission.

In some embodiments, the queue of objects to be asynchronously replicated may be restricted to a maximum count of data objects. Thus, when the queue has reached its maximum, the site may stop accepting new data objects.

To the accomplishment of the foregoing and related ends, the one or more embodiment of the disclosures may comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the drawings set forth in detail certain illustrative features of the one or more embodiment of the disclosures. These features are indicative, however, of but a few of the various ways in which the principles of various embodiments of the disclosures may be employed, and this description is intended to include all such embodiment of the disclosures and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:

FIG. 1 shows a simplified structure of a computer network infrastructure.

FIG. 2 shows a simplified scenario for migration of Metadata Operational Processor (MOP) and gateway services.

FIG. 3 shows a scenario for migration of a main gateway.

FIG. 4 shows a graphical representation of connection states of a gateway service in another site.

FIG. 5 shows data replication using different tiers.

FIG. 6 shows a graphical representation of data replication between sites.

FIG. 7A shows a site failure.

FIG. 7B shows actions taken upon the site failure.

FIG. 8 is a flow chart illustrating a method for replication of data.

FIG. 9 shows a sample system for replication of data.

FIG. 10 shows a diagrammatic representation of a computing device in the exemplary electronic form of a computer system.

DETAILED DESCRIPTION

The following detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show illustrations in accordance with exemplary embodiments. These exemplary embodiments, which are also referred to herein as “examples,” are described in enough detail to enable those skilled in the art to practice the present subject matter. The embodiments can be combined, other embodiments can be utilized, or structural, logical and electrical changes can be made without departing from the scope of what is claimed. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope is defined by the appended claims and their equivalents.

The approaches and principles disclosed herein relate to computer-implemented methods and systems for replication of data objects within a computer network infrastructure. The computer network infrastructure includes a plurality of nodes each having storage resources for storing various data objects and enabling access to them from other nodes. Moreover, the storage resources of a single node may include one or a plurality of hard drives or other memory devices such as RAM (random-access memory), ROM (read-only memory), hard disk drive (HDD), or solid state drive (SSD). Each data object (e.g., a file, a collection of files, or any other entity which can be manipulated by an operating system or an application) may be replicated to each of the nodes in the infrastructure.

Logical connections between devices may form various topologies. The topology may depend on the layout of physical network, a number of devices in the network, and other factors. One conventional topology includes a site where data nodes are connected in a circle and each data node is connected to the neighboring data nodes. Another conventional topology includes a “mesh” topology where every data node is connected to every other data node in the site. Such site can be referred to as a “ring.” Rings may be connected in a mesh, using point-to-point communications channels between each pair of nodes. Data nodes are said to be connected in a mesh when two or more rings are connected together and every pair of nodes in each ring are connected using point-to-point communications channels. Thus, if a ring contains 3 nodes, A, B, and C, there will be 3 network connections: A-B, B-C, and C-A. If there is a mesh with one member ring containing 2 nodes, A, B, and the other containing 3 nodes, D, E, F, then we have the following 10 network connections: A-B, D-E, E-F, F-D, A-D, A-E, A-F, B-D, B-E, and B-F.

In more complex designs, both of the described topologies may be used. For example, data sites may communicate with each other using a mesh connection. With this connection, every node in a data site can replicate data to every node in another data site. Thus, a mesh topology can exist between the data sites. This topology is illustrated by FIG. 1.

FIG. 1 is a simplified structure of a computer network infrastructure 100, in accordance with some example embodiments. The computer network infrastructure 100 includes at least two sites, site A and site B, both of which are active. Each site may include a plurality of nodes. Each node is a network device (which may include further component data components) that maintains a network address and can store and share data objects. In an example, the node may refer to a device having a plurality of hard drives, a computer, a server, a laptop, a tablet computer, a thin client, or any other electronic computing device suitable for replicating data. Furthermore, each node may include multiple storage resources such as RAM, ROM, flash memory, and the like.

As shown in FIG. 1, site A may include four nodes: 1A, 2A, 3A, and 4A. Site B may also include four nodes: 1B, 2B, 3B, and 4B. Logical connections between site A and site B may form a mesh topology. Thus, each node of site A may replicate data to each node of site B. This is illustrated by the example of node 2A. Communications associated with node 2A are shown by arrows leading to each node of site B. Although such communications are shown only for node 2A, all nodes of the site A may have similar communications with all nodes of site B.

Site A may be associated with a configuration space 140, while site B may be associated with a configuration space 160. Configuration spaces 140 and 160 of the sites A and B in the mirror may be coordinated by a system for replication between data sites 150. The system 150 may coordinate configuration of sites A and B connected in the mesh to provide a single common view of a file system.

Replication of data between mirroring data sites may employ both synchronous and asynchronous data paths. Synchronous data paths may be used for metadata communication, while asynchronous paths may be used to transmit data itself. This approach is illustrated in FIG. 1.

In a multi-site infrastructure, one site may control replication and operate a Metadata Operational Processor (MOP). This site may be referred to as a master site and may be used to control metadata and resolve conflicts. Other sites may each operate a MOP proxy. These sites may be referred to as subservient sites.

FIG. 2 illustrates a simplified structure of a computer network infrastructure 200, in accordance with some example embodiments. The computer network infrastructure 200 may include site A and site B, each having 4 nodes and connected in a mesh. In site A, node 2A may include MOP 220. Therefore, node 2A may be referred to as an arbitrator node and arbitrate data replications between the mirrored data site A and mirrored data site B. MOP 220 may synchronize all metadata operations in order to present a consistent view of the file system. Metadata operations from multiple nodes may be funneled through the MOP 220. MOP 220 may run on a single node within a site, but the node on which MOP 220 runs, may change depending on the state of the site and the data model. In some embodiments, the MOP functionality may be distributed across the nodes in the site, or between sites.

A site B not hosting MOP 220 may run a MOP proxy 230. The MOP proxy 230 may receive requests from nodes of the site B, just as MOP 220 receives requests from site A. However, by acting as a proxy, MOP proxy 230 may relay requests to MOP 220, in site A, and relay responses back to the nodes initiating the request. The MOP proxy 230 may act as a forwarding agent and relay remote procedure calls (RPC) between nodes of site B and the node running the MOP 220 in the site A.

Site A and site B may be connected using a bidirectional connection between the nodes in each site. This connection may be called a main gateway 210. The main gateway 210 may leverage a distributed messaging protocol for connection and events. The main gateway 210 may operate over a Local Area Network (LAN) or a Wide Area Network (WAN).

Referring now to FIG. 3, the main gateway 210 and MOP services may be collocated. MOP 220 may be located in node 2A of site A with gateway services, whereas MOP proxy 230 is located in node 2B together with gateway services. Since any node in the site A may be selected to run the MOP 220, and since any node in a site B may be selected to run the MOP proxy 230, all nodes may be provisioned ahead of time so that they connect to each node in the other site.

This may accommodate the node failover scenario in which a node 2A hosting the MOP 220 (or MOP proxy 230) fails over to another node in the site. The gateway service may follow the MOP 220 in a node failover. This may be performed using a pre-provisioned path to establish an alternate gateway between the mirrored sites. The state of the connections may be used to limit the possible MOP and gateway failover locations.

MOP proxy 230 that is associated with node 2B may migrate to another node of site B (for example, node 4B). This may be a result of a failure of the main gateway 210. Because the main gateway 210 and MOP services are co-located, the main gateway 210 may also migrate to node 4B.

Connection States

FIG. 4 shows a graphical representation of connection states 400 of a site awaiting connection from another site. Nodes in another site may initiate connections according to the mirror configuration published in a configuration space of each site.

Thus, connection states of a site may include awaiting connection 410, which may be initiated by nodes in another site. When connection is established, synchronizing 420 between the sites may start. The synchronizing may continue until either a synchronization error occurs 430 or the synchronization finishes 440. When either of the states 430 or 440 occurs, connection between sites no longer exists, and the site may go to the state 410 again and try to restore connection 410 and continue synchronizing 420 until the synchronization is finished 440.

Initial Synchronization

When sites connect or reconnect, the gateway service enters a phase called initial synchronization. The gateway services in each site may exchange object identifiers of objects known to exist on their respective sites. Object identifiers corresponding to objects unknown to the site may be pulled by the gateway service using a data receive operation, then written to the site using a data transfer operation. These operations may allow the gateway service to perform the initial synchronization of objects with more efficient use of the network link.

In some embodiments, status keys related to initial synchronization may be published in the configuration space, since initial synchronization is a long term operation. In such a way, the progress of the initial synchronization may be monitored.

When a mirror is connected, file system updates made on one site may be relayed to the other site. At the object level, this may be achieved by echoing updates made locally in one site to the other site.

Tier Architecture

Data objects may be replicated between nodes within a site and between the sites. Intra-site and inter-site operations may be performed at different levels or tiers as shown by FIG. 5.

Updates may be persisted on site A locally via tier 0510 (or local tier). Correspondingly, intra-site operations, such as operations between nodes 1A and 2A, 1A and 3A, 3A and 4A, and 2A and 4A may be performed at tier 0510.

Updates may then be pushed to site B via tier 1520 (or remote tier). Operations within site B, i.e., 1B and 2B, 1B and 3B, 3B and 4B, and 2B and 4B, may then also be performed at tier 0510.

By associating intra-site and inter-site operations with different tiers, looping of operations may be avoided. For example, a data object write may be replicated to all of the tiers in a list at the originating site. One of those tiers may contain a gateway to another site, which causes the data object to be replicated to that site. Within that site, a new list of tiers to store the data object may be generated, and the originating tier may be eliminated from the list in order to avoid the gateway on this end looping the data object back to the originating end.

In accordance to some embodiments, before a generic update operation completes successfully, it must successfully complete on all tiers.

Since the mirror connection may have slow WAN-like performance with high latencies, an update between sites may start with forming a data reference informing the other site about data objects that will be transferred. For example, an RPC may be handled by the gateway service on the other site, at which point an entry may be made in a proxy object database, called DB_MB. Such an entry may indicate a promise by the other site that an object corresponding to this entry will eventually arrive. The object transfer may be then queued on the node initiating the request for eventual delivery to the other site.

In some example embodiments, a configurable queue length maximum may be enforced. When the limit is reached, an alarm may be triggered, and the original operation will not complete until the queue can be appended.

In other example embodiments, when the queue limit is reached, backpressure may be applied to the clients, such that new writes are not accepted. Via this and similar schemes, the differences between the mirrors may be minimized and bounded in time.

To support a read, an object receive may be performed only after an object is not seen within any previous tier. In order to support a consistent file system view, if the requested object corresponds to an entry in the local proxy object database, then a tier 0 read operation may be performed in the other site via the gateway service node in each site. Meanwhile, the corresponding data object may not get persisted in that site until the write operation is eventually de-queued and processed successfully. Upon successful completion, the corresponding entry in the DB_MB is removed. It is possible that the write operation will fail when the mirror is compromised. This is discussed in more detail below.

The length of this write queue, when combined with the length of the read queue for initial synchronization, may provide information concerning synchronization of two sites. These lengths may be periodically recorded in a mesh status key in the configuration space.

Asynchronous Data Object Write Operations

When write operations within a site, for example, site A as shown by FIG. 6, for data objects 620 are successfully completed at tier 0510, the operations may be promoted at inter-site level to tier 1520. For this purpose, proxy 640 associated with the data objects 620 of the update operations may be synchronously sent to the other site. The proxy 640 may include a site identification (ID) identifying the site that currently contains the data object, with which the proxy 640 is associated.

On receiving the proxy 640, proxy reference database 630 of site B may be updated. Thus, proxy references may be created informing site B about the data objects 620 that will be transferred.

When a data delivery queue 610 allows, the data objects 620 may be compressed and sent to site B. Data objects 620 may be transferred asynchronously. After receiving the data objects 620 in site B, data objects 620 may be decompressed and written to nodes of site B.

After the update operation in all tiers is successfully completed, the proxy object info 640 in proxy reference database 630 may be removed.

Data Object Read Operations

When a read operation for a data object is initiated, the data object may be searched for at tier 0510. If the data object is found at tier 0510, then the read operation is successfully completed. However, if the data object is not written in the site yet, the data object will not be found, so the read operation may be retried to tier 1. This process may repeat until either the object is found or all tiers are exhausted. In the latter case, the object cannot be found so an error is returned.

Referring to FIG. 6, if the read operation is associated with data object 2, a proxy reference related to data object 2 may be found in proxy reference database 630. Then, a request may be issued at tier 1 and data object 2 may be pulled forward and written to site B.

In some embodiments, data objects may be associated with object identifiers. An object identifier may uniquely identify a data object based on the content of the data object. Thus, a data object may be found in any location, at any tier using the object identifier, despite replication policies local to a site that involve dynamically relocating data replicas within a site.

Site Failure Scenarios

In some cases, one of the sites may experience a failure due to various reasons (for example, a power outage). If a remote site fails, a new remote site may be provisioned, and after establishing a connection, an initial synchronization may be initiated. If the site hosting the MOP fails, the system for remote replication may designate a new site to host the MOP.

FIG. 7A illustrates a failure of a site hosting MOP, in accordance with some example embodiments. Site A is the site hosting MOP, and site B is a site hosting a MOP proxy. The sites are in the process of data delivery. Data delivery queue 710 contains data objects, and proxy reference database 730 has some proxies 740. If site A fails, the system for replication between data sites may promote site B to be a site hosting the MOP as shown in FIG. 7B.

If site A fails, proxy reference database 730 has proxies, so the system for replication between data sites may perform a rollback. The system may scan the database and roll back to a snapshot that will support the data that was written in site B.

Other actions may include emptying data delivery queue 710, removing proxies from proxy reference database 730, and so forth.

If site A fails and subsequently recovers (e.g., recovering from a temporary power outage), it may be demoted to a site running a MOP proxy. When the connection between the sites is established, an initial synchronization procedure may be initiated. Thus, access to data residing on site A will not be lost.

FIG. 8 is a process flow diagram showing a method 800 for replication of data between mirrored data sites within a computer network infrastructure 100. The method 800 may be performed by logic that may comprise hardware (e.g., dedicated logic, programmable logic, and microcode), software (such as software run on a general-purpose computer system or a dedicated machine), or a combination of both. In one exemplary embodiment, the processing logic resides at a system for replication between data sites 900, and the various elements of the system 900 can perform the method 800.

It will be appreciated by one of ordinary skill in the art that examples of the foregoing modules may be virtual, and instructions said to be executed by a module may, in fact, be retrieved and executed by the system 900. Although various elements may be configured to perform some or all of the various operations described herein, fewer or more elements may be provided and still fall within the scope of various embodiments.

As shown in FIG. 8, the method 800 may commence at operation 810 with receiving a data object at a mirrored data site. The data site may include one or more data nodes. One of these data nodes may be designated as the node running MOP, that is, an arbitrator node to arbitrate data replications between sites. The received data object may be replicated to the one or more data nodes internally at operation 820. After the data object is replicated to all data nodes, a data object reference associated with the data object may be generated at operation 830.

In some embodiments, an object identifier may be generated by running a cryptographic hash function over a content associated with the data object. Thereafter, the data object may be found based on the content associated with the data object.

At operation 840, the data object reference may be transmitted to one or more of other mirrored data sites including one or more nodes. Each of these nodes may be interconnected with each node in the other mirrored data sites to form a complete mesh. In some example embodiments, the data object reference may be transmitted to a data object reference database associated with the other mirrored data site. Then, the data object may be queued for transmission to the other mirrored data site at operation 850.

Upon transmission of the data object to the other mirrored data site, the data object may be replicated to one or more nodes of that data site. After completion of replication of the data object to the mirrored data site, the data object reference may be discarded.

In some embodiments, replication of the data object to the nodes within a mirrored data site may be performed at an intra-site operation tier, whereas transmitting the data object reference and the data object between mirrored data sites may be performed at an inter-site operation tier. Operations at both operation tiers may be performed using the same data logic.

Additionally, the method 800 may optionally comprise synchronizing data between mirrored data sites. The synchronizing may include comparing data object references and data objects associated with the mirrored data site internally to data object references and data objects associated with one or more of the other mirrored data sites. Delivery of the data objects corresponding to the object references may be requested.

In some embodiments, the method 800 may optionally comprise receiving a request for the data object at the other mirrored data site. If that site does not have the requested data object, it may be determined based on the data object reference associated with the other mirrored data site. In this case, the mirrored data site may be requested to serve the data object at a higher priority.

FIG. 9 shows a system for replication between data sites 900, in accordance with some example embodiments. The system 900 may include a processor 910 and a database 920. The processor 910 may be configured to receive a data object at a mirrored data site. That data site may include one or more data nodes, and the received data object may be replicated to the one or more data nodes internally, at a local tier. Then, a data object reference, for example, a proxy reference, associated with the data object may be generated and transmitted to the database 920. The database 920 may include a data object reference, which in some embodiments is referred to as a “proxy reference” that identifies the site in which the data resides.

The data object may be then queued for a transmission to one or more of the other mirrored data sites. Upon the transmission of the data object to the one or more of the other mirrored data sites, the data object may be replicated to the nodes of the one or more of the other mirrored data sites and the data object reference may be discarded.

FIG. 10 shows a diagrammatic representation of a computing device for a machine in the exemplary electronic form of a computer system 1000, within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein can be executed. In various exemplary embodiments, the machine operates as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a cellular telephone, a digital camera, a portable music player (e.g., a portable hard drive audio device, such as an Moving Picture Experts Group Audio Layer 3 (MP3) player), a web appliance, a network router, a switch, a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example computer system 1000 includes a processor or multiple processors 1002, a hard disk drive 1004, a main memory 1006 and a static memory 1008, which communicate with each other via a bus 1010. The computer system 1000 may also include a network interface device 1012, and coprocessors dedicated for data compression and object identifier cryptographic calculation. The hard disk drive 1004 may include a computer-readable medium 1020, which stores one or more sets of instructions 1022 embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 1022 can also reside, completely or at least partially, within the main memory 1006 and/or within the processors 1002 during execution thereof by the computer system 1000. The main memory 1006 and the processors 1002 also constitute machine-readable media such as, for example, an HDD or SSD.

While the computer-readable medium 1020 is shown in an exemplary embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methodologies of the present application, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such a set of instructions. The term “computer-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media. Such media can also include, without limitation, hard disks, floppy disks, NAND or NOR flash memory, digital video disks, RAM, ROM, HDD, SSD, and the like.

The exemplary embodiments described herein can be implemented in an operating environment comprising computer-executable instructions (e.g., software) installed on a computer, in hardware, or in a combination of software and hardware. The computer-executable instructions can be written in a computer programming language or can be embodied in firmware logic. If written in a programming language conforming to a recognized standard, such instructions can be executed on a variety of hardware platforms and for interfaces to a variety of operating systems. Although not limited thereto, computer software programs for implementing the present method can be written in any number of suitable programming languages such as, for example, C, C++, C# or other compilers, assemblers, interpreters or other computer languages or platforms.

Thus, computer-implemented methods and systems for replication of data between mirrored data sites are described. Although embodiments have been described with reference to specific exemplary embodiments, it will be evident that various modifications and changes can be made to these exemplary embodiments without departing from the broader spirit and scope of the present application. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for replication of data between mirrored data sites, the method comprising: receiving a data object at a first mirrored data site including one or more data nodes;replicating the data object to the one or more data nodes internally within the first mirrored data site;generating a data object reference indicating a forthcoming transmission of the data object to a second mirrored data site, wherein the data object reference includes an identification of the first mirrored data site, which stores one or more replicas of the data object;transmitting the data object reference to a proxy reference database; andqueuing the data object referred to by the data object reference in a data delivery queue for the forthcoming transmission to the second mirrored data site, wherein upon transmission of the data object referred to by the data object reference to the second mirrored data site, the data object is replicated to one or more data nodes associated with the second mirrored data site; anddiscarding the data object reference from the proxy reference database in response to the data object being successfully written to the second mirrored data site; andreceiving at the second mirrored data site, a request for the data object;determining, based on the data object reference in the proxy reference database, that the data object has not been received by the second mirrored data site; andbased on the determination, requesting the first mirrored data site referred to by the data object reference in the proxy reference database to serve the data object.

2. The method of claim 1, further comprising retrying the data object for transmission when the data object cannot be immediately written to the nodes within the second mirrored data site due to current storage capacity constraints, wherein the data object reference is not discarded until available capacity is realized in the second mirrored data site.

3. The method of claim 1, wherein the data object reference is generated by running a cryptographic hash function over a content associated with the data object.

4. The method of claim 3, wherein the data object reference allows finding the data object based on the content associated with the data object.

5. The method of claim 1, wherein the first mirrored data site and the second mirrored data site support an intra-site tier operation and an inter-site tier operation using the same data logic, the replicating of the data object to the one or more data nodes being performed by selecting the intra-site tier operation, and the replicating of the data object to the one or more data nodes in the second mirrored data site being performed by specifying the inter-site operation.

6. The method of claim 1, further comprising synchronizing mirrored data sites, the synchronizing including: comparing one or more data object references and data objects associated with the first mirrored data site to one or more data object references in the second mirrored data site and data objects associated with the second mirrored data site; andrequesting deliveries of the one or more data objects corresponding to the one or more data object references.

7. The method of claim 1, wherein the mirrored data sites provide a consistent view of a file system.

8. The method of claim 1, wherein each node in the first mirrored data site and in the second mirrored data sites are interconnected to form a complete mesh.

9. The method of claim 1, wherein one data site includes an arbitrator node to arbitrate data replications between the mirrored data sites.

10. The method of claim 9, wherein upon failure of a node hosting a Metadata Operational Processor (MOP), another node in that site is designated as the arbitrator node.

11. The method of claim 10, wherein each mirrored data site includes respective gateway nodes configured to establish a gateway between the mirrored data sites.

12. The method of claim 11, wherein upon a failure of one of the gateway nodes, an alternate gateway node is designated, the alternate gateway node using a pre-provisioned path to establish an alternate gateway between the mirrored data sites.

13. A system for replication between mirrored data sites, the system comprising: a proxy reference database configured to store one or more data object references associated with one or more data objects;a processor configured to: receive a data object at a first mirrored data site including one or more data nodes;replicate the data object to the one or more data nodes internally within the first mirrored data site;generate a data object reference indicating a forthcoming transmission of the data object to a second mirrored data site, wherein the data object reference includes an identification of the first mirrored data site, which stores one or more replicas of the data object;transmit the data object reference to the proxy reference database, wherein the proxy reference database is associated with the second mirrored data site, the first mirrored data site and the second mirrored data site having one or more pre-provisioned interconnections; andqueue the data object referred to by the data object reference in a data delivery queue for the forthcoming transmission to the second mirrored data site, wherein upon transmission of the data object referred to by the data object reference to the second mirrored data site, the data object reference is discarded from the proxy reference database in response to the data object being successfully written to the second mirrored data site; andreceive at the second mirrored data site, a request for the data object;determine, based on the data object reference in the proxy reference database, that the data object has not been received by the second mirrored data site; andbased on the determination, request the first mirrored data site referred to by the data object reference in the proxy reference database to serve the data object.

14. The system of claim 13, wherein the data object reference is generated by running a cryptographic hash function over a content associated with the data object.

15. The system of claim 14, wherein the data object reference allows finding the data object based on the content associated with the data object.

16. The system of claim 13, wherein the first mirrored data site and the second mirrored data sites support an intra-site tier operation and an inter-site tier operation using the same data logic, the replicating of the data object to the one or more data nodes being performed by selecting the intra-site tier operation and the replicating of the data object to the one or more data nodes in the second mirrored data sites being performed by selecting the inter-site tier operation.

17. The system of claim 13, wherein the processor is further configured to synchronize the first mirrored data site and the second mirrored data site, the synchronizing including: comparing one or more data object references and data objects associated with one of the mirrored data sites to one or more data object references and data objects associated with the other mirrored data site; andrequesting deliveries of the one or more data objects corresponding to the one or more data object references.

18. The system of claim 13, wherein one of the mirrored data sites includes an arbitrator node to arbitrate data replications between all mirrored data sites.

19. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, causes the one or more processors to: receive a data object at a first mirrored data site including one or more data nodes;replicate the data object to the one or more data nodes internally within the first mirrored data site;generate a data object reference indicating a forthcoming transmission of the data object to a second mirrored data site, wherein the data object reference includes an identification of the first mirrored data site, which stores one or more replicas of the data object;transmit the data object reference to a proxy reference database; andqueue the data object referred to by the data object reference in a data delivery queue for the forthcoming transmission to the second mirrored data site, wherein upon transmission of the data object referred to by the data object reference to the second mirrored data site, the data object is replicated to one or more data nodes associated with the second mirrored data site; anddiscard the data object reference from the proxy reference database in response to the data object being successfully written to the second mirrored data site; andreceive at the second mirrored data site, a request for the data object;determine, based on the data object reference in the proxy reference database, that the data object has not been received by the second mirrored data site; andbased on the determination, request the first mirrored data site referred to by the data object reference in the proxy reference database to serve the data object.