Method and apparatus for reading and writing committed data

Description

BACKGROUND

Databases may include multiple disks to store redundant data. The redundant storage of data in multiple disks may improve the rate at which data can be read from the database by spreading out read requests over multiple disks or systems. Further, the redundant storage of data may improve uptime of the database by making the database more robust to failures.

SUMMARY

According to some aspects, systems and methods are provided to read and write commit data in a database that redundantly stores data. Committed data may include, for example, data that has been written to at least a predetermined number of storage devices in the database. In some embodiments, the system may identify committed data within the database and process certain read requests (e.g., read commit requests) by reading the identified committed data. Thereby, the database may provide an indication of what data has been replicated on at least a certain number of storage devices in the database. Further, the system may allow certain write requests (e.g., write commit requests) that provide a confirmation once the particular changes in the write request have been replicated on at least a predetermined number of storage devices in the database.

According to at least one aspect, a database system is provided. The database system includes at least one processor configured to execute a plurality of system components. The plurality of system components may include an interface component configured to receive a read commit command and provide a result, a snapshot component configured to generate a plurality of snapshots of data stored in a first data storage node of a plurality of data storage nodes and identify a committed snapshot representative of data that has been replicated on a majority of the plurality of data storage nodes, and a command processing component configured to read the committed snapshot responsive to receipt of the read commit command and generate the result using the committed snapshot.

In one embodiment, each snapshot of the plurality of snapshots on a first storage node corresponds to the data stored in the first storage node of the plurality of data storage nodes at unique point in time. In one embodiment, the interface is further configured to receive a confirmation from a second data storage node of the plurality of data storage nodes that the second data storage node has replicated data from the first data storage node. In one embodiment, the snapshot component is further configured to identify a new committed snapshot responsive to receipt of the confirmation from the second data storage node. In one embodiment, the snapshot component is further configured to remove at least one snapshot from the plurality of snapshots responsive to receipt of the confirmation from the second data storage node and, for example, determination that a newer committed snapshot exists.

In one embodiment, the snapshot component is further configured to generate a new snapshot of the data stored in the first data storage node over time, for example, periodically. In one embodiment, the snapshot component is further configured to generate the new snapshot of the data stored in the first data storage node every 10 milliseconds (ms).

In one embodiment, the interface is further configured to receive a read command. In one embodiment, the command processing component is further configured to read data stored in a second data storage node of the plurality of data storage nodes responsive to receiving the read command.

In one embodiment, the command processing component is configured to read the committed snapshot at least in part by identifying a second data storage node that stores data consistent with committed snapshot and reading from the identified second data storage node. In one embodiment, the interface is further configured to receive a write commit command. In one embodiment, the command processing component is further configured write data to the first data storage node, determine whether a majority of the data storage nodes in the plurality of data storage nodes have replicated the data written to the first data storage node, and generate a write commit confirmation responsive to receipt of the write commit command. In one embodiment, the interface component is further configured to provide the write commit confirmation.

In one embodiment, the interface component is further configured to receive a write command. In one embodiment, the command processing component is further configured to write data to the first data storage node responsive to receipt of the write command.

In one embodiment, the system further includes a database including the plurality of data storage nodes. In one embodiment, the database follows an eventual consistency model. In one embodiment, the first data storage node is a primary data storage node and a remainder of the data storage nodes of the plurality of data storage nodes are secondary data storage nodes. In one embodiment, the first data storage node includes an operation log and a remainder of the data storage nodes of the plurality of data storage nodes are configured to retrieve the operation log and replicate operations in the operation log.

According to at least one aspect, a method of performing operations in a computer database is provided. The method includes receiving, by an interface component executed by at least one processor, a read commit command, generating, by a snapshot component executed by the at least one processor, a plurality of snapshots of data stored in a first data storage node of a plurality of data storage nodes and identify a committed snapshot representative of data that has been replicated on a majority of the plurality of data storage nodes, reading, by a command processing component executed by the at least one processor, the committed snapshot responsive to receipt of the read commit command, generating, by the command processing component, generate a result to the read commit command, and providing, by the interface component, the result.

In one embodiment, each snapshot of the plurality of snapshots on a first storage node corresponds to the data stored in the first storage node of the plurality of data storage nodes at unique point in time. In one embodiment, the method further includes receiving a confirmation from a second data storage node of the plurality of data storage nodes that the second data storage node has replicated data from the first data storage node. In one embodiment, the method further includes identifying a new committed snapshot from the plurality of snapshots responsive to receipt of the confirmation from the second data storage node. In one embodiment, the method further includes removing at least one snapshot from the plurality of snapshots responsive to receipt of the confirmation from the second data storage node.

In one embodiment, the method further includes generating new snapshots of the data stored in the first data storage node periodically. In one embodiment, the method further includes generating the new snapshots of the data stored in the first data storage node includes generate the new snapshots of the data stored in the first data storage node every 10 milliseconds.

In one embodiment, the method further includes receiving a read command. In one embodiment, the method further includes reading data stored in a second data storage node of the plurality of data storage nodes responsive to receiving the read command.

In one embodiment, reading the committed snapshot includes identifying a second data storage node that stores data consistent with committed snapshot and reading from the identified second data storage node. In one embodiment, the method further includes receiving a write commit command. In one embodiment, the method further includes writing data to the first data storage node, determining whether a majority of the data storage nodes in the plurality of data storage nodes have replicated the data written to the first data storage node, and generating a write commit confirmation responsive to receipt of the write commit command. In one embodiment, the method further includes providing the write commit confirmation.

In one embodiment, the method further includes receiving a write command. In one embodiment, the method further includes writing data to the first data storage node responsive to receipt of the write command.

In one embodiment, the plurality of data storage nodes are organized in a database. In one embodiment, the database follows an eventual consistency model. In one embodiment, the first data storage node is a primary data storage node and a remainder of the data storage nodes of the plurality of data storage nodes are secondary data storage nodes. In one embodiment, the first data storage node includes an operation log and a remainder of the data storage nodes of the plurality of data storage nodes are configured to retrieve the operation log and replicate operations in the operation log.

According to at least one aspect, a non-transitory computer readable medium comprising instructions consistent with any methods and/or combination of methods described or claimed herein.

According to at least one aspect, a database system is provided. The database system includes a database comprising a plurality of data storage nodes, the plurality of data storage nodes including a primary storage node and a plurality of secondary storage nodes and at least one processor configured to execute a plurality of system components. The plurality of system components may include an interface component configured to receive a read commit command and provide a result, a snapshot component configured to generate a plurality of snapshots of data stored in a secondary data storage node of the plurality of secondary storage nodes and identify a committed snapshot representative of data that has been replicated on a majority of the plurality of data storage nodes, and a command processing component configured to read the committed snapshot responsive to receipt of the read commit command and generate the result.

In one embodiment, the plurality of snapshots are stored in one or more of the plurality of data storage nodes. For example, the plurality of snapshots may be stored in a primary data storage node and/or a secondary data storage node.

According to at least one aspect, a database system is provided. The database system comprises at least one processor configured to execute a plurality of system components. The plurality of system components comprises an interface component configured to receive a read commit command and provide a result; a snapshot component configured to generate a plurality of snapshots of data stored in a first data storage node of a plurality of data storage nodes and identify a committed snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of the plurality of data storage nodes; and a command processing component configured to read the committed snapshot responsive to receipt of the read commit command and generate the result using the committed snapshot.

In some embodiments, each snapshot of the plurality of snapshots corresponds to data stored in the first storage node at a unique point in time. In some embodiments, the interface component is further configured to receive at least one confirmation from at least one data storage node of the plurality of data storage nodes that the at least one data storage node has replicated data from the first data storage node. In some embodiments, the snapshot component is further configured to identify a new committed snapshot responsive to receipt of the at least one confirmation from the at least one data storage node. In some embodiments, the snapshot component is further configured to remove at least one snapshot from the plurality of snapshots responsive to receipt of the at least one confirmation from the at least one data storage node.

In some embodiments, the snapshot component is further configured to generate a new snapshot of the data stored in the first data storage node periodically. In some embodiments, the snapshot component is further configured to generate the new snapshot of the data stored in the first data storage node every 10 milliseconds.

In some embodiments, the interface is further configured to receive a read command. In some embodiments, the command processing component is further configured to read data stored in a second data storage node of the plurality of data storage nodes responsive to receiving the read command.

In some embodiments, the command processing component is configured to read the committed snapshot at least in part by identifying a data storage node from the plurality of data storage nodes that stores data consistent with committed snapshot and reading from the identified data storage node.

In some embodiments, the system further comprises a database including the plurality of data storage nodes and wherein the database follows an eventual consistency model. In some embodiments, the first data storage node is a primary data storage node and a remainder of the data storage nodes of the plurality of data storage nodes are secondary data storage nodes. In some embodiments, the first data storage node includes an operation log and a remainder of the data storage nodes of the plurality of data storage nodes are configured to retrieve the operation log and replicate operations in the operation log.

According to at least one aspect, a method of performing operations in a computer database. The method comprises receiving, by an interface component executed by at least one processor, a read commit command; generating, by a snapshot component executed by the at least one processor, a plurality of snapshots of data stored in a first data storage node of a plurality of data storage nodes; identifying, by the snapshot component executed by the at least one processor, a committed snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of the plurality of data storage nodes; reading, by a command processing component executed by the at least one processor, the committed snapshot responsive to receipt of the read commit command; generating, by the command processing component, a result to the read commit command using the committed snapshot; and providing, by the interface component, the result.

In some embodiments, each snapshot of the plurality of snapshots corresponds to the data stored in the first storage node of the plurality of data storage nodes at unique point in time. In some embodiments, the method further comprises receiving at least one confirmation from at least one data storage node of the plurality of data storage nodes that the at least one data storage node has replicated data from the first data storage node. In some embodiments, the method further comprises identifying a new committed snapshot from the plurality of snapshots responsive to receipt of the at least one confirmation from the at least one data storage node. In some embodiments, the method further comprises removing at least one snapshot from the plurality of snapshots responsive to receipt of the at least one confirmation from the at least one data storage node.

In some embodiments, the method further comprises generating new snapshots of the data stored in the first data storage node periodically.

According to at least one aspect, a database system is provided. The database system comprises at least one processor configured to execute a plurality of system components, wherein the plurality of system components comprises an interface component configured to receive a write commit command and provide a write commit confirmation; a snapshot component configured to generate a plurality of snapshots of data stored in a first data storage node of a plurality of data storage nodes and identify a first snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of the plurality of data storage nodes as a committed snapshot; and a command processing component configured to: determine whether the write commit command modifies at least one data element in the first data storage node; and responsive to the write commit command modifying the at least one data element in the first data storage node, modify the at least one data element in the first data storage node based on the write commit command; determine whether the majority of the plurality of storage nodes have replicated the modification to the at least one data element using the committed snapshot; and generate the write commit confirmation responsive to a determination that the majority of the plurality of data storage nodes have replicated the modification to the at least one data element.

In some embodiments, the command processing component is further configured to generate the write commit confirmation responsive to the write commit command not modifying the at least one data element in the first data storage node. In some embodiments, the command processing component is further to send a command to at least one data storage node in the plurality of data storage nodes to replicate the modification of the at least one data element in the first data storage node responsive to modifying the at least one data element in the first data storage node.

In some embodiments, the interface component is further configured to receive a confirmation from a second data storage node of the plurality of data storage nodes that the second data storage node has replicated the modification to the at least one data element. In some embodiments, the snapshot component is further configured to identify a second snapshot from the plurality of snapshots as the committed snapshot responsive to receipt of the confirmation from the second data storage node. In some embodiments, the snapshot component is further to remove the first snapshot from the plurality of snapshots responsive to identifying the second snapshot as the committed snapshot.

In some embodiments, the first data storage node stores an operation log comprising a plurality of database operations performed on the first data storage node and wherein the command processing component is further configured to determine whether at least one entry in the operation log has been replicated by the majority of data storage nodes in the plurality of data storage nodes using the committed snapshot responsive to the write commit command not modifying the at least one data element in the first data storage node. In some embodiments, the command processing component is further configured to generate the write commit confirmation responsive to a determination that the majority of the plurality of data storage nodes replicated the at least one entry in the operation log. In some embodiments, the at least one entry in the operation log comprises a newest entry in the operation log that occurred before receipt of the write commit command.

In some embodiments, the interface component is further configured to receive a write command and wherein the command processing component is further configured to write data to the first data storage node responsive to receipt of the write command.

According to at least one aspect, a method of performing operations in a computer database is provided. The method comprises generating, by a snapshot component executed by at least one processor, a plurality of snapshots of data stored in a first data storage node of a plurality of data storage nodes; identifying, by the snapshot component executed by the at least one processor, a first snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of the plurality of data storage nodes as a committed snapshot; receiving, by an interface component executed by the at least one processor, a write commit command; determining, by a command processing component executed by the at least one processor, whether the write commit command modifies at least one data element in the first data storage node; responsive to the write commit command modifying the at least one data element in the first data storage node, modifying the at least one data element in the first data storage node based on the write commit command; determining whether the majority of the plurality of storage nodes have replicated the modification to the at least one data element using the committed snapshot; and generating a write commit confirmation responsive to a determination that the majority of the plurality of data storage nodes have replicated the modification to the at least one data element; and providing, by the interface component, the write commit confirmation.

In some embodiments, the method further comprises generating the write commit confirmation responsive to the write commit command not modifying the at least one data element in the first data storage node. In some embodiments, the method further comprises receiving a confirmation from a second data storage node of the plurality of data storage nodes that the second data storage node has replicated the modification to the at least one data element. In some embodiments, the method further comprises identifying a second snapshot from the plurality of snapshots as the committed snapshot responsive to receipt of the confirmation from the second data storage node.

In some embodiments, the first data storage node stores an operation log comprising a plurality of operations performed on the first data storage node and wherein the method further comprises determining whether at least one entry in the operation log has been replicated by the majority of data storage nodes in the plurality of data storage nodes using the committed snapshot responsive to the write commit command not modifying the at least one data element in the first data storage node. In some embodiments, the method further comprises generating the write commit confirmation responsive to a determination that the majority of the plurality of data storage nodes replicated the at least one entry in the operation log.

According to at least one aspect, a database system is provided. The database system comprises a database configured to follow an eventual consistency model and comprising a primary data storage node storing an operation log and a plurality of secondary data storage nodes configured to retrieve the operation log from the primary data storage node and replicate operations in the operation log; and at least one processor configured to execute a plurality of system components. The plurality of system components comprises an interface component configured to receive a write commit command and provide a write commit confirmation; a snapshot component configured to generate a plurality of snapshots of data stored in the primary data storage node, identify a first snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of the plurality of secondary data storage nodes as a committed snapshot, and identify a second snapshot from the plurality of snapshots as the committed snapshot responsive to one or more secondary data storages nodes replicating at least one operation in the operation log; and a command processing component configured to: determine whether the write commit command modifies at least one document in a primary data storage node; and responsive to the write commit command modifying the at least one document in the primary data storage node, modify the at least one document in the primary data storage node; determine whether the majority of the secondary storage nodes have replicated the modification to the at least one document in the primary data storage node using the committed snapshot; and generate the write commit confirmation responsive to a determination that the majority of the secondary storage nodes replicated the modification to the at least one document.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of at least one embodiment are discussed herein with reference to the accompanying figures, which are not intended to be drawn to scale. The figures are included to provide illustration and a further understanding of the various aspects and embodiments, and are incorporated in and constitute a part of this specification, but are not intended as a definition of the limits of the invention. Where technical features in the figures, detailed description or any claim are followed by references signs, the reference signs have been included for the sole purpose of increasing the intelligibility of the figures, detailed description, and/or claims. Accordingly, neither the reference signs nor their absence are intended to have any limiting effect on the scope of any claim elements. In the figures, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component may be labeled in every figure. In the figures:

FIG. 1 illustrates a block diagram of an example commit engine configured to process read and write commit commands, according to some embodiments;

FIG. 2 illustrates a block diagram of an example distributed database system, according to some embodiments;

FIG. 3 illustrates a block diagram of an example replica set hosting a distributed database, according to some embodiments;

FIG. 4 illustrates a block diagram of another example distributed database system, according to some embodiments;

FIG. 5 is a flowchart showing an example snapshot generation process, according to some embodiments;

FIG. 6 is a flowchart showing an example snapshot commit process, according to some embodiments;

FIG. 7 is a flowchart showing an example read commit process, according to some embodiments;

FIGS. 8A and 8B are flowcharts showing example write commit processes, according to some embodiments;

FIG. 9 is a block diagram of an example special-purpose computer system, according to some embodiments;

FIG. 10 is a block diagram of an example disk or flash memory, according to some embodiments;

FIG. 11 is a block diagram of an example a distributed system, according to some embodiments;

FIG. 12 is a state diagram showing the generation and maintenance of snapshots, according to some embodiments; and

FIG. 13 is a table showing the relationship between the snapshot view and the operation log view of data, according to some embodiments.

DETAILED DESCRIPTION

According to one aspect, systems and methods are provided for reading and writing committed data in a database. In some embodiments, the database may follow an eventual consistency model and committed data may be data that has been replicated by more than a predetermined number of secondary nodes (e.g., half the secondary nodes) in the database. In these embodiments, data may be replicated from a primary node in the database to secondary nodes in the database over time, and the database may contain many groups of primary and secondary nodes. Thereby, one or more of the secondary nodes may have data that is not completely up-to-date relative to a respective primary node. Further, a client sending a read request to the database may receive data from a secondary node that is not completely up-to-date. The systems, as described herein according to some embodiments, provide clients an ability to read only committed from the database. The term “read commit” may refer to the command to read committed data from the database. For example, users may provide command line or execution instructions to read only committed data. Thereby, the system provides the client a view of the data that is being provided by a majority of the secondary nodes in response to read requests. Further, the systems may provide clients an ability to write data to the database and receive a confirmation once the data is replicated on a sufficient number of nodes in the database so as to be committed. The term “write commit” may refer to such a command.

In some embodiments, the system may monitor the latest data that has been committed to the database through the generation and management of snapshots. Snapshots may be captured periodically (or aperiodically) and be representative of the state of the data in the database at a particular point in time. The system may identify a committed snapshot that is representative of committed data (e.g., replicated by a threshold number of nodes (e.g., majority of nodes)). These snapshots may advantageously simplify processing of read commit and/or write commit commands. For example, read commit commands may be processed by simply reading from the committed snapshot. Similarly, write commit commands may be performed by determining whether the operation(s) included in the write commit command is included in the committed snapshot and returning a confirmation one the operation(s) are included in the committed snapshot.

Examples of the methods, devices, and systems discussed herein are not limited in application to the details of construction and the arrangement of components set forth in the following description or illustrated in the accompanying drawings. The methods and systems are capable of implementation in other embodiments and of being practiced or of being carried out in various ways. Examples of specific implementations are provided herein for illustrative purposes only and are not intended to be limiting. In particular, acts, components, elements and features discussed in connection with any one or more examples are not intended to be excluded from a similar role in any other examples.

Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. Any references to examples, embodiments, components, elements or acts of the systems and methods herein referred to in the singular may also embrace embodiments including a plurality, and any references in plural to any embodiment, component, element or act herein may also embrace embodiments including only a singularity. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements. The use herein of “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms.

Example Database Commit Engine

FIG. 1 shows an example commit engine 102 according to some embodiments. The commit engine 102 may be designed to process various received commands on a database 108. In some embodiments, the database 108 may include a plurality of nodes and replicate data from primary node to one or more secondary nodes. The primary and respective secondary nodes are configured to host at least a portion of the database data, and other replica sets (e.g., primary and respective secondary nodes) can host other portions. In these embodiments, the primary node may handle commands that change the data stored in the database and the secondary nodes may replicate the data in the primary node over time and process read requests. Thereby, the secondary nodes may have data that is not completely up-to-date relative to the primary nodes. Committed data may include data that has been replicated to at least a predetermined number of secondary nodes in the database (e.g., at least half of the secondary nodes).

In some embodiments, the commit engine 102 receives and processes read commit commands 104 and/or write commit commands 106. The commit engine 102 may process a read commit command 104 by identifying committed data in the database 108, processing a read request on the identified committed data, and providing the results as committed data 110. The commit engine 102 may process a write commit command 106 by writing data to the database 108 and providing a write commit confirmation 112 once the changes in the write to the database 108 have been committed.

As shown in FIG. 1, the commit engine 102 includes an interface component 114. The interface component 114 may be configured to receive and provide data to various systems, such as the database 108. In some embodiments, the interface component 114 receives the read commit command 104 and provides the committed data 110 once the read commit command 104 has been processed by the commit engine 102. The interface component 114 may also receive the write commit command 106 and provide the write commit confirmation 112 once the write commit command 106 has been processed by the commit engine 102.

The commit engine 102 may include a snapshot component 116 to generate snapshots of data in the database 108. The snapshots are illustrative of the data in the database 108 at a particular point in time (e.g., a unique point in time) on a logical clock. For example, the snapshot component 116 may generate a snapshot of the database 108 every 10 milliseconds (ms) and each snapshot may be representative of the data in the database 108 at the time the snapshot was taken. For example, the snapshots may be indicative of the data in a primary node and/or a secondary node or any replica set at a particular point in time. Thereby, the snapshots generated by the snapshot component 116 may provide a different view of the data in the database 108 that illustrates changes over time, and/or from node to node.

It should be appreciated that the snapshots may be captured at unique points in time on both a logical clock and a physical clock in cases where the time difference between the captured snapshots is larger than the resolution of the physical clock. For example, the snapshots may be captured every 10 ms and the resolution of the physical clock may be 1 ms. In this example, each of the snapshots may be generated at both a unique point in time on a logical clock and a unique point in time on the physical clock.

As described above, secondary nodes in the database 108 may include data that is not completely up-to-date relative to the primary node. Thereby, there is generally a delay between changes being made to the primary node and the changes being replicated in a requisite number of secondary nodes so as to be committed to the database 108. In some embodiments, the snapshot component 116 may identify a committed snapshot from the generated snapshots. In one example, the commit engine evaluates changes in state within the snapshots to determine if replication has occurred to a threshold number of nodes, if so the snapshot can be identified or labelled as a committed snapshot. In another example, a snapshot is only updated to include new data once the new data has been committed. Thus, in one implementation committed snapshots are only updated to include committed data.

A committed snapshot may be a latest snapshot of the database 108 that is representative of only committed data. Stated differently, the committed snapshot may be the most recent snapshot that only contains committed data. Further, the snapshot component 116 may update the snapshot that is the committed snapshot as the secondary nodes replicate changes made to the primary node. For example, the latest changes to the primary node that have been committed may be the changes made up to 12:30 pm on May 6, 2016. Subsequently, the secondary nodes may replicate data from the primary node and move the commit forward from 12:30 pm on May 6, 2016 to 2:00 pm on May 6, 2016. The snapshot component 116 may, in response, select a more recent snapshot as the committed snapshot and remove any snapshots that are older than the new committed snapshot. Thereby, the snapshot component 116 updates which snapshot is the committed snapshot in tandem with the secondary nodes in the database 108 replicating the data from the primary node.

The read commit commands 104 may be processed by a command processing component 118 of the commit engine 102. In some embodiments, the command processing component 118 may process read commit commands 104 by reading data from the committed snapshot generated and identified by the snapshot component 116. The particular method employed by the command processing component 118 to read from the committed snapshot may vary depending upon the particular implementation of the snapshot. For example, the snapshot may be a copy of the data and the command processing component 118 may read directly from the snapshot. In other examples, the snapshots may only be representative of operations made on the database to obtain data in the same state as the data in the database 108 at the time of the snapshot. In these examples, the command processing component 118 may identify a secondary node in the database 108 that has data stored in the same state as the committed snapshot and read from that secondary node. In some embodiments, each node may maintain a respective snapshot and reads can be distributed to secondary nodes based on respective snapshot information. It should be appreciated that the command processing component 118 may identify a secondary node in the database 108 that has data that deviates from the committed snapshot so long as the data relevant to the read request (e.g., the portion of the data being read) is in the same state as the committed snapshot. The results from the read performed by the command processing component 118 may be provided as the committed data 110 via the interface component 114.

The write commit commands 106 may also be processed by the command processing component 118. In some embodiments, the command processing component 118 may process the write commit commands 106 by writing the data to the primary node of the database 108 and waiting for a requisite number of secondary nodes (e.g., at least half of the secondary nodes) to have replicated the changes written to the primary node. The command processing component 118 may, in some implementations, expedite the process of replicating the changes by, for example, communicating with the secondary nodes to trigger replication of the changes. Once the changes have been replicated to a sufficient number of secondary nodes, the command processing component 118 may generate the write commit confirmation 112 that may be provided via the interface component 114.

It should be appreciated that, in some embodiments, the commit engine 102 may also be able to process regular read and/or write commands in addition to the read commit command 104 and the write commit command 106. For example, the command processing component 118 may read data from a secondary node in the database 108 and provide the result via the interface component 114 responsive to receipt of a read command. Similarly, the command processing component 118 may write data to a primary node in the database 108 in response to receipt of the write command.

Example Database Systems

An example of a database subsystem 200 is shown in FIG. 2. The database subsystem 200 includes an interface 202 for sending and receiving information (including database requests and responses thereto) to router processes, database clients, or other components or entities in the system. In one embodiment, the backend architecture is configured to interact with any data model provided by a managed database. For example, the managed database can include a non-relational data model. In another embodiment, the data model can be implemented in the form of replica sets as described in U.S. patent application Ser. No. 12/977,563, which is hereby incorporated by reference in its entirety. The database subsystem 200 includes a storage application. In one implementation described in greater detail below, a base unit of data is a document.

In some embodiments, a storage application programming interface (API) 208 receives database requests, including requests to perform read and write operations. When a write operation is requested, the storage API 208 in response selectively triggers a first storage engine 204 or a second storage engine 206 configured to store data in a first data format or second data format, respectively, in node 210. As discussed in more detail below, a database monitor 211 may track a number of analytics about the database. In some embodiments, the database monitor 211 is configured to track the operations performed on the data over time, and stores that information as analytics data 213. In some examples, analytic data may be stored in a separate database. In other examples, the analytics data is stored as a name collection (i.e., a logical grouping of data). These analytics may be provided to the storage API 208, which relies on the analytics to selectively actuate an appropriate storage engine. In further embodiments, although multiple storage engines are provided, not all storage engines may operate with snapshots and/or not all storage engines may provide read commit (and/or write commit) functionality. Responsive to a command execution that includes read commit settings, the system may force use of a particular storage engine or alternatively provide error information that the current storage engine does not support the functionality. Thus, the system can be configured to check capability of storage engines to support read commit (and/or write commit) functions and report on the same to end users.

In one example, the database monitor 211 tracks the relative number of read and write operations performed on a collection within the database. In another example, the database monitor 211 is configured to track any operations (e.g., reads, writes, etc.) performed on any base unit of data (e.g., documents) in the database.

In some embodiments, the storage API 208 uses the tracked data (e.g., analytics data) collected by the database monitor 211 and/or the analytics data 213 to select an optimal storage engine for a database, a collection, or a document having the observed read/write ratio. In one example, the storage API 208 is mapped to the selected storage engine. For example, an identifier of the selected storage engine may be stored in a location in memory or on disk; when a write operation request is received by the storage API 208, the identifier is used to identify and activate the storage engine. Alternatively, elements of the database can specify a mapping or association with a storage engine that can be manually edited, edited through an administrative interface, or automatically changed responsive to system monitoring. In other embodiments, the database monitor 211 itself is configured to determine an optimal storage engine based on the analytics data 213 and other aspects of the data, for example, stored in the database, database collection, or in a document. This determination may be passed to the storage API 208, or otherwise used to map the storage API 208 to a determined storage engine.

The storage API 208 receives database write requests (e.g., from a database API (not shown)) via a network interface 202, and carries out the requested operations by selectively triggering one of the first storage engine 204 and the second storage engine 206. The first storage engine 204 and the second storage engine 206 are executable software modules configured to store database data in the data node 210 in a particular data format. For example, the first storage engine 204 may be configured to store data in a row-store format, and the second storage engine 206 may be configured to store data in a LSM-tree format. In one example, the first storage engine 204 and/or the second storage engine 206 are configured store primary database data (i.e., the data being stored and queried) in a particular data format in the primary data memory 212, and may store database index data in a particular data format in index data memory 214. In one embodiment, the first storage engine 204 and/or the second storage engine 206 are configured store an operation log (referred to as an “oplog”) 216 in a particular data format. As discussed in more detail below, a database monitor 211 may track a number of analytics about the database, and the operations performed on it over time, and stores that information as analytics data 213.

One advantage of using the storage API 208 as an abstraction layer between the database API and the storage engines is that the identity and selection of a particular storage engine can be transparent to the database API and/or a user interacting with the database API. For example, the database API may pass a “write” function call to the storage API 208 instructing the storage API to write a particular set of data to the database. The storage API 108 then determines, according to its own analysis and/or user input, which storage engine should perform the write operation. Different storage engines may be appropriate for different types of data stored in different collections that may undergo a variety of different operations. Thus, the choice and implementation of calls to an appropriate storage engine are made by the API 208, freeing the database API calls to simply request a “write” of certain data. This abstraction level allows for the implementation of the system on large filesystems that may be stored across machines in a database cluster, such as the Hadoop Filesystem offered by the Apache Software Foundation.

Another advantage of using the storage API 208 is the ability to add, remove, or modify storage engines without modifying the requests being passed to the API 208. The storage API 208 is configured to identify the available storage engines and select the appropriate one based on one or more factors discussed below. The database API requesting write operations need not know the particulars of the storage engine selection or operation, meaning that storage engines may be embodied in pluggable modules that may be swapped out or modified. Thus, users are able to leverage the same query language, data model, scaling, security and operational tooling across different applications, each powered by different pluggable storage engines.

The embodiment shown and discussed with respect to FIG. 2 depicts a single database node 210. Yet in some embodiments, multiple database nodes may be provided and arranged in a replica set. FIG. 3 shows a block diagram of an exemplary replica set 300. Replica set 300 includes a primary node 302 and one or more secondary nodes 308 and 310, each of which is configured to store a dataset that has been inserted into the database. The primary node 302 may be configured to store all of the documents currently in the database, and may be considered and treated as the authoritative version of the database in the event that any conflicts or discrepancies arise, as will be discussed in more detail below. While two secondary nodes 308, 310 are depicted for illustrative purposes, any number of secondary nodes may be employed, depending on cost, complexity, and data availability requirements. In a preferred embodiment, one replica set may be implemented on a single server. In other embodiments, the nodes of the replica set may be spread among two or more servers.

The primary node 302 and secondary nodes 308, 310 may be configured to store data in any number of database formats or data structures as are known in the art. In a preferred embodiment, the primary node 302 is configured to store documents or other structures associated with non-relational databases. The embodiments discussed herein relate to documents of a document-based database, such as those offered by MongoDB, Inc. (of New York, N.Y. and Palo Alto, Calif.), but other data structures and arrangements are within the scope of the disclosure as well.

In some embodiments, the replica set primary node 302 only accepts write requests (disallowing read requests) from client systems 304, 306 and the secondary nodes 308, 310 only accept reads requests (disallowing write requests) from client systems 304, 306. In such embodiments, the primary node 302 receives and processes write requests against the database, and replicates the operation/transaction asynchronously throughout the system to the secondary nodes 308, 310. In one example, the primary node 302 receives and performs client write operations and generates an oplog. Each logged operation is replicated to, and carried out by, each of the secondary nodes 308, 310, thereby bringing those secondary nodes into synchronization with the primary node 302. In some embodiments, the secondary nodes 308, 310 may query the primary node 302 to receive the operation log and identify operations that need to be replicated. In other embodiments, the operation log may be transmitted from the primary node 302 to the secondary nodes 308, 310 periodically or in response to the occurrence of a predefined condition, such as accruing a threshold number of operations in the operation log that have not yet been sent to the secondary nodes 308, 310. Other implementations can be configured to provide different levels of consistency, and, for example, by restricting read requests. According to one embodiment, read requests can be restricted to systems having up to date data, read requests can also in some settings be restricted to primary systems, among other options.

In some embodiments, both read operations may be permitted at any node (including primary node 302 or secondary nodes 308, 310) and write operations limited to primary nodes in response to requests from clients. The scalability of read operations can be achieved by adding nodes and database instances. In some embodiments, the primary node 302 and/or the secondary nodes 308, 310 are configured to respond to read operation requests by either performing the read operation at that node or by delegating the read request operation to another node (e.g., a particular secondary node 308). Such delegation may be performed based on load-balancing and traffic direction techniques. In other embodiments, read distribution can be managed based on a respective snapshot available at various nodes within a distributed database. For example, the system can determine based on analyzing client requested data what snapshot is associated with the requested data and what node hosts the respective data or snapshot that can be used to provide the requested data. In one example, a data routing processor accesses configuration files for respective replica sets to determine what node can respond to a data request, and further analysis of respective snapshots can determine, for example, what node within a replica set needs to be accessed.

In some embodiments, the primary node 302 and the secondary nodes 308, 310 may operate together to form a replica set 300 that achieves eventual consistency, meaning that replication of database changes to the secondary nodes 308, 310 may occur asynchronously. When write operations cease, all replica nodes of a database will eventually “converge,” or become consistent. The eventually consistent model provides for a loose form of consistency. In one particular example, (assuming >3 secondary nodes) client systems (e.g. 304, 306) request write operations: W(x=3); W(x=7); W(x=5). As the replication of the write requests occurs asynchronously, at some point all of the secondary nodes (e.g. 308, 310) will respond to a read request with 5. However, in the short term (during replication operations) client systems randomly reading from secondary nodes can see [read operations designated by R (variable and actual value)]: R(x==7); R(x==0); R(x==5); and R(x==3). In such a configuration, replica set 300 provides eventual consistency and can permit out of order reads (in the short term). Other example implementations can increase the strength of consistency, and for example, can include monotonic read consistency (no out of order reads). Eventual consistency may be a desirable feature where high availability is important, such that locking records while an update is stored and propagated is not an option. In such embodiments, the secondary nodes 308, 310 may handle the bulk of the read operations made on the replica set 300, whereas the primary node 308, 310 handles the write operations. For read operations where a high level of accuracy is important (such as the operations involved in creating a secondary node), read operations may be performed against the primary node 302. In some embodiments, replica set 300 can be configured to perform according to a single writer eventually consistent model.

It will be appreciated that the difference between the primary node 302 and the one or more secondary nodes 308, 310 in a given replica set may be largely the designation itself and the resulting behavior of the node; the data, functionality, and configuration associated with the nodes may be largely identical, or capable of being identical (e.g., secondary nodes can be elevated to primary nodes in the event of failure). Thus, when one or more nodes within a replica set 300 fail or otherwise become available for read and/or write operations, other nodes may change roles to address the failure. For example, if the primary node 302 were to fail, a secondary node 308 may assume the responsibilities of the primary node, allowing operation of the replica set to continue through the outage. This failover functionality is described in U.S. application Ser. No. 12/977,563, the disclosure of which is hereby incorporated by reference in its entirety.

Each node in the replica set 300 may be implemented on one or more server systems. Additionally, one server system can host more than one node. Each server can be connected via a communication device to a network, for example the Internet, and each server can be configured to provide a heartbeat signal notifying the system that the server is up and reachable on the network. Sets of nodes and/or servers can be configured across wide area networks, local area networks, intranets, and can span various combinations of wide area, local area and/or private networks. Various communication architectures are contemplated for the sets of servers that host database instances and can include distributed computing architectures, peer networks, virtual systems, among other options.

The primary node 302 may be connected by a LAN, a WAN, or other connection to one or more of the secondary nodes 308, 310, which in turn may be connected to one or more other secondary nodes in the replica set 300. Connections between secondary nodes 308, 310 may allow the different secondary nodes to communicate with each other, for example, in the event that the primary node 302 fails or becomes unavailable and a secondary node must assume the role of the primary node.

According to one embodiment, a plurality of nodes (e.g., primary nodes and/or secondary nodes) can be organized in groups of nodes in which data is stored and replicated across the nodes of the set. Each group can be configured as a replica set. In another embodiment, one or more nodes are established as primary nodes that host a writable copy of the database. Each primary node can be responsible for a portion of the database, e.g. a database shard. Database sharding breaks up sections of the database into smaller portions based on, for example, ranges of the data. In some implementations, database sharding facilitates scaling a primary-secondary architecture over a large number of nodes and/or large database implementations. In one embodiment, each database shard has one primary node which replicates its data to its secondary nodes. Database shards can employ location preferences. For example, in a database that includes user records, the majority of accesses can come from specific locations. Migrating a shard primary node to be proximate to those requests can improve efficiency and response time. For example, if a shard for user profile includes address information, shards can be based on ranges within the user profiles, including address information. If the nodes hosting the shard and/or the shard primary node are located proximate to those addresses, improved efficiency can result, as one may observe the majority of requests for that information to come from locations proximate to the addresses within the shard.

An example of a database subsystem 400 incorporating a replica set 410 is shown in FIG. 4. As can be seen, database subsystem 400 incorporates many of the elements of database subsystem 200 of FIG. 2 including the network interface 202, the storage engines 204, 206, the storage API 208, the database monitor 211, and the analytics database 212. Relative to the database subsystem 200 shown in FIG. 2, the database subsystem 400 replaces the single node 210 with a replica set 410 comprising primary node 420 and secondary nodes 430 and 440. In one example, the replica set 410 functions in much the same manner as the replica set 300 discussed with respect to FIG. 3. While only two secondary nodes 430 and 440 are shown for illustrative purposes, it will be appreciated that the number of secondary nodes may be scaled up or down as desired or necessary.

In one example, database operation requests directed to the replica set 410 may be processed by the primary node 420 and either performed by the primary node 420 or directed to a secondary node 430, 440 as appropriate. In one embodiment, both read and write operations are permitted at any node (including primary node 420 or secondary nodes 430, 440) in response to requests from clients. The scalability of read operations can be achieved by adding nodes and database instances. In some embodiments, the primary node 420 and/or the secondary nodes 430, 440 are configured to respond to read operation requests by either performing the read operation at that node or by delegating the read request operation to another node (e.g., a particular secondary node 430). Such delegation may be performed based on various load-balancing and traffic direction techniques.

In some embodiments, the database only allows write operations to be performed at the primary node 420, with the secondary nodes 430, 440 disallowing write operations. In such embodiments, the primary node 420 receives and processes write requests against the database, and replicates the operation/transaction asynchronously throughout the system to the secondary nodes 430, 440. In one example, the primary node 420 receives and performs client write operations and generates an oplog. Each logged operation is replicated to, and carried out by, each of the secondary nodes 430, 440, thereby bringing those secondary nodes into synchronization with the primary node 420 under an eventual-consistency model.

In one example, primary database data (i.e., the data being stored and queried) may be stored by one or more data storage engines in one or more data formats in the primary data memory 422, 432, 442 of nodes 420, 430, 440, respectively. Database index data may be stored by one or more data storage engines in one or more data formats in the index data memory 424, 434, 444 of nodes 420, 430, 440, respectively. Oplog data may be stored by a data storage engine in a data format in oplog data memory 426 of node 420.

Example Methods for Processing Read/Write Commit Commands

As discussed above, various systems may be configured to process read commit commands. The processing of these commands may be facilitated by the generation of snapshots of the database at different points in time on a logical clock. FIG. 5 shows an example snapshot generation process 500 according to some embodiments. The snapshot generation process 500 may be performed by a system (e.g., snapshot component 116 shown in FIG. 1) to generate snapshots of the data in the database that may be used to facilitate execution of read commit requests. In some embodiments, one or more storage nodes in a database may generate snapshots. For example, all of the storage nodes in a database may generate snapshots. Further, the storage nodes may generate snapshots of their own data (e.g., data in the respective storage node) and/or snapshots of data in another storage node (e.g., data in a primary storage node). As shown in FIG. 5, the snapshot generation process 500 includes an act 502 of determining whether it is time to take a snapshot, an act 504 of determining whether a snapshot limit has been reached, and an act 506 of taking a snapshot.

In act 502, the system determines whether it is time to take a snapshot. The system may make the determination as to whether it is appropriate to take a snapshot based on a predetermined policy. For example, the system may periodically or aperiodically capture snapshots of the data. In other examples, the system may change the timing of taking snapshots based on client interaction with the database. For example, the system may take snapshots every 10 milliseconds during time periods where the database is receiving change requests and pause snapshot generation during periods where no changes to the database are taking place. Thereby, the system may reduce the number of snapshots being taken and stored by avoiding multiple successive snapshots that do not include any changes to the database. If the system determines that it is an appropriate time to take a snapshot, the system proceeds to act 504 to determine whether a snapshot limit has been reached. Otherwise the snapshot generation process 500 ends.

In act 504, the system determines whether a snapshot limit has been reached. The system may have a predefined maximum number of snapshots that can be maintained at any given time. For example, the system may have a maximum number of snapshots of 10,000. If the snapshot limit has been reached, then process 500 ends. Otherwise, the system proceeds to act 506 and takes a snapshot.

FIG. 6 shows an example snapshot commit process 600. A system (e.g., snapshot component 116 shown in FIG. 1) may perform the snapshot commit process 600 to continually update which snapshot is the latest committed snapshot and purge older snapshots. In some embodiments, one or more of the storage nodes that generate snapshots may perform process 600 to identify a committed snapshot. For example, all of the storage nodes may identify a committed snapshot. As shown in FIG. 6, the snapshot commit process 600 includes an act 602 of receiving confirmation from a secondary node, an act 604 of determining whether there is a new commit point in the operation log, an act 606 of determining whether to advance the committed snapshot, an act 608 of advancing the committed snapshot, and an act 610 of removing old snapshot(s).

In act 602, the system receives confirmation from a secondary node. The confirmation from the secondary node may indicate that a particular secondary node has performed various actions consistent with the operation log to make the data in that secondary node more up-to-date. The secondary node may provide the confirmation without explicit instructions from a primary node.

In act 604, the system determines whether there is a new commit point in the operation log. The system may determine whether there is a new commit point in the operation log by, for example, identifying the latest operation in the operation log that at least 50% of the secondary nodes have replicated. If the latest operation in the operation log is more up-to-date than the previous commit point, the commit point in the operation log advances. If the commit point in the operation log advances, the system proceeds to act 606 to determine whether the advance the committed snapshot. Otherwise the snapshot commit process 600 ends.

In act 606, the system determines whether to advance the committed snapshot based on the change in the commit point in the operation log. The system may determine whether to advance the committed snapshot by determining whether there is a more recent snapshot that only captures committed operations in the operation log. For example, the advance in the commit point in the operation log may be very small (e.g., 2 operations). In this example, the system may determine not to advance the snapshot by determining that the most recent snapshot that does not include any uncommitted operations (e.g., operations after the commit point in the operation log) is the same snapshot that has already been committed. In another example, the advance in the commit point in the operation log may be very large (e.g., 30 operations). In this example, the system may determine to advance the snapshot by determining that the most recent snapshot that does not include any uncommitted operations is the 10^thsnapshot after the previously committed snapshot. If the system determines that is appropriate to advance the committed snapshot, the system proceeds to act 608 and advances the committed snapshot (e.g., to the 10^thsnapshot). Otherwise the snapshot commit process 600 ends.

In act 608, the system advances the committed snapshot. The system may advance the committed snapshot by flagging or otherwise marking the identified snapshot as the committed snapshot.

In act 610, the system removes old snapshots. The old snapshots that may be removed may be the snapshots that are older than the new committed snapshot. For example, a previous committed snapshot may be removed.

As discussed above, various processes may be performed by the system to generate and maintain the snapshots. FIG. 12 is a diagram showing the generation and maintenance of snapshots. A first collection of snapshots 1202 is shown in the top left section of the diagram. The first collection of snapshots 1202 includes a committed snapshot illustrated in bold (Snapshot_5-10-15_1000). A new snapshot (Snapshot_5-10-15_1011) is added to the first collection of snapshots 1202 to form a second collection of snapshots 1204. The committed snapshot in the second collection of snapshots 1204 is advanced (now Snapshot_5-10-15_1002) to form the third collection of snapshots 1206. The committed snapshot may advance responsive to a sufficient number of secondary nodes replicating the data in the new committed snapshot. The snapshots that are older than the new committed snapshot (Snapshot_5-10-15_1002) are removed from the third collection of snapshots 1206 to form the fourth collection of snapshots 1208. These old snapshots may be removed to limit the number of snapshots that need to be stored at any given time.

Each of the snapshots in the various snapshot collections may be representative of data stored at various points in time that may correspond to one or more actions in the operation log. Thereby, the snapshots create a new view of the data in the database that illustrates changes over discrete chunks of time as opposed to changes over operations performed (as shown by the operation log). FIG. 13 is a table showing the relationship between the snapshot view 1302 and the operation log view 1304 of data. As shown, the snapshot view 1302 includes three snapshots (Snapshot_5-10-15_1000, Snapshot_5-10-15_1001, and Snapshot_5-10-15_1002) ordered from oldest (at the top) to newest (at the bottom). Each of the three snapshots were taken at different points in time on a database receiving a sequence of write requests shown in the operation log view 1304. The requests are ordered from the oldest (at the top) to the newest (at the bottom). Each of the three snapshots corresponds to a specific point in time in the operation log and, thereby, a portion of the operation log. For example, Snapshot_5-10-15_1000 was taken just after the “Create VEGETABLE collection” operation was performed. Thereby, the data in the database at the time Snapshot_5-10-15_1000 was taken may be recreated by performing the “Create VEGETABLE collection” operation. The second snapshot (Snapshot_5-10-15_1001) was taken just after the “Insert BROCCOLI into VEGETABLE collection” operation was performed. Thereby, the data in the database at the time Snapshot_5-10-15_1001 was taken may be replicated by performing all of the operations prior to (and including) the “Insert BROCCOLI into VEGETABLE collection” operation. The third snapshot (Snapshot_5-10-15_1002) was taken just after the “Insert TOMATO into FRUIT collection” operation was performed. Thereby, the data in the database at the time Snapshot_5-10-15_1002 was taken may be replicated by performing all of the operations prior to (and including) the “Insert TOMATO into FRUIT collection” operation.

As shown in FIG. 13, a particular snapshot may be associated with more than one operations relative to the previous snapshot. For example, the third snapshot (Snapshot_5-10-15_1002) is associated with four additional operations relative to the second snapshot (Snapshot_5-10-15_1001). Further, operations may be performed after the latest snapshot (Snapshot_5-10-15_1002) has been taken as shown by the last two operations in the operation log view 1304. These two operations may be captured in the next snapshot to be captured (e.g., Snapshot_5-10-15_1003).

As discussed above, the generation and maintenance of snapshots may facilitate the system to process read commit commands. FIG. 7 shows an example read commit process 700 that leverages the generated snapshots. The read commit process 700 may be performed by a system (e.g., a command processing component 118) to process read commit commands received from, for example, client. In some embodiments, a storage node (e.g., a primary or secondary storage node) that receives a read commit request may perform the read commit process 700. As shown in FIG. 7, the read commit process 700 includes an act 702 of receiving a read commit command, an act 704 of identifying the latest committed snapshot, an act 706 of reading from the latest committed snapshot, and an act 708 of returning the read results.

In some embodiments, read requests are permitted against uncommitted data by default. In one example, if no read commit parameter is specified in a data command, the system is configured to process against any available node and data. Whereas, if a read commit parameter is specified the system is configured to perform a read commit process, such as read commit process 700. For example, the system may determine where to access and return committed data.

In act 702, the system receives a read commit command. The read commit command may identify, for example, particular collections or documents that should be read.

In act 704, the system identifies the latest committed snapshot. The latest snapshot may be flagged or otherwise marked by, for example, the snapshot advance process 600 described above with reference to FIG. 6.

In act 706, the system reads from the latest committed snapshot. It should be appreciated that the particular method employed to read from the latest committed snapshot may vary depending upon the particular implementation of the snapshot employed. For example, the snapshot may not include data from the database but rather include a set of operations that have been performed (e.g., a section of the operation log). In this example the system may identify an appropriate secondary node to read that has the relevant data to be read in the same state as the committed snapshot. In other examples, the snapshot may store a copy of the data in the database. The snapshot may be stored in, for example, the primary storage node and/or any combination of one or more storage nodes. In these examples the system may directly read from the latest committed snapshot. In act 708, the system returns the data from the committed snapshot.

As discussed above, the generation and maintenance of snapshots may facilitate the system to process write commit commands. FIG. 8A illustrates an example write commit process 800A. The write commit process 800A described an example set of acts that may be performed by a system (e.g., command processing component 118 shown in FIG. 1) when a write commit command is received. In some embodiments, the write commit process 800A may be performed by, for example, a storage node that receives a write commit command (such as a primary storage node). It should be appreciated that, in some database models, only one data storage node (e.g., the primary storage node) may receive write and/or write commit requests. As shown in FIG. 8A, the process 800A includes the acts of: receiving a write commit command 802, determining whether a document is being modified 804, returning a write commit confirmation 806, writing data 808, and determining whether the write was committed 810.

In some embodiments, write requests (without a confirmation) are permitted by default (sometimes called “fire and forget write requests”). In one example, if no write commit parameter is specified in a data command, the system is configured to process the write command as a write request and not issue a confirmation once the write is committed. Whereas, if a write commit parameter is specified the system may be configured to perform a write commit process, such as write commit process 800A.

In act 802, the system receives a write commit command. The write commit command may include various document modifications that need to be performed by the system.

In act 804, the system determines whether the received write commit command includes any data element (e.g., document) modifications. Write commit commands may fail to make any document modifications for a variety of reasons. For example, a write commit command may request the addition of an entry to documents that do not already contain the entry. In this example, the write commit command may not have to make any document modifications if all of the documents already contain the entries. In these circumstances, the system may, for example, analyze data in the primary node of the database to determine whether any document modifications are necessary. If no document modifications are necessary, the system proceeds to act 806 and returns a write commit confirmation. Otherwise, the system proceeds to act 808 to write the data.

In act 808, the system writes the appropriate data to the primary and/or secondary nodes based on the requested changes in the write commit command. In some embodiments, the system may write the data by writing the data to the primary node and adding the appropriate operations to the oplog for the secondary nodes to perform. The system may also send explicit commands to all (or a portion) of the secondary nodes to reduce the time required for a sufficient number of secondary nodes to replicate the changes in the oplog.

In act 810, the system determines whether the write was committed. In some embodiments, the system may determine whether a write was committed using the committed snapshot. For example, the system may determine whether the operation(s) in the write commit command are included in the committed snapshot. If the operation(s) are included in the committed snapshot, the system may determine that the write has been committed. Otherwise, the system may determine that the write has not been committed. Additionally (or alternatively), the system may determine whether a write was committed based on a number of confirmations received from secondary nodes indicating that the operations added to the oplog in act 808 have been completed. For example, the system may have 11 secondary nodes and the system may determine that the write was committed once at least 6 confirmations have been received from secondary nodes. If the write has been committed, the system proceeds to act 806 and returns a write commit confirmation. Otherwise, the system repeats act 810 until the write has been committed.

It should be appreciated that various alterations may be made to the write-commit process 800A without departing from the scope of this disclosure. For example, the system may wait for the last oplog entry to be committed before returning a write commit confirmation regardless of the write commit command received. The write commit process 800B in FIG. 8B illustrates such an example process. As shown, the write commit process 800B adds an act 812 of determining whether the last oplog entry was committed in cases where the system determines that the write commit command does not modify documents in act 804.

In act 812, the system determines whether one or more oplog entries have been committed. For example, the system may determine whether the latest oplog entry that occurred before the write commit command was made has been committed. In another example, the system may determine whether the oplog entry that made the data in the primary data storage node consistent with the change requested in the write commit command has been committed. If the oplog entry has been committed, the system proceeds to act 806 and returns a write commit confirmation. Otherwise, the system repeats act 812. The determination of whether the last oplog entry has been committed in act 812 may be performed in a similar fashion to the determination of whether the write was committed in act 810 described above. For example, the system may wait for a sufficient number of confirmations from secondary nodes that have replicated the latest oplog entry so as to commit the last oplog entry.

Example Special-Purpose Computer System

A special-purpose computer system can be specially configured as disclosed herein. According to one embodiment the special-purpose computer system is configured to perform any of the described operations and/or algorithms. The operations and/or algorithms described herein can also be encoded as software executing on hardware that defines a processing component, that can define portions of a special purpose computer, reside on an individual special-purpose computer, and/or reside on multiple special-purpose computers.

FIG. 9 shows a block diagram of an example special-purpose computer system 900 on which various aspects of the present invention can be practiced. For example, computer system 900 may include a processor 906 connected to one or more memory devices 910, such as a disk drive, memory, or other device for storing data. Memory 910 is typically used for storing programs and data during operation of the computer system 900. Components of computer system 900 can be coupled by an interconnection mechanism 908, which may include one or more busses (e.g., between components that are integrated within a same machine) and/or a network (e.g., between components that reside on separate discrete machines). The interconnection mechanism enables communications (e.g., data, instructions) to be exchanged between system components of system 900.

Computer system 900 may also include one or more input/output (I/O) devices 902-904, for example, a keyboard, mouse, trackball, microphone, touch screen, a printing device, display screen, speaker, etc. Storage 912, typically includes a computer readable and writeable nonvolatile recording medium in which computer executable instructions are stored that define a program to be executed by the processor or information stored on or in the medium to be processed by the program.

The medium can, for example, be a disk 1002 or flash memory as shown in FIG. 10. Typically, in operation, the processor causes data to be read from the nonvolatile recording medium into another memory 1004 that allows for faster access to the information by the processor than does the medium. This memory is typically a volatile, random access memory such as a dynamic random access memory (DRAM) or static memory (SRAM). According to one embodiment, the computer-readable medium comprises a non-transient storage medium on which computer executable instructions are retained.

Referring again to FIG. 9, the memory can be located in storage 912 as shown, or in memory system 910. The processor 906 generally manipulates the data within the memory 910, and then copies the data to the medium associated with storage 912 after processing is completed. A variety of mechanisms are known for managing data movement between the medium and integrated circuit memory element and the invention is not limited thereto. The invention is not limited to a particular memory system or storage system.

The computer system may include specially-programmed, special-purpose hardware, for example, an application-specific integrated circuit (ASIC). Aspects of the invention can be implemented in software, hardware or firmware, or any combination thereof. Although computer system 900 is shown by way of example, as one type of computer system upon which various aspects of the invention can be practiced, it should be appreciated that aspects of the invention are not limited to being implemented on the computer system as shown in FIG. 9. Various aspects of the invention can be practiced on one or more computers having a different architectures or components than that shown in FIG. 9.

It should be appreciated that the invention is not limited to executing on any particular system or group of systems. Also, it should be appreciated that the invention is not limited to any particular distributed architecture, network, or communication protocol.

Various embodiments of the invention can be programmed using an object-oriented programming language, such as Java, C++, Ada, or C# (C-Sharp). Other programming languages may also be used. Alternatively, functional, scripting, and/or logical programming languages can be used. Various aspects of the invention can be implemented in a non-programmed environment (e.g., documents created in HTML, XML or other format that, when viewed in a window of a browser program, render aspects of a graphical-user interface (GUI) or perform other functions). The system libraries of the programming languages are incorporated herein by reference. Various aspects of the invention can be implemented as programmed or non-programmed elements, or any combination thereof.

Various aspects of this invention can be implemented by one or more systems similar to system 1100 shown in FIG. 11. For instance, the system can be a distributed system (e.g., client server, multi-tier system) that includes multiple special-purpose computer systems. In one example, the system includes software processes executing on a system associated with hosting database services, processing operations received from client computer systems, interfacing with APIs, receiving and processing client database requests, routing database requests, routing targeted database request, routing global database requests, determining global a request is necessary, determining a targeted request is possible, verifying database operations, managing data distribution, replicating database data, migrating database data, identifying committed data, etc. These systems can also permit client systems to request database operations transparently, with various routing processes handling and processing requests for data as a single interface, where the routing processes can manage data retrieval from database partitions, merge responses, and return results as appropriate to the client, among other operations.

There can be other computer systems that perform functions such as hosting replicas of database data, with each server hosting database partitions implemented as a replica set, among other functions. These systems can be distributed among a communication system such as the Internet. One such distributed network, as discussed below with respect to FIG. 11, can be used to implement various aspects of the invention. Various replication protocols can be implemented, and in some embodiments, different replication protocols can be implemented, with the data stored in the database replication under one model, e.g., asynchronous replication of a replica set, with metadata servers controlling updating and replication of database metadata under a stricter consistency model, e.g., requiring two phase commit operations for updates.

FIG. 11 shows an architecture diagram of an example distributed system 1100 suitable for implementing various aspects of the invention. It should be appreciated that FIG. 11 is used for illustration purposes only, and that other architectures can be used to facilitate one or more aspects of the invention.

System 1100 may include one or more specially configured special-purpose computer systems 1104, 1106, and 1108 distributed among a network 1102 such as, for example, the Internet. Such systems may cooperate to perform functions related to hosting a partitioned database, managing database metadata, monitoring distribution of database partitions, monitoring size of partitions, splitting partitions as necessary, migrating partitions as necessary, identifying sequentially keyed collections, optimizing migration, splitting, and rebalancing for collections with sequential keying architectures.

CONCLUSION

Having thus described several aspects and embodiments of this invention, it is to be appreciated that various alterations, modifications and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the spirit and scope of the invention. Accordingly, the foregoing description is by way of example only.

Use of ordinal terms such as “first,” “second,” “third,” “a,” “b,” “c,” etc., in the claims to modify or otherwise identify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements.

Claims

1. A database system comprising: at least one processor configured to execute a plurality of system components, wherein the plurality of system components comprise: an interface component configured to receive a write commit command and provide a write commit confirmation;a snapshot component configured to: generate a plurality of snapshots of data stored in a first data storage node of a plurality of data storage nodes, wherein the first data storage node stores an operation log comprising a plurality of database operations performed on the first data storage node; andidentify a first snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of the plurality of data storage nodes as a committed snapshot; anda command processing component configured to: determine whether the write commit command modifies at least one data element in the first data storage node; andresponsive to the write commit command not modifying the at least one data element in the first data storage node: determine whether at least one entry in the operation log has been replicated by the majority of data storage nodes in the plurality of data storage nodes using the committed snapshot; andgenerate the write commit confirmation responsive to determining that the at least one entry in the operation log in the first data storage node has been replicated by the majority of data storage nodes in the plurality of data storage nodes.
2. The system of claim 1, wherein the command processing component is further to send a command to at least one data storage node in the plurality of data storage nodes to replicate the modification of the at least one data element in the first data storage node responsive to modifying the at least one data element in the first data storage node.
3. The system of claim 1, wherein the interface component is further configured to receive a confirmation from a second data storage node of the plurality of data storage nodes that the second data storage node has replicated the modification to the at least one data element.
4. The system of claim 3, wherein the snapshot component is further configured to identify a second snapshot from the plurality of snapshots as the committed snapshot responsive to receipt of the confirmation from the second data storage node.
5. The system of claim 4, wherein the snapshot component is further to remove the first snapshot from the plurality of snapshots responsive to identifying the second snapshot as the committed snapshot.
6. The system of claim 1, wherein the command processing component is configured to, responsive to the write commit command modifying the at least one data element in the first data storage node: modify the at least one data element in the first data storage node based on the write commit command;determine whether the majority of the plurality of storage nodes have replicated the modification to the at least one data element using the committed snapshot; andgenerate the write commit confirmation responsive to a determination that the majority of the plurality of data storage nodes have replicated the modification to the at least one data element.
7. The system of claim 6, wherein the command processing component is further configured to generate the write commit confirmation responsive to a determination that the majority of the plurality of data storage nodes replicated the at least one entry in the operation log.
8. The system of claim 6, wherein the at least one entry in the operation log comprises a newest entry in the operation log that occurred before receipt of the write commit command.
9. The system of claim 1, wherein the interface component is further configured to receive a write command and wherein the command processing component is further configured to write data to the first data storage node responsive to receipt of the write command.
10. The system of claim 1, further comprising a database including the plurality of data storage nodes and wherein the database follows an eventual consistency model.
11. The system of claim 10, wherein the first data storage node is a primary data storage node and a remainder of the data storage nodes of the plurality of data storage nodes are secondary data storage nodes.
12. The system of claim 10, wherein the first data storage node includes an operation log and a remainder of the data storage nodes of the plurality of data storage nodes are configured to retrieve the operation log and replicate operations in the operation log.
13. A method of performing operations in a computer database, comprising: generating, by a snapshot component executed by at least one processor, a plurality of snapshots of data stored in a first data storage node of a plurality of data storage nodes, wherein the first data storage node stores an operation log comprising a plurality of database operations performed on the first data storage node;identifying, by the snapshot component executed by the at least one processor, a first snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of the plurality of data storage nodes as a committed snapshot;receiving, by an interface component executed by the at least one processor, a write commit command and providing, by the interface component, a write commit confirmation;determining, by a command processing component executed by the at least one processor, whether the write commit command modifies at least one data element in the first data storage node;responsive to the write commit command not modifying the at least one data element in the first data storage node: determining whether at least one entry in the operation log has been replicated by the majority of data storage nodes in the plurality of data storage nodes using the committed snapshot; andgenerating the write commit confirmation responsive to determining that the at least one entry in the operation log in the first data storage has been replicated by the majority of data storage nodes in the plurality of data storage nodes.
14. The method of claim 13, further comprising receiving a confirmation from a second data storage node of the plurality of data storage nodes that the second data storage node has replicated the modification to the at least one data element.
15. The method of claim 14, further comprising identifying a second snapshot from the plurality of snapshots as the committed snapshot responsive to receipt of the confirmation from the second data storage node.
16. The method of claim 13, further comprising, responsive to the write commit command modifying the at least one data element in the first data storage node: modify the at least one data element in the first data storage node based on the write commit command;determine whether the majority of the plurality of storage nodes have replicated the modification to the at least one data element using the committed snapshot; andgenerate the write commit confirmation responsive to a determination that the majority of the plurality of data storage nodes have replicated the modification to the at least one data element.
17. The method of claim 16, further comprising generating the write commit confirmation responsive to a determination that the majority of the plurality of data storage nodes replicated the at least one entry in the operation log.
18. A database system comprising: a database configured to follow an eventual consistency model and comprising a primary data storage node storing an operation log and a plurality of secondary data storage nodes configured to retrieve the operation log from the primary data storage node and replicate operations in the operation log; andat least one processor configured to execute a plurality of system components, wherein the plurality of system components comprise: an interface component configured to receive a write commit command and provide a write commit confirmation;a snapshot component configured to: generate a plurality of snapshots of data stored in the primary data storage node, wherein the primary data storage node stores an operation log comprising a plurality of database operations performed on the primary data storage node; andidentify a first snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of the plurality of secondary data storage nodes as a committed snapshot; andidentify a second snapshot from the plurality of snapshots as the committed snapshot responsive to one or more secondary data storages nodes replicating at least one operation in the operation log; anda command processing component configured to: determine whether the write commit command modifies at least one document in the primary data storage node; andresponsive to the write commit command not modifying the at least one document in the primary data storage node: determine whether at least one entry in the operation log has been replicated by the majority of data storage nodes in the plurality of data storage nodes using the committed snapshot; andgenerate the write commit confirmation responsive to determining that the at least one entry in the operation log in the primary data storage has been replicated by the majority of data storage nodes in the plurality of secondary data storage nodes.

RELATED APPLICATIONS

This Application is a continuation of U.S. application Ser. No. 15/605,276, entitled “METHOD AND APPARATUS FOR READING AND WRITING COMMITTED DATA” filed May 25, 2017, which claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Application Ser. No. 62/343,494, entitled “METHOD AND APPARATUS FOR READING AND WRITING COMMITTED DATA” filed on May 31, 2016, which is herein incorporated by reference in its entirety.

US Referenced Citations (388)

Number	Name	Date	Kind
4918593	Huber	Apr 1990	A
5379419	Heffernan et al.	Jan 1995	A
5416917	Adair et al.	May 1995	A
5471629	Risch	Nov 1995	A
5551027	Choy et al.	Aug 1996	A
5598559	Chaudhuri	Jan 1997	A
5710915	McElhiney	Jan 1998	A
5884299	Ramesh et al.	Mar 1999	A
5999179	Kekic et al.	Dec 1999	A
6065017	Barker	May 2000	A
6088524	Levy et al.	Jul 2000	A
6112201	Wical	Aug 2000	A
6115705	Larson	Sep 2000	A
6212529	Boothby et al.	Apr 2001	B1
6240406	Tannen	May 2001	B1
6240514	Inoue et al.	May 2001	B1
6249866	Brundett et al.	Jun 2001	B1
6324540	Khanna et al.	Nov 2001	B1
6324654	Wahl et al.	Nov 2001	B1
6339770	Leung et al.	Jan 2002	B1
6351742	Agarwal et al.	Feb 2002	B1
6363389	Lyle et al.	Mar 2002	B1
6385201	Iwata	May 2002	B1
6385604	Bakalash et al.	May 2002	B1
6427230	Goiffon et al.	Jul 2002	B1
6438538	Goldring	Aug 2002	B1
6496843	Getchius et al.	Dec 2002	B1
6505187	Shatdal	Jan 2003	B1
6611850	Shen	Aug 2003	B1
6687846	Adrangi et al.	Feb 2004	B1
6691101	MacNicol et al.	Feb 2004	B2
6748393	Kapoor et al.	Jun 2004	B1
6801905	Andrei	Oct 2004	B2
6823474	Kampe et al.	Nov 2004	B2
6920460	Srinivasan et al.	Jul 2005	B1
6959369	Ashton et al.	Oct 2005	B1
6973452	Metzger et al.	Dec 2005	B2
7020649	Cochrane et al.	Mar 2006	B2
7032089	Ranade et al.	Apr 2006	B1
7082473	Breitbart et al.	Jul 2006	B2
7177866	Holenstein et al.	Feb 2007	B2
7181460	Coss et al.	Feb 2007	B2
7191299	Kekre et al.	Mar 2007	B1
7246345	Sharma et al.	Jul 2007	B1
7447807	Merry et al.	Nov 2008	B1
7467103	Murray et al.	Dec 2008	B1
7469253	Celis et al.	Dec 2008	B2
7472117	Dettinger et al.	Dec 2008	B2
7486661	Van den Boeck et al.	Feb 2009	B2
7529834	Birrell et al.	May 2009	B1
7548928	Dean et al.	Jun 2009	B1
7552356	Waterhouse et al.	Jun 2009	B1
7558481	Jenkins et al.	Jul 2009	B2
7567991	Armangau et al.	Jul 2009	B2
7617369	Bezbaruah et al.	Nov 2009	B1
7634459	Eshet et al.	Dec 2009	B1
7647329	Fischman et al.	Jan 2010	B1
7657570	Wang et al.	Feb 2010	B2
7657578	Karr et al.	Feb 2010	B1
7668801	Koudas et al.	Feb 2010	B1
7761465	Nonaka et al.	Jul 2010	B1
7827144	Saito	Nov 2010	B1
7957284	Lu et al.	Jun 2011	B2
7962458	Holenstein et al.	Jun 2011	B2
8005804	Greer	Aug 2011	B2
8005868	Saborit et al.	Aug 2011	B2
8037059	Bestgen et al.	Oct 2011	B2
8078825	Oreland et al.	Dec 2011	B2
8082265	Carlson et al.	Dec 2011	B2
8086597	Balmin et al.	Dec 2011	B2
8099572	Arora et al.	Jan 2012	B1
8103906	Alibakhsh et al.	Jan 2012	B1
8108443	Thusoo	Jan 2012	B2
8126848	Wagner	Feb 2012	B2
8170984	Bakalash et al.	May 2012	B2
8185505	Blitzer et al.	May 2012	B1
8260840	Sirota et al.	Sep 2012	B1
8296419	Khanna et al.	Oct 2012	B1
8305999	Palanki et al.	Nov 2012	B2
8321558	Sirota et al.	Nov 2012	B1
8352450	Mraz et al.	Jan 2013	B1
8352463	Nayak	Jan 2013	B2
8363961	Avidan et al.	Jan 2013	B1
8370857	Kamii et al.	Feb 2013	B2
8386463	Bestgen et al.	Feb 2013	B2
8392482	McAlister et al.	Mar 2013	B1
8539197	Marshall et al.	Sep 2013	B1
8572031	Merriman et al.	Oct 2013	B2
8589382	Betawadkar-Norwood	Nov 2013	B2
8589574	Cormie et al.	Nov 2013	B1
8615507	Varadarajulu et al.	Dec 2013	B2
8712044	MacMillan et al.	Apr 2014	B2
8712993	Ordonez	Apr 2014	B1
8751533	Dhavale et al.	Jun 2014	B1
8775070	Bhatia	Jul 2014	B1
8843441	Rath et al.	Sep 2014	B1
8869256	Sample	Oct 2014	B2
8996463	Merriman et al.	Mar 2015	B2
9015431	Resch et al.	Apr 2015	B2
9069827	Rath et al.	Jun 2015	B1
9116862	Rath et al.	Aug 2015	B1
9141814	Murray	Sep 2015	B1
9183254	Cole et al.	Nov 2015	B1
9262462	Merriman et al.	Feb 2016	B2
9268639	Leggette et al.	Feb 2016	B2
9274902	Morley et al.	Mar 2016	B1
9313604	Holcombe	Apr 2016	B1
9317576	Merriman et al.	Apr 2016	B2
9350633	Cudak et al.	May 2016	B2
9350681	Kitagawa et al.	May 2016	B1
9442995	Pareek et al.	Sep 2016	B2
9460008	Leshinsky et al.	Oct 2016	B1
9495427	Abadi et al.	Nov 2016	B2
9569481	Chandra et al.	Feb 2017	B1
9628404	Goffinet et al.	Apr 2017	B1
9660666	Ciarlini et al.	May 2017	B1
9715433	Mu et al.	Jul 2017	B2
9740762	Horowitz et al.	Aug 2017	B2
9792322	Merriman et al.	Oct 2017	B2
9800685	Neerincx et al.	Oct 2017	B2
9805108	Merriman et al.	Oct 2017	B2
9881034	Horowitz et al.	Jan 2018	B2
9959308	Carman et al.	May 2018	B1
10031931	Horowitz et al.	Jul 2018	B2
10031956	Merriman et al.	Jul 2018	B2
10262050	Bostic et al.	Apr 2019	B2
10303570	Nakajima	May 2019	B2
10346430	Horowitz et al.	Jul 2019	B2
10346434	Morkel et al.	Jul 2019	B1
10366100	Horowitz et al.	Jul 2019	B2
10372926	Leshinsky et al.	Aug 2019	B1
10394822	Stearn	Aug 2019	B2
10423626	Stearn et al.	Sep 2019	B2
10430433	Stearn et al.	Oct 2019	B2
10467245	Sirer et al.	Nov 2019	B2
10474645	Freedman et al.	Nov 2019	B2
10489357	Horowitz et al.	Nov 2019	B2
10496669	Merriman et al.	Dec 2019	B2
10614098	Horowitz et al.	Apr 2020	B2
10621050	Horowitz et al.	Apr 2020	B2
10621200	Merriman et al.	Apr 2020	B2
10671496	Horowitz et al.	Jun 2020	B2
10673623	Horowitz et al.	Jun 2020	B2
10698775	Horowitz et al.	Jun 2020	B2
10713275	Merriman et al.	Jul 2020	B2
10713280	Horowitz et al.	Jul 2020	B2
10740353	Horowitz et al.	Aug 2020	B2
10740355	Horowitz et al.	Aug 2020	B2
10776220	Horowitz et al.	Sep 2020	B2
10846305	Merriman et al.	Nov 2020	B2
10846411	Horowitz et al.	Nov 2020	B2
10866868	Horowitz	Dec 2020	B2
10872095	Horowitz et al.	Dec 2020	B2
10977277	Merriman et al.	Apr 2021	B2
10990590	Merriman et al.	Apr 2021	B2
10997211	Merriman et al.	May 2021	B2
11012806	Tan et al.	May 2021	B2
11222043	Horowitz et al.	Jan 2022	B2
20010021929	Lin et al.	Sep 2001	A1
20020029207	Bakalash et al.	Mar 2002	A1
20020065675	Grainger et al.	May 2002	A1
20020065676	Grainger et al.	May 2002	A1
20020065677	Grainger et al.	May 2002	A1
20020143901	Lupo et al.	Oct 2002	A1
20020147842	Breitbart et al.	Oct 2002	A1
20020184239	Mosher, Jr. et al.	Dec 2002	A1
20020198872	MacNicol et al.	Dec 2002	A1
20030046307	Rivette et al.	Mar 2003	A1
20030212668	Hinshaw et al.	Apr 2003	A1
20030084073	Hotti et al.	May 2003	A1
20030088659	Susarla et al.	May 2003	A1
20030182427	Halpern	Sep 2003	A1
20030187864	McGoveran	Oct 2003	A1
20040078569	Hotti	Apr 2004	A1
20040133591	Holenstein et al.	Jul 2004	A1
20040168084	Owen et al.	Aug 2004	A1
20040186817	Thames et al.	Sep 2004	A1
20040186826	Choi et al.	Sep 2004	A1
20040205048	Pizzo et al.	Oct 2004	A1
20040220937	Bickford et al.	Nov 2004	A1
20040236743	Blaicher et al.	Nov 2004	A1
20040254919	Giuseppini	Dec 2004	A1
20050027796	San Andres et al.	Feb 2005	A1
20050033756	Kottomtharayil et al.	Feb 2005	A1
20050038833	Colrain et al.	Feb 2005	A1
20050192921	Chaudhuri et al.	Sep 2005	A1
20050216923	Krebs	Sep 2005	A1
20050234841	Miao et al.	Oct 2005	A1
20050283457	Sonkin et al.	Dec 2005	A1
20060004746	Angus et al.	Jan 2006	A1
20060020586	Prompt et al.	Jan 2006	A1
20060085541	Cuomo et al.	Apr 2006	A1
20060090095	Massa et al.	Apr 2006	A1
20060168154	Zhang et al.	Jul 2006	A1
20060209782	Miller et al.	Sep 2006	A1
20060218123	Chowdhuri et al.	Sep 2006	A1
20060224603	Correll, Jr.	Oct 2006	A1
20060235905	Kapur	Oct 2006	A1
20060259160	Hood et al.	Nov 2006	A1
20060287998	Folting et al.	Dec 2006	A1
20060288232	Ho et al.	Dec 2006	A1
20060294129	Stanfill et al.	Dec 2006	A1
20070050436	Chen et al.	Mar 2007	A1
20070061487	Moore et al.	Mar 2007	A1
20070094237	Mitchell et al.	Apr 2007	A1
20070124317	Dettinger et al.	May 2007	A1
20070203944	Batra et al.	Aug 2007	A1
20070226640	Holbrook et al.	Sep 2007	A1
20070233746	Garbow et al.	Oct 2007	A1
20070240129	Kretzschmar et al.	Oct 2007	A1
20080002741	Maheshwari et al.	Jan 2008	A1
20080005475	Lubbers et al.	Jan 2008	A1
20080016021	Gulbeden et al.	Jan 2008	A1
20080071755	friedman et al.	Mar 2008	A1
20080098041	Chidambaran et al.	Apr 2008	A1
20080140971	Dankel et al.	Jun 2008	A1
20080162590	Kundu et al.	Jul 2008	A1
20080288646	Hasha et al.	Nov 2008	A1
20090030986	Bates	Jan 2009	A1
20090055350	Branish et al.	Feb 2009	A1
20090077010	Muras et al.	Mar 2009	A1
20090094318	Gladwin et al.	Apr 2009	A1
20090222474	Alpern et al.	Sep 2009	A1
20090240744	Thomson et al.	Sep 2009	A1
20090254572	Redlich et al.	Oct 2009	A1
20090271412	Lacapra et al.	Oct 2009	A1
20100011026	Saha et al.	Jan 2010	A1
20100030793	Cooper et al.	Feb 2010	A1
20100030800	Brodfuehrer et al.	Feb 2010	A1
20100049717	Ryan et al.	Feb 2010	A1
20100057764	Williamson	Mar 2010	A1
20100058010	Augenstein et al.	Mar 2010	A1
20100094851	Bent et al.	Apr 2010	A1
20100106934	Calder et al.	Apr 2010	A1
20100161492	Harvey et al.	Jun 2010	A1
20100198791	Wu et al.	Aug 2010	A1
20100205028	Johnson et al.	Aug 2010	A1
20100223078	Willis et al.	Sep 2010	A1
20100235606	Oreland et al.	Sep 2010	A1
20100250497	Redlich et al.	Sep 2010	A1
20100250930	Csaszar et al.	Sep 2010	A1
20100257142	Murphy et al.	Oct 2010	A1
20100333111	Kothamasu	Dec 2010	A1
20100333116	Prahlad et al.	Dec 2010	A1
20110022642	deMilo et al.	Jan 2011	A1
20110032571	Kitada	Feb 2011	A1
20110125704	Mordinova et al.	May 2011	A1
20110125766	Carozza	May 2011	A1
20110125894	Anderson et al.	May 2011	A1
20110138148	Friedman et al.	Jun 2011	A1
20110202792	Atzmony	Aug 2011	A1
20110225122	Denuit et al.	Sep 2011	A1
20110225123	D'Souza et al.	Sep 2011	A1
20110231447	Starkey	Sep 2011	A1
20110246717	Kobayashi et al.	Oct 2011	A1
20110258225	Taylor et al.	Oct 2011	A1
20110258317	Sinha et al.	Oct 2011	A1
20110295686	Martin-Cocher	Dec 2011	A1
20110307338	Carlson	Dec 2011	A1
20120054155	Darcy	Mar 2012	A1
20120054249	Batra et al.	Mar 2012	A1
20120054755	Evans	Mar 2012	A1
20120076058	Padmanabh et al.	Mar 2012	A1
20120078848	Jennas et al.	Mar 2012	A1
20120078896	Nixon et al.	Mar 2012	A1
20120079224	Clayton et al.	Mar 2012	A1
20120084414	Brock et al.	Apr 2012	A1
20120084789	Iorio	Apr 2012	A1
20120109892	Novik et al.	May 2012	A1
20120109935	Meijer	May 2012	A1
20120130973	Tamm et al.	May 2012	A1
20120130988	Nica et al.	May 2012	A1
20120131278	Chang et al.	May 2012	A1
20120136835	Kosuru et al.	May 2012	A1
20120138671	Gaede et al.	Jun 2012	A1
20120158655	Dove et al.	Jun 2012	A1
20120158949	Lee	Jun 2012	A1
20120159097	Jennas, II et al.	Jun 2012	A1
20120166390	Merriman et al.	Jun 2012	A1
20120166517	Lee et al.	Jun 2012	A1
20120179833	Kenrick et al.	Jul 2012	A1
20120198200	Li et al.	Aug 2012	A1
20120209625	Armstrong et al.	Aug 2012	A1
20120215740	Vaillant et al.	Aug 2012	A1
20120215763	Hughes et al.	Aug 2012	A1
20120221540	Rose et al.	Aug 2012	A1
20120254175	Horowitz et al.	Oct 2012	A1
20120274664	Fagnou	Nov 2012	A1
20120320914	Thyni et al.	Dec 2012	A1
20130019296	Brandenburg	Jan 2013	A1
20130151477	Tsaur et al.	Jun 2013	A1
20130151558	Chércoles Sánchez et al.	Jun 2013	A1
20130179450	Chitiveli	Jul 2013	A1
20130290249	Merriman et al.	Oct 2013	A1
20130290471	Venkatesh	Oct 2013	A1
20130332484	Gajic	Dec 2013	A1
20130339379	Ferrari et al.	Dec 2013	A1
20130346366	Ananthanarayanan et al.	Dec 2013	A1
20140013334	Bisdikian et al.	Jan 2014	A1
20140032525	Merriman et al.	Jan 2014	A1
20140032579	Merriman et al.	Jan 2014	A1
20140032628	Cudak et al.	Jan 2014	A1
20140074790	Berman et al.	Mar 2014	A1
20140101100	Hu et al.	Apr 2014	A1
20140164831	Merriman et al.	Jun 2014	A1
20140180723	Cote et al.	Jun 2014	A1
20140201267	Yang	Jul 2014	A1
20140258343	Nikula	Sep 2014	A1
20140279929	Gupta et al.	Sep 2014	A1
20140280380	Jagtap et al.	Sep 2014	A1
20140289197	Webber	Sep 2014	A1
20150012797	Leggette et al.	Jan 2015	A1
20150016300	Devireddy et al.	Jan 2015	A1
20150074041	Bhattacharjee et al.	Mar 2015	A1
20150081766	Curtis et al.	Mar 2015	A1
20150242531	Rodniansky	Aug 2015	A1
20150278295	Merriman et al.	Oct 2015	A1
20150301901	Rath et al.	Oct 2015	A1
20150331755	Morgan	Nov 2015	A1
20150341212	Hsiao et al.	Nov 2015	A1
20150378786	Suparna et al.	Dec 2015	A1
20150378825	Resch	Dec 2015	A1
20160005423	Neppalli et al.	Jan 2016	A1
20160042008	Tripathy et al.	Feb 2016	A1
20160048345	Vijayrao et al.	Feb 2016	A1
20160110284	Athalye et al.	Apr 2016	A1
20160110414	Park et al.	Apr 2016	A1
20160162354	Singhai et al.	Jun 2016	A1
20160162374	Mutha et al.	Jun 2016	A1
20160188377	Thimmappa et al.	Jun 2016	A1
20160198044	Gruchala et al.	Jul 2016	A1
20160203202	Merriman et al.	Jul 2016	A1
20160246861	Merriman et al.	Aug 2016	A1
20160253109	Litke et al.	Sep 2016	A1
20160306709	Shaull	Oct 2016	A1
20160323378	Coskun et al.	Nov 2016	A1
20160364440	Lee et al.	Dec 2016	A1
20170032007	Merriman	Feb 2017	A1
20170032010	Merriman	Feb 2017	A1
20170091327	Bostic et al.	Mar 2017	A1
20170109398	Stearn	Apr 2017	A1
20170109399	Stearn et al.	Apr 2017	A1
20170109421	Stearn et al.	Apr 2017	A1
20170169059	Horowitz et al.	Jun 2017	A1
20170177658	Lee et al.	Jun 2017	A1
20170262516	Horowitz et al.	Sep 2017	A1
20170262517	Horowitz et al.	Sep 2017	A1
20170262519	Horowitz et al.	Sep 2017	A1
20170262638	Horowitz et al.	Sep 2017	A1
20170264432	Horowitz et al.	Sep 2017	A1
20170270176	Horowitz et al.	Sep 2017	A1
20170286510	Horowitz et al.	Oct 2017	A1
20170286516	Horowitz et al.	Oct 2017	A1
20170286517	Horowitz et al.	Oct 2017	A1
20170286518	Horowitz et al.	Oct 2017	A1
20170322954	Horowitz et al.	Nov 2017	A1
20170322996	Horowitz et al.	Nov 2017	A1
20170344290	Horowitz et al.	Nov 2017	A1
20170344441	Horowitz et al.	Nov 2017	A1
20170344618	Horowitz et al.	Nov 2017	A1
20170371750	Horowitz et al.	Dec 2017	A1
20170371968	Horowitz et al.	Dec 2017	A1
20180004801	Burchall et al.	Jan 2018	A1
20180004804	Merriman et al.	Jan 2018	A1
20180095852	Keremane et al.	Apr 2018	A1
20180096045	Merriman et al.	Apr 2018	A1
20180096066	Venkataramanappa et al.	Apr 2018	A1
20180150230	Schreter	May 2018	A1
20180165338	Kumar et al.	Jun 2018	A1
20180173745	Balasubramanian et al.	Jun 2018	A1
20180176300	Chen	Jun 2018	A1
20180300203	Kathpal et al.	Oct 2018	A1
20180300209	Rahut	Oct 2018	A1
20180300381	Horowitz et al.	Oct 2018	A1
20180300385	Merriman et al.	Oct 2018	A1
20180314750	Merriman et al.	Nov 2018	A1
20180343131	George et al.	Nov 2018	A1
20180365114	Horowitz	Dec 2018	A1
20190102410	Horowitz et al.	Apr 2019	A1
20190303382	Bostic et al.	Oct 2019	A1
20200097486	Horowitz et al.	Mar 2020	A1
20200099392	Hecker et al.	Mar 2020	A1
20200160297	Munk	May 2020	A1
20200285549	Horowitz et al.	Sep 2020	A1
20200295925	Horowitz et al.	Sep 2020	A1
20200327021	Horowitz et al.	Oct 2020	A1
20200341867	Horowitz et al.	Oct 2020	A1
20200409804	Horowitz et al.	Dec 2020	A1

Non-Patent Literature Citations (13)

Entry
[No Author Listed], Automated Administration Tasks (SQL Server Agent). https://docs.microsoft.com/en-us/sql/ssms/agent/automated-adminsitration-tasks-sql-server-agent. 2 pages, [downloaded Mar. 4, 2017].
Chang et al., Bigtable: a distributed storage system for structured data. OSDI'06: Seventh Symposium on Operating System Design and Implementation. Nov. 2006.
Cooper et al., PNUTS: Yahoo!'s hosted data serving platform. VLDB Endowment. Aug. 2008.
Decandia et al., Dynamo: Amazon's highly available key-value store. SOSP 2007. Oct. 2004.
Nelson et al., Automate MongoDB with MMS. PowerPoint Presentation. Published Jul. 24, 2014. 27 slides. http://www.slideshare.net/mongodb/mms-automation-mongo-db-world.
Ongaro et al., In Search of an Understandable Consensus Algorithm. Proceedings of USENIX ATC '14: 2014 USENIX Annual Technical Conference. Philadelphia, PA. Jun. 19-20, 2014; pp. 305-320.
Poder, Oracle living books. 2009. <http://tech.e2sn.com/oracle/sql/oracle-execution-plan-operation-reference >.
Stirman, Run MongoDB with Confidence using MMS. PowerPoint Presentation. Published Oct. 6, 2014. 34 slides. http://www.slideshare.net/mongodb/mongo-db-boston-run-mongodb-with-mms-20141001.
Van Renesse et al., Chain replication for supporting high throughput and availability. OSDI. 2004: 91-104.
Walsh et al., Xproc: An XML Pipeline Language. May 11, 2011. <https://www.w3.org/TR/xproc/>.
Wikipedia, Dataflow programming. Oct. 2011. <http://en.wikipedia.org/wiki/Dataflow_programming>.
Wikipedia, Pipeline (Unix). Sep. 2011. <http://en.wikipedia.org/wiki/Pipeline (Unix)>.
Wilkins et al., Migrate DB2 applications to a partitioned database. developerWorks, IBM. Apr. 24, 2008, 33 pages.

Related Publications (1)

	Number	Date	Country
	20200285549 A1	Sep 2020	US

Provisional Applications (1)

	Number	Date	Country
	62343494	May 2016	US

Continuations (1)

	Number	Date	Country
Parent	15605276	May 2017	US
Child	16883653		US

Method and apparatus for reading and writing committed data

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

US

International Classifications

Disclaimer

Term Extension

Abstract