Patent Grant
11403269
The present disclosure generally relates to data storage, and in a more particular example, to version control for data transfer between heterogeneous data stores.
Often, distributed storage systems are used to store large amounts (e.g., terabytes, petabytes, exabytes, etc.) of data, such as objects or files in a distributed and fault tolerant manner with a predetermined level of redundancy.
Some existing object storage systems store data objects referenced by an object identifier versus file systems. This can generally allow object storage systems to surpass the maximum limits for storage capacity of file systems in a flexible way such that, for example, storage capacity can be added or removed as a function of the applications, systems, and/or enterprise needs, while reducing degradation in performance as the system grows. As a result, object storage systems are often selected for large-scale storage systems.
These large-scale storage systems may include a variety of storage systems in varying locations and of varying ages and configurations. As a result, not all object data stores within a large-scale storage system may be configured to manage data objects and their related metadata in the same way. Such disparate configurations within the same storage system may be referred to as heterogeneous data stores. Within or between object storage systems, individual buckets may also be configured differently and described as heterogeneous.
Large-scale storage systems generally distribute the stored data over multiple storage elements, such as for example solid state drives, or multiple components such as storage nodes comprising a plurality of such storage elements. However, as the number of storage elements in such a distributed storage system increase, the probability of failure of one or more of these storage elements increases, as does the probability of failure or interruption of controller, network, and interface components. To cope with these issue, distributed storage systems generally use some level of redundancy at the data, storage node, controller, interface, client, and network levels. In some cases, redundancy may include moving data objects between object data stores, such as using one or more replication agents, for mirroring, load balancing, and/or disaster recovery.
Version control among heterogeneous data stores may present a challenge. For example, when a versioned object is transferred to an object data store or bucket that does not have versioning enabled, version information may be lost. Also, because many replication agents use parallel and/or asynchronous replication paths, it is possible for versions of the same versioned object to be transferred at different times. In dynamic storage environments, it may be possible for the configuration of a destination data store to change during the transfer of object versions. This could result in incomplete versioned data objects being stored in the destination object store, if versioning is disabled or suspended during the transfer.
In addition, data objects can be large and use substantial computing and network resources to transfer between storage systems. Initiating a large data object transfer, which may include multi-part data object transfers, that may ultimately be invalidated because the destination object store did not have versioning enabled, may result in wasted resources that add to system overhead and latency.
As large-scale storage systems scale and create more heterogeneous object data stores within storage systems, reliable and efficient implementations for managing version control may be needed. A need exists for at least versioning validation for data transfer between heterogeneous data stores.
Various aspects for versioning validation, particularly, versioning validation for data transfer between heterogeneous data stores are described.
One general aspect includes a system that includes a first object data store configured to include a first plurality of versioned data objects. A versioned data object of the first plurality of versioned data objects includes a plurality of sequential versions corresponding to the versioned data object. A second object data store is configured to include a versioning-enabled status. A replication manager includes at least one processor, where the replication manager is configured to: transfer the versioned data object from the first object data store to the second object data store; send a version write request message to the second object data store for the versioned data object; receive a response message from the second object data store; determine from the response message whether the versioning-enabled status is invalid; and send, responsive to determining the versioning-enabled status is invalid, a delete request for the versioned data object.
Implementations may include one or more of the following features. The replication manager may be further configured to add, responsive to determining the versioning-enabled status is invalid, a version write request for the versioned data object to a retry queue. The replication manager may be further configured to generate, responsive to determining the versioning-enabled status is invalid, a notification event configured to request enabling versioning on the second object data store. The second object data store may be configured to store, responsive to the version write request message, a residual version of the versioned data object in the second object data store when the versioning-enabled status is invalid and delete, responsive to the delete request, the residual version of the versioned data object. The version write request message may be a multi-part write request including multiple write transactions between the first object data store and the second object data store to transfer the versioned data object. The replication manager may be further configured to send a test message to the second object data store prior to initiating the version write request message for the multi-part write request. Initiating the version write request message may be responsive to determining the versioning-enabled status is valid. The test message may be a test version write request message that includes a host data size of 0. The replication manager may be further configured to retry, responsive to determining the versioning-enabled status is invalid, the test message a plurality of times. The replication manager may further include a versioning-invalid list of destination object data buckets with an invalid versioning-enabled status and may be further configured to add, responsive to determining the versioning-enabled status is invalid, a versioning-invalid entry for a destination object data bucket in the second object data store. The replication manager may be further configured to: search the versioning-invalid list for the destination object data bucket for the version write request message; send, responsive to finding the versioning-invalid entry for the destination object data bucket, a test version write request message to the destination object data bucket; and, responsive to receiving a test response message indicating the versioning-enabled status is valid, remove the versioning-invalid entry for the destination object data bucket; and initiate the version write request message.
One general aspect includes A computer-implemented method including: sending a version write request message for a versioned data object from a first object data store to a second object data store; receiving a response message from the second object data store; determining from the response message whether a versioning-enabled status is invalid; and sending, responsive to determining the versioning-enabled status is invalid, a delete request for the versioned data object. The versioned data object includes a plurality of sequential versions corresponding to the versioned data object and the second object data store includes the versioning-enabled status.
Implementations may include one or more of the following features. The computer-implemented method may further include adding, responsive to determining the versioning-enabled status is invalid, a version write request for the versioned data object to a retry queue. The computer-implemented method may further include generating, responsive to determining the versioning-enabled status is invalid, a notification event configured to request enabling versioning on the second object data store. The computer-implemented method may further include storing, responsive to the version write request message, a residual version of the versioned data object in the second object data store when the versioning-enabled status is invalid and deleting, responsive to the delete request, the residual version of the versioned data object. The computer-implemented method may further include sending a test message to the second object data store and receiving a test response message from the second object data store, where sending the version write request message is responsive to determining the versioning-enabled status is valid from the test response message. The computer-implemented method may further include retrying, responsive to determining the versioning-enabled status is invalid, the test message a plurality of times. The computer-implemented method may further include adding, responsive to determining the version enabled status is invalid, a versioning-invalid entry for a destination object data bucket in the second object data store to a versioning-invalid list. The computer-implemented method may further include: searching the versioning-invalid list for the destination object data bucket for the version write request message; sending, responsive to finding the versioning-invalid entry for the destination object data bucket, a test message to the destination object data bucket; and responsive to receiving a test response message indicating the versioning-enabled status is valid, removing the versioning-invalid entry for the destination object data bucket and initiating the version write request message.
One general aspect includes a system that includes a first object data store, including a first plurality of versioned data objects. A versioned data object includes a plurality of sequential versions corresponding to the versioned data object. A second object data store includes a versioning-enabled status. Means are provided for sending a version write request message for the versioned data object from the first object data store to the second object data store. Means are provided for determining, responsive to a response message from the second object data store, whether the versioning-enabled status is invalid. Means are provided for sending, responsive to determining the versioning-enabled status is invalid, a delete request for the versioned data object.
Implementations may include one or more of the following features. The system may further include: means for sending a test message to the second object data store and means for receiving a test response message from the second object data store, where sending the version write request message is responsive to determining the versioning-enabled status is valid from the test response message.
The various embodiments advantageously apply the teachings of distributed storage networks and/or systems to improve the functionality of such computer systems. The various embodiments include operations to overcome or at least reduce the issues in the previous storage networks and/or systems discussed above and, accordingly, are more reliable and/or efficient than other computing networks. That is, the various embodiments disclosed herein include hardware and/or software with functionality to improve the version control for data transfer between heterogeneous data stores, such as by using versioning validation and automatic removal of residual objects when transferring versioned data objects between data stores. Accordingly, the embodiments disclosed herein provide various improvements to storage networks and/or storage systems.
It should be understood that language used in the present disclosure has been principally selected for readability and instructional purposes, and not to limit the scope of the subject matter disclosed herein.
As discussed above, out of order versioned object write ingestion can occur in heterogeneous object storage systems between versioning enabled heterogeneous buckets.
Object storage systems have a notion of data objects. Data objects may be contained in buckets and each bucket may be associated with a specific object storage system or object data store. Specific object storage systems may include a number of configuration parameters, including support for versioning. In some systems, individual buckets may include different configuration parameters, including enabling and disabling version control within an object storage system that is otherwise configured for versioning.
In many object storage systems, a data object is written using either a PUT function, for single part objects, or a multi-part write process, such as INIT, multiple PARTPUT, and COMPLETE functions. Data objects written to a versioning not enabled or versioning suspended system or bucket, may create a data object with a version value set to null and a creation timestamp. Subsequent read functions, such as GET, of the data object with a defined object name (ex. “object1”) may return the data object (“object1”). If an object is later written with the same object name (“object1”), the original object may be overwritten. Subsequent GET operations with the object name (“object1”) may return the overwritten object and the original object may be lost.
However, in a versioning enabled system or bucket, earlier versions of the object may not get overwritten. A new write of an existing data object (“object1”) may create a new version of that object. The object name does not change, but an additional version identifier may be associated with the new write. For example, when the object is initially written, it may receive a version identifier (ex. “1111”). Subsequent GET operations for the object name (“object1”) return the object (“object1”) with the most recent version identifier (“1111”). Each subsequent time an object with the same object name is written with versioning enabled, a new object is created with the same object name (“object1”) but a new creation timestamp and a new version identifier (e.g. “2222”, “3333”. etc.). The set of data objects with the same object name and different version identifiers in a given storage system or bucket may be referred to collectively as a versioned data object, comprised of a plurality of data object versions. Ordering of versions may be important for GET operations that do not specify a version and are intended to retrieve the latest version of the object.
Data object creation time and versioning depends on specific implementations in different object storage systems, resulting in heterogeneous versioning configurations. Versioning may be different among different object storage systems, implementing entirely different versioning schemes that may not be recognized by other object storage systems. For example, the versioning identifiers may be in a format or location that is unknown to the other system. In such a case, a GET function from the other system intended to include a version request may not comply with the necessary operation syntax and may return an error, an empty object, or a default version, such as the most recent according to the local creation timestamp or versioning scheme.
In particular, a versioned write to a non-versioning-enabled bucket or data store may result in only a single, non-versioned data object being transferred and stored in the destination data store. As a result, all other object versions and associated metadata may be lost without generating any sort of system error or notice to system administrators. Even if an initial check of versioning-enabled status is made, it may not be an atomic operation with the actual versioned write and the status may be modified between the initial check and completion of the versioned data object transfer.
While validation of the versioning-enabled status of the destination bucket as part of the versioned write operation may be advantageous, there may still be a risk of transferring unnecessary data (using valuable compute and network resources) before receiving notice that the versioning-enabled status is invalid. This may be particularly wasteful for large data objects using multipart data transfers, where versioning status may not be confirmed until the last operation of the multipart sequence is complete. Additional safeguards to limit large, unnecessary data transfers may be implemented within versioning-enabled data write requests.
As further shown in
These access nodes 20 and storage nodes 30 may be built as general-purpose computers. Alternatively, they may be physically adapted for arrangement in large data centers, where they are arranged in modular racks 40.1-40.n comprising standard dimensions. Exemplary access nodes 20 and storage nodes 30 may be dimensioned to take up a single unit of such racks 40, which is generally referred to as 1U. Such an exemplary storage node may use a low-power processor and may be equipped with ten or twelve high capacity serial advanced technology attachment (SATA) disk drives and is connectable to the network over redundant Ethernet network interfaces. An exemplary access node 20 may comprise high-performance servers and provide network access to clients 10 over multiple high bandwidth Ethernet network interfaces. Data can be transferred between clients 10 and such access nodes 20 by means of a variety of network protocols including hypertext transfer protocol (HTTP)/representational state transfer (REST) object interfaces, language-specific interfaces such as Microsoft .Net, Python or C, etc. Additionally, such access nodes may comprise additional high bandwidth Ethernet ports to interface with the storage nodes 30. In some embodiments, HTTP/REST protocols complying with the Amazon Simple Storage Service (S3) object storage service may enable data transfer through a REST application protocol interfaces (API). Such access nodes 20 may operate as a highly available cluster of controller nodes with one or more integrated and/or independent interface systems, and provide for example shared access to the storage nodes 30, metadata caching, protection of metadata, etc.
As shown in
A client device 60 can be any computing hardware and/or software (e.g., a thick client, a thin client, or hybrid thereof) capable of accessing storage system 80 utilizing network 70. Each client device 60, as part of its respective operation, relies on sending input/output (I/O) requests to storage system 80 to write data, read data, and/or modify data. Specifically, each client device 60 can transmit I/O requests to read, write, store, communicate, propagate, and/or transport instructions, data, computer programs, software, code, routines, etc., to storage system 80. Client device(s) 60 and storage system 80 may comprise at least a portion of a client-server model. In general, storage system 80 can be accessed by client device(s) 60 and/or communication with storage system 80 can be initiated by client device(s) 60 through a network socket (not shown) utilizing one or more inter-process networking techniques. In some embodiments, client devices 60 may access one or more applications to use or manage a distributed storage system, such as distributed storage system 1 in
Storage element interface 370 may comprise a storage interface such as for example a SATA interface or a small computer system interface (SCSI) for connecting bus 310 to one or more storage elements 300, such as one or more local disks, for example 3 terabyte (TB) SATA disk drives, and control the reading and writing of data to/from these storage elements 300. In one exemplary embodiment as shown in
As is clear from
As shown in
The storage elements 300 or a portion thereof may be redundant and operate independently of one another. This means that if one particular storage element 300 fails its function it can easily be taken on by another storage element 300 in the distributed storage system 1. However, as will be explained in more detail further below, the storage elements 300 are capable of providing redundancy without having to work in synchronism, as is for example the case in many well-known redundant array of independent disks (RAID) configurations, which sometimes even require disc spindle rotation to be synchronised. Furthermore, the independent and redundant operation of the storage elements 300 may allow a suitable mix of types of storage elements 300 to be used in a particular distributed storage system 1. It is possible to use for example storage elements 300 with differing storage capacity, storage elements 300 of differing manufacturers, using different hardware technology such as for example conventional hard disks and solid state storage elements, using different storage interfaces such as for example different revisions of SATA, parallel advanced technology attachment (PATA), and so on. This may result in advantages relating to scalability and flexibility of the distributed storage system 1 as it allows for adding or removing storage elements 300 without imposing specific requirements to their design in correlation to other storage elements 300 already in use in the distributed object storage system.
Bus 210 may include one or more conductors that permit communication among the components of access node 20. Processor 220 may include any type of conventional processor or microprocessor that interprets and executes instructions. Local memory 230 may include a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor 220 and/or a read only memory (ROM) or another type of static storage device that stores static information and instructions for use by processor 320 and/or any suitable storage element such as a hard disc or a solid state storage element. An optional input unit 240 may include one or more conventional mechanisms that permit an operator to input information to the access node 20 such as a keyboard, a mouse, a pen, voice recognition and/or biometric mechanisms, etc. Optional output unit 250 may include one or more conventional mechanisms that output information to the operator, such as a display, a printer, a speaker, etc. Communication interface 260 may include any transceiver-like mechanism that enables access node 20 to communicate with other devices and/or systems, for example mechanisms for communicating with other storage nodes 30 or access nodes 20 such as for example two 10 Gb Ethernet interfaces.
According to an alternative embodiment, the access node 20 could have an identical design as a storage node 30, or according to still a further alternative embodiment one of the storage nodes 30 of the distributed object storage system could perform both the function of an access node 20 and a storage node 30. According to still further embodiments, the components of the access node 20 as described in more detail below could be distributed amongst a plurality of access nodes 20 and/or storage nodes 30 in any suitable way. According to still a further embodiment, the clients 10 may run an access node 20. According to still further embodiments, access node 20 may be embodied in separate controller nodes and interface nodes with or without redundancy among the controller nodes and/or interface nodes.
Object storage system 500 may include a bus 510 interconnecting at least one communication unit 512, at least one processor 514, and at least one memory 516. Bus 510 may include one or more conductors that permit communication among the components of access system 500. Communication unit 512 may include any transceiver-like mechanism that enables access system 500 to communicate with other devices and/or systems. For example, communication unit 512 may include wired or wireless mechanisms for communicating with file system clients, other access systems, and/or one or more object storage systems or components, such as storage nodes or controller nodes. Processor 514 may include any type of processor or microprocessor that interprets and executes instructions. Memory 516 may include a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor 514 and/or a read only memory (ROM) or another type of static storage device that stores static information and instructions for use by processor 514 and/or any suitable storage element such as a hard disc or a solid state storage element.
Object storage system 500 may include or have access to one or more databases and/or specialized data stores, such metadata store 550 and object store 560. Databases may include one or more data structures for storing, retrieving, indexing, searching, filtering, etc. of structured and/or unstructured data elements. In some embodiments, metadata store 550 may be structured as reference data entries and/or data fields indexed by metadata key value entries related to data objects stores in object store 560. Object store 560 may include data objects comprised of object data (such as host data), some amount of metadata (stored as metadata tags), and a GUID. Metadata store 550, object store 560, and/or other databases or data structures may be maintained and managed in separate computing systems, such as storage nodes, with separate communication, processor, memory, and other computing resources and accessed by access system 500 through data access protocols. Metadata store 550 and object store 560 may be shared across multiple object storage systems 500.
Object storage system 500 may include a plurality of modules or subsystems that are stored and/or instantiated in memory 516 for execution by processor 514. For example, memory 516 may include a storage interface 520 configured to receive, process, and respond to object data requests from client systems or other nodes in distributed storage system 1. Memory 516 may include a version control manager 530 for creating and managing object versions in object store 560. Memory 516 may include a replication manager 540 for processing data replication between storage nodes or storage systems, such as transferring or replicating data objects to other object stores that may be heterogeneous. In some embodiments, version control manager 530 and/or replication manager 540 may be integrated into storage interface 520 and/or managed as separate libraries or background processes (e.g. daemon) through an API or other interface.
Storage interface 520 may include an interface protocol or set of functions and parameters for storing, reading, and otherwise managing data requests to an associated distributed storage system and the corresponding metadata data source for mapping file data to persistent storage data elements. For example, storage interface 520 may include functions for reading, writing, modifying, or otherwise manipulating data objects and their respective client or host data and metadata in accordance with the protocols of an object storage system.
In some embodiments, storage interface 520 may include a plurality of hardware and/or software modules configured to use processor 514 and memory 516 to handle or manage defined operations of storage interface 520. For example, storage interface 520 may include a client request handler 522, a metadata manager 524, a storage manager 526, and a client response handler 528. For any given client request, storage interface 520 may receive a client request through client request handler 522 and determine one or more operations based on the content of the request. These operations may include metadata operations handled by metadata manager 524 and/or object data operations handled by storage manager 526. The results of these operations may be processed, formatted, and returned by client response handler 528.
Storage interface 520 may include one or more object storage configurations that define a storage protocol used for validating, managing, and processing object data requests. For example, object storage configurations may generally define the availability of version control for any given object or bucket, as well as specifying how the object storage system creates and manages versioning information. Object storage configurations may also define what metadata is collected and stored, as well as whether it is stored in a metadata database, such as metadata store 550, in object metadata tags stored in the data objects, and/or a combination thereof.
Client request handler 522 may include an interface and/or communication event-based condition for receiving object data requests from one or more clients. For example, client systems may send an object data request over a network connection and addressed to object storage system 500 or a port or component thereof. Client request handler 522 may receive these requests and parse them according to the appropriate communication and object storage protocols. For example, client request handler 522 may identify a transaction identifier, a client identifier, an object identifier (object name or GUID), a data operation, and additional parameters for the data operation, if any, from the received message or messages that make up the object data request.
Metadata manager 524 may include interfaces, functions, and/or parameters for creating, modifying, deleting, accessing, and/or otherwise managing object metadata, such as object metadata 554 stored in metadata store 550. For example, when a new object is written to object store 560, at least one new metadata entry may be created in metadata store 550 to represent parameters describing or related to the newly created object. Metadata manager 524 may generate and maintain an object data index 552 that enables metadata manager 524 to locate object metadata within metadata store 550. For example, metadata store 550 may be organized as a key-value store and object data index 552 may include key values for data objects and/or operations related to those objects that are indexed with a key value that include the object identifier or GUID for each object. In some embodiments, metadata manager 524 may also manage object metadata stored in object store 560 with object data 564. Metadata manager 524 may work in conjunction with storage manager 526 to create, modify, delete, access or otherwise manage object metadata stored as object tags 566 within object store 560.
Storage manager 526 may include interfaces, functions, and/or parameters for reading, writing, and deleting object data elements in object store 560. For example, object PUT commands may be configured to write object identifier 562, object data 564, and/or object tags 566 to object store 560. Object GET commands may be configured to read data from object store 560. Object DELETE commands may be configured to delete data from object store 560, or at least mark a data object for deletion until a future garbage collection or similar operation actually deletes the data or reallocates the physical storage location to another purpose.
Other object storage commands may be handled by storage manager 526, including special functions defined for version control. Object storage commands processed by storage manager 526 may include parameters for accessing special functions and/or metadata resources stored with host or client data in the data objects. Storage manager 526 may work in conjunction with metadata manager 524 and version control manager 530 for managing object metadata, including versioning information. Storage manager 526 may work in conjunction with replication manager 540 to replicate or migrate data from object store 560 to another data store. For example, storage manager 526 may read the object store 560 for transfer by one or more replication engines managed by replication manager 540.
Client response handler 528 may include an interface and/or communication logic for sending response messages, such as result, status, or error messages, to one or more clients related to object data requests received. For example, client response handler 528 may wait for processing by metadata manager 524 and/or storage manager 526 to complete or generate an error, and then provide an appropriate result or error message to the client system(s) for each object data request received
In some embodiments, metadata store 550 may be distributed across multiple systems, such as a plurality of access systems. Metadata store 550 and/or portions thereof may be sharded data stores, wherein the data stores are partitioned into segments stored in different computing systems. Storage interface 520 may include the functions for locating and accessing relevant portions of the sharded data base.
Version control manager 530 may be invoked by storage interface 520 to manage version control for one or more data objects, buckets, or all of object store 560. For example, if versioning is enabled, each time the same object name is written, a new object version may be created. Similarly, which object version is returned from a read operation may be determined by a version identifier included as a parameter of the read operation and/or may default to the most recent version. Version control manager 530 may provide interfaces, functions, and parameters for managing the use of persistent version fields to version control across heterogeneous data stores. In some embodiments, version control manager 530 may be included within storage interface 520, such as within library functions used by client request handler 522, metadata manager 524, storage manager 526, and/or client response handler 528.
In some embodiments, version control manager 530 may include one or more hardware and/or software modules or data structures for executing specific operations. For example, version control manager 530 may include a versioning enabler 532, persistent version fields 534, version field writer/reader 536, and response handler 538.
Versioning enabler 532 may include one or more parameters, data structures, and/or related methods for manipulating those parameters or data structures. Versioning enabler 532 may be configured to manage versioning-enabled status 532.1 for object store 560 and/or specific buckets or data object contained therein. In some embodiments, each bucket may include a versioning-enabled field to determine whether versioning is enabled for that bucket. For example, versioning-enabled field may be a first value if versioning is enabled for a target data object, bucket, or object store and a second value if versioning is not enabled for the target data object, bucket, or object store. In some embodiments, versioning enabled fields may include more values to convey whether versioning is available (supported but not enabled), versioning is unsupported, or versioning is suspended (previously enabled but inoperative due to an interruption, conflict, or configuration choice at the bucket or object level).
In some contexts, versioning-enabled status 532.1 may be referred to as valid for a specific write transaction if versioned data objects may be written to object store 560 with their sequential object versions and related versioning metadata. There may be multiple values for versioning-enabled fields that translate into valid versioning-enabled status for a specific write transaction. Similarly, versioning-enabled status 532.1 may be referred to as invalid for a specific write transaction if versioned data objects may not be written to object store 560 with their sequential object versions and related versioning metadata. There may be multiple values for versioning-enabled fields that translate into invalid versioning-enabled status for a specific write transaction.
Versioning enabler 532 may also include one or more configurable parameters that define how versioning is handled by version control manager 530. For example, an object storage system may support both a default, local version control configuration and a persistent version control configuration for use across object storage systems. In some embodiments, versioning enabler 532 may include a parameter for determining whether version fields are stored in a metadata database, such as in object metadata 554 in metadata store 550, or with object data, such as in object tags 566 in object store 560.
Persistent version fields 534 may include one or more parameter, data structures, and/or related methods for configuring and generating persistent version fields for object versions. For example, a persistent version field configuration may include bucket name 534.1, bucket identifier 534.2, version identifier 534.3, and creation timestamp 534.3. In some embodiments, persistent version fields 534 may each include a series of values corresponding to the subfields and stored and presented according to a defined syntax. For example, an example persistent version field could be “bucket1.1234.1111.05/12/2018-01:00:00”. In some embodiments, persistent version fields 534 may be originated when the object is initially created by a first object storage system and are transferred with the object versions whenever they are migrated or replicated. As a result, the original version information, including creation timestamp, may be determined from the object metadata, regardless of where that version of the object ends up or how many heterogeneous object stores it may have been transferred through (or when or in what order).
Version field writer/reader 536 may include functions, methods, or operations for providing the persistent version fields 556, 568 to be written to metadata and read from metadata. For example, for any given object version being created and stored to object store 560, version field writer may use persistent version fields 534 as a template to populate persistent version fields 556, 568 with corresponding values for the write operation, such as bucket name, bucket identifier, version identifier, and creation timestamp values related to the write operation. For any given read operation for a versioned data object or object version thereof, version field reader may identify the persistent version field from the metadata for each object version of interest and use persistent version fields 534 as a template to extract and return the corresponding values for the fields. In some embodiments, version field writer/reader 536 may be called by storage interface 520 and/or replication manager 540.
A special instance of version field writer/reader 536 may include a delete marker writer. For example, when a delete event is identified for a versioned data object, such as processing a delete command for the versioned data object that is received by storage interface 520, the delete marker writer may generate a data object version with no host data (or a host data size of 0) and a persistent version field that identifies the object version as a delete marker. This delete marker object version may have the same object name (like any other object version) and be managed and transferred like any other object version in for the versioned data object. In some embodiments, the delete marker may include a delete marker identifier, such as a flag, tag, or similar value, added to the persistent version field to identify the object version as a delete marker.
Write response handler 538 may include functions, methods, or operations for providing the responses to version write requests received from other object storage systems. For example, when a version write request is received by storage interface 520 to write a versioned data object into object store 560 from another object store, the version write request may be configured to receive a response regarding versioning status for the write request. Write response handler 538 may read versioning-enabled status 532.1 and determine whether the version write request was completed with a valid versioning-enabled status 532.1. In some embodiments, the valid/invalid status value determined by write response handler 538 may be appended to a write complete status message generated by storage interface 520.
Replication manager 540 may include functions, methods, or operations for managing the transfer of data objects, including versioned data objects, to other object stores or object storage systems. For example, replication manager 540 may manage one or more replication engines that move data objects from object store 560 to a destination object store that may be remote from object store 560. Replication manager 540 may be configured to transfer data objects between heterogeneous data stores. In some embodiments, replication manager 540 transfers each versioned data object, including object identifiers 562, object data 564, and associated object tags 566, which may include persistent version fields 568. In some embodiments, replication manager 540 may also transfer metadata associated with the data object from a corresponding metadata store, such as metadata store 550, using object data index 552 to identify associated object metadata 554, which may include persistent version fields 568.
In some embodiments, replication manager 540 may be configured for asynchronous data transfers. For example, a scheduler may select object for transfer based on criteria other than the order in which they were ordered within object store 560, such as object size, availability, etc. In some embodiments, multiple replication engines may operate in parallel and transfer object versions for the same versioned data object at similar times and using resources and paths that may result in different transfer rates. Thus, the object versions of versioned data objects may be ingested at the destination system in a different order than they were stored in object store 560 or even a different order than they were sent. Replication manager 540 may also support multipart data object transfers that include a plurality of write transactions to complete a single object replication. For example, a multipart write process may include a series of commands or messages, such as INIT, multiple PARTPUT, and COMPLETE functions.
In some embodiments, replication manager 540 may include one or more hardware and/or software modules or data structures for executing specific operations. For example, replication manager 540 may include a version write request 542, a versioning-invalid list 544, a version response handler 546, and retry queue 548.
Version write request 542 may a include a command, interface protocols, and/or a set of functions and parameters for a special write operation to provide validated writing of a versioned data object to another object store. For example, when replication manager is tasked with replicating a versioned data object to a heterogeneous data store with unknown versioning configuration, version write request 542 may enable the versioned data object to be transferred and written while validating that a valid versioning configuration was operational on the destination data store before confirming completion of the transfer to storage interface 520 or another system resource or client. In some embodiments, version write request 542 may operate as a PUT or multipart PUT command that awaits a response message that includes versioning related information for determining whether versioning status was valid for the PUT request. Version write request 542 may operate in conjunction with version response handler 546 to receive and validate the versioning status.
In some embodiments, version write request 542 may be configured to use a test object 542.1 to validate the versioning status of the destination object store. For example, version write request 542 may generate a test version write request or PUT request to the destination object store with the test object 542.1 rather than an object version or object part (for multipart PUTs). In some embodiments, test object 542.1 may include one or more object versions for the test object with no host data payload and a small object data size relative to the versioned data object being replicated. A test request for test object 542.1 may be executed prior to initiating the version write request for the versioned data object selected for replication. In some embodiments, a test request may be initiated in response to the destination object store or bucket having an entry in versioning-invalid list 544 and/or in response to determining an invalid versioning-enabled status from a response to a prior request to the same destination. Test objects 542.1 that are successfully written to the destination data store may still be followed with a delete request from version response handler 546 to remove test objects from the destination data store in a manner similar to the removal of residual data objects from failed versioned write requests.
Versioning-invalid list 544 may include a data structure maintained by replication manager 540 to improve the efficiency of initial version write requests and monitor versioning status during iterative test message and retry cycles. For example, versioning-invalid list 544 may include a searchable data structure with entries for destination object stores that have returned invalid versioning-enabled status. In some embodiments, bucket identifiers may be organized in a data table and used to index the bucket-invalid status for buckets in one or more destination object stores. Versioning-invalid list 544 may include a function or method to search the data structure for a bucket identifier of interest.
Version response handler 546 may a include a command, interface protocols, and/or a set of functions and parameters for receiving and parsing responses from the destination object store to validate writing of a versioned data object. For example, version response handler 546 may receive version write response messages that include parameters indicating the versioning-enabled status of the destination object store at the time of the version write request. In some embodiments, the version write response message may include a value for the versioning-enabled status. In some embodiments, version response handler 546 may include logic for parsing the parameters of the version write response message to determine the versioning-enabled status as valid or invalid.
A valid versioning-enabled status may trigger an operation complete or similar success message and/or enable continued processing of subsequent operations within replication manager 540, such as sending a delete request 546.1 for successful versioned writing of test object 542.1. An invalid versioning-enabled status may trigger delete request 546.1 to remove any residual object version(s) written to the destination object store but not completing the version write request. An invalid versioning-enabled status may trigger a notification event 546.2 back to the initiating service or client. In some embodiments, notification event 546.2 may indicate the invalid versioning-enabled status and/or request a versioning configuration change for the destination object store.
Retry queue 548 may include a data structure of maintaining a list of pending version write requests that may be retried through an iterative retry process. For example, an iterative retry process may be enabled for a fixed number of retries, retry period, or other limit on retry attempts, with or without the use of test messages to await a change in versioning-enabled status. In some embodiments, retry queue 548 may be maintained in a data structure, such as an operations table, with entries for each pending version write request and one or more parameters for determining priority, number of attempts, and other version write request data relevant to managing retry attempts.
In some embodiments, version write request 542, versioning-invalid list 544, version response handler 546, and/or retry queue 548 may provide one or more atomic functions for executing a valid versioned data object replication (or returning an appropriate error for a replication that could not be completed as described above). An atomic function may be invoked or requested by or through storage interface 520 and then maintain an operating state until the result or response is returned without further interaction with the initiating service. For example, version write request 542 may send the PUT request, version response handler 546 may receive the version write response and determine whether a delete request needs to be processed, and replication manager 540 may return a result, complete, or error message only once the delete request has been successfully executed. Similarly, if a version write request 542 is configured to use test requests or iterative attempts using retry queue 548, those operations may similarly be executed by replication manager until a resolution to the version write request is achieved without further input from the initiating service or client.
Memory 516 may include additional logic and other resources (not shown) for processing object data requests, such as modules for generating, queueing, and otherwise managing object data requests. Processing of an object data request by storage interface 520 may include any number of intermediate steps that yield at least one data request to the distributed storage system.
Through an attempted version write request 640 from object storage system 630.1 to object storage system 630.2, the object versions were transferred to object storage system 630.2. Because versioning-enabled status 638.2 was invalid, such as versioning being disabled or unavailable for bucket 632.2, only version 3333 was successfully stored for object 634.2. This results in version 3333 being a residual data object for object 634.2, which may create errors if it remains in bucket 632.2. Response 642 may include versioning-enabled status 638.2 or other parameters from which versioning-enabled status 638.2 may be determined. As a result, when object storage system 630.1 receives and processes response 642, it may identify the error in writing versioned data object 634.2.
Object storage system 630.1 may identify the invalid version write attempt. In response, object storage system 630.1 may immediately and automatically issue a delete message 644 to delete version 3333 and object 634.2. It may await successful completion of the delete operation and response 646 from object storage system 630.2 before completing the failed replication attempt and providing appropriate notifications to the initiating services or clients. In an end state, object 634.2 and version 3333 may be removed from object storage system 630.2 and another attempt may be made after versioning status 638.2 has been updated without creating conflicts with the residual data object 634.2.
As shown in
At block 710, a version write request message may be sent to a destination object storage system including a destination object store. For example, a replication manager may generate a version write request for a versioned data object and send the version write request. In some embodiments, the version write request message may include a series of version write request messages for each of the object versions and/or parts of a multipart object write that involve multiple write transactions. The replication manager may then wait for a response.
At block 720, a response message may be received by the object storage system that sent the version write request. For example, the object storage system may receive a response message responsive to a PUT request sent as the version write request message.
At block 730, the versioning-enabled status for the destination object store may be determined from the response message. For example, the response message may include a flag or other parameter value for versioning-enabled status. In some embodiments, the replication manager may extract and parse one or more parameters in the response message to determine the versioning-enabled status. At block 735, whether or not the versioning-enabled status is valid may be determined? If yes, method 700 may proceed to block 740. If no, method 700 may proceed to block 750.
At block 740, valid versioning-enabled status has been determined for the version write request, for example, as described for block 730. At block 745, a success notification may be generated responsive to the valid versioning-enabled status. For example, a success notification may be sent to the initiating system service or client that the versioned data object was successfully replicated with versioning.
At block 750, invalid versioning-enabled status has been determined for the version write request, for example, as described for block 730. At block 760, a delete request for the versioned data object is sent to the destination object store. For example, a delete request to delete any residual object version(s) may be sent to the destination object store. At block 770, the version write request may optionally be added to a retry queue to be attempted again. At block 780, a versioning-invalid notification event may be generated. For example, a failure notification that includes an error code for versioning not enabled may be sent to the initiating system service or client. In some embodiments, the versioning-invalid notification event may generate a message to the destination object storage system or a related storage management system to enable versioning on the destination object store. For example, the object storage system may be configured to request that a versioning enabler on the destination object storage system change the versioning enabled status for the destination object data bucket.
As shown in
At block 810, a version write request is received for a versioned data object. For example, the originating object storage system may send a version write request with one or more object versions for a versioned data object.
At block 820, the write request may be processed without versioning due to the versioning-enabled status being invalid. For example, each of the object versions received may overwrite the prior object version for the versioned data object.
At block 830, the residual version of the versioned data object is stored with that object name. For example, after each prior object version has been overwritten, only the last version may remain as the residual version and it will be stored on the object store using the object name and without regard to any versioning information provided in the request or the object itself.
At block 840, a response message is sent upon completion of the version write request. For example, completion of a write request may generate a completion message including parameters to match the completion message with the original request message. In some embodiments, the response message may include an indicator related to versioning-enabled status. In some embodiments, the response message may include parameters describing the residual data object stored without versioning information, from which the invalid versioning-enabled status may be determined.
At block 850, a delete request may be received for the residual object. For example, responsive to determining the invalid versioning-enabled status, the originating object storage system may send a delete request for the residual version(s).
At block 860, the residual version is deleted in response to the delete request. For example, the destination object storage system may process the delete request to delete the residual version.
At block 870, a response message is sent by the destination object storage system to confirm completion of the delete operation. For example, completion of a delete request may generate a completion message including parameters to match the completion message with the original request message.
As shown in
At block 910, a replication request is received for a versioned data object. For example, a client or system service may initiate a replication operation for one or more data objects, buckets, or object stores in the originating data storage system.
At block 920, a versioning-invalid list is searched for the destination object store. For example, the replication request may include a destination bucket for the replication request and the bucket identifier may be used as an index to search a versioning-invalid list maintained by the originating storage system. At block 925, whether the destination bucket is in the versioning-invalid list may be determined. If yes, method 900 may proceed to block 930. If no, method 900 may proceed to block 980.
At block 930, a test request may be sent to the destination object store. For example, a test request that is a version write request for a versioned test object may be sent to the destination object storage system targeting the destination bucket. In some embodiments, a version status request message may be sent as a test message.
At block 940, a test response message to the test request may be received from the destination object storage system. For example, the version write request for the test object may generate a response message from which valid versioning-enabled status may be determined. Alternatively, a response message to a version status request may include versioning-enabled status information from which valid or invalid status may be determined. At block 945, whether the versioning-enabled status is valid may be determined. If yes, method 900 may proceed to block 950. If no, method 900 may proceed to block 960.
At block 950, the destination object store may be removed from the invalid list in response to the valid versioning test. For example, when the valid versioning-enabled status is determined from the response message, the originating object storage system may remove the entry for the destination bucket from the versioning-invalid list. Method 900 may proceed to block 970 to initiate a retry.
At block 960, the versioning-invalid list may be updated in response to invalid versioning. For example, if an entry already exists for the destination bucket, then that entry may be updated with the most recent attempt information or other related data. If no entry exists for the destination bucket, then a new entry for the destination bucket may be created.
At block 970, a retry iteration may be initiated. For example, whether a test iteration has passed or failed valid versioning or a version write request for the versioned data object being replicated has failed valid versioning, a retry may be initiated. In some embodiments, the retry may include a set of limit conditions that determine whether the retry proceeds back to block 920.
At block 980, a version write request for the versioned data object to be replicated is initiated. For example, the version write request may be processed as described above for method 700. At block 985, whether the versioning-enabled status is valid may be determined. If yes, method 900 may proceed to block 990. If no, method 900 may proceed to block 960. At block 990, the version write request for the versioned data object may be completed as described above with regard to method 700.
While at least one exemplary embodiment has been presented in the foregoing detailed description of the technology, it should be appreciated that a vast number of variations may exist. It should also be appreciated that an exemplary embodiment or exemplary embodiments are examples, and are not intended to limit the scope, applicability, or configuration of the technology in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing an exemplary embodiment of the technology, it being understood that various modifications may be made in a function and/or arrangement of elements described in an exemplary embodiment without departing from the scope of the technology, as set forth in the appended claims and their legal equivalents.
As will be appreciated by one of ordinary skill in the art, various aspects of the present technology may be embodied as a system, method, or computer program product. Accordingly, some aspects of the present technology may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.), or a combination of hardware and software aspects that may all generally be referred to herein as a circuit, module, system, and/or network. Furthermore, various aspects of the present technology may take the form of a computer program product embodied in one or more computer-readable mediums including computer-readable program code embodied thereon.
Any combination of one or more computer-readable mediums may be utilized. A computer-readable medium may be a computer-readable signal medium or a physical computer-readable storage medium. A physical computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, crystal, polymer, electromagnetic, infrared, or semiconductor system, apparatus, or device, etc., or any suitable combination of the foregoing. Non-limiting examples of a physical computer-readable storage medium may include, but are not limited to, an electrical connection including one or more wires, a portable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a Flash memory, an optical fiber, a compact disk read-only memory (CD-ROM), an optical processor, a magnetic processor, etc., or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program or data for use by or in connection with an instruction execution system, apparatus, and/or device.
Computer code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to, wireless, wired, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing. Computer code for carrying out operations for aspects of the present technology may be written in any static language, such as the C programming language or other similar programming language. The computer code may execute entirely on a user's computing device, partly on a user's computing device, as a stand-alone software package, partly on a user's computing device and partly on a remote computing device, or entirely on the remote computing device or a server. In the latter scenario, a remote computing device may be connected to a user's computing device through any type of network, or communication system, including, but not limited to, a local area network (LAN) or a wide area network (WAN), Converged Network, or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider).
Various aspects of the present technology may be described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus, systems, and computer program products. It will be understood that each block of a flowchart illustration and/or a block diagram, and combinations of blocks in a flowchart illustration and/or block diagram, can be implemented by computer program instructions. These computer program instructions may be provided to a processing device (processor) of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which can execute via the processing device or other programmable data processing apparatus, create means for implementing the operations/acts specified in a flowchart and/or block(s) of a block diagram.
Some computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device(s) to operate in a particular manner, such that the instructions stored in a computer-readable medium to produce an article of manufacture including instructions that implement the operation/act specified in a flowchart and/or block(s) of a block diagram. Some computer program instructions may also be loaded onto a computing device, other programmable data processing apparatus, or other device(s) to cause a series of operational steps to be performed on the computing device, other programmable apparatus or other device(s) to produce a computer-implemented process such that the instructions executed by the computer or other programmable apparatus provide one or more processes for implementing the operation(s)/act(s) specified in a flowchart and/or block(s) of a block diagram.
A flowchart and/or block diagram in the above figures may illustrate an architecture, functionality, and/or operation of possible implementations of apparatus, systems, methods, and/or computer program products according to various aspects of the present technology. In this regard, a block in a flowchart or block diagram may represent a module, segment, or portion of code, which may comprise one or more executable instructions for implementing one or more specified logical functions. It should also be noted that, in some alternative aspects, some functions noted in a block may occur out of an order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or blocks may at times be executed in a reverse order, depending upon the operations involved. It will also be noted that a block of a block diagram and/or flowchart illustration or a combination of blocks in a block diagram and/or flowchart illustration, can be implemented by special purpose hardware-based systems that may perform one or more specified operations or acts, or combinations of special purpose hardware and computer instructions.
While one or more aspects of the present technology have been illustrated and discussed in detail, one of ordinary skill in the art will appreciate that modifications and/or adaptations to the various aspects may be made without departing from the scope of the present technology, as set forth in the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201811020720 | Jun 2018 | IN | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5649200 | Leblang et al. | Jul 1997 | A |
| 5729744 | Gerken | Mar 1998 | A |
| 5813009 | Johnson et al. | Sep 1998 | A |
| 6615223 | Shih | Sep 2003 | B1 |
| 8504758 | McHugh et al. | Aug 2013 | B1 |
| 8533170 | McHugh | Sep 2013 | B1 |
| 8650156 | McHugh et al. | Feb 2014 | B1 |
| 8731994 | Kass | May 2014 | B2 |
| 20030145199 | Miller | Jul 2003 | A1 |
| 20050027757 | Kiessig et al. | Feb 2005 | A1 |
| 20070208790 | Reuter et al. | Sep 2007 | A1 |
| 20080005077 | Eggebraaten et al. | Jan 2008 | A1 |
| 20080059608 | Wood | Mar 2008 | A1 |
| 20100191774 | Mason, Jr. et al. | Jul 2010 | A1 |
| 20120109915 | Kamekawa | May 2012 | A1 |
| 20130275379 | Trebas et al. | Oct 2013 | A1 |
| 20140143208 | Kohl et al. | May 2014 | A1 |
| 20140149353 | Lee et al. | May 2014 | A1 |
| 20140244708 | Taine et al. | Aug 2014 | A1 |
| 20150363271 | Haustein et al. | Dec 2015 | A1 |
| 20160092526 | Kothari et al. | Mar 2016 | A1 |
| 20160140133 | Lu et al. | May 2016 | A1 |
| 20160330299 | Hu | Nov 2016 | A1 |
| 20170032006 | Anglin | Feb 2017 | A1 |
| 20170359400 | Demulder et al. | Dec 2017 | A1 |
| 20180019867 | Davis | Jan 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 2017023709 | Feb 2017 | WO |
| Entry |
|---|
| International Search Report and Written Opinion of Application No. PCT/US2019/023745, dated Jul. 2, 2019, p. 1-12. |
| International Search Report and Written Opinion of Application No. PCT/US2019/023751, dated Jul. 2, 2019, p. 1-12. |
| Number | Date | Country | |
|---|---|---|---|
| 20190370237 A1 | Dec 2019 | US |
