Patent Grant
11775193
Computing devices may include any number of internal components such as processors, memory, and persistent storage. Each of the internal components of a computing device may be used to generate data. Given the large volume of data that is generated, a significant number of resources may be required to manage the generated data.
In general, in one aspect, the invention relates to a method for managing data. The method includes obtaining data from a host, wherein the data is associated with an object identifier (ID), initiating a classification mapping update to obtain a classification entry, wherein the classification entry is associated with a classification ID, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk, generating an object entry associated with the plurality of data chunks, and the at least one parity chunk, wherein the object entry comprises the object ID and the classification ID, storing the storage metadata and the object entry in an accelerator pool, and storing, across a plurality of fault domains, the plurality of deduplicated data chunks and the at least one parity chunk.
In general, in one aspect, the invention relates to a non-transitory computer readable medium which includes computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing data. The method includes obtaining data from a host, wherein the data is associated with an object identifier (ID), initiating a classification mapping update to obtain a classification entry, wherein the classification entry is associated with a classification ID, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk, generating an object entry associated with the plurality of data chunks, and the at least one parity chunk, wherein the object entry comprises the object ID and the classification ID, storing the storage metadata and the object entry in an accelerator pool, and storing, across a plurality of fault domains, the plurality of deduplicated data chunks and the at least one parity chunk.
In general, in one aspect, the invention relates to a data cluster that includes a host and an accelerator pool that includes a plurality of data nodes, wherein a data node of the plurality of data nodes includes a processor and memory that includes instructions, which when executed by the processor perform a method for managing data. The method includes obtaining data from a host, wherein the data is associated with an object identifier (ID), initiating a classification mapping update to obtain a classification entry, wherein the classification entry is associated with a classification ID, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk, generating an object entry associated with the plurality of data chunks, and the at least one parity chunk, wherein the object entry comprises the object ID and the classification ID, storing the storage metadata and the object entry in an accelerator pool, and storing, across a plurality of fault domains, the plurality of deduplicated data chunks and the at least one parity chunk.
Certain embodiments of the invention will be described with reference to the accompanying drawings. However, the accompanying drawings illustrate only certain aspects or implementations of the invention by way of example and are not meant to limit the scope of the claims.
Specific embodiments will now be described with reference to the accompanying figures. In the following description, numerous details are set forth as examples of the invention. It will be understood by those skilled in the art that one or more embodiments of the present invention may be practiced without these specific details and that numerous variations or modifications may be possible without departing from the scope of the invention. Certain details known to those of ordinary skill in the art are omitted to avoid obscuring the description.
In the following description of the figures, any component described with regard to a figure, in various embodiments of the invention, may be equivalent to one or more like-named components described with regard to any other figure. For brevity, descriptions of these components will not be repeated with regard to each figure. Thus, each and every embodiment of the components of each figure is incorporated by reference and assumed to be optionally present within every other figure having one or more like-named components. Additionally, in accordance with various embodiments of the invention, any description of the components of a figure is to be interpreted as an optional embodiment, which may be implemented in addition to, in conjunction with, or in place of the embodiments described with regard to a corresponding like-named component in any other figure.
Throughout this application, elements of figures may be labeled as A to N. As used herein, the aforementioned labeling means that the element may include any number of items and does not require that the element include the same number of elements as any other item labeled as A to N. For example, a data structure may include a first element labeled as A and a second element labeled as N. This labeling convention means that the data structure may include any number of the elements. A second data structure, also labeled as A to N, may also include any number of elements. The number of elements of the first data structure and the number of elements of the second data structure may be the same or different.
In general, embodiments of the invention relate to a method and system for storing data and metadata in a data cluster. Embodiments of the invention may utilize a data processor, operating in an accelerator pool, which applies an erasure coding procedure on data obtained from a host to divide the data into data chunks and to generate parity chunks using the data chunks. The data processor may then perform deduplication on the data chunks to generate deduplicated data that includes deduplicated data chunks. The deduplicated data chunks and the parity chunks are subsequently distributed to nodes in the data cluster in accordance with an erasure coding procedure.
In one or more embodiments of the invention, the accelerator pool stores storage metadata that specifies the nodes in which each data chunk and parity chunk is stored and object metadata that specifies an object associated with each data chunk. The storage metadata and object metadata may also be distributed to nodes in the non-accelerator pool. In this manner, if the storage metadata or object metadata stored in the accelerator pool becomes unavailable, the storage metadata and/or object metadata may be reconstructed using the storage metadata stored in the non-accelerator pool.
In one or more embodiments of the invention, a data classification engine of the accelerator pool obtains classification attributes about the object. The classification attributes may relate to a geolocation in which the data associated with the object was obtained, an organization or legal entity that owns the object, any policies or regulations associated with the object, and/or other classification attributes without departing from the invention. The attributes associated with the object may be stored in an entry of a classification mapping of the data classification engine. The entry may include a classification (ID) that may be stored in the object metadata. In this manner, embodiments of the invention enable the classification to be associated (or bound) to the object (or more specifically to the data chunks (described below) that make up the object). By binding the classification at a granular level, embodiments of the invention enable granular use and updating of the classification. Moreover, embodiments of the invention, bind the classification to the object (or portions thereof) without requiring the object's contents to be analyzed. In this manner, embodiments of the invention enable more efficient classification and limit the exposure of potentially sensitive data that is included within the objects.
In one or more embodiments of the invention, the host (100) utilizes the data cluster (110) to store data. The data stored may be backups of databases, files, applications, and/or other types of data without departing from the invention.
In one or more embodiments of the invention, the host (100) is implemented as a computing device (see e.g.,
In one or more embodiments of the invention, the host (100) is implemented as a logical device. The logical device may utilize the computing resources of any number of computing devices and thereby provide the functionality of the host (100) described throughout this application.
In one or more embodiments of the invention, the data cluster (110) stores data, metadata, and/or backups of data generated by the host (100). The data and/or backups may be deduplicated versions of data obtained from the host. The data cluster may, via an erasure coding procedure, store portions of the data (which may or may not be deduplicated) across nodes operating in the data cluster (110).
As used herein, deduplication refers to methods of storing only portions (also referred to as file segments or segments) of files (which are a type of object) that are not already stored in persistent storage. For example, when multiple versions of a large file, having only minimal differences between each of the versions, are stored without deduplication, storing each version will require approximately the same amount of storage space of a persistent storage. In contrast, when the multiple versions of the large file are stored with deduplication, only the first version of the multiple versions stored will require a substantial amount of storage. Once the first version is stored in the persistent storage, the subsequent versions of the large file subsequently stored will be deduplicated before being stored in the persistent storage resulting in much less storage space of the persistent storage being required to store the subsequently stored versions when compared to the amount of storage space of the persistent storage required to store the first stored version.
Continuing with the discussion of
In one or more embodiments of the invention, the data processor (122) is a device that includes functionality to perform deduplication on data obtained from a host (e.g., 100,
In one or more embodiments of the invention, the storage metadata (124) is stored in a data node (126A, 126N) of the accelerator pool (120). A copy of the storage metadata (124) may be distributed to one or more data nodes (132, 134) of the non-accelerator pool (130). In this manner, if the storage metadata (124) stored in the accelerator pool (120) experiences a failure (e.g., it becomes unavailable, corrupted, etc.), the storage metadata (124) may be reconstructed using the copies of storage metadata stored in the non-accelerator pool (130). For additional detail regarding the distribution on storage metadata, see e.g.,
In one or more embodiments of the invention, the data processor (122) updates object metadata (128) after storing data chunks (which may be deduplicated) and parity chunks. In one or more embodiments of the invention, the object metadata is a data structure, stored in a computing device (e.g., a data node (126A, 126N)) of the accelerator pool (120), includes object information about the data stored in the data cluster (110A). An object may be, for example, a file, a set of files, a portion of a file, a backup of any combination thereof, and/or any other type of data without departing from the invention. For additional details regarding the object metadata, see, e.g.,
In one or more of embodiments of the invention, the data processor (122) is implemented as computer instructions, e.g., computer code, stored on a persistent storage that when executed by a processor of a data node (e.g., 126A, 126N) of the accelerator pool (120) cause the data node to provide the aforementioned functionality of the data processor (122) described throughout this application and/or all, or a portion thereof, of the method illustrated in
In one or more embodiments of the invention, the data processor (122) is implemented as a computing device (see e.g.,
In one or more embodiments of the invention, the data processor (122) is implemented as a logical device. The logical device may utilize the computing resources of any number of computing devices and thereby provide the functionality of the data processor (122) described throughout this application and/or all, or a portion thereof, of the method illustrated in
In one or more embodiments of the invention, the data classification engine (129) manages the data classification of objects stored in the data cluster (110A). The data classification engine may manage the data classification based on classification information provided by the host it is attempting to store an object from and/or by obtaining data classification information from a host (e.g., 100,
In one or more of embodiments of the invention, the data classification engine (129) is implemented as computer instructions, e.g., computer code, stored on a persistent storage that when executed by a processor of a data node (e.g., 126A, 126N) of the accelerator pool (120) cause the data node to provide the aforementioned functionality of the data classification engine (129) described throughout this application and/or all, or a portion thereof, of the method illustrated in
In one or more embodiments of the invention, the data classification engine (129) is implemented as a computing device (see e.g.,
In one or more embodiments of the invention, the data classification engine (129) is implemented as a logical device. The logical device may utilize the computing resources of any number of computing devices and thereby provide the functionality of the data classification engine (129) described throughout this application and/or all, or a portion thereof, of the method illustrated in
Continuing with the discussion of
The heterogeneous population of nodes may be logically divided into: (i) an accelerator pool (120) including nodes that have more computing resources, e.g., high performance nodes (126A, 126N), than other nodes and (ii) a non-accelerator pool (130) including nodes that have fewer computing resources, e.g., low performance nodes (132, 134) than the nodes in the accelerator pool (120). For example, nodes of the accelerator pool (120) may include enterprise-class solid state storage resources that provide very high storage bandwidth, low latency, and high input-outputs per second (IOPS). In contrast, the nodes of the non-accelerator pool (130) may include hard disk drives that provide lower storage performance. While illustrated in
In one or more embodiments of the invention, the data nodes (126A, 126N, 132, 134) store data chunks and parity chunks along with storage metadata (as described below). The data nodes (126A, 126N, 132, 134) may include persistent storage that may be used to store the data chunks, parity chunks and storage metadata. The generation of the data chunks and parity chunks as well as the storage metadata is described below with respect to
In one or more embodiments of the invention, the non-accelerator pool (130) includes any number of fault domains. In one or more embodiments of the invention, a fault domain is a logical grouping of nodes (e.g., data nodes) that, when one node of the logical grouping of nodes goes offline and/or otherwise becomes inaccessible, the other nodes in the same logical grouping of nodes are directly affected. However, nodes in a different fault domain may be unaffected. For additional details regarding fault domains, see, e.g.
In one or more embodiments of the invention, each data node (126A, 126N, 132, 134) is implemented as a computing device (see e.g.,
In one or more embodiments of the invention, each of the data nodes (126A, 126N, 132, 134) is implemented as a logical device. The logical device may utilize the computing resources of any number of computing devices and thereby provide the functionality of the data nodes (126A, 126N, 132, 134) described throughout this application and/or all, or a portion thereof, of the method illustrated in
In one or more embodiments of the invention, the processor (142) is a component that processes data and processes of requests. The processor (142) may be, for example, a central processing unit (CPU). The processor may process a request to rebuild data and/or metadata using data stored in memory (144) and/or the persistent storage devices (146A, 146N). The processor (142) may process other requests without departing from the invention.
In one or more embodiments of the invention, the data node includes memory (144) which stores data that is more accessible to the processor (142) than the persistent storage devices (146A, 146N). The memory (144) may be volatile storage. Volatile storage may be storage that stores data that is lost when the storage loses power. The memory may be, for example, Random Access Memory (RAM). In one or more embodiments of the invention, a copy of the storage metadata discussed in
In one or more embodiments of the invention, the persistent storage devices (146A, 146N) store data. The data may be data chunks and/or parity chunks. In addition, the data may also include storage metadata. The persistent storage devices (146A, 146N) may be non-volatile storage. In other words, the data stored in the persistent storage devices (146A, 146N) is not lost or removed when the persistent storage devices (146A, 146N) lose power. Each of the persistent storage devices (146A, 146N) may be, for example, solid state drives, hard disk drives, and/or tape drives. The persistent storage devices may include other types of non-volatile or non-transitory storage mediums without departing from the invention. For additional details regarding the persistent storage devices, see, e.g.,
In one or more embodiments of the invention, a data chunk (152A, 152M) is a data structure that includes a portion of data that was obtained from a host. The data chunks (152A, 152M) may be deduplicated by a data processor and obtained by the data node (140) from the data processor. Each of the data chunks (152A, 152M) may be used by the data node (140) (or another data node) to reconstruct another data chunk or a parity chunk based on an erasure coding algorithm that was applied to the other data chunk or parity chunk.
In one or more embodiments of the invention, a parity chunk (154A, 154P) is a data structure that includes a parity value generated using an erasure coding algorithm. The parity value may be generated by applying the erasure coding algorithm to one or more data chunks stored in the data node (140) or other data nodes. Each of the parity chunks (154A, 154P) may be used by the data node (140) (or another data node) to reconstruct another parity chunk or a data chunk based on an erasure coding algorithm that was applied to the other parity chunk or data chunk.
As discussed above, a fault domain (160A, 160N) is a logical grouping of data nodes (164A, 164B) that, when one data node of the logical grouping of data nodes goes offline and/or otherwise becomes inaccessible, the other nodes in the logical grouping of nodes are directly affected. The effect of the node going offline to the other nodes may include the other nodes also going offline and/or otherwise inaccessible. The non-accelerator pool (130) may include multiple fault domains. In this manner, the events of one fault domain in the non-accelerator pool (130A) may have no effect to other fault domains in the non-accelerator pool (130A).
For example, two data nodes may be in a first fault domain (e.g., 160A). If one of these data nodes in the first fault domain (160A) experiences an unexpected shutdown, other nodes in the first fault domain may be affected. In contrast, another data node in a second fault domain may not be affected by the unexpected shutdown of a data node in the first fault domain. In one or more embodiments of the invention, the unexpected shutdown of one fault domain does not affect the nodes of other fault domains. In this manner, data may be replicated and stored across multiple fault domains to allow high availability of the data.
As discussed above, the data chunks and parity chunks of a slice (e.g., generated using the erasure coding described in
In one or more embodiments of the invention, each fault domain (160A, 160N) stores a copy of storage metadata (162) and a copy of object metadata (166) obtained from an accelerator pool and/or from another fault domain (160A, 160N) distributing a copy of the storage metadata. The copy of storage metadata (162) and the copy of the object metadata (166) in a fault domain (e.g., 160A) may each be stored in one or more data nodes (164A, 164B) of the fault domain. Further, a copy of a classification mapping (e.g., 129A,
In one or more embodiments of the invention, a metadata slice entry (200A, 200N) is an entry that specifies metadata associated with chunks of data generated using an erasure coding procedure. The metadata slice entry (200A, 200N) includes chunk metadata (202, 204). Each chunk metadata (202, 204) may correspond to metadata for a data chunk or a parity chunk. Each chunk metadata (202, 204) may include information about a chunk such as, for example, a unique identifier (e.g., a fingerprint) and a storage location of the chunk, e.g., the non-accelerator pool. The unique identifier of a chunk may be generated using the chunk (e.g., calculated using the data of the chunk). The chunk metadata may also include a classification ID. In this scenario, the classification ID is the same classification ID associated with the object, where the chunk metadata corresponds to a chunk that is part of the object.
In one or more embodiments of the invention, the object ID (212) is an identifier that specifies an object associated with the object entry (210A, 210N). The object ID (212) may be, for example, a string of numbers, letters, symbols, or any combination thereof that uniquely identifies the object.
In one or more embodiments of the invention, the classification information (218) specifies a classification ID associated with classification attributes of the object. The classification information (218) may be used to map the object entry (210A, 210N) to a classification mapping entry of a classification mapping. The classification mapping entry may specify one or more classification attributes associated with the object of the object entry (210A, 210N). The classification ID may be, for example, a string of numbers, letters, symbols, or any combination thereof that uniquely identifies the classification mapping entry.
In one or more embodiments of the invention, the timestamp (214) specifies a point in time that corresponds to a state of the object as specified by a set of chunk metadata. The timestamp (214) may be used to replay the object to a point in time. In one or more embodiments of the invention, the object is replayed to a point in time when the data associated with the object that was part of the object at the point in time is reconstructed to generate the object at the point in time. Said another way, the content of each object may vary over time and each time the object is modified a corresponding object entry is created where the object entry specifies chunk metadata for the chunks that make up the object at that point in time.
For example, at a first point in time, the object may include a first set of data, of which there is a first chunk and a second chunk. At a second point in time, the object may include a second set of data, of which there is a first chunk and a third chunk. The third chunk may be a modified version of the second chunk. The object may be replayed to the first point in time by obtaining the first chunk and the second chunk. The object may be replayed to the second point in time by obtaining the first chunk and the third chunk. For each point in time, there may be an object entry that specifies the object, the point in time, and each chunk used to replay the object.
In one or more embodiments of the invention, the chunk metadata (216A, 216M) each corresponds to a data chunk or parity chunk associated with the object at the point in time specified by the timestamp (214). The chunk metadata may include information about the data chunk or parity chunk such as, for example, a unique identifier (e.g., a fingerprint). The unique identifier may be, for example, a string of numbers, letters, symbols, or any combination thereof that uniquely identifies the chunk.
In one or more embodiments of the invention, an object entry (210A) is associated with more than one timestamp (214). In such embodiments, each chunk metadata (216A, 216M) may specify multiple chunks associated with a point in time. For example, after every iteration of an object (i.e., an object is associated with a new point in time), an object entry (210A, 210N) is updated with new chunk metadata (216A, 216M) that specifies the chunks of that iteration. In this manner, each object is associated with one object entry (210A, 210N) and each chunk metadata (202, 204) is associated with multiple chunks of an object at a point in time.
The object metadata (210) may be organized using other schemes without departing from the invention.
In one or more embodiments of the invention, the classification ID (222) is a data structure that uniquely identifies the classification mapping entry (220A, 220N). The classification ID (222) may be, for example, a string of numbers, letters, symbols, or any combination thereof that uniquely identifies the classification mapping entry.
In one or more embodiments of the invention, the object ID(s) is a data structure that specifies one or more objects. Each of the object IDs may be, for example, a string of numbers, letters, symbols, or any combination thereof that uniquely identifies the object.
In one or more embodiments of the invention, each classification mapping entry (220A, 220N) is associated with one object. In such a scenario, the classification mapping entry would include one object ID (224). Further to this embodiment, if a second object shared a classification ID (222) and the classification attributes (226, 228) to a first object, the classification mapping (220) would include two classification mapping entries, one associated with the first object and a second associated with the second object.
In one or more embodiments of the invention, each classification mapping entry (220A, 220N) is associated with one or more objects. In such a scenario, the classification mapping entry may include more than one object ID (224). For example, further to this embodiment, a classification mapping entry specifies a first classification ID and a first object ID associated with a first object. A classification mapping engine may update the classification mapping to specify a second object with the same classification ID as the first object. The classification mapping may be updated by updating the classification mapping entry to specify a second object ID associated with the second object.
In one or more embodiments of the invention, the classification mapping entry (220A, 220N) further includes classification attributes (226, 228). In one or more embodiments of the invention, a classification attribute (226, 228) is an attribute that further describes the nature of the objects of the classification mapping entry (220A). Non-limiting examples of classification attributes may include a geographical location in which the data associated with the objects was obtained, a legal entity that has ownership and/or access to the data, a retention period in which to store the data, one or more regulations with which the data is to be in compliance, and/or industry (e.g., healthcare, financial services, etc.).
In one or more embodiments of the invention, a legal entity is a person, a group of people, a partnership, corporation, any other business entity, or any combination thereof. The legal entity may be specified using, for example, an identifier that uniquely species the legal entity.
In one or more embodiments of the invention, a regulation is a set of legal standards set in place for the geographical location associated with the data in which the legal entity is to comply. For example, the state of California has a California Consumer Privacy Act (CCPA), which is a law that specifies a set of legal standards intended to protect the privacy of data owned or accessed by residents of the state of California. Legal entities accessing data that is associated with a classification mapping entry that specifies the CCPA will need to comply with the legal standards of the CCPA.
In step 300, a write request is obtained from a host. The write request may specify an object, e.g., a file segment, a collection of files, or any other type of data without departing from the invention.
In step 302, a determination is made about whether an object entry associated with the object (i.e., the object specified with write request) is in the object metadata. In one or more embodiments of the invention, the data processor analyzes the object metadata to determine if any object entry includes an object ID that corresponds to the object specified in the write request. Further, the data processor analyzes a corresponding object entry (if any) that is associated with the object specified in the write request to determine whether the object entry includes a classification ID. If an object entry is stored in object metadata associated with the data in the write request (and the object entry includes a classification ID), the method proceeds to step 310; otherwise, the method proceeds to step 306.
In step 306, a classification attributes update is initiated. In one or more embodiments of the invention, the classification attributes update is a process for obtaining classification attributes associated with the data and storing the attributes in a classification mapping. In one or more embodiments of the invention, the classifications attributes update is initiated by prompting a data classification engine of the accelerator pool for additional details regarding the classification attributes associated with the object (i.e., the object specified in Step 300), see, e.g.,
In step 308, the object metadata is updated based on the classification attributes update. In one or more embodiments of the invention, the result of step 306 is an updated classification mapping. The update may include generating a classification mapping entry. The classification mapping entry may include a classification ID. The object metadata may be updated by updating the object entry associated with the data by specifying the classification ID generated in step 306 or by creating an object entry corresponding the object, where the newly created object entry includes the aforementioned classification ID.
In step 310, an erasure coding procedure is performed on the data to generate data chunks and parity chunks. In one or more embodiments of the invention, the erasure coding procedure includes dividing the obtained data into portions, referred to as data chunks. Each data chunk may include any number of data segments associated with the obtained data. The individual data chunks may then be combined (or otherwise grouped) into slices (also referred to as Redundant Array of Independent Disks (RAID) slices). One or more parity values are then calculated for each of the aforementioned slices. The number of parity values may vary based on the erasure coding algorithm that is being used as part of the erasure coding procedure. Non-limiting examples of erasure coding algorithms are RAID-3, RAID-4, RAID-5, and RAID-6. Other erasing coding algorithms may be used without departing from the invention. Continuing with the above discussion, if the erasing code procedure is implementing RAID-3, then a single parity value is calculated. The resulting parity value is then stored in a parity chunk. If erasure coding procedure algorithm requires multiple parity values to be calculated, then the multiple parity values are calculated with each parity value being stored in a separate data chunk.
As discussed above, the data chunks are used to generate parity chunks in accordance with the erasure coding procedure. More specifically, the parity chunks may be generated by applying a predetermined function (e.g., P Parity function, Q Parity Function), operation, or calculation to at least one of the data chunks. Depending on the erasure coding procedure used, the parity chunks may include, but are not limited to, P parity values and/or Q parity values.
In one embodiment of the invention, the P parity value is a Reed-Solomon syndrome and, as such, the P Parity function may correspond to any function that can generate a Reed-Solomon syndrome. In one embodiment of the invention, the P parity function is an XOR function.
In one embodiment of the invention, the Q parity value is a Reed-Solomon syndrome and, as such, the Q Parity function may correspond to any function that can generate a Reed-Solomon syndrome. In one embodiment of the invention, a Q parity value is a Reed-Solomon code. In one embodiment of the invention, Q=g0·D0+g1·D1+g2D2+ . . . +gn-1·Dn-1, where Q corresponds to the Q parity, g is a generator of the field, and the value of D corresponds to the data in the data chunks.
In one or more embodiments of the invention, the number of data chunks and parity chunks generated is determined by the erasure coding procedure, which may be specified by the host, by the data cluster, and/or by another entity.
In one embodiment of the invention, each of the chunks generated in step 310 are associated with the classification ID.
In step 312, deduplication operation is performed on the data chunks to obtain deduplicated data chunks. In one or more embodiments of the invention, the deduplication operation is performed in the accelerator pool by identifying the data chunks of the obtained data and assigning a fingerprint to each data chunk. A fingerprint is a unique identifier that may be stored in metadata of the data chunk. The data processor performing the deduplication may generate a fingerprint for a data chunk and identify whether the fingerprint matches an existing fingerprint stored in storage metadata stored in the accelerator pool. If the fingerprint matches an existing fingerprint, the data chunk may be deleted, as it is already stored in the data cluster. If the fingerprint does not match any existing fingerprints, the data chunk may be stored as a deduplicated data chunk. Additionally, the fingerprint of each deduplicated data chunk is stored in a storage metadata slice entry of the storage metadata. A fingerprint (or other unique identifier) of each parity chunk is also generated and stored in the storage metadata slice entry.
In one or more embodiments of the invention, the deduplicated data chunks collectively make up the deduplicated data. In one or more embodiments of the invention, the deduplicated data chunks are the data chunks that were not deleted during deduplication.
In step 314, a storage metadata entry and the object metadata are updated based on the deduplicated data chunks. In one or more embodiments of the invention, the storage metadata is updated by generating a storage metadata slice entry that specifies the deduplicated data chunks and the parity chunks and their respective storage locations. In one or more embodiments of the invention, the object metadata is updated by updating an object entry associated with the object to specify the data chunks (which are not deduplicated) and the parity chunks.
In step 316, the deduplicated data chunks and parity chunk(s) are stored across data nodes in different fault domains in a non-accelerator pool. As discussed above, the deduplicated data chunks and the parity chunk(s) are stored in a manner that minimizes reads and writes from the non-accelerator pool. In one embodiment of the invention, this minimization is achieved by storing data chunks and parity chunks, which are collective referred to as a slice (or slice), in the same manner as a prior version of the slice. The data processor may use, as appropriate, storage metadata for the previously stored data chunks and parity chunks to determine where to store the data chunks and parity chunks in step 306.
More specifically, in one embodiment of the invention, if the deduplicated data chunks and parity chunks are the first version of a slice (as opposed to a modification to an existing/previously stored slice), then the deduplicated data chunks and parity chunks may be stored across the data nodes (each in a different fault domain) in the non-accelerator pool. The location in which the data chunk or parity chunk is stored is tracked using the storage metadata. The scenario does not require the data processor to use location information for previously stored data chunks and parity chunks.
However, if the deduplicated data chunks and parity chunks are the second version of a slice (e.g., a modification to a previously stored slice), then the deduplicated data chunks and parity chunks are stored across the nodes (each in a different fault domain) in the non-accelerator pool using prior stored location information. The information about the location in which the data chunk or parity chunk for the second version of the slice is stored in the storage metadata.
For example, consider a scenario in which the first version of the slice includes three data chunks (D1, D2, D3) and one parity chunk (P1) that were stored as follows: Data Node 1 stores D1, Data Node 2 stores D2, Data Node 3 stores D3, and Data Node 4 stores P1. Further, in this example, a second version of the slice is received that includes three data chunks (D1, D2′, D3) and one newly calculated parity chunk (P1′). After deduplication only D2′ and P1′ need to be stored. Based on the prior storage locations (also referred to as locations) of the data chunks (D1, D2, and D3) and parity chunks (P1) for the first version of the slice, D2′ is stored on Node 2 and P1′ is stored on Node 4. By storing the D2′ on Node 2 and P1′ on Node 4 the data chunks and parity chunks associated with the second slice satisfy the condition that all data chunks and parity chunks for the second version of the slice are being stored in separate fault domains. If the location information was not taken into account, then the entire slice (i.e., D1, D2′, D3, and P1′) would need to be stored in order to guarantee that the requirement that all data chunks and parity chunks for the second version of the slice are being stored in separate fault domains is satisfied.
In one or more embodiments of the invention, if the data node that obtains the deduplicated data chunk, which is a modified version of a prior stored deduplicated data chunk, then the data node may: (i) store the modified version of the deduplicated data chunk (i.e., the data node would include two versions of the data chunk) or (ii) store the modified version of the deduplicated data chunk and delete the prior version of the deduplicated data chunk.
In one embodiment of the invention, the data processor includes functionality to determine whether a given data chunk is a modified version of a previously stored data chunk. Said another way, after the data is received from a host divided into data chunks and grouped into slices, the data processor includes functionality to determine whether a slice is a modified version of a prior stored slice. The data processor may use the fingerprints of the data chunks within the slice to determine whether the slice is a modified version of a prior stored slice. Other methods for determining whether a data chunk is a modified version of a prior stored data chunk and/or whether a slice is a modified version of a prior slice without departing from the invention.
In step 318, a distribution of storage metadata and object metadata is initiated. In one or more embodiments of the invention, the storage metadata and the object metadata are distributed by generating a copy of the storage metadata that includes the storage metadata slice entry generated in step 304 and a copy of object metadata which includes the object entry and sending the copy of storage metadata and the copy of object metadata in the non-accelerator pool.
In one or more embodiments of the invention, the copy of storage metadata, and the copy of object metadata are sent to a data node of a fault domain by the data processor. The data processor may further instruct the data node to distribute the copy of storage metadata and the copy of object metadata to other data nodes in the fault domain or to other data nodes in other fault domains. In this manner, a copy of the storage metadata is stored in multiple fault domains in the event of a storage metadata failure.
In one or more embodiments of the invention, the copy of storage metadata, and the copy of object metadata are sent to multiple fault domains by the data processor. The data processor may send a copy of storage metadata to one or more data nodes of each of the multiple fault domains. In this manner, a copy of the storage metadata is stored in multiple fault domains in the event of a storage metadata failure.
Though not shown in
While
In step 320, a classification attributes request is sent to the host. The classification attributes request may request that the host to provide one or more classification attributes associated with the object (i.e., the object that was sent in the write request in step 300). The classification attributes request may specify, for example, that a geographical location associated with the data be specified, a legal entity that owns the data to be specified, one or more regulations that is to be complied when accessing the data, etc. The classification attributes request may specify other types of classification attributes without departing from the invention.
In step 322, a classification attributes response is obtained from the host based on the classification attributes request. In one or more embodiments of the invention, the classification attributes response may specify one or more classification attributes associated with the data as requested in the classification attributes request.
In one or more embodiments of the invention, the classification attributes response includes a classification ID. The classification ID may be associated with a classification mapping entry.
In step 324, a classification mapping is updated based on the attributes. In one or more embodiments of the invention, the classification mapping is updated by generating a classification mapping entry that specifies a classification ID and the object ID corresponding to the object in specified in the write request in step 300. The classification mapping entry may further specify the classification attributes obtained in the classification attributes response.
In one or more embodiments of the invention, if the classification attributes response includes classification attributes that match classification attributes of an existing classification mapping entry, then the object ID is added to an existing classification mapping entry. In this manner, the classification mapping entry may be associated with multiple distinct objects.
In one or more embodiments of the invention, after the classification mapping is updated, the data classification engine may provide a classification ID associated with the object to the data processor to be used for updating object metadata based on the classification attributes update.
In step 340, a determination is made about whether classification attributes associated with the object need to be changed. If classification attributes associated with the object need to be changed, the method proceeds to step 342; otherwise, the method returns to step 340. In one embodiment of the invention, the determination in step 340 may be made based on one or more of the following scenarios: (a) a change in the law requires objects that were previously associated with a certain regulation in a geographic region to be updated to be associated with a new regulation in the geographic region; (b) a change in company policy of a company that owns an object requires a retention policy of four years for all of its objects. Other scenarios may trigger a change in classification attributes without departing from the invention.
In step 342, the classification mapping is updated. In one or more embodiments of the invention, the classification mapping is updated by analyzing the classification mapping to identify one or more classification mapping entries associated with the object ID. The classification mapping entry may be modified to reflect the changes to the classification attributes. A classification mapping entry may be modified by adding, removing, and/or changing a classification attribute of the classification mapping entry. For example, if a new regulation is established that requires the object to comply with a new set of standards, the classification mapping entry may be updated to specify the new regulation and to remove the prior regulation.
In one or more embodiments of the invention, if a classification mapping entry specifies multiple objects, and only one object is to be changed based on the determination of step 340, the classification mapping may be updated so that the object ID of the affected object is associated with a new classification mapping entry that specifies the new classification attributes of the affected object. The new classification mapping entry may include a new classification ID. Further, the object ID is removed from the classification mapping entry that specified the multiple objects.
In step 344, one or more object entries impacted by the update to the classification mapping are updated. In one or more embodiments of the invention, if the affected object is associated with a new classification mapping entry, the object entries are updated by updating the classification IDs of the object entries with the new classification ID. Further, though not shown in FIG. 3C, the storage metadata may also be updated to associate a new classification ID with each of the data chunks that are associated with the aforementioned object(s) (i.e., the object(s) associated with the updated object entry(ies)).
The following section describes an example. The example is not intended to limit the invention. The example is illustrated in
The data processor (412) uses object metadata (416) to determine if an object entry is associated with the object [2]. After determining that no object entry in the object metadata (416) is associated with the object, the data processor generates an object entry with an object ID associated with the object [3]. The data processor (412) further prompts a data classification engine (414) to perform a classification attributes update for classification attributes associated with the object.
The data classification engine (414), in response to the prompting, sends a classification attributes request to a host (400) to obtain one or more classification attributes associated with the object [4]. The data classification engine (414) subsequently obtains classification attributes in response from the host (400) [5]. The classification attributes specify that the object was generated in Europe (“EU”), specifically in the United Kingdom (“UK”), and it is to be stored, accessed, and/or used in compliance with a General Data Protection Regulation (“GDPR”).
The data classification engine (416) uses the obtained classification attributes to generate a classification mapping entry for a classification mapping (not shown) stored in the data classification engine (414) [6]. The classification mapping entry may specify a classification ID, the object ID associated with the object, and the obtained classification attributes “EU,” “UK,” and “GDPR”.
After the classification mapping is updated and a classification ID is generated, the data classification engine (414) sends the classification ID to the object metadata [7]. The classification ID is subsequently stored in the object entry associated with the object [8].
The data processor (412) performs the method of
The data chunks and the parity chunk are stored in the non-accelerator pool [10]. Specifically, each of the three data chunk and the parity chunk is stored in a unique fault domain. In other words, a first data chunk is stored in fault domain A (420A), a second data chunk is stored in fault domain B (420B), a third data chunk is stored in fault domain C (420C), and the parity chunk is stored in fault domain D (420D).
In addition to storing the data chunks and the parity chunks, the data processor generates a storage metadata slice entry in storage metadata (not shown) and an object entry in object metadata (416). A unique identifier of each deduplicated data chunk and parity chunk are stored in the storage metadata slice entry and in the object entry.
End of Example
As discussed above, embodiments of the invention may be implemented using computing devices.
In one embodiment of the invention, the computer processor(s) (502) may be an integrated circuit for processing instructions. For example, the computer processor(s) may be one or more cores or micro-cores of a processor. The computing device (500) may also include one or more input devices (510), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. Further, the communication interface (512) may include an integrated circuit for connecting the computing device (500) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and/or to another device, such as another computing device.
In one embodiment of the invention, the computing device (500) may include one or more output devices (508), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device), a printer, external storage, or any other output device. One or more of the output devices may be the same or different from the input device(s). The input and output device(s) may be locally or remotely connected to the computer processor(s) (502), non-persistent storage (504), and persistent storage (506). Many different types of computing devices exist, and the aforementioned input and output device(s) may take other forms.
One or more embodiments of the invention may be implemented using instructions executed by one or more processors of the data management device. Further, such instructions may correspond to computer readable instructions that are stored on one or more non-transitory computer readable mediums.
One or more embodiments of the invention may improve the operation of one or more computing devices. More specifically, embodiments of the invention improve the efficiency of storing data that is in compliance with one or more regulations. Embodiments of the invention improve efficiency by storing a classification mapping that specifies the data, a geolocation associated with the data, and the regulations that are to be followed by the entities storing the data. The classification mapping may be updated to keep up with changes to real-world regulations that specify a set of standards for using, accessing, and/or storing data. Embodiments of the invention provide a data cluster with a way to effectively track data to ensure it is in compliance with the set of standards.
While the invention has been described above with respect to a limited number of embodiments, those skilled in the art, having the benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4780809 | Woffinden | Oct 1988 | A |
| 5689678 | Stallmo et al. | Nov 1997 | A |
| 6098098 | Sandahi et al. | Aug 2000 | A |
| 6223252 | Bandera et al. | Apr 2001 | B1 |
| 6516425 | Belhadj et al. | Feb 2003 | B1 |
| 7636814 | Karr | Dec 2009 | B1 |
| 7882386 | Potnis et al. | Feb 2011 | B1 |
| 7987353 | Holdaway et al. | Jul 2011 | B2 |
| 8037345 | Iyer et al. | Oct 2011 | B1 |
| 8099571 | Driscoll et al. | Jan 2012 | B1 |
| 8161255 | Anglin et al. | Apr 2012 | B2 |
| 8190835 | Yueh | May 2012 | B1 |
| 8364917 | Bricker et al. | Jan 2013 | B2 |
| 8386930 | Dillenberger et al. | Feb 2013 | B2 |
| 8583769 | Peters et al. | Nov 2013 | B1 |
| 8788466 | Anglin | Jul 2014 | B2 |
| 8868987 | Wagner | Oct 2014 | B2 |
| 8874892 | Chan et al. | Oct 2014 | B1 |
| 8898114 | Feathergill et al. | Nov 2014 | B1 |
| 8949208 | Xu et al. | Feb 2015 | B1 |
| 9122501 | Hsu et al. | Sep 2015 | B1 |
| 9201751 | Muthirisavenugopal et al. | Dec 2015 | B1 |
| 9256381 | Fultz et al. | Feb 2016 | B1 |
| 9278481 | Hull | Mar 2016 | B2 |
| 9355036 | Beard et al. | May 2016 | B2 |
| 9521198 | Agarwala et al. | Dec 2016 | B1 |
| 9710367 | Nagineni | Jul 2017 | B1 |
| 9749480 | Katano | Aug 2017 | B1 |
| 9830111 | Patiejunas et al. | Nov 2017 | B1 |
| 9898224 | Marshak et al. | Feb 2018 | B1 |
| 10002048 | Chennamsetty et al. | Jun 2018 | B2 |
| 10013323 | Puhov et al. | Jul 2018 | B1 |
| 10013325 | Garrett, Jr. et al. | Jul 2018 | B1 |
| 10031672 | Wang et al. | Jul 2018 | B2 |
| 10057184 | Prahlad et al. | Aug 2018 | B1 |
| 10091295 | Savic et al. | Oct 2018 | B1 |
| 10097620 | Reddy et al. | Oct 2018 | B2 |
| 10152254 | Kang et al. | Dec 2018 | B1 |
| 10185624 | Akutsu | Jan 2019 | B2 |
| 10241695 | Baptist | Mar 2019 | B2 |
| 10339455 | Parush-Tzur et al. | Jul 2019 | B1 |
| 10409778 | Zhao et al. | Sep 2019 | B1 |
| 10452301 | Farhan | Oct 2019 | B1 |
| 10503413 | Gal et al. | Dec 2019 | B1 |
| 10503611 | Srivastav | Dec 2019 | B1 |
| 10528429 | Vempati et al. | Jan 2020 | B1 |
| 10817392 | Mcauliffe et al. | Oct 2020 | B1 |
| 10929256 | Danilov et al. | Feb 2021 | B2 |
| 10956601 | Surla et al. | Mar 2021 | B2 |
| 10963345 | Patel et al. | Mar 2021 | B2 |
| 10990480 | Bernat et al. | Apr 2021 | B1 |
| 11005468 | Subramanian et al. | May 2021 | B1 |
| 11112990 | Bernat et al. | Sep 2021 | B1 |
| 20010044879 | Moulton | Nov 2001 | A1 |
| 20030036882 | Harper et al. | Feb 2003 | A1 |
| 20030058277 | Bowman-Amuah | Mar 2003 | A1 |
| 20030065873 | Collins et al. | Apr 2003 | A1 |
| 20030172145 | Nguyen | Sep 2003 | A1 |
| 20030177205 | Liang et al. | Sep 2003 | A1 |
| 20040128587 | Kenchammana-hosekote | Jul 2004 | A1 |
| 20040153844 | Ghose et al. | Aug 2004 | A1 |
| 20040260967 | Guha et al. | Dec 2004 | A1 |
| 20050076049 | Qubti et al. | Apr 2005 | A1 |
| 20050086557 | Sato et al. | Apr 2005 | A1 |
| 20050182797 | Adkins et al. | Aug 2005 | A1 |
| 20050262385 | Mcneill et al. | Nov 2005 | A1 |
| 20050283655 | Ashmore | Dec 2005 | A1 |
| 20070214255 | Spitz et al. | Sep 2007 | A1 |
| 20070283011 | Rakowski et al. | Dec 2007 | A1 |
| 20080244204 | Cremelie et al. | Oct 2008 | A1 |
| 20080282105 | Deenadhayalan et al. | Nov 2008 | A1 |
| 20090077165 | Rhodes et al. | Mar 2009 | A1 |
| 20090094250 | Dhuse | Apr 2009 | A1 |
| 20090235022 | Bates et al. | Sep 2009 | A1 |
| 20090254572 | Redlich et al. | Oct 2009 | A1 |
| 20090265360 | Bachwani | Oct 2009 | A1 |
| 20100061207 | Trantham | Mar 2010 | A1 |
| 20100138604 | Noguchi et al. | Jun 2010 | A1 |
| 20100241616 | Murphy | Sep 2010 | A1 |
| 20110099351 | Condict | Apr 2011 | A1 |
| 20110173484 | Schuette et al. | Jul 2011 | A1 |
| 20110197024 | Thomas | Aug 2011 | A1 |
| 20110246597 | Swanson et al. | Oct 2011 | A1 |
| 20110302446 | Becker-szendy et al. | Dec 2011 | A1 |
| 20120096309 | Kumar et al. | Apr 2012 | A1 |
| 20130036277 | Szczepkowski et al. | Feb 2013 | A1 |
| 20130047029 | Ikeuchi et al. | Feb 2013 | A1 |
| 20130067459 | Sannidhanam et al. | Mar 2013 | A1 |
| 20130086006 | Colgrove | Apr 2013 | A1 |
| 20130151683 | Jain et al. | Jun 2013 | A1 |
| 20130339818 | Baker et al. | Dec 2013 | A1 |
| 20140020083 | Fetik | Jan 2014 | A1 |
| 20140032834 | Cudak et al. | Jan 2014 | A1 |
| 20140059311 | Oberhofer et al. | Feb 2014 | A1 |
| 20140089265 | Talagala et al. | Mar 2014 | A1 |
| 20140089630 | Pignatelli | Mar 2014 | A1 |
| 20140282824 | Lango et al. | Sep 2014 | A1 |
| 20140285917 | Cudak et al. | Sep 2014 | A1 |
| 20140297603 | Kim et al. | Oct 2014 | A1 |
| 20150012775 | Cudak et al. | Jan 2015 | A1 |
| 20150046756 | Sreekumaran et al. | Feb 2015 | A1 |
| 20150058582 | Baldwin et al. | Feb 2015 | A1 |
| 20150095596 | Yang | Apr 2015 | A1 |
| 20150161000 | Kim et al. | Jun 2015 | A1 |
| 20150205657 | Clark | Jul 2015 | A1 |
| 20150205669 | Sundaram et al. | Jul 2015 | A1 |
| 20150220400 | Resch | Aug 2015 | A1 |
| 20150227602 | Ramu et al. | Aug 2015 | A1 |
| 20150286545 | Brown | Oct 2015 | A1 |
| 20150355980 | Volvovski | Dec 2015 | A1 |
| 20150379430 | Dirac et al. | Dec 2015 | A1 |
| 20160013988 | Andrews | Jan 2016 | A1 |
| 20160057226 | Bestler | Feb 2016 | A1 |
| 20160062674 | Benight et al. | Mar 2016 | A1 |
| 20160070652 | Sundararaman et al. | Mar 2016 | A1 |
| 20160077933 | Ventura et al. | Mar 2016 | A1 |
| 20160085630 | Gardner | Mar 2016 | A1 |
| 20160170871 | Hyun | Jun 2016 | A1 |
| 20160196324 | Haviv | Jul 2016 | A1 |
| 20160246537 | Kim | Aug 2016 | A1 |
| 20160313916 | Sivananainthaperumal | Oct 2016 | A1 |
| 20160334998 | George et al. | Nov 2016 | A1 |
| 20170031761 | Horan et al. | Feb 2017 | A1 |
| 20170090767 | Poston et al. | Mar 2017 | A1 |
| 20170099187 | Dale et al. | Apr 2017 | A1 |
| 20170147437 | Borlick et al. | May 2017 | A1 |
| 20170160983 | Fiske et al. | Jun 2017 | A1 |
| 20170192865 | Pan | Jul 2017 | A1 |
| 20170192868 | Vijayan et al. | Jul 2017 | A1 |
| 20170199893 | Aronovich et al. | Jul 2017 | A1 |
| 20170201545 | Nicodemus et al. | Jul 2017 | A1 |
| 20170206034 | Fetik | Jul 2017 | A1 |
| 20170235609 | Wires et al. | Aug 2017 | A1 |
| 20170262185 | Long | Sep 2017 | A1 |
| 20170272209 | Yanovsky et al. | Sep 2017 | A1 |
| 20170308436 | Agombar et al. | Oct 2017 | A1 |
| 20170308536 | Azzam et al. | Oct 2017 | A1 |
| 20180018227 | Kazi | Jan 2018 | A1 |
| 20180018235 | Arslan | Jan 2018 | A1 |
| 20180060894 | Beveridge et al. | Mar 2018 | A1 |
| 20180101305 | Kazi et al. | Apr 2018 | A1 |
| 20180157532 | Kumar et al. | Jun 2018 | A1 |
| 20180165169 | Camp et al. | Jun 2018 | A1 |
| 20180173730 | Copenhaver | Jun 2018 | A1 |
| 20180189109 | Nagai et al. | Jul 2018 | A1 |
| 20180205785 | Caulfield et al. | Jul 2018 | A1 |
| 20180217898 | Tormasov | Aug 2018 | A1 |
| 20180260123 | Andresen et al. | Sep 2018 | A1 |
| 20180278597 | Helms et al. | Sep 2018 | A1 |
| 20180284987 | Lazier | Oct 2018 | A1 |
| 20180307560 | Vishnumolakala et al. | Oct 2018 | A1 |
| 20180314607 | Deshpande et al. | Nov 2018 | A1 |
| 20180322558 | Padmanabh et al. | Nov 2018 | A1 |
| 20180356998 | Wu et al. | Dec 2018 | A1 |
| 20190050263 | Patel et al. | Feb 2019 | A1 |
| 20190104398 | Owen et al. | Apr 2019 | A1 |
| 20190108099 | Mazumdar | Apr 2019 | A1 |
| 20190109772 | Lipstone et al. | Apr 2019 | A1 |
| 20190188079 | Kohli | Jun 2019 | A1 |
| 20190197023 | Chennamsetty et al. | Jun 2019 | A1 |
| 20190332473 | Yang et al. | Oct 2019 | A1 |
| 20190332502 | Ma et al. | Oct 2019 | A1 |
| 20190361850 | Uno et al. | Nov 2019 | A1 |
| 20190379588 | Rao | Dec 2019 | A1 |
| 20200026439 | Gao et al. | Jan 2020 | A1 |
| 20200034257 | Mahmood et al. | Jan 2020 | A1 |
| 20200042388 | Roberts | Feb 2020 | A1 |
| 20200050689 | Tal et al. | Feb 2020 | A1 |
| 20200065191 | Zhou | Feb 2020 | A1 |
| 20200133503 | Sun et al. | Apr 2020 | A1 |
| 20200175038 | Guan et al. | Jun 2020 | A1 |
| 20200201837 | Motwani et al. | Jun 2020 | A1 |
| 20200257457 | Matsugami et al. | Aug 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 2015114643 | Aug 2015 | WO |
| Entry |
|---|
| Extended European Search Report issued in corresponding European Application No. 20152195.2, dated Jun. 2, 2020. |
| Suzhen Wu et al. ;“Improving Reliability of Deduplication-based Storage Systems with Per-File Parity”; 2018 IEEE 37th International Symposium on Reliable Distributed Systems (SRDS); Salvador, Brazil; 2018; pp. 171-180 (DOI:10.1109/SRDS.2018.00028). |
| “Features—RoboHead”; Project Management Software for Marketing & Creative Teams, Aquent; 2018 (https://www.robohead.net/features). |
| “Make more time for the work that matters most”; Asana, Inc.; 2017 (https://asana.com/). |
| “Online Project Management Tools & Features”; ProWorkflow; Jun. 2015 (https://www.proworkflow.com/features-project-management-tools/). |
| Number | Date | Country | |
|---|---|---|---|
| 20210034264 A1 | Feb 2021 | US |
