Patent Application
20240045872
Data processing and protection have undergone transformational change with the increased availability of inexpensive processors and storage media. Users now have the option to process and store their data locally, or to store their data on servers connected over a network, in computing clusters, or in the cloud. In addition, cloud computing options include both public cloud and private cloud offerings.
With the era of big data upon us, users wish to store and process ever more voluminous data objects. For example, it is not uncommon for tabular data, tree-based data, and audio and/or video data to reach sizes in the gigabyte range or above. Processing, protecting, and storing such large data objects presents unique challenges.
A common approach is to divide a large object into separate portions and to store the portions on respective computers. Programs may divide an object by identifying byte boundaries in the object and producing portions of equal size, or nearly so. To perform data processing on a data object once it has been stored in a distributed manner, a computer may gather particular portions or groups of portions of the original object, perform desired processing tasks on the gathered portions, and generate results.
Unfortunately, the above-described distributed approach can be inefficient. For example, the practice of dividing large data objects into equal or nearly equal portions can ignore structural features and can introduce dependencies between or among different data portions. As a simple example, consider a data object containing many rows of tabular data. Dividing the object to form equal-sized portions may mean cutting off a row in the middle. Any subsequent query that involves access to the cut-off row may thus require access to two portions of the data object, one that stores the beginning of the row and one that stores the end. The two portions may typically be stored on different computers on a network.
Continuing with the above example, it may further be necessary to transfer both portions (containing both parts of the cut-off row) back to the requester or to some other node, where the portions are reassembled and a query is performed. These acts introduce large inefficiencies as they involve large copies of data over the network.
In addition to the above, the prior approach may be oblivious to content. For example, a split-off portion of a data object may lose its association with the data object as a whole. Field names may be missing for tabular data (e.g., if only row data are stored). Extracting meaningful data from a distributed object may thus involve directing many network accesses to different computers, in an effort to collect all the pieces needed to complete a desired processing task.
To address these deficiencies, a technique for managing data objects in a storage cluster includes splitting a data object into multiple portions at boundaries within the data object. The technique further includes transforming the portions of the data object into segments that provide individually processable units, and distributing the segments among multiple computing nodes of the storage cluster for storage therein
Advantageously, providing segments as individually-processable units means that the workload associated with performing a processing task on the data object can be pushed down efficiently to the computing nodes that store the segments of the data object locally. The technique thus enables true parallel processing, with each computing node performing the processing task on only the segment or segments of the data object stored therein. It also greatly reduces network traffic as compared with prior schemes. For example, high-speed connections of computing nodes to their local storage greatly enhances overall efficiency. Further, the independent nature of segments means that little or no communication is required among computing nodes (e.g., to resolve dependencies) in order to complete a processing task.
Particular challenges arise when partitioning and processing multi-dimensional data. First, such data can be voluminous, with datasets commonly being in the terabyte range, with some extending into the petabyte range. Consider weather data, for example, where datasets track temperature, humidity, air pressure, wind speed, precipitation, and other factors. The sizes of these datasets can grow exceedingly large, particularly when taken over large geographical regions. Second, multi-dimensional data is not naturally amenable to being divided for parallel processing. Although various software programs exist which support data extraction, extracted data represent only parts of the original datasets and are not reflective of the datasets in their entirety. What is needed, therefore, is a way of rendering large datasets as smaller segments that can be processed independently and in parallel to produce information reflective of the datasets as a whole.
To address this need at least in part, an improved technique of managing multi-dimensional data includes providing an original dataset containing data arranged along multiple dimensions, each dimension covering a respective original range of dimensional units. The technique further includes extracting multiple portions of data from the original dataset, each portion extending over a reduced range of dimensional units, smaller than the original range, in at least one dimension, and all extracted portions together covering the original ranges of the original dataset in all dimensions.
Advantageously, the improved technique enables a large dataset to be rendered in multiple smaller portions, which together contain the same data as the original dataset but which are independently much more efficient to process. In some examples, portions may be stored in respective computing nodes in a storage cluster and processed in parallel. Results of parallel processing may be aggregated to produce meaningful results that are reflective of the original dataset as a whole, but at much higher efficiency than could be realized by processing the original dataset as one large object.
Certain embodiments are directed to a method of managing multi-dimensional data. The method includes providing an original dataset that contains data arranged along multiple dimensions of an N-dimensional space. Each dimension of the N-dimensional space has a respective original range of dimensional units, and the original dataset has a data format. The method further includes extracting multiple portions of data from the original dataset. Each portion extends over a reduced range of dimensional units in at least one dimension of the N-dimensional space. The extracted portions together cover all original ranges of the N-dimensional space. The method still further includes rendering the extracted portions in respective segments that provide data of the extracted portions in the same data format as the original dataset.
Additional embodiments are directed to a computerized apparatus constructed and arranged to perform a method of managing multi-dimensional data, such as the method described above. Still other embodiments are directed to a computer program product. The computer program product stores instructions which, when executed on control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of managing multi-dimensional data, such as the method described above.
The foregoing summary is presented for illustrative purposes to assist the reader in readily grasping example features presented herein; however, this summary is not intended to set forth required elements or to limit embodiments hereof in any way. One should appreciate that the above-described features can be combined in any manner that makes technological sense, and that all such combinations are intended to be disclosed herein, regardless of whether such combinations are identified explicitly or not.
The foregoing and other features and advantages will be apparent from the following description of particular embodiments, as illustrated in the accompanying drawings, in which like reference characters refer to the same or similar parts throughout the different views.
Embodiments of the improved technique will now be described. One should appreciate that such embodiments are provided by way of example to illustrate certain features and principles but are not intended to be limiting.
A technique for managing data objects in a storage cluster includes splitting a data object into multiple portions at boundaries within the data object. The technique further includes transforming the portions of the data object into segments that provide individually processable units, and distributing the segments among multiple computing nodes of the storage cluster for storage therein.
In the following description:
This application discloses multiple embodiments. One embodiment is directed to splitting a data object into portions for distributed storage in the storage cluster. Another embodiment is directed to performing a distributed processing task by the storage cluster. Yet another embodiment is directed to protecting data of a data object stored in a storage cluster. These embodiments may be realized as respective aspects of a single system, as shown and described in the examples that follow. Alternatively, embodiments may be practiced independently, such that an implementation supporting any one of the embodiments need not also support the other embodiments.
Preferably, each node 120 has one or more high-speed connections to its respective persistent storage. For example, connections between nodes 120 and their storage devices (e.g., SSDs) may have bandwidths that exceed those of connections between nodes over network 140 by an order of magnitude or more.
In an example, the storage cluster 130 is configured as an object store, which may be compatible with commercially-available cloud-based object stores, such as AWS (Amazon Web Services) S3 (Simple Storage Service), Microsoft Azure Data Lake, and/or Google Cloud Storage. In a particular example, the storage cluster 130 is configured as an S3-compatible object store. To this end, each node 120 may include an API (application program interface) 122 that enables the node 120 to participate as a member of the object store.
The cluster 130 may be implemented in a data center, which may occupy a room or multiple rooms of a building, in which the nodes 120 are networked together. Other implementations may span multiple buildings, and metro-cluster arrangements are feasible.
In other examples, the storage cluster 130 may be implemented within a cloud service 150, e.g., using physical or virtual machines provided therein. For instance, the entire storage cluster 130 may be disposed entirely within the cloud service 150.
As yet another example, the cloud service 150 may act as a primary repository of data, with the storage cluster 130 acting as a cache for the cloud service 150. The storage cluster 130 may thus store commonly accessed data but typically not all data available from the cloud service 150.
Implementations may be suitable for individuals, small organizations, and/or enterprises, and may be delivered according to a SaaS (software as a service) model or according to other models. Embodiments are particularly suitable for managing large data objects, which may have sizes in the hundred-megabyte range or above. This feature makes embodiments a good match for big data applications, such as those involving data lakes. One should appreciate, though, that embodiments are not limited to any particular users, service model, data size, or application.
In example operation, gateway 110 (which may be part of the storage cluster 130 or separate therefrom) accesses one or more data objects 160 to be managed by the storage cluster 130. The data objects 160 may reside in the cloud service 150, e.g., within buckets or blobs, or they may be provided by one or more separate sources. For example, data objects 160 may be generated by real-time activities, such as industrial or scientific processes which may produce the data objects 160 as data logs or other records of ongoing activities. The data objects 160 may be presented as files, streams, memory ranges, or in any other manner.
The data objects 160 may be structured in accordance with particular object types. For example, data objects 160 may be provided as tabular objects such as CSV (comma-separated values) or log files, as tree-based objects such as JSON (JavaScript Object Notation) or XML (extensible markup language) documents, as column-oriented objects such as Apache Parquet files, as video files or streams, as audio files or streams, or as collections of pictures, for example. Although certain types of data are particularly shown and/or described, one should appreciate that embodiments are intended to encompass any type of data, with the ones shown and/or described merely providing concrete examples used to illustrate operating principles.
To initiate management of a data object 160, gateway 110 may scan the data object, e.g., starting from the beginning of the data object and proceeding forward. Normally, the gateway 110 may be oblivious to the data object's type when it first accesses the object and may perform an initial scan of the object 160 to identify its type. The scan may involve sampling a set of regions of the data object, typically at the beginning of the object, and searching for sequences or characters that are specific to particular object types. For instance, CSV and log files typically use NewLine characters to denote ends of records, and may use commas, spaces, or other characters to separate adjacent fields. Some data objects may include headers that directly identify the type of object. For example, Parquet files start with a 4-byte header that designates a so-called “magic number,” which provides the code “PAR1” to identify the file as a Parquet file. Most file types provide clear indications that enable them to be identified without much effort. Some types may be harder to identify. Should one wish to recognize such less-easily identifiable types, more advanced algorithms may be applied, which may include machine learning or other types of artificial intelligence.
Once the gateway 110 has identified the type of the data object 160, the gateway 110 may proceed to start splitting the data object 160 into portions. For example, gateway 110 may search for boundaries in the data object that provide separators between adjacent processable units of the data object. The exact nature of the boundaries may vary from one object type to another. For example, CSV files may use NewLine characters to identify boundaries, whereas video files or streams may use I-frames (intra-coded pictures). Some object types specify boundaries using embedded metadata. For instance, Parquet files contain footers that identify boundaries between adjacent row groups.
The “processable units” of a data object are regions which are amenable to independent processing, in the sense that they contain few if any dependencies on other processable units. Splitting a data object into processable units thus promotes efficient parallel processing by nodes 120 of the storage cluster 130.
Although splitting is a first step in promoting independent processing of split-off portions, it is not always sufficient for optimal performance. For example, split-off portions may lack certain metadata (e.g., headers, footers, or other content) that cause them to retain dependencies on other parts of the data object 160. Thus, the gateway 110 preferably performs an additional step of transforming the split-off portions into segments 170. In an example, the transformed segments 170 can be processed as if they were complete, self-contained objects of the same type as the data object 160.
The segments 170 are similar to the portions from which they were created, but they are adjusted to reduce or eliminate dependencies on other portions. For example, if the first portion of a CSV file contains a header but subsequent portions do not, then the gateway 110 may copy the header of the first portion to each of the segments 170 that are formed from the subsequent portions. In this manner, each segment 170 has its own header and can be processed as if it were an independent CSV file. Corresponding adjustments may be performed for other object types, with the particulars of the adjustments depending on the object type. Various examples are provided below.
With the segments 170 thus formed as independently-processable units of the same type as the data object 160, gateway 110 may distribute the segments 170 to various nodes 120 of the storage cluster 130, which nodes 120 store the segments therein, e.g., in persistent storage locally connected to the respective nodes 120. To keep track of segment locations, gateway 110 may update object metadata 112.
As shown in an expanded view of
In some examples, object metadata 112 is stored redundantly to promote reliability. For instance, object metadata 112 may be stored on multiple nodes 120 of the storage cluster 130, e.g., using a multi-way mirror and/or other RAID (Redundant Array of Independent Disks) or erasure-coding techniques. Also, activities attributed herein to the gateway 110 may be performed by any number of computers, and such computers may include nodes 120 of the storage cluster 130. For example, a particular node of the storage cluster 130 may be designated as a load balancer and may take the workload of nodes 120 into account when segments 170 are distributed among nodes of the cluster.
As still further shown in
As with object metadata 112, segment metadata 124 may also be stored redundantly to promote reliability. In some examples, nodes 120 may store segment metadata 124 along with the segments 170 that the metadata describe. For example, segment metadata for segment A may be stored with Segment A Likewise, segment metadata for segment B may be stored with Segment B. Segment metadata 124 may then be protected in the same ways that the segments 170 themselves are protected. Various examples of segment protection are described hereinbelow.
As shown, the gateway 110 includes a type detector 210, a splitter 220, a transformer 230, and a distributer 240. The type detector 210 performs the function of reading a set of regions of a data object 160, e.g., by sampling bytes at the beginning of the object, and identifying the object type of the data object 160 based on the sampling. The type detector 210 may inform the splitter 220 and the transformer 230 of the determined object type.
Splitter 220 performs the function of splitting the data object 160 into portions 250. The portions 250 include respective processable units of the data object 160 and are defined by boundaries 252 in the data object. A boundary detector 222 of the splitter 220 scans the data object 160 for boundaries 252, i.e., separators between the processable units, and notes the locations of the boundaries 252 relative to the data object 160 (e.g., based on byte locations). As mentioned earlier, the nature of the boundaries 252 depends upon the object type of the data object 160, which is preferably known based on operation of the type detector 210.
In some examples, such as when splitting Parquet files, the boundary detector 222 may identify every boundary 252 in the data object 160 and define a new portion 250 between each pair of boundaries. Detecting every boundary works well for Parquet files, where boundaries 252 are based on row groups, which tend to be large (e.g., in the megabyte range). If a row group is found to be unusually small, however, then a boundary may be skipped, such that multiple row groups may be included within a single portion 250. In other examples, such as when splitting CSV files, boundary detector 222 does not mark every single boundary of the data object 160, as doing so would produce an undesirably large number of small portions 250. In such cases, boundary detector 222 may wait to start detecting boundaries 252 when scanning a current portion 250 until the scanned size of the portion 250 exceeds some desired target size. Once the scan passes the target size, the boundary detector 222 may start detecting boundaries, preferably identifying the first boundary that the object contains beyond the target size. The current portion may thus end and a new portion may begin at the first detected boundary.
As the boundary detector 222 scans the object 160 for boundaries 252, a feature detector 224 may scan the object for additional features that may provide helpful information relevant to later processing. It is recognized that certain processing tasks run faster if it is known in advance that certain content is present or absent. As a particular example, certain queries of CSV files run more quickly if it is known in advance that there are no quotation marks in the data. Feature detector 224 may thus check CSV files for the presence or absence of quotation marks and update the object metadata 112 (“Features”) accordingly.
With portions 250 of the data object 160 identified based on boundaries 252, transformer 230 transforms the portions 250 into respective segments 170. For example, transformer 230 modifies at least some of the portions 250 by adding metadata found in some portions to one or more other portions, so as to make such portions more amenable to independent processing, i.e., by removing dependencies between portions 250. The nature of the adjustments depends on the object type, which is known based on operation of the type detector 210. The results of operation of transformer 230 are segments 170, which provide individually processable units of the data object. For example, each of the segments 170 is rendered as the same object type as the data object 160. The segments 170 can thus be processed the same way that data objects can be processed, with the primary difference being that segments 170 are much smaller and more easily handled.
Distributor 240 then distributes the segments 170 to selected nodes 120 of the storage cluster 130 for storage in such nodes. At this time, gateway 110 updates object metadata 112 to record the locations to which the segments 170 are sent, e.g., the identities of particular nodes 120. In the manner described, the data object 160 is thus split, transformed, and distributed among nodes 120 of the storage cluster 130.
As shown in
When splitting the data object 160a, the splitter 220 may apply a target size 320, which defines a minimum size for portions 350 of the data object 160a. For example, the splitter 220 may identify a location (shown as a dotted line) along the data object 160a that corresponds to the target size 320, and then split the data object 160a at the first boundary that follows the identified location. In the example shown, the splitter 220 detects the NewLine character at the end of the sixth row as a first boundary 252 following the target size 320, and splits the object 160a at this location. As a result, the first six rows of object 160a form a first portion 350a, and the next two rows form the first two rows of a second portion 350b. Additional rows may be added to the second portion 350b as the splitter 220 continues to scan the object 160a.
Even though the splitter 220 has successfully separated the object 160a at a row boundary (thus avoiding having different parts of the same row assigned to different portions 350), the result of splitting may still be inefficient. For example, if the first row 310 of object 160a is a header row (e.g., a row that contains text indicating column names), then the second portion 350b would lack that header and its later processing might be compromised. For example, the header may be required for responding to certain queries or other activities. This deficiency may be addressed by transformer 230, however.
It is noted that some CSV files do not use header rows, such that the first row 310 may contain data, rather than text-based field names. In such cases, replication of the first row 310 of the first segment 370a to other segments 370 of object 160a may merely propagate redundant data. Such cases can be handled easily, however. For instance, queries or other processing tasks (e.g., arriving from clients of the storage cluster) may specify whether the CSV file represented by object 160a contains a header. If it does, then no change needs to be made, as copying the header was proper. But if the task specifies that the CSV file contains no header, then the copying turns out to have been unnecessary. In such cases, the nodes 120 that perform the distributed processing task on the CSV file may be directed simply to ignore the first row of all but the first segment 370a of segments 370. Little will have been lost as a result of copying the first row 310, which is typically negligible in size compared with that of a segment 370.
As seen in
Unlike the CSV example, where boundaries 252 may be detected directly while scanning forward through an object, boundaries between row groups 410 can be detected easily only by reading the footer 420. This means that splitter 220 typically makes a pass through the entire file 160a before reaching the footer 420, and then splits retrospectively. Splitting is generally performed at every row-group boundary, such that each portion 260 of the Parquet file 160b is made to contain a single row group 410. Given that row groups 410 may vary in size based on content, it may occasionally be worthwhile to place two or more row groups 410 into a single portion 260. This is a matter of design preference.
As shown in
The modifications shown in
In some examples, an additional segment 470-(N+1) may be provided as a final segment of the Parquet file 160b. Segment 470-(N+1) contains no row groups but rather provides a persisted version of parts of the original footer 420 of file 160b, i.e., the “File Metadata (for all Row Groups)” and the “Length of File Metadata.” This segment is provided for reference and may be useful for speeding up certain processing tasks, but it is not intended to be treated as a self-contained Parquet file. Nor is it intended to be used as a source of data when performing queries.
As seen in
Splitting video data in object 160c works much like splitting CSV data in object 160a (
Splitting the video just before I-frame 510c creates a problem, however, as B-frame 510b references I-frame 510c and thus cannot be rendered without it. If splitter 220 were to split the video immediately after B-frame 510b, then a gap in the video would appear in the segment that contains B-frame 510b. That segment would thus be incomplete, as it would have a dependency on another segment.
As shown in
In example operation, the task requestor 610 initiates a request 650 for performing a processing task on a specified data object 160 (or set of objects 160). Various types of tasks are contemplated. These may include, for example, reads and/or queries of specified data (e.g., for tabular or tree-based data objects). Types of queries may include SQL (Simple Query Language) queries, key-value lookups, noSQL queries, and the like. Tasks for video data objects may include distributed video-processing tasks, such as searches for specified graphical content (e.g., faces, license plates, geographical features, and the like). Tasks for audio data objects may include searches for spoken words, voice characteristics (e.g., tone, accent, pitch, etc.), particular sounds, or the like. Essentially, any task that is amenable to splitting among multiple nodes 120 and involves access to potentially large amounts of data is a good candidate for processing in the arrangement of
Upon issuance of the request 650, dispatcher 620 begins distributing components of the requested task to the respective nodes 120. For example, dispatcher 620 checks object metadata 112 to identify segments 170 of the specified data object 160 (or set of objects) and their respective locations in the storage cluster 130. In the simplified example shown, the object metadata 112 identifies three segments 170 (e.g., S1, S2, and S3), which make up the data object 160 (typical results may include tens or hundreds of segments) and three computing nodes 120-1, 120-2, and 120-3 that store the respective segments 170.
Dispatcher 620 then transmits requests 650-1, 650-2, and 650-3 to the identified nodes 120-1, 120-2, and 120-3, respectively. Requests 650-1, 650-2, and 650-3 may be similar or identical to request 650, e.g., they may provide the same query or other task as specified in request 650. Such requests 650-1, 650-2, and 650-3 need not be identical to one another, however. For example, some requests may include segment-specific metadata (e.g., stored in object metadata 112) that differs from that sent in other requests, and which may be used to guide a processing task on a particular node.
The identified nodes 120-1, 120-2, and 120-3 receive the requests 650-1, 650-2, and 650-3, respectively, and each of these nodes begins executing the requested task on its respective segment. For example, node 120-1 executes the task on segment S1, node 120-2 executes the task on segment S2, and node 120-3 executes the task on segment S3. In an example, each node 120 independently executes its respective task on its respective segment 170, without needing to contact any other node 120. For instance, node 120-1 completes its work by accessing only S1, without requiring access to S2 or S3. Likewise for the other nodes.
As the nodes 120-1, 120-2, and 120-3 perform their respective work, such nodes produce respective output 660, shown as output 660-1 from node 120-1, output 660-2 from node 120-2, and output 660-3 from node 120-3. The participating nodes send their respective output 660 back to the gateway 110, which collects the output in output receiver 630.
As shown in the expanded view near the bottom of
In a second scenario, nodes 120-1, 120-2, and 120-3 are configured to return their output in increments, such as immediately upon such increments becoming available. In this second scenario, each participating node may return its output 660 in multiple transmissions, which may be spread out over time. Output data 664 shows example results according to this scenario. Here, output data 664 is seen to include six different batches (660-1a, 660-1b, 660-2a, 660-2b, 660-3a, and 660-3b), i.e., two batches of output from each of nodes 120-1, 120-2, and 120-3. The batches appear in output data 664 in the order received, which thus may be interleaved at finer granularity than was seen in the first scenario.
Of course, gateway 110 may sort the output 660 in any desired manner, and any node 120 of the storage cluster 130 may be called upon to perform this task. In some examples, both the affected nodes and the gateway 110 may participate in sorting the output 660. For example, each of the nodes may sort its respective output, such that each of the results 660-1, 660-2, or 660-3 arrives individually in sorted order. The gateway 110 may then complete the work, e.g., by employing the aggregator 640 for sorting among the sorted sets of returned results.
Sorting takes time, and many processing tasks value speed more highly than sorted output. To further promote high-speed operation, the computing nodes 120 may in some examples employ RDMA (remote direct memory access) when returning output 660 to the gateway 110.
For some processing tasks, dispatcher 620 may send processing requests to all involved nodes (i.e., to all nodes that store segments of the subject data object). In other examples, dispatcher 620 may limit the nodes to which requests are sent, e.g., based on knowledge of a priori segment contents, byte ranges of segments, or other factors. Limiting the number of involved nodes in this manner helps to reduce traffic over the network 140 (
Some processing tasks may involve aggregation. For example, a query may request a count of records that meet specified criteria, rather than the records themselves. A query may also request an average value, a maximum value, a minimum value, or some other aggregate value. Nodes 120 may perform certain aggregate functions themselves (e.g., count, total, max, min, etc.), but individual nodes 120 do not typically aggregate output across multiple nodes. Rather, this function may be performed by the data aggregator 640. For example, aggregator 640 may receive counts from multiple nodes, with each providing partial aggregate results derived from its processing on a respective segment. Aggregator 640 may then sum the counts from the responding nodes to produce an aggregate total for the entire data object 160. To produce an aggregated average for a data object, for example, aggregator 640 may direct each participating node to provide both a count and a total. It may then sum all counts returned to produce an aggregate count, sum all totals to produce an aggregate total, and then divide the aggregate total by the aggregate count to produce the desired aggregate average. Other types of aggregate functions may be performed in a similar way.
One should appreciate that the arrangement of
Although the gateway 110 has been shown and described as the originator of task requests 650, as the dispatcher of requests to affected nodes, and as the collector of output 660 from the nodes, these functions may alternatively be performed by other computers, or by multiple computers. Indeed, they may be performed by one or more nodes 120 of the storage cluster 130. The example shown is thus intended to be illustrative rather than limiting.
Notably, the segments 170 have different respective lengths. It is thus possible to rank the segments 170 in order of length, e.g., from longest to shortest, as shown at the top-right of the figure.
The depicted repair group 802 allows for damage to up to M elements prior to experiencing data loss. The damaged elements may be any elements of the repair group 802, which may include data segments 170 and/or repair elements 810, in any combination. Complete recovery and repair can be achieved as long as no greater than M total elements are damaged. One should appreciate that the choices of K=9 and M=3 may be varied, based upon a desired level of data protection, among other factors. In an example, repair elements 810 are generated using a computationally efficient procedure 800 that appears to be entirely new.
Prior erasure-coding schemes may require all K data elements to have equal length. If data elements have unequal lengths, then zero padding may be used to make the lengths equal. Parity calculations are then performed using the full length of all K data elements, producing M parity elements having the same length as the K data elements.
In contrast with the usual erasure-coding approach, the procedure 800 generates repair elements from data elements that have unequal lengths. No zero-padding is required. In an example, procedure 800 proceeds by logically aligning the segments 170, i.e., the K=9 data elements. For example, the segments 170 may be aligned at their respective tops, as shown. Alternatively, the segments 170 may be aligned at their respective bottoms (not shown) or may be aligned in some other known way. Note that such alignment is logical rather than physical, as no actual movement of any segment 170 is required. Also, the depicted ranking of segments 170 should be understood to be logical rather than physical.
With the segments 270 logically aligned, the procedure 800 proceeds by identifying the shortest segment 170 (labeled “1”) and identifying a corresponding range (Rng1). Rng1 aligns with Segment 1 and has the same size and limits. As Segment 1 is the shortest segment and the segments 170 are logically aligned, all of the K segments 170 (Segments 1-9) have data within Rng1. Using the Rng1 data across Segments 1-9, the procedure computes M sets of repair data, one set for each of the M repair elements 810, and places the repair data in the respective repair elements 810 at the location of Rng1. Repair data for Rng1 is thus complete, and such repair data is based on all K segments 170. One should appreciate that the computations herein of repair data may be similar to what is used in conventional K+M erasure coding, the details of which are not critical to embodiments and are not described further.
The procedure 800 then continues in a similar manner for additional ranges. For example, Rng2 corresponds to the part of Segment 2 that extends beyond Segment 1, i.e., the part of Segment 2 for which no repair data has yet been computed. As Segment 1 has no data in Rng2, repair data for Rng2 may be computed using only the corresponding parts of Segments 2-9 (i.e., a total of K−1 segments). As before, the procedure computes M sets of repair data, one set for each of the M repair elements 810, and places the repair data in the respective repair elements 810, this time at the location of Rng2. Repair data for Rng2 is thus complete, but such repair data is based on only K−1 segments 170.
The procedure 800 may continue in this manner for each of ranges Rng3 through Rng8, with the computations of repair data for each range involving one fewer segment than do the computations for the immediately preceding range. Thus, the computations for Rng3 involve K−2 segments, the computations for Rng4 involve K−3 segments, and so on, with the computations for Rng8 involving only K−7 segments, i.e., Segments 8 and 9. It is noted that no computation is needed for Rng9, as Rng9 intersects only a single segment (Segment 9). Rather than computing repair data for Rng9, the procedure 800 instead stores replicas (copies) of the affected data, i.e., the portion of Segment 9 within Rng9. A separate copy of the Rng9 data may be provided at the Rng9 location of each of the repair elements 810.
The erasure-coding procedure 800 is typically faster to compute than conventional erasure coding. Instead of requiring all K data elements for computing repair data of M repair elements 810, the procedure 800 requires K data elements for only the shortest data element. For each next-shortest data element, the procedure 800 requires one fewer data element, eventually requiring only two data elements, and thus reduces computational complexity and execution time.
One should appreciate that segments 170 as produced from objects 160 may be protected using the erasure-coding procedure 800. For example, when distributing segments 170 to computing nodes 120 for storage in the cluster 130, gateway 110 (or some other computer) may perform the procedure 800 to generate repair elements 810 at reduced computational cost. The procedure 800 may operate with K segments 170 at a time, producing M repair elements for each, and forming respective repair groups 802 for each set of K+M elements.
It should be appreciated that erasure coding may place certain constraints on data placement. For example, no two segments 170 that belong to the same repair group 802 should normally be stored on the same disk drive (e.g., SSD, magnetic disk drive, etc.), as doing so would undermine the redundancy of the erasure coding and subject the segments to an increased risk of data loss. For similar reasons, no two segments 170 that belong to the same repair group 802 should normally be stored on the same computing node 120, as doing so would reduce redundancy, e.g., in the event of a failure of the computing node 120. These rules do not typically apply across different repair groups 802, however. For example, no substantial loss of redundancy results from storing segments 170 that belong to different repair groups 802 on the same computing node 120, as long as no two segments belong to the same repair group 802. For example, it may be permissible for a single computing node 120 to store one segment 170 from each of the R repair groups that protect a given data object 160 (a total of R segments of the same data object).
It should further be appreciated that erasure coding is but one way to protect data, with another way being replication. In an example, data objects 160 and their associated repair data and/or replicas reside in buckets of an object store, and data protection schemes are applied on a per-bucket basis. A bucket that uses replication for its data protection will thus use replication for protecting all of its contents, including all objects 160 contained therein. Likewise, a bucket that uses erasure coding for its data protection will use erasure coding for all of its contents. Erasure coding parameters K and M may also be selected and applied on a per-bucket basis. Thus, the arrangement in
At 1010, the method 1000 establishes a maximum size SMAX of segments 170 that can be processed efficiently by nodes 120. The maximum size may be based on practical considerations, such as hardware specifications of nodes 120 (e.g., clock speed, number of cores, amount of memory, and so forth), as well as expected latency to processing tasks and expectations of users. Typical ranges of SMAX may fall between several hundred kilobytes and several megabytes, for example.
At 1012, the method computes an average number of bytes per column, BC. In an example, the value of BC may be based upon the size “ObjectSize” of the data object 160 and on the number K used in the K+M erasure coding used to protect the data object 160. For example, BC=ObjectSize/K. Referring briefly back to
At 1014, the method 1000 calculates a number R of repair groups, e.g., by dividing BC by SMAX and rounding up to the nearest integer. More specifically, the number of repair groups may be calculated as R=BC/SMAX, rounded up.
At 1016, the method calculates the target segment size 320 as STAR=BC/R. The resulting quantity STAR may be provided to splitter 220, e.g., in determining where to start searching for boundaries 252 when splitting the data object 160.
At 1018, the method 1000 directs the splitter 220 to split the data object 160 in a way that produces portions 250 that are at least as large as STAR, e.g., to produce portions 250 that extend to the next boundary 252 beyond STAR.
Method 1000 thus provides useful guidelines for establishing the target segment size 320 and the number R of repair groups to be used for a particular data object 160. Actual selections of these quantities may involve the discretion of administrators and may be driven by other factors besides those described. Thus, the method 1000 is intended to be advisory rather than required.
As shown, computing node 120 includes one or more communication interfaces, such as one or more network interface cards (NICs) 1110, a set of processors 1120, such as one or more processing chips and/or assemblies, memory 1130, such as volatile memory for running software, and persistent storage 1140, such as one or more solid-state disks (SSDs), magnetic disk drives, or the like. The set of processors 1120 and the memory 1130 together form control circuitry, which is constructed and arranged to carry out various methods and functions as described herein. Also, the memory 1130 includes a variety of software constructs, such as those shown in
An improved technique for managing data objects 160 in a storage cluster 130 includes splitting a data object 160 into multiple portions 250 at boundaries 252 within the data object 160. The technique further includes transforming the portions 250 of the data object 160 into segments 170 that provide individually processable units, and distributing the segments 170 among multiple computing nodes 120 of the storage cluster 130 for storage therein.
This section describes examples of managing multi-dimensional data. One should appreciate that any of the features and methodology as described in the above Section I may also be used in embodiments described in this Section II. Certain embodiments of Section II may be used independently of those described in Section I, however. Thus, and unless specifically indicated to the contrary, the features of Section I should not be regarded as required or necessary for any of the Section-II features described below.
The massive volumes of multi-dimensional array-oriented data generated by the scientific community are predominantly stored in industry standard Network Common Data Form (NetCDF). Key challenges exist in making use of data stored in netCDF: datasets are often too large to be copied and transferred across networks for every user; and each time data is accessed by an analytics tool it must be retrieved, subsets must be extracted, and extracted subsets must be formatted, among other requirements. These activities can account for 80-90% of the total time needed to insight.
To unlock the enormous potential of terabyte scale netCDF-formatted data stored at different locations, we have been developing solutions for integrating in-situ analysis capabilities for multi-dimensional data (e.g., netCDF) into our highly innovative real-time smart data lake solution. Dramatically accelerated data analytics performed at the storage layer addresses key challenges: reduced data traffic between sites, reduced compute resources, and lowered costs—all while accelerating data analyses by large margins, with the potential to bring great benefits to the scientific community.
Along these lines, a novel technique for managing multi-dimensional data includes providing an original dataset containing data arranged along multiple dimensions, each dimension covering a respective original range of dimensional units. The technique further includes extracting multiple portions of data from the original dataset, each portion extending over a reduced range of dimensional units, smaller than the original range, in at least one dimension, and all extracted portions together covering the original ranges of the original dataset in all dimensions.
As shown, the example dataset 1502 has three dimensions, labeled X, Y, and Time, thus providing an N-dimensional space 1510 where N equals 3 (Time is considered a dimension of the N-dimensional space for our purposes). The N-dimensional space 1510 may include any number of dimensions. In an example, the dataset 1502 is provided as a NetCDF file, such as a NetCDF4 (or later) file. The file format of the NetCDF4 file may be HDF (Hierarchical Data Format), such as HDF5 or later. These are merely examples, however. A header 1504 may be included with the dataset 1502, e.g., for defining dimensions, variables, and other features.
The dataset 1502 can be split along any of its dimensions. In an example, splitting is performed by the splitter 220 described in connection with
In an example, the splitter 220 is configured to split the dataset 1502 to create portions 250 (
Once the desired level of splitting has been achieved and the portions 250 have been identified, the transformer 230, e.g., running in gateway 110 (
In an example, both splitting and transforming as described above may be facilitated through the use of NetCDF4 and HDF5 libraries. For example, HDF5 libraries may be used to extract information needed to construct the segments 170. Also, the netcdf-c library may be used to construct a header 1530 appropriate for each segment. In an example, the data for each segment 170 may be injected into a determined location, and adjustments may be made to data indices and header information.
The header 1530 in each segment 170 reflects the dimensional ranges of the respective portion 250, rather than that of the original dataset 1502. Using the simple example shown in
Once the segments 170 have been formed from the respective portions 250, distributor 240 may place the segments 170 on respective nodes 120 of the storage cluster 130, such as Node 0 through Node 4, as indicated. In an example, each of these nodes is made to include a segment 170 containing a NetCDF4-compliant dataset, such as a file, which may be accessed using NetCDF4 and HDF5 libraries. Access to the segments 170 using these libraries enables in-situ access to data and independent data analysis. For example, the portions 250 shown in
In some examples, the header 1504 and other metadata of the original dataset 1502 may itself be stored in the cluster 130, e.g., along with one or more of the portions 250 or in a separate segment 170 or other location. Also, one should appreciate that placing segments 170 on respective nodes 120 may include protecting the segments using erasure coding, e.g., as described in connection with
The example depicted in
All chunks 1610 in a dataset 1502a have uniform dimensional proportions. For example, all chunks 1610 have the same dimensions, such as 1×1, 1×2, 200×50, or the like. Chunks 1610 may have non-uniform data size, however. For example, one chunk may be 4 MB, whereas another chunk may be 5 kB or less, e.g., reflecting different levels of compression and/or filtering. Thus, the aim of creating portions 250 having a desired target size is not simply a matter of splitting an array.
In an example, the splitter 220 processes chunked data by selectively combining chunks 1610 to produce portions 250 having a desired target size. For example, the splitter 220 may assign chunks 1610 close to the target size to their own respective portions 250, but the splitter 220 may also combine smaller chunks into single portions, such that the sum of the sizes of the smaller chunks adds up to approximately the desired target size.
In an example, chunks 1610 may be combined not merely based on size. As stated above, NetCDF4 requires that datasets be provided in sub-tensors. Only groups of chunks are arranged in sub-tensors meet this requirement. In the example of
To illustrate, groupings G1, G2, G3, and G4 are all legal groupings of chunks, as their constituent chunks form complete sub-tensors—rectangles with no holes or outcroppings. By contrast, group G5 is not a legal grouping, as one of the chunks is outside the rectangle formed by the other chunks. More formally, group G5 is not a sub-tensor.
Given this constraint, splitter 220 is configured to assemble groupings of chunks that form sub-tensors whose sizes add up approximately to the target size. Such groupings may then be provided in respective portions 250, which may be transformed into respective segments 170 and stored in the cluster 130, as described above.
In some examples, the splitter 220 applies additional constraints when grouping chunks 1610 into portions 250. For example, the splitter 220 may further impose a requirement that any chunks combined within portions must be physically consecutive in the original NetCDF4 file.
In some examples, the above-described selection of physically consecutive chunks does not prevent chunks 1610 from being grouped together when they are separated in the file layout simply by metadata or other non-chunk data. As shown in the figure, metadata 1720a is located between two consecutive (but not contiguous) regions 1750a and 1750b of chunk data for variable V2. The presence of such metadata 1720a does not prevent the splitter 220 from grouping together regions 1750a and 1750b, however. Indeed, grouping may proceed past 1750b and may continue until a complete grouping (which forms a sub-tensor) is created.
For example, the object metadata 112 may associate segments 170 (SegID) with respective locations in the cluster 130 where those segments can be found, e.g., the particular nodes 120 and in some cases locations within those nodes, such as pathnames. In some examples, the object metadata 112 may associate segments with respective dimensional ranges of multi-dimensional data stored within the segments, such as a range of X values, a range of Y values, and a range of Time values (e.g., for the
As before, the task requestor 610 initiates a request 650 for performing a processing task on a specified data object 160, which in this case may be a multi-dimensional dataset 1502 or 1502a (referred to henceforth using single reference 1502), such as a NetCDF4 file. Upon issuance of the request 650, dispatcher 620 begins distributing components of the requested task to the respective nodes 120. In this example, five nodes 120 are identified, i.e., Nodes 0-4. Dispatcher 620 then transmits requests 1910-1 through 1910-4 to the respective nodes 0-4. Requests 1910-1 through 1910-4 may include read requests, select queries, aggregate queries, or analytic processing requests, for example. In some examples, the requests 1910-1 through 1910-4 may all be the same, e.g., the same as the request 650. In such cases, each node 120 may respond based on the contents that it stores. In other examples, the requests 1910-1 through 1910-4 are individually tailored to the particular segments being accessed, such as by limiting query ranges for different segments based on ranges contained within the respective segments, as read from object metadata 112.
The identified nodes 0-4 receive the respective requests 1910-1 through 1910-4, and each of these nodes begins executing the requested task on its respective segment 170. For example, Node 0 executes a task on segment S0, Node 1 executes a task on segment S1, and so on. Execution of a task may be facilitated by a local agent 1902. For example, agent 1902 may respond to a request by accessing the local segment using NetCDF4 and/or HDF5 libraries, such as by extracting data that satisfies the parameters of the request. The local agent 1902 may also perform local aggregation, computing values such as count, average, total, maximum, minimum, or the like, based on local data. Such local processing avoids having to send large data selections over the network. In an example, each node 120 independently executes its respective task on its respective segment 170, without needing to contact any other node 120. For instance, Node 0 completes its work by accessing only S0, without requiring access to S1, S2, S3, or S4.
As the nodes 120 perform their respective work, such nodes produce respective output 1920, shown as output 1920-0 through 1920-4. The participating nodes send their respective output 1920 back to the gateway 110, which collects the output in receiver 630. As mentioned previously, nodes 120 may perform certain aggregate functions themselves, but individual nodes 120 do not typically aggregate output across multiple nodes. This function may be performed instead by the data aggregator 640. For example, aggregator 640 may receive partial aggregate results from multiple nodes. Aggregator 640 may combine the partial aggregate results to produce an overall aggregate value, which may be representative of an entire NetCDF4 file.
Some data processing tasks may involve creating objects, such as NetCDF4 files. For these kinds of tasks, the gateway 110 may include a formatter 1930. For example, the formatter 1930 is configured to place received data (e.g., from responses 1920) into a NetCDF4-compliant file. Such operation of the formatter 1930 may thus facilitate reconstruction of original NetCDF4 files (datasets 1502), as well as construction of other datasets, such as those which combine output from multiple segments.
In some examples, the query criteria are expressed directly as predicates in units that match dimensions of the dataset 1502, e.g., the NetCDF4 file. In other examples, query criteria are expressed in non-dimensional units, such as units of certain variables stored in a NetCDF4 file. In these latter examples, the method 2000 may include a step 2020, which provides a first stage of query processing. Here, the gateway 110 contacts the nodes 120 (e.g., all nodes) requesting dimensional predicates that correspond to the query criteria. The nodes reply with predicate information. The gateway 110 then transforms the query criteria from the request 650 into corresponding predicates expressed in terms of dimensions defined by the NetCDF4 file.
Regardless of whether a first stage of querying is needed, operation proceeds to 2030, whereupon the gateway 110 accesses object metadata 112 to get locations of relevant segments, such as those segments which might contain portions of the requested data. The identified segments 170 may be precisely those which store the requested data, or they may be overinclusive, meaning that more segments are identified than actually contain portions of the requested data. One should appreciate that step 2030 may also be optional, as the request 650 may simply be broadcast to all nodes in the cluster 130 that contain any portions of the NetCDF4 file.
At 2040, the request 650 or modified versions thereof (e.g., requests 1910-0 through 1910-4) is sent to the nodes 120 that contain the relevant segments, which may include sending the request 650 to all nodes in the cluster 130. This step 2040 may provide a second stage of query processing (if a first stage of processing was done at 2020), or it may be the only stage of querying (if no first stage of querying was needed).
At 2050, the gateway 110 receives partial query results from the respective nodes, and at 2060 the gateway 110 merges the partial query results to produce an overall result, which is representative of the NetCDF4 file as a whole.
At 2120, the data extracted from the segments at 2110 is injected into a template dataset 2122, such as a template NetCDF4 file. At 2130, the gateway 110 retrieves the original header 1504 from the cluster 130, e.g., from a dedicated metadata segment, from the object metadata 112, or from some other location in the cluster 130. At 2130, the gateway 110 copies the retrieved original header 1504 into the template dataset 2122. With the data and header of the original NetCDF4 file assembled as described, the original NetCDF4 file is fully reconstructed. Preferably, the NetCDF4 file is reconstructed such that it is identical in every respect to the original file (assuming no updates have been done since the initial splitting).
At 2220, multiple portions 250 of data are extracted from the original dataset 1502. Each portion 250 extends over a reduced range of dimensional units in at least one dimension of the N-dimensional space (such as one or more Time units, one or more Y units, etc.), and the extracted portions 250 together cover all original ranges of the N-dimensional space 1510.
At 2230, the extracted portions 250 are rendered in respective segments 170 that provide data of the extracted portions 250 in the same data format (e.g., NetCDF4) as the original dataset 1502.
An improved technique has been described for managing multi-dimensional data. The technique includes providing an original dataset 1502 containing data arranged along multiple dimensions, each dimension covering a respective original range of dimensional units. The technique further includes extracting multiple portions 250 of data from the original dataset 1502, each portion 250 extending over a reduced range of dimensional units, smaller than the original range, in at least one dimension, and all extracted portions 250 together covering the original ranges of the original dataset 1502 in all dimensions.
Having described certain embodiments, numerous alternative embodiments or variations can be made. For example, although embodiments have been described in connection with NetCDF4 and HDF5, these are merely examples. As technology evolves, additional versions are expected to be released, and nothing herein is limited only to current versions. Previous versions may also be used. In addition, the technique presented herein applies to datasets constructed using other formats. The disclosure is not limited to any particular format or version.
Further, although features have been shown and described with reference to particular embodiments hereof, such features may be included and hereby are included in any of the disclosed embodiments and their variants. Thus, it is understood that features disclosed in connection with any embodiment are included in any other embodiment.
Further still, the improvement or portions thereof may be embodied as a computer program product including one or more non-transient, computer-readable storage media, such as a magnetic disk, magnetic tape, compact disk, DVD, optical disk, flash drive, solid state drive, SD (Secure Digital) chip or device, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), and/or the like (shown by way of example as medium 1250 in
As used throughout this document, the words “comprising,” “including,” “containing,” and “having” are intended to set forth certain items, steps, elements, or aspects of something in an open-ended fashion. Also, as used herein and unless a specific statement is made to the contrary, the word “set” means one or more of something. This is the case regardless of whether the phrase “set of” is followed by a singular or plural object and regardless of whether it is conjugated with a singular or plural verb. Also, a “set of” elements can describe fewer than all elements present. Thus, there may be additional elements of the same kind that are not part of the set. Further, ordinal expressions, such as “first,” “second,” “third,” and so on, may be used as adjectives herein for identification purposes. Unless specifically indicated, these ordinal expressions are not intended to imply any ordering or sequence. Thus, for example, a “second” event may take place before or after a “first event,” or even if no first event ever occurs. In addition, an identification herein of a particular element, feature, or act as being a “first” such element, feature, or act should not be construed as requiring that there must also be a “second” or other such element, feature or act. Rather, the “first” item may be the only one. Also, and unless specifically stated to the contrary, “based on” is intended to be nonexclusive. Thus, “based on” should not be interpreted as meaning “based exclusively on” but rather “based at least in part on” unless specifically indicated otherwise. Although certain embodiments are disclosed herein, it is understood that these are provided by way of example only and should not be construed as limiting.
Those skilled in the art will therefore understand that various changes in form and detail may be made to the embodiments disclosed herein without departing from the scope of the following claims.
This invention claims the benefit of U.S. Provisional Application No. 63/394,466, filed on Aug. 2, 2022, the contents and teachings of which are incorporated herein by reference in their entirety.
This invention was made with government support under NA22OAR0210591 awarded by the National Oceanic and Atmospheric Administration. The government has certain rights in the invention.
| Number | Date | Country | |
|---|---|---|---|
| 63394466 | Aug 2022 | US |
