Patent Grant
11630729
This disclosure relates to information storage and processing packets of information, for example, in the fields of networking, storage, and cryptography.
In a typical cloud-based data center, a large collection of interconnected servers provides computing and/or storage capacity for execution of various applications. For example, a data center may comprise a facility that hosts applications and services for subscribers, i.e., customers of the data center. The data center may, for example, host all of the infrastructure equipment, such as compute nodes, networking and storage systems, power systems, and environmental control systems. In most data centers, clusters of storage systems and application servers are interconnected via a high-speed switch fabric provided by one or more tiers of physical network switches and routers. Data centers vary greatly in size, with some public data centers containing hundreds of thousands of servers, and are usually distributed across multiple geographies for redundancy.
In a large scale fabric, storage systems or data within a storage system may become unavailable from time to time, due to hardware error, software error, or another reason. Data durability procedures may be employed to ensure access to critical data.
This disclosure describes techniques that include data durability or data reliability coding, including network-efficient data durability or data reliability coding on a network. In some examples, techniques described herein may involve a data processing unit or access node (e.g., a master node) storing data in fragments across multiple fault domains in a manner that enables efficient recovery of the data, even if only a subset of the data is available. Data fragments within each fault domain may be further processed when stored, by data durability logic within the fault domain, to implement an additional or second level of data durability. Corrupted, lost, or otherwise unavailable data may be reconstructed using various data fragments while, in some cases, also minimizing or reducing network traffic.
In some examples, techniques described herein tend to reduce the number of read-modify-write cycles by accumulating data at a receiving or master node, and then storing data across the network once a sufficient amount of data has been accumulated. The master node may generate data blocks from the accumulated data using an erasure coding algorithm, and then store the data blocks across the network in other nodes. Each node may shard a data block to be stored at the node and store the shards and additional parity information (i.e., secondary data fragments) using a data durability scheme that protects against a storage failure at the node. Such a technique may enable the node to recover data lost due to a failure of one or more storage devices at the node. In some cases, such data may be recovered through operations performed within a node, and without requiring data fragments stored at other nodes.
The techniques described herein may provide some technical advantages. For instance, in examples where a second-level or separately-implemented reliability coding process is implemented for data stored at a node, reliability of data across the system or network may be enhanced, since a node may be able to regenerate lost data without relying on data durability coding implemented across multiple nodes or across durability schemes implemented using nodes spanning a network. Further, by avoiding use of data durability coding that may be implemented network-wide, network traffic generated by at least some data rebuilding operations may be reduced, thereby enabling more network bandwidth to be consumed by users of the network for productive purposes. In addition, aspects of the present disclosure may enable faster rebuild times in some situations, including in situations in which one or more components of a node (e.g., a storage drive) has failed.
In some examples, this disclosure describes operations performed by a network node or other network device in accordance with one or more aspects of this disclosure. In one specific example, this disclosure describes a method comprising generating a plurality of data fragments (which may include parity fragments) from a set of data to enable reconstruction of the set of data from a subset of the plurality of data fragments, wherein the plurality of data fragments includes a first fragment and a second fragment; storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node; generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments; storing the plurality of secondary fragments (which may include parity fragments) from the first fragment across a plurality of storage devices included within the first node, wherein storing the plurality of secondary fragments includes storing each of the plurality of secondary fragments in a different one of the plurality of storage devices; and reconstructing the set of data from a subset of the plurality of data fragments, wherein reconstructing the set of data includes reconstructing the first fragment from a subset of the plurality of secondary fragments.
In another example, this disclosure describes a storage system comprising a plurality of nodes connected by a network, wherein the storage system is configured to be capable of performing operations comprising: generating a plurality of data fragments from a set of data to enable reconstruction of the set of data from a subset of the plurality of data fragments, wherein the plurality of data fragments includes a first fragment and a second fragment; storing, across the plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node, wherein the first node and the second node are included within the plurality of nodes; generating a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments; storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node, wherein storing the plurality of secondary fragments includes storing each of the plurality of secondary fragments in a different one of the plurality of storage devices; and reconstructing the set of data from a subset of the plurality of data fragments, wherein reconstructing the set of data includes reconstructing the first fragment from a subset of the plurality of secondary fragments.
In another example, this disclosure describes a computer-readable storage medium comprising instructions that, when executed, configure processing circuitry of a storage system to perform operations comprising: generating a plurality of data fragments from a set of data to enable reconstruction of the set of data from a subset of the plurality of data fragments, wherein the plurality of data fragments includes a first fragment and a second fragment; storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node; generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments; storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node, wherein storing the plurality of secondary fragments includes storing each of the plurality of secondary fragments in a different one of the plurality of storage devices; and reconstructing the set of data from a subset of the plurality of data fragments, wherein reconstructing the set of data includes reconstructing the first fragment from a subset of the plurality of secondary fragments.
This disclosure describes techniques that include implementing network-efficient data durability or data reliability coding on a network. Storage systems may be implemented in various ways, including as scale-up storage systems or scale-out storage systems. In both types of storage systems, data durability procedures, such as replication, erasure coding, RAID or other procedures may be employed to make data reliably available.
Replication, RAID or Erasure coding (EC) may be used to protect data on drives. Typically, extra space is introduced to the storage system for data protection. In replication, a second or third copy of the data is also stored, and the overhead, or extra space needed is on the order of one hundred percent or more. In RAID schemes, the extra space stores parity information. Overhead or extra space for RAID may be of the order of ⅛. In erasure coding schemes, the extra storage space is used to store Reed Solomon codes. The overhead for erasure coding tends to be similar to RAID schemes. In general, RAID schemes are sometimes limited in the number of simultaneous failures that can be handled, whereas erasure coding schemes can be more flexible. For example, a RAID-5 scheme protects against a single disk failure and a RAID-6 scheme can protect against the failure of 2 simultaneous drives. On the other hand, a 72+8 EC scheme allows the failure of up to 8 drives simultaneously from among a group of 80 drives.
In scale-up storage systems, individual nodes are built to be highly available, and drive and processor failures are repaired generally without involving other nodes. Processor failures are handled by another processor in the node taking over the work of the failed processor. Drive failures are handled by rebuilding the data to spare space on other surviving drives in the node.
In scale-out storage systems, individual nodes are often not designed to be highly available. Node failures or drive failures are handled by rebuilding data from the failed node to spare space on drives in other nodes. When a node fails, the data from all the drives in the node are rebuilt elsewhere, often by other nodes rebuilding data using data available from other nodes on the network.
Although techniques described herein may apply to both types of systems, many of the techniques described herein may be particularly applicable to scale-out systems. In scale-out systems, the replication, RAID, or erasure coding may be performed across various failure domains, such as across nodes within a storage system. For a 4+2 erasure coding system, 6 nodes are involved. The failure of a block of data on a drive will typically require reading corresponding blocks of data from drives on 4 other nodes to rebuild the lost data. This incurs network traffic on the node to node network. The failure of an entire drive will typically require rebuilding all the blocks of data on that failed drive, and each block can be rebuilt as described above. The failure of a node or of the processor in the node will require rebuilding all the blocks of data on all the drives in that node.
Data center 10 represents an example of a system in which various techniques described herein may be implemented. In general, data center 10 provides an operating environment for applications and services for customers 11 coupled to the data center by service provider network 7 and gateway device 20. Data center 10 may, for example, host infrastructure equipment, such as compute nodes, networking and storage systems, redundant power supplies, and environmental controls. Service provider network 7 may be coupled to one or more networks administered by other providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet.
In some examples, data center 10 may represent one of many geographically distributed network data centers. In the example of
In the illustrated example, data center 10 includes a set of storage systems and application servers 12 interconnected via a high-speed switch fabric 114. In some examples, servers 12 are arranged into multiple different server groups, each including any number of servers up to, for example, n servers 12l-12n. Servers 12 provide computation and storage facilities for applications and data associated with customers 11 and may be physical (bare-metal) servers, virtual machines running on physical servers, virtualized containers running on physical servers, or combinations thereof.
In the example of
In the example shown in
In general, each node group 19 of rack 70-1 may be configured to operate as a high-performance I/O hub designed to aggregate and process network and/or storage I/O for multiple servers 12. As mentioned above, the set of nodes 17 within each of the node groups 19 provide programmable, specialized I/O processing circuits for handling networking and communications operations on behalf of servers 12. In addition, in some examples, each of node groups 19 may include storage devices 27, such as solid state drives (SSDs) and/or hard disk drives (HDDs), configured to provide network accessible storage for use by applications executing on the servers 12. In some examples, one or more of the SSDs may comprise non-volatile memory (NVM) or flash memory. Although illustrated as logically within node groups 19 and external to nodes 17, storage devices may alternatively or in addition be included within one or more nodes 17 or within one or more servers 12.
Other nodes 17 may serve as storage nodes that might not be directly connected to any of servers 12. For instance,
Rack 70-3 is illustrated as being implemented in a manner similar to rack 70-2, with storage nodes 17 configured to store data within storage devices. Although for ease of illustration, only racks 70-1, 70-2, 70-3, through 70-N are illustrated or represented in
Nodes 17 of rack 70-2 (or rack 70-3) may be devices or systems that are the same as or similar to nodes 17 of rack 70-1. In other examples, nodes 17 of rack 70-2 may have different capabilities than those of rack 70-1 and/or may be implemented differently. In particular, nodes 17 of rack 70-2 may be somewhat more capable than nodes 17 of rack 70-1, and may have more computing power, more memory capacity, more storage capacity, and/or additional capabilities. For instance, each of nodes 17 of rack 70-2 may be implemented by using a pair of nodes 17 of rack 70-1. To reflect such an example, nodes 17 of rack 70-2 and 70-3 are illustrated in
In a large scale fabric, storage systems (e.g., represented by nodes 17 of rack 70-2 or even NCSUs 40 of rack 70-1) may become unavailable from time to time. Failure rates of storage systems are often significant, even if single component failure rates are quite small. Further, storage systems may become unavailable for reasons other than a software error or hardware malfunction, such as when a storage system or other device is being maintained or the software on such a device is being modified or upgraded. Accordingly, as further described herein, data durability procedures may be employed to ensure access to critical data stored on a network when one or more storage systems are unavailable.
In some examples, one or more hardware or software subsystems may serve as a failure domain or fault domain for storing data across data center 10. For instance, in some examples, a failure domain may be chosen to include hardware or software subsystems within data center 10 that are relatively independent, such that a failure (or unavailability) of one such subsystem is relatively unlikely to be correlated with a failure of another such subsystem. Storing data fragments in different failure domains may therefore reduce the likelihood that more than one data fragment will be lost or unavailable at the same time. In some examples, a failure domain may be chosen at the node level, where each node represents a different failure domain. In another example, a failure domain may be chosen at a logical or physical grouping level, such that each group or unit of nodes 17 represents a different failure domain. In other examples, failure domains may be chosen more broadly, so that a failure domain encompasses a logical or physical rack 70 comprising many nodes 17. Broader or narrower definitions of a failure domain may also be appropriate in various examples, depending on the nature of the network 8, data center 10, or subsystems within data center 10.
As further described herein, in one example, each node 17 may be a highly programmable I/O processor specially designed for performing storage functions and/or for offloading certain functions from servers 12. In one example, each node 17 includes a number of internal processor clusters, each including two or more processing cores and equipped with hardware engines that offload cryptographic functions, compression and regular expression (RegEx) processing, data durability functions, data storage functions and networking operations. In such an example, each node 17 may include components for processing and storing network data (e.g., nodes 17 of rack 70-2) and/or for and processing network data on behalf of one or more servers 12 (e.g., nodes 17 of rack 70-1). In addition, some or all of nodes 17 may be programmatically configured to serve as a security gateway for its respective servers 12, freeing up other computing devices (e.g., the processors of the servers 12) to dedicate resources to application workloads.
In some example implementations, some nodes 17 may be viewed as network interface subsystems that serve as a data storage node configured to store data across storage devices 227. Other nodes 17 in such implementations may be viewed as performing full offload of the handling of data packets (with, in some examples, zero copy in server memory) and various data processing acceleration for the attached server systems.
In one example, each node 17 may be implemented as one or more application-specific integrated circuit (ASIC) or other hardware and software components, each supporting a subset of storage devices 227 or a subset of servers 12. In accordance with the techniques of this disclosure, any or all of nodes 17 may include a data durability module or unit, which may be implemented as a dedicated module or unit for efficiently and/or quickly performing data durability operations. In some examples, such a module or unit may be referred to as an “accelerator” unit. That is, one or more computing devices may include a node including one or more data durability, data reliability, and/or erasure coding accelerator units, according to the techniques of this disclosure.
The data durability module or unit of the node, according to the techniques of this disclosure, may be configured to store data in fragments across multiple fault domains in a manner that enables efficient recovery of the data using or based on a subset of the data fragments. When storing data, the data durability accelerator unit may encode data using any of a variety of data durability, RAID, or erasure coding schemes that enable recovery of data when one or more of such fragments are unavailable due to software or hardware error, or for another reason, such as modifications (e.g., software upgrades) being performed on the storage unit where a data fragment is being stored. Further, the data durability accelerator unit may provide a flexible and/or configurable data durability system by applying a unified approach to implementing a variety of data durability coding schemes. In some examples, the data durability accelerator may implement multiple data durability coding schemes or algorithms through a common matrix approach, such as that described in U.S. patent application Ser. No. 16/265,606, filed Feb. 1, 2019, entitled “FLEXIBLE RELIABILITY CODING FOR STORAGE ON A NETWORK,” which is hereby incorporated by reference.
In the example of
Example architectures of nodes 17 are described herein with respect to
More details on how nodes 17 may operate are available in U.S. Provisional Patent Application No. 62/589,427, filed Nov. 21, 2017, entitled “Work Unit Stack Data Structures in Multiple Core Processor System,” and U.S. Provisional Patent Application No. 62/625,518, entitled “EFFICIENT WORK UNIT PROCESSING IN A MULTICORE SYSTEM”, filed Feb. 2, 2018, and in U.S. patent application Ser. No. 16/031,676, filed Jul. 10, 2018, entitled “Access Node Integrated Circuit for Data Centers which Includes a Networking Unit, a Plurality of Host Units, Processing Clusters, a Data Network Fabric, and a Control Network Fabric,”. All of these applications are hereby incorporated by reference.
In
Node 17-1 of rack 70-2 may store data across data center 10. For instance, in an example that can be described in the context of
Within each of nodes 17 receiving a different data fragment, each node 17 may perform additional or secondary data durability processes on the received data fragment. For instance, again with reference to an example that can be described in the context of
After storing the data fragments, node 17-1 of rack 70-2 may receive a request (e.g., a “read” request) for a portion of or all of the stored data that was stored across data center 10 as data fragments. For instance, in the example of
If one or more of the data fragments is not available, however, node 17-1 of rack 70-2 accesses one or more of the parity data fragments and uses the parity data fragments, along with the available data fragments, to reconstruct the original data. To do so, node 17-1 performs a data durability decoding operation to reconstruct the data. If the data was encoded using a Reed Solomon erasure coding algorithm, for example, the decoding operation involves a corresponding Reed Solomon decoding operation. As with the encoding operation, the decoding operation may be a computationally intensive operation. When the decode operation is complete, the requested data, which may be a subset of the reconstructed data, is output to the requesting server 12 over switch fabric 114 as a response to the read request.
In some examples, if data is lost at a given storage node 17, but the node 17 is still operational, that node 17 may be able to reconstruct the data from other data stored at the node 17, without relying on the erasure coding system implemented across network 108. As further described herein, by storing additional parity data at a storage node 17, each storage node 17 may have the capability of independently recovering from some storage failures using other data stored at the node 17, without relying on fragments stored at other nodes. By not relying on fragment stored at other nodes (i.e., relying on the erasure coding system implemented across network 108), that node 17 may be able to avoid generating network traffic on network 108 when recovering data.
Further details relating to techniques for reliability coding and storage of data to support erasure coding are available in U.S. patent application Ser. No. 16/215,178, filed Dec. 10, 2018, entitled “Durable Block Storage in Data Center Access Nodes with Inline Erasure Coding,”, U.S. patent application Ser. No. 16/169,736, filed Oct. 24, 2018, entitled “INLINE RELIABILITY CODING FOR STORAGE ON A NETWORK,”, and U.S. patent application Ser. No. 16/265,606, filed Feb. 1, 2019, entitled “FLEXIBLE RELIABILITY CODING FOR STORAGE ON A NETWORK,”. The entire content of all of these applications is incorporated herein by reference.
Thus, DPU 217 may be communicatively coupled to one or more network devices, server devices (e.g., servers 12), random access memory, storage media (e.g., solid state drives (SSDs)), storage devices 227, a data center fabric (e.g., switch fabric 114), or the like, e.g., via PCI-e, Ethernet (wired or wireless), or other such communication media. Moreover, DPU 217 may be implemented as one or more application-specific integrated circuit (ASIC), may be configurable to operate as a component of a network appliance or may be integrated with other DPUs within a device.
In the illustrated example of
Memory unit 134 may include two types of memory or memory devices, namely coherent cache memory 136, non-coherent buffer memory 138, and non-volatile memory 139 (e.g., NVDIMM memory). Processor 132 also includes a networking unit 142, work unit (WU) queues 143, a memory controller 144, and accelerators 146. Although not shown, processor 132 may also include a storage device controller used when accessing storage devices 127. As illustrated in
In this example, DPU 217 represents a high performance, hyper-converged network, storage, and data processor and input/output hub. For example, networking unit 142 may be configured to receive one or more data packets from and transmit one or more data packets to one or more external devices, e.g., network devices. Networking unit 142 may perform network interface card functionality, packet switching, and the like, and may use large forwarding tables and offer programmability. Networking unit 142 may expose Ethernet ports for connectivity to a network, such as switch fabric 114 of
Processor 132 further includes accelerators 146 configured to perform acceleration for various data-processing functions, such as look-ups, matrix multiplication, cryptography, compression, data durability and/or reliability, regular expressions, or the like. For example, accelerators 146 may comprise hardware implementations of look-up engines, matrix multipliers, cryptographic engines, compression engines, or the like. In accordance with the techniques of this disclosure, at least one of accelerators 146 may represent a data durability unit that may be used to implement one or more data durability and/or reliability schemes. In some examples, such a data durability unit may be configured to perform matrix multiplication operations commonly performed in erasure coding schemes, such as Reed Solomon erasure coding schemes. Such a data durability unit may be configured to efficiently perform operations, such as those relating to Galois Field mathematics, that might be difficult and/or inefficient to perform using commonly available processors or other processing hardware. Further, such a data durability unit may be designed to perform and/or implement multiple different types of data durability schemes by configuring different matrices specific to each implementation.
Memory controller 144 may control access to on-chip memory unit 134 by cores 140, networking unit 142, and any number of external devices, e.g., network devices, servers, external storage devices, or the like. Memory controller 144 may be configured to perform a number of operations to perform memory management in accordance with the present disclosure. For example, memory controller 144 may be capable of mapping accesses from one of the cores 140 to either of coherent cache memory 136 or non-coherent buffer memory 138. More details on a bifurcated memory system that may be included in DPU 217 are available in U.S. Provisional Patent Application No. 62/483,844, filed Apr. 10, 2017, and titled “Relay Consistent Memory Management in a Multiple Processor System,” the entire content of which is incorporated herein by reference.
Cores 140 may comprise one or more microprocessors without interlocked pipeline stages (MIPS) cores, advanced reduced instruction set computing (RISC) machine (ARM) cores, performance optimization with enhanced RISC—performance computing (PowerPC) cores, RISC Five (RISC-V) cores, or complex instruction set computing (CISC or x86) cores. Each of cores 140 may be programmed to process one or more events or activities related to a given data packet such as, for example, a networking packet or a storage packet. Each of cores 140 may be programmable using a high-level programming language, e.g., C, C++, or the like.
In
Through techniques in accordance with one or more aspects of the present disclosure, such as by employing a second-level or separately-implemented reliability coding process for data stored at a node, reliability of data across the system or network may be enhanced, since a node may be able to regenerate lost data without relying on data durability coding implemented across nodes or network-wide. Further, by avoiding use of data durability coding that may be implemented network-wide, network traffic generated by at least some data rebuilding operations may be reduced, thereby enabling more network bandwidth to be consumed by users of the network for productive purposes. In addition, aspects of the present disclosure may enable faster rebuild times in some situations, including in situations in which a component of a node (e.g., a storage drive) has failed.
In general, DPU 317 may represent a high performance, hyper-converged network, storage, and data processor and input/output hub. As illustrated in
As shown in
Networking unit 152 has Ethernet interfaces 164 to connect to the switch fabric, and interfaces to the data network formed by grid links 160 and the signaling network formed by direct links 162. Networking unit 152 provides a Layer 3 (i.e., OSI networking model Layer 3) switch forwarding path, as well as network interface card (NIC) assistance. One or more hardware direct memory access (DMA) engine instances (not shown) may be attached to the data network ports of networking unit 152, which are coupled to respective grid links 160. The DMA engines of networking unit 152 are configured to fetch packet data for transmission. The packet data may be in on-chip or off-chip buffer memory (e.g., within buffer memory of one of processing clusters 156 or external memory 170), or in host memory.
Host units 154 each have PCI-e interfaces 166 to connect to servers and/or storage devices, such as SSD devices. This allows DPU 317 to operate as an endpoint or as a root. For example, DPU 317 may connect to a host system (e.g., a server) as an endpoint device, and DPU 317 may connect as a root to endpoint devices (e.g., SSD devices). Each of host units 154 may also include a respective hardware DMA engine (not shown). Each DMA engine is configured to fetch data and buffer descriptors from host memory, and to deliver data and completions to host memory.
DPU 317 may provide optimizations for stream processing. For instance, DPU 317 may execute an operating system that facilitates run-to-completion processing, which may eliminate interrupts, thread scheduling, cache thrashing, and associated costs. For example, an operating system may run on one or more of processing clusters 156. Central cluster 158 may be configured differently from processing clusters 156, which may be referred to as stream processing clusters. In one example, central cluster 158 executes the operating system kernel (e.g., Linux kernel) as a control plane. Processing clusters 156 may function in run-to-completion thread mode of a data plane software stack of the operating system. That is, processing clusters 156 may operate in a tight loop fed by work unit queues associated with each processing core in a cooperative multi-tasking fashion.
Each of racks 470 include one or more nodes 417 and may include one or more servers 412. Rack 470A is illustrated in
In the example of
In
Typically, an erasure coding algorithm splits data blocks into “d” data blocks and “p” parity blocks. A Reed Solomon 4+2 erasure coding scheme, for example, uses d=4 data blocks to generate p=2 parity blocks. Many other Reed Solomon implementations are possible, including 12+3, 10+4, 8+2, and 6+3 schemes. Other types of erasure encoding schemes beyond Reed Solomon schemes include parity array codes (e.g., EvenOdd codes, X codes, HoVer codes, WEAVER codes), Low Density Parity Check (LDPC) codes, or Local Reconstruction Codes (LRC). In some cases, such as for parity array codes, reliability schemes may be more restrictive in terms of an ability to recover from failure for a given set of unavailable data fragments or data blocks. Further, data recovery for parity array codes may be iterative if more than one data fragment or data block is unavailable; such iterative data recovery may involve time-consuming and/or inefficient processing, thereby leading to latency and/or poor performance.
Examples described herein principally are described in the context of a 4+2 erasure coding scheme. In such a scheme, two erasure coding data blocks or parity blocks storing Reed Solomon codes are used for every four blocks of data, as described herein. Although various examples herein are principally described in the context of a 4+2 erasure coding scheme, techniques described herein are applicable to other erasure coding or Reed Solomon formulations beyond the 4+2 scheme described in various examples herein. Further, although principally described with respect to Reed Solomon erasure coding scheme, techniques described herein may be applicable to replication, various RAID variants, or other erasure coding schemes. Such RAID variants may include RAID-5, RAID-6, RAID RDP (row-diagonal parity), RAID TP (triple parity), RAID 3D, and others.
In
In some examples, data 801 may be received by node 417A as a series of segments of data 801. In such an example, node 417A outputs each segment of data 801 to data durability module 406A. Data durability module 406A stores each segment within non-volatile memory 408A. In some examples, data durability module 406A may compress each segment of data 801 before storing the segment within non-volatile memory 408A. Once each segment of data 801 is stored within non-volatile memory 408A, data durability module 406A may acknowledge the write operation, thereby enabling the device sending that segment of data 801 (e.g., server 412A or server 512) to release or reallocate storage that previously held the segment.
Node 417A may accumulate data within non-volatile memory 408A before writing the data across network 108. For instance, continuing with the example being described with reference to
Node 417A may prepare data 801 for storage across network 108. For instance, still continuing with the example being described with reference to
Node 417A may send data fragments 802 across network 108 for storage. For instance, still continuing with the example being described with reference to
In the example of
In some examples, the storage system illustrated in
The erasure coding scheme illustrated in
One of the drawbacks of erasure coding systems is complexity, and encoding and decoding data using an erasure coding scheme may require high computing resources, complexity, and/or costs. For example, a Reed Solomon erasure coding scheme is typically implemented using Galois Field mathematics, and many current processors are not well equipped to perform Galois Field mathematics operations efficiently. Complexity, computing resources, and/or inefficiency may affect performance, and/or increase latency of operations on network 108. To address these issues, data durability modules 406 may be configured and/or equipped, in some examples, to process Galois Field mathematical operations efficiently, and may include specialized circuitry or logic that enables efficient performance of operations involved in encoding and/or decoding Reed Solomon erasure codes. In examples where a server (e.g., server 412A of
Included within each of nodes 417 illustrated in
In
Node 417A may employ a parity scheme to store data fragment 802D1 within rack 470A. For instance, continuing with the example being described in the context of
In a similar manner, node 417B may employ a parity scheme (e.g., a RAID or erasure coding scheme) to store data fragment 802D2 within rack 470B. For instance, again with reference to the example being described in the context of
Each of node 417E and node 417F may also employ similar parity schemes to store each of data fragment 802P1 and data fragment 802P2 across storage devices 427 within respective nodes 417. For instance, data durability module 406E shards data fragment 802P1 into three smaller segments (data fragments 802P11, 802P12, and 802P13), each having a size that is equal to or approximately equal to D/3. Data durability module 406E computes data fragment 802P1P, and stores each of data fragments 802P11, 802P12, 802P13, and 802P1P in a different one of storage devices 427E as illustrated in
In the flush operation described in connection with
In some examples, choosing which of storage devices 427 within a given node 417 to store a data fragment or a secondary data fragment may be based on a load balancing scheme across each of the storage devices 427 within a given node. Although in the example described in connection with
In the scheme described in
Note that the parity scheme employed by each of nodes 417 to store data fragments 802 across storage devices 427 within each node 417 provides additional protection against data loss. A standard 4+2 erasure coding scheme can protect only against the failure of one or more drives in each of 2 different nodes. However, without further data reliability measures, a storage system employing a 4+2 scheme cannot recover data if there is an additional drive or node failure in a third node. As further described herein, however (see, e.g.,
In
In some examples, node 417A may reconstruct data 801 if the request by server 512 requires a significant portion or all of the data 801. For instance, in another example that can be described in the context of
In
In some examples, data durability module 406A may determine that the required fragment is not available, or that the requested data is for a significant part of or all of data 801. For instance, in the example of
Node 417A may reconstruct data 801. For instance, continuing with the example being described in the context of
In
In
Once online, new node 417D′ may reconstruct data fragment 802D4. For instance, referring again to the example being described in the context of
New node 417D′ may reconstruct data previously stored across storage devices 427D. For instance, still referring to the example being described in the context of
In
In
Once storage device 427E3′ is deployed and operational, node 417E may reconstruct data fragment 802P13. For instance, referring again to the example being described in the context of
One drawback of scale-up storage systems, where node or drive failures are handled by rebuilding data by retrieving data from other nodes, is that recovering from a failure tends to increase network traffic. For example, for a given failed node 417, such as that described and illustrated in connection with
However, in the example of a single failed storage device 427 on a given node, such as that described and illustrated in connection with
Further, in cases in which data can be recovered and rebuilt without requiring network traffic (e.g., as in
In a typical 4+2 erasure coding scheme, network 108 would normally be unable to recover from a loss of three of data fragments 802. However, if one or more of nodes 417 within racks 470 employ the additional parity coding and storage procedures described in connection with
In
Node 417E may reconstruct data fragment 802P1. For instance, referring again to the example being described in connection with
Node 417A may reconstruct data to comply with the request from server 512. For instance, still referring to the example being described in connection with
In the example of
Although techniques described and illustrated herein have been primarily described in the context of erasure coding used across nodes 417 and parity used within nodes 417 (e.g., across storage devices 427), other variations are possible. For instance, in other examples, erasure coding may be used across nodes 417 and erasure coding may also be used within nodes 417 (across storage devices 427). In another example, parity coding may be used across nodes 417 and erasure coding may be used within nodes 417 and across storage devices 427. In yet another example, parity may be used both across nodes 417 and within nodes 417. Each of these variations may require different storage space overheads for data protection, and they may also differ in the number of drive failures they can recover from.
Although techniques described and illustrated herein have primarily described flushing operations writing each of four shards from data fragments 802 to one of storage devices 427, other variations for selection of the destination for such shards are possible. In particular, where one or more of nodes 417 have sixteen storage devices 427, at least two possible variations may be considered.
In the first variation, data durability module 406 within a given node 417 may divide the 16 storage devices 427 into 4 pre-selected groups of 4 storage devices 427 each. Data durability module 406 may pick one of the 4 groups to write to, based on the volume whose data is being flushed.
In a second variation, data durability module 406 may, based on the volume whose data is being flushed, pick a set of 4 storage devices 427 from among the 16 storage devices 427. In such an example, there are 1820 ways to choose 4 storage devices 427 out of 16, and each of these combinations can be configured to be equally likely to be picked.
The advantage of the first variation is that up to 4 storage devices 427 can fail per nodes 417, as long as the 4 storage devices 427 are in different groups. In such an example, network 108 may still recover without any network traffic. In the second variation, network 108 may recover from one storage device 427 failure per node without network traffic. However, rebuild is much faster for this second variation, as all the remaining 15 storage devices 427 can be used to complete the rebuild. In the first variation, on the other hand, 3 storage devices 427 are involved in a rebuild, so the rebuild may take longer to complete.
An alternative implementation is also possible relating to how the sharding, parity and erasure coding calculation during a flush operation is performed by a given data durability module 406 of a given node 417. In the scheme primarily described herein, a master node uses data fragments 802D1, 802D2, 802D3, and 802D4 to compute data fragments 802P1 and 802P2. Node 417A then transmits data fragments 802D2, 802D3, 802D4, data fragments 802P1, and 802P2 to 5 other nodes 417 and each of these receiving nodes 417 shard the received information and compute parity on the shards. A variation is to have the master node (e.g., node 417A) shard the data, compute erasure coding on the shards, then send the sharded data and sharded EC to 5 other nodes, where parity is computed on the shards. Specifically, at the master node, data fragment 802D1 is sharded into Data11, Data12 and Data13; data fragment 802D2 into Data21, Data22, Data23; data fragment 802D3 into Data31, Data32 and Data33; and data fragment 802D4 into Data41, Data42, and Data43. EC11 and EC21 is computed by node 417A using Data11, Data21, Data31 and Data41. Similarly, EC12 and EC22 are also computed by node 417A node using Data12, Data22, Data32, and Data42. Finally, EC13 and EC23 are computed by node 417A using Data 13, Data23, Data33, and Data43. Node 417 sends the data fragment 802D2 shards, the data fragment 802D3 shards, the data fragment 802D4 shards, the EC1 shards and the EC2 shards to 5 other nodes which compute the respective parities. This variation can have benefits during rebuild, possibly allowing for more parallelism and also allowing for recovery from individual shard failures without reading all the 3 shards.
In some examples, locally decodable erasure codes (LRC) can be used in a manner consistent with techniques described herein. In one such example, parity can be used to recover from storage devices 427 failures and LRC may be used to recover from node failures.
One possible scheme that may be used to avoid network traffic on failures of storage devices 427 is to add a parity drive to each of nodes 417. For example, each of nodes 417 could have 15 storage devices 427 for data and one storage device 427 for parity. When one of storage devices 427 fails, the drive storing parity can be used to recover the failed data drive. However, with this approach, on every flush where 6 nodes write to one drive each, a given node 417 may have to also update the parity drive. This requires a read modify write operation on both the data drive and the parity drive. Each of the 6 writes turns into 4 drive operations (read old data, read old parity, write new data, write new parity), so every flush operation goes from needing 6 drive operations to 24 drive operations. At least some of the techniques described herein avoid this disadvantage.
In the process illustrated in
Node 417A may store data fragments 802 across nodes 417 (902). For example, again with reference to
Each of nodes 417 may generate secondary data fragments 802 (903). For example, each of nodes 417 that receive one of data fragments 802 shards the respective data fragment 802 into secondary data fragments (e.g., three secondary data fragments or segments for each of data fragments 802). Each of data durability modules 406 within respective nodes 417 also compute a secondary parity data fragment from the other secondary data fragments.
Node 417A may store secondary data fragments 802 across storage devices 427 at each node 417 (904). For example, each of data durability modules 406 store the three secondary data fragments across storage devices 427 included within respective nodes 417 (904). Each of data durability modules 406 also store the parity data fragment in a fourth storage device 427 (see
Node 417A may receive a request for data 801 (YES path from 905). For example, and with reference to
Node 417A may, responsive to the request, identify a subset of fragments sufficient to satisfy the request. Node 417 may access such fragment(s) and output the data, over switch fabric 114, to server 512. However, in some examples, node 417A may reconstruct data 801 (906). For example, data durability module 406A may determine that all of the data fragments are needed to fulfill the request from server 512. To respond to the request, data durability module 406A within node 417A accesses and/or retrieves data fragments 802D1, 802D2, 802D3, and 802D4 over switch fabric 114 within network 108. If data durability modules 406A determines that any of data fragments 802D are not available, data durability module 406A accesses one or more of data fragments 802P. In some examples, each of nodes 417 may reconstruct any data fragment 802 that is not available by using the reliability coding incorporated within the data stored within storage devices 427 included in that node 417. In such an example, any such data fragment 802 that is not available may be recovered using data internal to that node 417, and without incurring additional network traffic to retrieve one or more of data fragments 802 from other nodes 417.
In some examples, perhaps most examples, write and read requests might not be as cleanly separated as illustrated in
For ease of illustration, only a limited number of devices (e.g., data durability modules 406, as well as others) are shown within the Figures and/or in other illustrations referenced herein. However, techniques in accordance with one or more aspects of the present disclosure may be performed with many more of such systems, components, devices, modules, and/or other items, and collective references to such systems, components, devices, modules, and/or other items may represent any number of such systems, components, devices, modules, and/or other items.
The Figures included herein each illustrate at least one example implementation of an aspect of this disclosure. The scope of this disclosure is not, however, limited to such implementations. Accordingly, other example or alternative implementations of systems, methods or techniques described herein, beyond those illustrated in the Figures, may be appropriate in other instances. Such implementations may include a subset of the devices and/or components included in the Figures and/or may include additional devices and/or components not shown in the Figures.
The detailed description set forth above is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a sufficient understanding of the various concepts. However, these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form in the referenced figures in order to avoid obscuring such concepts.
Accordingly, although one or more implementations of various systems, devices, and/or components may be described with reference to specific Figures, such systems, devices, and/or components may be implemented in a number of different ways. For instance, one or more devices illustrated in the Figures herein (e.g.,
Further, certain operations, techniques, features, and/or functions may be described herein as being performed by specific components, devices, and/or modules. In other examples, such operations, techniques, features, and/or functions may be performed by different components, devices, or modules. Accordingly, some operations, techniques, features, and/or functions that may be described herein as being attributed to one or more components, devices, or modules may, in other examples, be attributed to other components, devices, and/or modules, even if not specifically described herein in such a manner.
Although specific advantages have been identified in connection with descriptions of some examples, various other examples may include some, none, or all of the enumerated advantages. Other advantages, technical or otherwise, may become apparent to one of ordinary skill in the art from the present disclosure. Further, although specific examples have been disclosed herein, aspects of this disclosure may be implemented using any number of techniques, whether currently known or not, and accordingly, the present disclosure is not limited to the examples specifically described and/or illustrated in this disclosure.
In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored, as one or more instructions or code, on and/or transmitted over a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another (e.g., pursuant to a communication protocol). In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media, which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.
By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the terms “processor” or “processing circuitry” as used herein may each refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described. In addition, in some examples, the functionality described may be provided within dedicated hardware and/or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.
The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, a mobile or non-mobile computing device, a wearable or non-wearable computing device, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of interoperating hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.
This application claims the benefit of U.S. Provisional Patent Application No. 63/016,137 filed on Apr. 27, 2020, which is hereby incorporated by reference herein in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5126739 | Whiting et al. | Jun 1992 | A |
| 5155484 | Chambers, IV | Oct 1992 | A |
| 5426779 | Chambers, IV | Jun 1995 | A |
| 5525982 | Cheng et al. | Jun 1996 | A |
| 5778255 | Clark et al. | Jul 1998 | A |
| 5805086 | Brown et al. | Sep 1998 | A |
| 6191782 | Mori et al. | Feb 2001 | B1 |
| 7308446 | Panigrahy et al. | Dec 2007 | B1 |
| 7711897 | Chatterjee | May 2010 | B1 |
| 7805392 | Steele et al. | Sep 2010 | B1 |
| 7949683 | Goyal | May 2011 | B2 |
| 8055601 | Pandya | Nov 2011 | B2 |
| 8086609 | Goyal et al. | Dec 2011 | B2 |
| 8156306 | Raizen et al. | Apr 2012 | B1 |
| 8176300 | Goyal et al. | May 2012 | B2 |
| 8180803 | Goyal | May 2012 | B2 |
| 8203805 | Huang et al. | Jun 2012 | B2 |
| 8350732 | Carlson | Jan 2013 | B2 |
| 8473523 | Goyal | Jan 2013 | B2 |
| 8426165 | Cho et al. | Apr 2013 | B2 |
| 8456331 | Carlson | Jun 2013 | B2 |
| 8516456 | Starovoitov et al. | Aug 2013 | B1 |
| 8726253 | Glendenning et al. | May 2014 | B2 |
| 8806296 | Lazier | Aug 2014 | B1 |
| 8819217 | Hussain et al. | Aug 2014 | B2 |
| 8832363 | Sundaram | Sep 2014 | B1 |
| 8886680 | Goyal | Nov 2014 | B2 |
| 8990259 | Billa et al. | Mar 2015 | B2 |
| 9054729 | Carlson | Jun 2015 | B2 |
| 9065482 | Johnston et al. | Jun 2015 | B1 |
| 9077368 | Miller et al. | Jul 2015 | B2 |
| 9083740 | Ma et al. | Jul 2015 | B1 |
| 9166620 | Jayaraman et al. | Oct 2015 | B2 |
| 9203805 | Goyal et al. | Dec 2015 | B2 |
| 9304768 | Ruehle | Apr 2016 | B2 |
| 9325813 | Jayaraman et al. | Apr 2016 | B2 |
| 9363339 | Bhaskar et al. | Jun 2016 | B2 |
| 9419943 | Goyal et al. | Aug 2016 | B2 |
| 9426165 | Billa et al. | Aug 2016 | B2 |
| 9426166 | Billa et al. | Aug 2016 | B2 |
| 9438561 | Goyal et al. | Sep 2016 | B2 |
| 9495479 | Goyal | Nov 2016 | B2 |
| 9507563 | Billa et al. | Nov 2016 | B2 |
| 9514246 | Billa et al. | Dec 2016 | B2 |
| 9547553 | Khermosh et al. | Jan 2017 | B1 |
| 9563399 | Goyal et al. | Feb 2017 | B2 |
| 9584155 | Gopal et al. | Feb 2017 | B1 |
| 9602532 | Goyal et al. | Mar 2017 | B2 |
| 9680500 | Bhaskar et al. | Jun 2017 | B2 |
| 9762544 | Goyal et al. | Sep 2017 | B2 |
| 9768802 | Gopal et al. | Sep 2017 | B2 |
| 9785403 | Goyal et al. | Oct 2017 | B2 |
| 9787693 | Goyal et al. | Oct 2017 | B2 |
| 9823895 | Goyal et al. | Nov 2017 | B2 |
| 9825648 | Gopal et al. | Nov 2017 | B1 |
| 9853660 | Gopal et al. | Dec 2017 | B1 |
| 9904630 | Goyal et al. | Feb 2018 | B2 |
| 10091295 | Savic et al. | Oct 2018 | B1 |
| 10110558 | Goyal et al. | Oct 2018 | B2 |
| 10224956 | Gopal et al. | Mar 2019 | B2 |
| 10224957 | Cassetti et al. | Mar 2019 | B1 |
| 10275312 | Barhate | Apr 2019 | B1 |
| 10277716 | Bhaskar et al. | Apr 2019 | B2 |
| 10339141 | Cafarella et al. | Jul 2019 | B2 |
| 10511324 | Goyal et al. | Dec 2019 | B1 |
| 10528539 | Guilford et al. | Jan 2020 | B2 |
| 10540288 | Noureddine et al. | Jan 2020 | B2 |
| 10841245 | Gray et al. | Jan 2020 | B2 |
| 10565112 | Noureddine et al. | Feb 2020 | B2 |
| 10567458 | Bhaskar et al. | Feb 2020 | B2 |
| 10635419 | Yang et al. | Apr 2020 | B2 |
| 10645187 | Goyal et al. | May 2020 | B2 |
| 10656949 | Billa et al. | May 2020 | B2 |
| 10659254 | Sindhu et al. | May 2020 | B2 |
| 10686729 | Sindhu et al. | Jun 2020 | B2 |
| 10725825 | Sindhu et al. | Jul 2020 | B2 |
| 10727865 | Beckman et al. | Jul 2020 | B2 |
| 10761931 | Goyal et al. | Sep 2020 | B2 |
| 10771090 | Goyal et al. | Sep 2020 | B2 |
| 10812630 | Beckman et al. | Oct 2020 | B2 |
| 10827191 | Dikshit et al. | Nov 2020 | B2 |
| 10827192 | Dikshit et al. | Nov 2020 | B2 |
| 10848775 | Dikshit et al. | Nov 2020 | B2 |
| 10862513 | Thomas et al. | Dec 2020 | B2 |
| 10922026 | Billa et al. | Feb 2021 | B2 |
| 10929175 | Goyal et al. | Feb 2021 | B2 |
| 10931958 | Dikshit | Feb 2021 | B2 |
| 10949303 | Kohli | Mar 2021 | B2 |
| 10983721 | Yang et al. | Apr 2021 | B2 |
| 10990478 | Goyal et al. | Apr 2021 | B2 |
| 10997123 | Beckman et al. | May 2021 | B2 |
| 11048634 | Noureddine et al. | Jun 2021 | B2 |
| 11269738 | Armangau | Mar 2022 | B2 |
| 11288138 | Freilich | Mar 2022 | B1 |
| 11356120 | Markle | Jun 2022 | B1 |
| 20030084020 | Shu | May 2003 | A1 |
| 20030204584 | Zeira et al. | Oct 2003 | A1 |
| 20050229251 | Chellapilla et al. | Oct 2005 | A1 |
| 20060069872 | Bouchard et al. | Mar 2006 | A1 |
| 20060075206 | Bouchard et al. | Apr 2006 | A1 |
| 20060123321 | Deenadhayalan | Jun 2006 | A1 |
| 20070150497 | De La Cruz et al. | Jun 2007 | A1 |
| 20080101371 | Law et al. | May 2008 | A1 |
| 20090077252 | Abdo et al. | Mar 2009 | A1 |
| 20090138440 | Goyal | May 2009 | A1 |
| 20100114973 | Goyal | May 2010 | A1 |
| 20100246663 | Citta et al. | Sep 2010 | A1 |
| 20100332401 | Prahlad et al. | Dec 2010 | A1 |
| 20110016154 | Goyal et al. | Jan 2011 | A1 |
| 20110107379 | LaJoie et al. | May 2011 | A1 |
| 20110307659 | Hans et al. | Dec 2011 | A1 |
| 20120262314 | Carlson | Oct 2012 | A1 |
| 20120286979 | Carlson | Nov 2012 | A1 |
| 20120331554 | Goyal et al. | Dec 2012 | A1 |
| 20130031446 | Kamiya | Jan 2013 | A1 |
| 20130046784 | Ruehle | Feb 2013 | A1 |
| 20130133064 | Goyal et al. | May 2013 | A1 |
| 20130249716 | Carlson | Sep 2013 | A1 |
| 20130324900 | Wariar et al. | Dec 2013 | A1 |
| 20140101187 | Chao et al. | Apr 2014 | A1 |
| 20140129775 | Ruehle | May 2014 | A1 |
| 20140324793 | Glazemakers | Oct 2014 | A1 |
| 20140380126 | Yekhanin et al. | Dec 2014 | A1 |
| 20150067123 | Goyal et al. | Mar 2015 | A1 |
| 20150067776 | Billa et al. | Mar 2015 | A1 |
| 20150067863 | Billa et al. | Mar 2015 | A1 |
| 20150244804 | Warfield et al. | Aug 2015 | A1 |
| 20150295889 | Goyal et al. | Oct 2015 | A1 |
| 20150295891 | Goyal et al. | Oct 2015 | A1 |
| 20160077746 | Muth et al. | Mar 2016 | A1 |
| 20160182088 | Sipos et al. | Jun 2016 | A1 |
| 20160239384 | Slik et al. | Aug 2016 | A1 |
| 20160314043 | Slik | Oct 2016 | A1 |
| 20160378401 | Savic et al. | Dec 2016 | A1 |
| 20170090765 | Balinski et al. | Mar 2017 | A1 |
| 20170187398 | Trusov et al. | Jun 2017 | A1 |
| 20170250712 | Chiu et al. | Aug 2017 | A1 |
| 20170277713 | Strauss et al. | Sep 2017 | A1 |
| 20180004600 | Danilov et al. | Jan 2018 | A1 |
| 20180018235 | Arslan et al. | Jan 2018 | A1 |
| 20180287965 | Sindhu et al. | Oct 2018 | A1 |
| 20180293168 | Noureddine et al. | Oct 2018 | A1 |
| 20180331784 | Chiu et al. | Nov 2018 | A1 |
| 20190012278 | Sindhu et al. | Jan 2019 | A1 |
| 20190012350 | Sindhu et al. | Jan 2019 | A1 |
| 20190013965 | Sindhu et al. | Jan 2019 | A1 |
| 20190104206 | Goel et al. | Apr 2019 | A1 |
| 20190104207 | Goel et al. | Apr 2019 | A1 |
| 20190158428 | Gray et al. | May 2019 | A1 |
| 20190026034 | Tashiro et al. | Jun 2019 | A1 |
| 20190182114 | Tavridis et al. | Jun 2019 | A1 |
| 20190188079 | Kohli | Jun 2019 | A1 |
| 20190243765 | Noureddine et al. | Aug 2019 | A1 |
| 20190312915 | LaJoie et al. | Oct 2019 | A1 |
| 20190370170 | Oltean et al. | Dec 2019 | A1 |
| 20200019339 | Yang et al. | Jan 2020 | A1 |
| 20200019391 | Yang et al. | Jan 2020 | A1 |
| 20200019404 | Billa et al. | Jan 2020 | A1 |
| 20200021664 | Goyal et al. | Jan 2020 | A1 |
| 20200119903 | Thomas et al. | Apr 2020 | A1 |
| 20200133771 | Goyal et al. | Apr 2020 | A1 |
| 20200142642 | Billa et al. | May 2020 | A1 |
| 20200145020 | Goyal et al. | May 2020 | A1 |
| 20200145680 | Dikshit et al. | May 2020 | A1 |
| 20200145681 | Dikshit et al. | May 2020 | A1 |
| 20200145682 | Dikshit et al. | May 2020 | A1 |
| 20200145683 | Dikshit | May 2020 | A1 |
| 20200151101 | Noureddine et al. | May 2020 | A1 |
| 20200159568 | Goyal et al. | May 2020 | A1 |
| 20200159840 | Beckman et al. | May 2020 | A1 |
| 20200159859 | Beckman et al. | May 2020 | A1 |
| 20200162100 | Beckman et al. | May 2020 | A1 |
| 20200162101 | Beckman et al. | May 2020 | A1 |
| 20200162584 | Beckman et al. | May 2020 | A1 |
| 20200169268 | Billa et al. | May 2020 | A1 |
| 20200183841 | Noureddine et al. | Jun 2020 | A1 |
| 20200228148 | Thomas et al. | Jul 2020 | A1 |
| 20200250032 | Goyal et al. | Aug 2020 | A1 |
| 20200278866 | Billa et al. | Sep 2020 | A1 |
| 20200280462 | Sindhu et al. | Sep 2020 | A1 |
| 20200314026 | Sindhu et al. | Oct 2020 | A1 |
| 20200394066 | Senthil Nayakam et al. | Dec 2020 | A1 |
| 20210097047 | Billa et al. | Apr 2021 | A1 |
| 20210097082 | Billa | Apr 2021 | A1 |
| 20210097108 | Goyal et al. | Apr 2021 | A1 |
| 20210255925 | Bent | Aug 2021 | A1 |
| 20210281394 | Thomas et al. | Sep 2021 | A1 |
| 20210294662 | Goyal et al. | Sep 2021 | A1 |
| 20210295181 | Goyal et al. | Sep 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 112012002624 | Apr 2014 | DE |
| 0913825 | Jun 1999 | EP |
| 2215563 | Aug 2010 | EP |
| 2215565 | Aug 2010 | EP |
| 2276217 | Jan 2011 | EP |
| 2750018 | Jul 2014 | EP |
| 2003023553 | Mar 2003 | WO |
| 2007079095 | Jul 2007 | WO |
| 2009070191 | Jun 2009 | WO |
| 2009070192 | Jun 2009 | WO |
| 2012177736 | Dec 2012 | WO |
| 2013078053 | May 2013 | WO |
| Entry |
|---|
| Evans, Chris, “Erasure coding vs Raid for disk rebuilds”, ComputerWeekly.com, May 14, 2015. (Year: 2015). |
| “Hash Table,” Wikipedia, the free encyclopedia, retrieved from https:l/en.wikipedla.orglw/Index.php?title=Hash_table&oldld=866368727, on Mar. 4, 2020, 18 pp. |
| “Lempel-Ziv-Markov chain algorithm,” Wikipedia, the free encyclopedia, accessed from https://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm on or about May 7, 2019, last edit made Mar. 17, 2019, 15 pp. |
| “Lloyd/ easylzma,” GitHub, accessed from https://github.com/lloyd/easylzma on or about May 7, 2019, last commit made Apr. 1, 2009, 2 pp. |
| “Range encoding,” Wikipedia, the free encyclopedia, accessed from https://en.wikipedia.org/wiki/Range_encoding on or about May 7, 2019, last edit made Jan. 2, 2019, 6 pp. |
| “Sort Benchmark Home Page,” retrieved from http://sortbenchmark.org/ on Feb. 4, 2020, 13 pp. |
| Becchi et al., “Efficient Regular Expression Evaluation: Theory to Practice,” Proceedings of the 2008 ACM/IEEE Symposium on Architecture for Networking and Communications Systems, Nov. 6, 2008, pp. 50-59. |
| Becchi et al., “Evaluating Regular Expression Matching Engines on Network and General Purpose Processors,” Proceedings of the 2009 ACM/IEEE Symposium on Architecture for Networking and Communications Systems, Oct. 19, 2009, pp. 30-39. |
| Bille et al., “Subsequence Automata with Default Transitions,” Journal of Discrete Algorithms, vol. 44, Jun. 13, 2017, pp. 48-55. |
| Deutsch, “DEFLATE Compressed Data Format Specification version 1.3,” Internet Draft Deflate 1.3, Feb. 1, 1996, 15 pp. |
| Deutsch, “DEFLATE Compressed Data Format Specification version 1.3,” Network Working Group, RFC 1951, May 1996, 15 pp. |
| Ficara et al., “An Improved FDA for Fast Regular Expression Matching,” ACM SIGCOMM Computer Communication Review, vol. 38, No. 5, Oct. 2008, pp. 29-40. |
| Xu et al., “Expander code: A scalable erasure-resilient code to keep up with data growth in distributed storage,” 2013 IEEE 32nd International Performance Computing and Communications Conference (IPCCC), San Diego, CA, Dec. 6-8, 2013, 9 pp. |
| Gogte et al., “HARE: Hardware Accelerator for Regular Expressions,” 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Oct. 2016, 12 pp. |
| Chen et al., “Improving Coding Performance and Energy Efficiency of Erasure Coding Process for Storage Systems-A Parallel and Scalable Approach,” 2016 IEEE 9th International Conference on Cloud Computing (CLOUD), San Francisco, CA, Jun. 27-Jul. 2, 2016, pp. 933-936. |
| Huang et al., “Erasure Coding in Windows Azure Storage,” Microsoft Corporation, USENIX ATC'12 Proceedings of the 2012 USENIX conference on Annual Technical Conference, Jun. 13-15, 2012, 12 pp. |
| Huang et al., “Pyramid Codes: Flexible Schemes to Trade Space for Access Efficiency in Reliable Data Storage Systems,” Microsoft Corporation, 2007 (Applicant points out, in accordance with MPEP 609.04(a), that the year of publication, 2007, is sufficiently earlier than the effective U.S. filing date, so that the particular month of publication is not in issue.) 14 pp. |
| Kumar et al., “Advanced Algorithms for Fast and Scalable Deep Packet Inspection,” Proceedings of the 2006 ACM/IEEE Symposium, Dec. 3, 2006, pp. 81-92. |
| Martin, “Range encoding: an algorithm for removing redundancy from a digitised message,” Video & Data Recording Conference, Mar. 1979, 11 pp. |
| Qi et al., “FEACAN: Front-End Acceleration for Content-Aware Network Processing,” 2011 Proceedings IEEE Infocom, Apr. 10-15, 2011, pp. 2114-2122. |
| Salomon, “Data Compression: The Complete Reference,” Chapter 3, Dictionary Methods, Jan. 1998, pp. 101-162. |
| Schroeder et al., “Flash Reliability in Production: The Expected and the Unexpected,” 14th USENIX Conference on File and Storage Technologies (FAST '16), Feb. 22-25, 2016, pp. 67-80. |
| Sidhu et al., “Fast Regular Expression Matching Using FPGAs,” Proceedings of the 9th Annual IEEE Symposium on Field-Programmable Custom Computing Machines (FCCM'01), Apr. 29, 2001, 12 pp. |
| Van Lunteren et al., “Designing a Programmable Wire-Speed Regular-Expression Matching Accelerator,” 2012 IEEE/ACM 45th Annual International Symposium on Microarchitecture, Dec. 1-5, 2012, pp. 461-472. |
| Zheng et al., “Algorithms to Speedup Pattern Matching for Network Intrusion Detection Systems,” Computer Communications, vol. 62, Feb. 2015. pp. 47-58. |
| U.S. Appl. No. 17/360,619, filed Jun. 28, 2021, naming inventors Noureddine et al. |
| Sathiamoorthy et al., “XORing Elephants: Novel Erasure Codes for Big Data,” arXiv: 1301.3791v1, Jan. 16, 2013, 16 pp. |
| Number | Date | Country | |
|---|---|---|---|
| 20210334184 A1 | Oct 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 63016137 | Apr 2020 | US |
