Method and System for Fast Ordered Writes with Atomic Multicast

Abstract

In general, embodiments of the technology relate to a method and system for performing fast ordered writes in a storage appliance that includes multiple separate storage modules. More specifically, embodiments of the technology enable multicasting of data to multiple storage modules in a storage appliance, where the order in which the write requests are processed is the same across all storage modules in the storage appliance.

Description

BACKGROUND

Storage appliances include storage media and provide a mechanism to enable clients to write data to and to read data from the storage media.

BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1A-1C show systems in accordance with one or more embodiments of the technology.

FIGS. 2-4B show flowcharts in accordance with one or more embodiments of the technology.

DETAILED DESCRIPTION

In the following detailed description of embodiments of the technology, numerous specific details are set forth in order to provide a more thorough understanding of the technology. However, it will be apparent to one of ordinary skill in the art that the technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

In the following description of FIGS. 1A-4B, any component described with regard to a figure, in various embodiments of the technology, may be equivalent to one or more like-named components described with regard to any other figure. For brevity, descriptions of these components will not be repeated with regard to each figure. Thus, each and every embodiment of the components of each figure is incorporated by reference and assumed to be optionally present within every other figure having one or more like-named components. Additionally, in accordance with various embodiments of the technology, any description of the components of a figure is to be interpreted as an optional embodiment which may be implemented in addition to, in conjunction with, or in place of the embodiments described with regard to a corresponding like-named component in any other figure.

In general, embodiments of the technology relate to a method and system for performing fast ordered writes in a storage appliance that includes multiple separate storage modules. More specifically, embodiments of the technology enable multicasting of data to multiple storage modules in a storage appliance, where the order in which the write requests are processed is the same across all storage modules in the storage appliance. Embodiments of the technology enable the aforementioned ordering without requiring the clients to coordinate the writing of data to the storage appliance. In particular, embodiments of the technology enforce ordering of write requests using a fabric (e.g., a PCIe fabric) in combination with storage modules that include global submission queues (see e.g., FIG. 4A) or per-client submission queues (see e.g., FIG. 4B).

In one embodiment of the technology, the ordering is consistent across the storage modules in the storage modules such that the data stored in the individual storage modules (or a subset of the storage modules) is the same. For example, consider a scenario in which the storage appliance includes three storage modules (1, 2, 3). Further, assume that client A issues a first write request (A) specifying an object and offset and Client B issues a second write request (B) specifying the same object and offset. Embodiments of the technology ensure that the write requests (A, B) are processed in the same order on each of the storage modules such that after both write requests have been processed on the three storage modules (1, 2, 3), the data stored for the specific object and offset is the same on each of the three storage modules (1, 2, 3).

FIG. 1A shows a system in accordance with one or more embodiments of the technology. Referring to FIG. 1A, the system includes one or more clients (client A (108A)-client M (108M)) that are operatively connected to a storage appliance (100).

In one embodiment of the technology, the clients (108A-108M) may be any type of physical system that includes functionality to issue a read request to the storage appliance (100) and/or to issue a write request to the storage appliance (100). Though not shown in FIG. 1A, each of the clients (108A-108M) may include a client processor (not shown), client memory (not shown), and any other software and/or hardware necessary to implement one or more embodiments of the technology.

In one embodiment of the technology, the clients (108A-108M) are configured to execute an operating system (OS) that includes a file system. The file system provides a mechanism for the storage and retrieval of files from the storage appliance (100). More specifically, the file system includes functionality to perform the necessary actions to issue read requests and write requests to the storage appliance (100). The file system may also provide programming interfaces to enable the creation and deletion of files, reading and writing of files, performing seeks within a file, creating and deleting directories, managing directory contents, etc. In addition, the file system also provides management interfaces to create and delete file systems. In one embodiment of the technology, to access a file, the operating system (via the file system) typically provides file manipulation interfaces to open, close, read, and write the data within each file and/or to manipulate the corresponding metadata. In another embodiment of the technology, the client is configured to execute a block driver that is used to access the storage appliance. In another embodiment of the technology, the client is configured to execute an application programming interface (API) that enables the client to access the storage appliance.

In one embodiment of the technology, the clients (108A-108M) interface with the fabric (106) of the storage appliance (100) to communicate with the storage appliance (100), as further described below in FIG. 2.

In one embodiment of the technology, the storage appliance (100) is a system that includes persistent storage, and is configured to service read requests and/or write requests from one or more clients (108A-108M).

The storage appliance (100), in accordance with one or more embodiments of the technology, includes one or more storage modules (104A-104N) organized in a storage array (102) and a fabric (106) that interfaces the storage module(s) (104A-104N) with the clients (108A-108M). Each of these components is described below.

The storage array (106), in accordance with an embodiment of the technology, accommodates one or more storage modules (104A-104N). The storage array may enable a modular configuration of the storage appliance, where storage modules may be added to or removed from the storage appliance (100), as needed or desired. A storage module (104), in accordance with an embodiment of the technology, is described below, with reference to FIG. 1C.

Continuing with the discussion of the storage appliance (100), the storage appliance includes the fabric (106). The fabric (106) may provide physical connectivity between the clients (108A-108M) and the storage module(s) (104A-104N) using one or more of the following protocols: Peripheral Component Interconnect (PCI), PCI-Express (PCIe), PCI-eXtended (PCI-X), Non-Volatile Memory Express (NVMe), Non-Volatile Memory Express (NVMe) over a PCI-Express fabric, Non-Volatile Memory Express (NVMe) over an Ethernet fabric, and Non-Volatile Memory Express (NVMe) over an Infiniband fabric. Those skilled in the art will appreciate that the technology is not limited to the aforementioned protocols. A fabric (106) in accordance with an embodiment of the technology, is described below, with reference to FIG. 1B.

FIG. 1B shows a fabric in accordance with one or more embodiments of the technology. The fabric (106) includes one or more leaf tier switches (110A, 110M) and one or more hub tier switches (112A, 112N). Each of these components is described below.

With respect to the leaf tier switches (110A, 110M), each leaf tier switch is a physical PCIe switch that is connected to one or more clients (108) and each client (108) may be connected to one or more leaf tier switches. Each leaf tier switch is also connected to one or more hub tier switches. In one embodiment of the technology, each leaf tier switch is connected to every hub tier switch in the fabric resulting in a fully connected mesh between the tiers in the fabric. Embodiments of the technology may be implemented without a fully connected mesh between the tiers without departing from the technology.

In one embodiment of the technology, each leaf tier switch is configured to: (i) receive transaction layer packets (TLPs) from the clients with which it is connected and route the TLPs to the appropriate hub tier switch using address routing and (ii) receive TLPs from one or more hub tier switches to which the leaf tier switch is connected and route the TLPs to the appropriate client using address routing.

With respect to the hub tier switches (112A-112N), each hub tier switch is a physical PCIe switch that is connected to one or more leaf tier switches (110A-110M) and to one or more storage modules (104). Further, each storage module may be connected to one or more hub tier switches. In one embodiment of the technology, each hub tier switch is connected to every other leaf tier switch in the fabric resulting in a fully connected mesh between the tiers in the fabric. Embodiments of the technology may be implemented without a fully-connected mesh between the tiers without departing from the technology.

In one embodiment of the technology, each hub tier switch is configured to: (i) receive TLPs from the storage modules (104) with which it is connected and route the TLPs to the appropriate leaf tier switch using address routing and (ii) receive TLPs from one or more leaf tier switches to which the hub tier switch is connected and route the TLPs to the appropriate storage modules (104) using address routing.

In addition, one or more hub tier switches are configured to generate multiple TLPs in response to receiving a single TLP from a leaf tier switch. For example, a client (108) may generate and send a TLP that includes a multicast address to a leaf tier switch. The leaf tier switch evaluates the multicast address and routes, based on the evaluation, the TLP to a hub tier switch. The hub tier switch subsequently receives the TLP and evaluates the address, which based on the evaluation, is identified as a multicast address. Based on the evaluation, N number of TLPs are generated. Each of the N number of TLPs is then transmitted to a different storage module.

In one embodiment of the technology, the fabric may concurrently receive TLPs from multiple clients. However, the fabric serializes the transmission to the storage modules. More specifically, in one embodiment of the technology, each hub tier switch serializes the processing of TLPs it receives from the various clients. As a result, the hub tier switch also serializes the transmission of TLPs to the storage modules. In this manner, hub tier switch ensures that the TLPs it issues to each storage module are issued in the same order.

In one embodiment of the technology, the fabric is configured to perform all or a portion of the methods shown in FIG. 3.

FIG. 1C shows a storage module (104), in accordance with one or more embodiments of the technology. The storage module (104) includes a storage module controller (120), a buffer (122), one or more submission queues (124A, 104N), and persistent storage (126). Each of these components is described below.

In one embodiment of the technology, the storage module controller (120) is configured to receive and service requests to read from and/or write data to the persistent storage (122). These requests may originate from the clients (108) and may be conveyed to the storage module controller (120) via the fabric (106). Further, the storage module controller (124) may be programmed to, or otherwise configured to, perform all or a portion of the methods shown in FIGS. 4A and 4B.

In one embodiment of the technology, the storage module controller (120) includes a processor (not shown) (e.g., one or more cores, or micro-cores of a processor that are configured to execute instructions), one or more hardware registers (not shown), and memory (shown) (e.g., volatile memory that may be, but is not limited to, dynamic random-access memory (DRAM), synchronous DRAM, SDR SDRAM, and DDR SDRAM) to perform all or a portion of the method described in FIGS. 4A and/or 4B. Alternatively, the storage module controller (120) may be implemented using a field-programmable gate array (FPGA) and/or an application-specific integrated circuit (ASIC).

In one embodiment of the technology, the persistent storage (126) of the storage module (104) may include, but is not limited to, Spin-Torque Magnetic RAM (ST-MRAM), Phase Change Memory (PCM), NAND Flash memory and NOR Flash memory. Those skilled in the art will appreciate that the technology is not limited to aforementioned types of persistent storage.

In one embodiment of the technology, the buffer (122) is volatile memory that temporarily stores data received by the storage module prior to the data being stored in the persistent storage.

In one embodiment of the technology, the storage module may include one or more submission queues (124A, 124N). The submission queues store commit requests in the order in which they are received by the storage module. In one embodiment of the technology, the storage module includes a single global submission queue in which all commit requests received from all clients are stored. In another embodiment of the technology, the storage module includes a set of submission queues, where each submission queue only stores commit requests received from a specific client.

One skilled in the art will recognize that the architecture of the system is not limited to the components shown in FIGS. 1A-1C. For example, the components of the storage appliance (100) may include any number of storage modules (104A-104N). Further, the storage module controller (120) of the storage module (104) may be equipped with central processing units (CPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs) and/or any other type of processor, without departing from the technology. In addition, the fabric may be based on communication standards other than those discussed above, without departing from the technology.

FIGS. 2-4A show flowcharts in accordance with one or more embodiments of the technology. While the various steps in the flowchart are presented and described sequentially, one of ordinary skill in the art will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel. In one embodiment of the technology, each of the steps shown in FIGS. 2-4A may be performed in parallel with any of the other steps shown in FIGS. 2-4A.

FIG. 2 shows a method for writing data to the storage modules from the perspective of a client in accordance with one or more embodiments of the technology.

In Step 200, a client generates a write request that includes an address and data. The address included in the write request may be a multicast address. More specifically, the address may appear, from the perspective of the client, as any other address; however, when this address is evaluated by the fabric, it results in the generation of N number of write requests (see e.g., FIG. 3, step 304).

In Step 202, the write request is issued by the client to the fabric. If the fabric is a PCIe fabric (i.e., a fabric that implements PCIe), then the write request may be issued using one or more TLPs.

In Step 204, the client generates a commit request which includes a multicast address (i.e., the same address that was included in the write request generated in Step 200). While the write request includes the data to be transmitted, the commit request includes the command which results in the data being stored in the persistent storage (see e.g., FIGS. 4A, 4B).

In Step 206, the commit request is issued by the client to the fabric. If the fabric is a PCIe fabric (i.e., a fabric that implements PCIe), then the commit request may be issued using one or more TLPs.

At some later point in time, in Step 208, the client receives N commit replies. More specifically, because the client issued a write request and a commit request with a multicast address, N number of copies of the data will be written to the storage array. For each copy of the data written to the storage array, the fabric will receive a commit reply (see e.g., FIG. 3, Step 316). Each time a commit reply is received by the fabric it is transmitted to the client that issued the commit request. The client will continue to receive commit replies associated with a given commit request until it has received N commit replies (assuming that the commit requests were successfully processed by the individual storage modules and the fabric remains operational).

FIG. 3 shows a method for writing data to the storage modules from the perspective of the fabric in accordance with one or more embodiments of the technology.

In Step 300, a write request that includes data and a multicast address is received by a leaf switch in the fabric. If the fabric is a PCIe fabric, then the write request may be received as one or a set of TLPs.

In Step 302, the leaf switch processes the write request. More specifically, the processing of the write request includes evaluating the address in the write request to identify a hub tier switch (or an egress port operatively connected to the hub tier switch). The result of the processing is the transmission of the write request to a hub tier switch. In one embodiment of the technology, a specific hub tier switch processes all write requests for a given multicast address. For example, all write requests received by the fabric that include address A are routed to hub tier switch A.

In Step 304, in response to receiving the write request, the hub tier switch processes the write request. The processing of the write request includes determining that the address in the write request is a multicast address and, in response to this determination, generating N number of write requests. The specific number (N) of write requests generated is pre-configured on the hub tier switch. In one embodiment of the technology, each of the generated write requests includes a unicast address.

In Step 306, the generated write requests (i.e., the write requests generated in step 304) are transmitted to the storage modules. More specifically, each of the N write requests is transmitted to a different storage module in the storage appliance.

In Step 308, a commit request that includes the multicast address is received by a leaf switch in the fabric. If the fabric is a PCIe fabric, then the commit request may be received as one or a series of TLPs.

In Step 310, the leaf switch processes the commit request. More specifically, the processing of the commit request includes evaluating the address in the commit request to identify a hub tier switch (or an egress port operatively connected to the hub tier switch). The result of the processing is the transmission of the commit request to a hub tier switch (i.e., the hub tier switch to which the write request in step 302 was sent).

In Step 312, in response to receiving the commit request, the hub tier switch processes the commit request. The processing of the commit request includes determining that the address in the commit request is a multicast address and, in response to this determination, generating N number of commit requests. The specific number (N) of commit requests generated is pre-configured on the hub tier switch. In one embodiment of the technology, each of the generated commit requests includes a unicast address.

In Step 314, the generated commit requests (i.e., the commit requests generated in step 312) are transmitted to the storage modules (i.e., to the same storage modules to which the write requests were sent in Step 306). More specifically, each of the N commit requests is transmitted to one of N a storage modules in the storage appliance, where a commit request is received by each of the storage modules after the corresponding write request is received by the storage module.

In one embodiment of the technology, a specific hub switch receives all commit requests associated with a specific multicast address. The order in which the hub switch receives commit requests for a specific multicast address is maintained such that the N commit requests generated in step 312 are generated in the same order to all target storage modules.

For example, consider a scenario in which there are two clients (C1, C2) and three storage modules (SM 1, SM 2, SM 3). In this scenario, C1 may issue a first commit request (CR1) that includes an address and C2 may issue a second commit request (CR2) that includes the same address. As discussed above, both of these commit requests will be routed through the fabric to the same hub switch. In this example, assume that CR1 is received prior to CR2. Accordingly, CR1 is processed first resulting in the generation and transmission of three commit requests (CR1₁, CR1₂, CR1₃). CRL₁ is transmitted to SM 1, CR1₂ is transmitted to SM 2, and CR1₃ is transmitted to SM 3. The hub switch then processes CR2 after CR1 has been processed. The result of processing CR 2 resulting in the generation and transmission of three commit requests (CR2₁, CR2₂, CR2₃). CR2₁ is transmitted to SM 1, CR2₂ is transmitted to SM 2, and CR2₃ is transmitted to SM 3. Because the hub tier switch processes CR1 and CR2 specific order, the resulting CRs issued from the hub tier switch to the storage modules are also issued in the same order (e.g., CR1₁ is issued to SM 1 before CR2¹ is issued to SM1). The order in which the CRs are issued to a given storage module also results in the CRs being received in this order and subsequently processed in this order.

At some later point in time, in Step 316, the fabric receives N commit replies. More specifically, because the fabric issued N write requests and N commit requests, N number of copies of the data will be written to the storage array. For each copy of the data written to the storage array, the fabric will receive a commit reply from a corresponding storage module (see e.g., FIG. 4A, step 410, and FIG. 4B, step 430). Each time a commit reply is received by the fabric it is transmitted to the client that issued the commit request. The fabric will continue to receive commit replies until it has received N commit replies (assuming that the commit requests were successfully processed by the individual storage modules).

FIG. 4A shows a method for writing data to the storage modules from the perspective of a storage module with a single global submission queue in accordance with one or more embodiments of the technology.

In Step 400, a unicast write request is received from the fabric (i.e., a write request issued by the fabric in Step 306).

In Step 402, the data associated with the write request is stored in a buffer of the storage module.

In Step 404, a unicast commit request is received from the fabric (i.e., a commit request issued by the fabric in Step 312).

In Step 406, the unicast commit request is placed in the single global submission queue. In one embodiment of the technology, the commit request includes or is otherwise associated with a client identifier. The client identifier is used in the generation and/or processing of the corresponding commit reply (see e.g., FIG. 4, step 410).

In Step 408, the commit requests in the global submission queue are processed based on a first-in first-out mechanism in order to preserve the processing order. The result of processing a commit request is the storing of data in the persistent storage. More specifically, consider a scenario in which the storage module received write request 1 (WR1) and subsequently commit request 1 (CR1). The result of processing CR1 is that data associated with WR1 is copied from the buffer in the storage module to a location in persistent storage.

In Step 410, a commit reply is generated and sent to the fabric. In one embodiment of the technology, the commit reply is generated based, as least in part, on the client identifier associated with the corresponding commit request.

In one embodiment of the technology, steps 400-406 are performed in parallel with steps 408-410. Specifically, steps 400-406 result in the population of the global submission queue while steps 408-410 result in the processing of commit requests in the global submission queue.

FIG. 4B shows a method for writing data to the storage modules from the perspective of a storage module with a per-client submission queue in accordance with one or more embodiments of the technology.

In Step 420, a unicast write request is received from the fabric (i.e., a write request issued by the fabric in Step 306).

In Step 422, the data associated with the write request is stored in the buffer of the storage module.

In Step 424, a unicast commit request is received from the fabric (i.e., a commit request issued by the fabric in Step 312).

In Step 426, the unicast commit request is placed in a per-client submission queue. In one embodiment of the technology, the commit request includes or is otherwise associated with a global sequencing tag. The global sequencing tags are used to maintain a global processing order of commit requests received by the storage module. The global sequencing tag may be added to (or otherwise associated with) the commit request by the storage module controller.

In Step 428, the commit requests in the per-client submission queues are processed based on the global sequencing tags in order to preserve the global processing order in the storage module. Said another way, the order in which the data is written to the persistent storage is dictated by the order in which the commit requests are processed and not by the order in which the write requests (and/or data) is received by the storage module. The result of processing a commit request is the storing of data in the persistent storage. More specifically, consider a scenario in which the storage module received write request 1 (WR1) and subsequently commit request 1 (CR1). The result of processing CR1 is that data associated with WR1 is copied from the buffer in the storage module to a location in persistent storage.

In Step 430, a commit reply is generated and sent to the fabric. In one embodiment of the technology, the commit reply is generated based, as least in part, on the client identifier associated with the per-client submission queue in which the commit request was stored.

In one embodiment of the technology, steps 420-426 are performed in parallel with steps 428-430. Specifically, steps 420-426 result in the population of the per-client submission queues while steps 428-430 result in the processing of commit requests in the per-client submission queues.

While the technology has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the technology as disclosed herein. Accordingly, the scope of the technology should be limited only by the attached claims.

Claims

1. A method for processing write requests comprising: receiving, from a first client and by a fabric, a first write request comprising a first data;transmitting, by the fabric and in response to the first write request, a second write request to a first storage module and a third write request to a second storage module;receiving, from a second client and by the fabric, a fourth write request comprising a second data;transmitting, by the fabric and in response to the fourth write request, a fifth write request to the first storage module and a sixth write request to the second storage module;transmitting, by the fabric, a first commit request associated with the second write request and a second commit request associated with the fifth write request to the first storage module;transmitting, by the fabric, a third commit request associated with the third write request and a fourth commit request associated with the sixth write request to the second storage module;processing, by the first storage module, the first commit request and the second commit request, wherein, based on an order, the first commit request is processed before the second commit request; andprocessing, by the second storage module, the third commit request and the fourth commit request, wherein based on the order, the third commit request is processed before the fourth commit request.

2. The method of claim 1, further comprising: receiving, by the fabric, a fifth commit request from the first client;generating, in response to the fifth commit request, the first commit request and the third commit request.

3. The method of claim 2, further comprising: receiving, after transmitting the first commit request, a first commit reply from the first storage module;receiving, after transmitting third commit request, a second commit reply from the second storage module;transmitting the first commit reply to the first client; andtransmitting the second commit reply to the first client.

4. The method of claim 1, wherein the fabric is a Peripheral Component Interconnect Express (PCIe) fabric.

5. The method claim 1, wherein the order is specified by the fabric.

6. The method of claim 1, wherein the first write request is received by a leaf tier switch in the fabric and the second write request is transmitted to the first storage module by a hub tier switch in the fabric.

7. The method of claim 6, wherein the fourth write request is received by a second leaf tier switch in the fabric and the fifth write request is transmitted to the first storage module by the hub tier switch.

8. The method of claim 6, wherein the leaf tier switch transmits the first write request to the hub tier switch,wherein the hub tier switch generates the second write request and the third write request in response to receiving the first write request from the leaf tier switch.

9. The method of claim 1, wherein the first storage module comprises phase change storage media.

10. The method of claim 1, wherein processing the first commit request and the second commit request comprises: storing, based on the order, the first commit request and the second commit request in a global submission queue,wherein the order is specified by a hub tier switch in the fabric.

11. The method of claim 10, wherein the first commit request and the second commit request are processed from the global submission queue based on the order,wherein processing the first commit request comprises storing a copy of the first data in persistent storage in the first storage module.

12. The method of claim 10, wherein the first commit request is associated with a first client identifier specifying the first client and the second commit request is associated with a second client identifier specifying the second client.

13. The method of claim 1, wherein processing the first commit request and the second commit request comprises: storing the first commit request in a first client submission queue associated with first client and storing the second commit request in a second client submission queue associated with the second client,wherein the first commit request is associated with a first global sequencing tag and the second commit request is associated with a second global sequencing tag,wherein the first global sequencing tag and the second global sequencing tag are based on the order,wherein the order is specified by a hub tier switch in the fabric.

14. The method of claim 13, wherein the first commit request and the second commit request are processed based on the first global sequencing tag and the second global sequencing tag,wherein processing the first commit request comprises storing a copy of the first data in persistent storage in the first storage module.

15. A storage appliance, comprising: a fabric;a storage array operatively connected to the fabric and comprising a first storage module and the second storage module,wherein the fabric is programmed to: receive, from a first client, a first write request comprising a first data;transmitting, in response to the first write request, a second write request to the first storage module and a third write request to the second storage module;receive, from a second client, a fourth write request comprising a second data;transmit, in response to the fourth write request, a fifth write request to the first storage module and a sixth write request to the second storage module;transmit a first commit request associated with the second write request and a second commit request associated with the fifth write request to the first storage module; andtransmit a third commit request associated with the third write request and a fourth commit request associated with the sixth write request to the second storage module;wherein the first storage module is configured to process the first commit request and the second commit request, wherein based on an order, the first commit request is processed before the second commit request, wherein the order is specified by the fabric; andwherein the second storage module is configured to process the third commit request and the fourth commit request, wherein based on the order, the third commit request is processed before the fourth commit request.

16. The storage appliance of claim 15, wherein the fabric is further configured to: receive, by the fabric, a fifth commit request from the first client;generate, in response to receiving the fifth commit request, the first commit request and the third commit request;receive, after transmitting the first commit request, a first commit reply from the first storage module;receive, after transmitting the third commit request, a second commit reply from the second storage module;transmit the first commit reply to the first client; andtransmit the second commit reply to the first client.

17. The storage appliance of claim 15, wherein the first write request is received by a leaf tier switch in the fabric,wherein the leaf tier switch transmits the first write request to the hub tier switch,wherein the hub tier switch generates the second write request and the third write request in response to receiving the first write request from the leaf tier switch,wherein the hub tier switch transmits the second write request to the first storage module.

18. The storage appliance of claim 15, wherein processing the first commit request and the second commit request comprises: storing, based on the order, the first commit request and the second commit request in a global submission queue,wherein the first commit request and the second commit request are processed from the global submission queue based on the order,wherein processing the first commit request comprises storing a copy of the first data in persistent storage in the first storage module.

19. The storage appliance of claim 15, wherein processing the first commit request and the second commit request comprises: storing the first commit request in a first client submission queue associated with first client and storing the second commit request in a second client submission queue associated with the second client,wherein the first commit request is associated with a first global sequencing tag and the second commit request is associated with a second global sequencing tag, wherein the first global sequencing tag and the second global sequencing tag are based on the order,wherein the first commit request and the second commit request are processed based on the first global sequencing tag and the second global sequencing tag,wherein processing the first commit request comprises storing a copy of the first data in persistent storage in the first storage module.

20. The storage appliance of claim 15, wherein the fabric is a Peripheral Component Interconnect Express (PCIe) fabric.