Storage protocol emulation in a peripheral device

Information

  • Patent Grant
  • 11892964
  • Patent Number
    11,892,964
  • Date Filed
    Thursday, March 25, 2021
    3 years ago
  • Date Issued
    Tuesday, February 6, 2024
    11 months ago
Abstract
A peripheral device includes a host interface and processing circuitry. The host interface is configured to communicate with a host over a peripheral bus. The processing circuitry is configured to expose on the peripheral bus a peripheral-bus device that communicates with the host using a bus storage protocol, to receive, using the exposed peripheral-bus device, Input/Output (I/O) transactions that are issued by the host, and to complete the I/O transactions for the host in accordance with a network storage protocol, by running at least part of a host-side protocol stack of the network storage protocol.
Description
FIELD OF THE INVENTION

The present invention relates generally to data storage, and particularly to emulation of storage protocols in peripheral devices.


BACKGROUND OF THE INVENTION

Various techniques for data storage using network adapters are known in the art. For example, U.S. Pat. Nos. 9,696,942 and 9,727,503 describe techniques for accessing remote storage devices using a local bus protocol. A disclosed method includes configuring a driver program on a host computer to receive commands in accordance with a protocol defined for accessing local storage devices connected to a peripheral component interface bus of the host computer. When the driver program receives, from an application program running on the host computer a storage access command in accordance with the protocol, specifying a storage transaction, a remote direct memory access (RDMA) operation is performed by a network interface controller (NIC) connected to the host computer so as to execute the storage transaction via a network on a remote storage device.


U.S. Pat. No. 10,657,077 describes a HyperConverged NVMF storage-NIC card. A storage and communication apparatus for plugging into a server, includes a circuit board, a bus interface, a Medium Access Control (MAC) processor, one or more storage devices and at least one Central Processing Unit (CPU). The bus interface is configured to connect the apparatus at least to a processor of the server. The MAC mounted on the circuit board and is configured to connect to a communication network. The storage devices are mounted on the circuit board and are configured to store data. The CPU is mounted on the circuit board and is configured to expose the storage devices both (i) to the processor of the server via the bus interface, and (ii) indirectly to other servers over the communication network.


SUMMARY OF THE INVENTION

An embodiment of the present invention that is described herein provides a peripheral device including a host interface and processing circuitry. The host interface is configured to communicate with a host over a peripheral bus. The processing circuitry is configured to expose on the peripheral bus a peripheral-bus device that communicates with the host using a bus storage protocol, to receive, using the exposed peripheral-bus device, Input/Output (I/O) transactions that are issued by the host, and to complete the I/O transactions for the host in accordance with a network storage protocol, by running at least part of a host-side protocol stack of the network storage protocol.


In some embodiments, in running at least part of the host-side protocol stack, the processing circuitry is configured to isolate the host from control-plane operations of the network storage protocol. In an embodiment, the processing circuitry is configured to complete at least some, or at least part, of the I/O transactions for the host in a local storage. In another embodiment, the peripheral device further includes a network port configured to communicate over a network, and the processing circuitry is configured to complete at least some, or at least part, of the I/O transactions for the host by communicating over the network with a storage system that operates in accordance with the network storage protocol.


In some embodiments, in completing an I/O transaction over the network, the processing circuitry is configured to transfer data directly between a memory of the host and the storage system using zero-copy transfer. In an example embodiment, the processing circuitry is configured to determine one or more addresses for the data in the storage system, and then to transfer the data directly between the one or more addresses and the memory of the host, without intermediate storage of the data in the network adapter.


In a disclosed embodiment, at least one of the bus storage protocol and the network storage protocol is a block storage protocol. In another embodiment, at least one of the bus storage protocol and the network storage protocol is a File-System (FS) protocol. In yet another embodiment, at least one of the bus storage protocol and the network storage protocol is an object storage protocol. In still another embodiment, at least one of the bus storage protocol and the network storage protocol is a Key-Value (KV) protocol.


In an embodiment, in exposing the peripheral-bus device, the processing circuitry is configured to emulate a hot-plug indication, notifying the host that a storage device has connected to the peripheral bus. In some embodiments, the processing circuitry is configured to receive from the host a doorbell indicative of a queue on which the host posted one or more work requests pertaining to an I/O transaction, and to read and execute the one or more work requests so as to complete the I/O transaction. In an example embodiment, the processing circuitry includes hardware that is configured to receive the doorbell and to read the one or more work requests from the queue in response co the doorbell.


In another embodiment, the processing circuitry is configured to issue a Message Signaled Interrupt to the host upon completing an I/O transaction. In a disclosed embodiment, the processing circuitry is configured to communicate with the host via one or more registers exposed on the peripheral bus. In another embodiment, the processing circuitry is further configured to perform one or more of storage virtualization and data manipulation operations.


There is additionally provided, in accordance with an embodiment of the present invention, a method including, in a peripheral device that communicates with a host over a peripheral bus, exposing on the peripheral bus a peripheral-bus device that communicates with the host using a bus storage protocol.


Input/Output (I/O) transactions, which are issued by the host, are received in the peripheral device using the exposed peripheral-bus device. The I/O transactions are completed for the host, by the peripheral device, in accordance with a network storage protocol, by running at least part of a host-side protocol stack of the network storage protocol.


The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:





BRIEF DESCRIPTION OF THE DRAWINGS


FIG. 1 is a block diagram that schematically illustrates a computing and storage system, in accordance with an embodiment of the present invention;



FIG. 2 is a block diagram that schematically illustrates an example use-case of filesystem emulation in the system of FIG. 1 above, in accordance with an embodiment of the present invention; and



FIG. 3 is a flow chart that schematically illustrates a method for emulating a storage protocol in a peripheral device, in accordance with an embodiment of the present invention.





DETAILED DESCRIPTION OF EMBODIMENTS
Overview

Embodiments of the present invention that are described herein provide improved data storage techniques in which a peripheral device provides storage services to a host over a peripheral bus. The host may comprise, for example, a server in a data center. The peripheral may comprise, for example, a high-performance network adapter, sometimes referred to as Data Processing Unit (DPU) or “Smart-NIC”. The embodiments described herein refer mainly to a DPU that provides storage services to a host over a Peripheral Component Interconnect express (PCIe) bus. Generally, however, the disclosed techniques are applicable to various other types of peripherals and buses.


In the disclosed embodiments, the peripheral device serves the host using a network storage protocol, e.g., a block storage protocol, a File-System (FS) protocol, an object storage protocol or a Key-Value (KV) storage protocol. In particular, the peripheral device (i) exposes to the host a dedicated PCIe device that emulates a bus storage protocol, and (ii) runs at least part of the host-side protocol stack of the network storage protocol. The peripheral device receives Input/Output (I/O) transactions that are issued by the host, and completes the I/O transactions for the host, in accordance with the network storage protocol, using the internally-run protocol stack.


When using the disclosed techniques, the host is completely isolated from the control plane (also referred to as management plane or orchestration plane) of the storage service. The dedicated PCIe device presents to the host a storage interface, which is by nature more specific and restricted than a network interface. The host's interaction with the storage service is confined to data-plane storage operations, i.e., to exchanging I/O transactions with the dedicated PCIe device. Communication between. the host and the dedicated PCIe device is typically implemented using a limited set of commands and virtually no security privileges. Control and management operations relating to storage services, for example login, management of identities, credentials and access privileges and other security-related operations, are carried out between the peripheral device (e.g., DPU) and any relevant (remote and/or local) storage system. The host, and therefore any untrusted software that might run on it, is completely isolated from these operations.


In some embodiments, being exposed only to the bus storage protocol, the host may be unaware of the type of network storage protocol used by the peripheral device. As such, it is even possible for the bus storage protocol and the network storage protocol to be of different types. For example, the dedicated PCIe device receive I/O transactions from the host in accordance with a File-System protocol, and complete the I/O transactions over the network in accordance with an object or Key-Value protocol.


As is evident from the description above, the disclosed architecture provides a high degree of security to the storage service. Isolating the host from the management and control of the storage service is important in many applications and use-cases. One example is a multi-tenant cloud application, in which the host does not always have control over the different applications it runs. Another example is a “bare metal” cloud application, in which a tenant is provided with full access privileges to the host. In such scenarios, the disclosed technique enables a storage provider to provide storage services to the various applications running on the host, in a well-controlled, secure and mutually-isolated manner.


The disclosed technique also improves performance, since the host is offloaded of most, if not all, of the network storage protocol stack. Storage tasks often exhibit unpredictable bursts of computational load, e.g., due to complex operations such as manipulation of metadata structures, garbage collection, data compaction and defragmentation. Some computational tasks that may be carried out by the host, e.g., some High-Performance Computing (HPC) workloads, are sensitive to such variations in computational load. Offloading the host-side protocol stack to a peripheral device is therefore highly advantageous.


Moreover, when using the disclosed techniques, maintenance and administration of the network storage protocol stack (e.g., installation, upgrade and configuration) are performed entirely within the peripheral device (e.g., DPU). No cooperation or awareness is required from the host or the host administrator in performing such actions.


Several example implementations and use-cases of the disclosed techniques are described herein. Complementary techniques, such as zero-copy completion of I/O transactions and special-purpose doorbell mechanisms, are also described.


System Description


FIG. 1 is a block diagram that schematically illustrates a computing and storage system 20, in accordance with an embodiment of the present invention. System 20 comprises a high-performance Network Interface Controller (NIC) 24, also referred to as a Data Processing Unit (DPU), which serves a host 28. DPU 24 communicates with host 28 over a peripheral bus, in the present example a Peripheral Component Interconnect express (PCIe) bus 36.


In the present context, DPU 24 is regarded as a peripheral device connected to PCIe bus 36. DPU 24 provides host 28 with data storage services, possibly among other tasks. In the example of FIG. 1, DPU 24 connects host 28 to a network that employs a networked/clustered filesystem (FS) 32. FS 32 operates in accordance with a certain network storage protocol. Non-limiting examples of network storage protocols comprise NFS, NFSoRDMA, SMB, SMB-direct, Lustre, GPFS, BeeGFS, CEPH, as well as proprietary protocols offered by vendors such as VastData and Weka.


Host 28 comprises a host CPU 40 that may run various software applications depending on the applicable use case. In one embodiment, host 40 comprises a server in a cloud-based data center, which hosts applications belonging to multiple customers (“tenants”). In another embodiment, host 40 comprises a server in a “bare metal” data center, in which a tenant “owns” the server, in the sense that the tenant is given full access privileges to the server.


Among other functions, the applications running on host CPU 40 issue Input/Output (I/O) transactions, e.g., transactions that write data to files, read data from files, or create, modify or delete files or directories. Generally, I/O transactions can be issued by any software that runs on host CPU 40, e.g., by Virtual Machines (VMs), processes, containers, or any other software.


In the present example, DPU 24 comprises a host interface 44 for communicating with host 28 over PCIe bus 36, a network port 48 for communicating with FS 32 over the network (e.g., using Ethernet packets), and processing circuitry 52 for carrying out the various networking and storage functions of the DPU. Processing circuitry 52 typically comprises one or more CPUs 56 that run suitable software, and dedicated hardware 60. The tasks of processing circuitry 52 may be partitioned between software (CPUs 56) and hardware (dedicated hardware 60) in any suitable way.


In some embodiments, processing circuitry provides storage services to host CPU 40 by running at least part of the host-side protocol stack of the network storage protocol of FS 32. In addition, processing circuitry 52 exposes to host CPU 40 a dedicated PCIe device 62 on PCIe bus 36. In some embodiments, processing circuitry 52 may perform additional processing that enhances the specified network storage protocol of FS 32. For example, if the network storage protocol does not provide cryptographic capabilities, processing circuitry 52 of DPU 24 may add this functionality on top of the network storage protocol.


For the sake of clarity, PCIe device 62 is depicted in the figure inside host 28, in order to emphasize the interaction between device 62 and host CPU 40. In reality, however, PCIe device 62 is a logical interface presented to host 28 by DPU 24 over bus 36. The terms “PCIe device” and “PCIe interface” can therefore be used interchangeably. PCIe device 62 may comprise a PCIe physical function or virtual function.


PCIe device 62 is configured to emulate a bus storage protocol vis-à-vis the host CPU. Host CPU 40 conducts the I/O transactions by communicating with PCIe device 62 using the bus storage protocol. Processing circuitry 52 of DPU 24 completes (i.e., executes) the I/O transactions for host CPU 40 in FS 32 (and/or in local storage as elaborated below), using the internally-run protocol stack of the network storage protocol. Host interaction with PCIe device 62 may be implemented using standard operating-system (OS) drivers, or as a vendor specific driver, as appropriate.


The protocol between PCIe device 62 and host CPU 40 is typically limited to a small dedicated set of storage-related commands, as opposed to arbitrary communication enabled by conventional network devices. Therefore, the security vulnerability of this protocol is considerably reduced, and the task of securing it is significantly simpler. For example, processing circuitry 56 in DPU 24 may analyze the transactions arriving via PCIe device 62 and apply a security policy that is specified per the storage protocol being used. For example, the security policy may examine attributes relating to the storage protocol (e.g., filenames, offsets, object identifiers and the like) and take actions depending of the attribute values. Actions may comprise, for example, permitting or denying access to certain files or objects, or any other suitable action.


In some embodiments, DPU 24 further comprises local storage 64, e.g., one or more Flash memory devices. In some embodiments, system 20 further comprises an additional peripheral device 68 that comprises local storage 72, e.g., one or more Flash memory devices. The additional peripheral device may be, for example, another NIC (DPU or otherwise) or a Solid State Drive (SSD). In some embodiments, completion of I/O transactions may involve storing data in local storage, e.g., storage 64 in DPU 24 or storage 72 in additional peripheral device 68.


Generally, DPU 24 may complete at least some, or at least part, of the I/O transactions over the network, and may complete at least some, or at least part, of the I/O transactions in the local storage. Thus, for example, the protocol stack running in the DPU may translate a given I/O transaction into multiple storage operations (read or write) of the network storage protocol. One or more of these storage operations may be performed over the network, and one or more of she storage operations may be performed in the local storage.


The configurations of system 20 and its components, e.g., DPU 24 and host 28, shown in FIG. 1, are example configurations that are depicted purely for the sake of conceptual clarity. Any other suitable configurations can be used in alternative embodiments.


For example, in alternative embodiments, the peripheral device that exposes PCIe device 62 and runs the host-side network storage protocol stack may have no network connection at all. In an example embodiment of this sort, the peripheral device is a storage device such as an SSD. In such embodiments, data storage for the host is performed locally with no network communication. Further alternatively, the peripheral device may perform both local storage and remote storage.


Moreover, the disclosed techniques are not limited to file-system protocols, or to any other type of storage protocol. In alternative embodiments, the bus storage protocol may comprise various other types of storage protocols. Example bus storage protocols include block-storage (“block device”) protocols such as NVMe, virtio-blk, SCSI, SATA and SAS, various object storage protocols, KV storage protocols such as NVMe-KV, or any other suitable protocol. Alternatively to FS 32, DPU 24 may complete I/O transactions using various network storage protocols, e.g., block-storage protocols such as NVMe-over-Fabrics, NVMe-over-TCP, iSCSI, iSER, SRP and Fibre-channel, object storage protocols such as Amazon S3, Microsoft Azure, OpenStack Swift and Google Cloud Storage, KV storage protocols such as NoSQL, Redis and RocksDB, or any other suitable storage system or protocol.


As noted above, it is not mandatory that the bus storage protocol (exposed toward the host) and the network storage protocol (used for transaction completion) be of the same type. For example, in an embodiment, the bus storage protocol may comprise a FS protocol, while the network storage protocol comprises an object or KV protocol. Any other suitable combination can also be used.


The various elements of system 20 and its components, e.g., DPU 24 and host 28, may be implemented using software, using suitable hardware such as in one or more Application-Specific Integrated Circuits (ASIC) or Field-Programmable Gate Arrays (FPGA), or using a combination of software and hardware elements.


Typically, host CPU 40 and CPUs 56 of DPU 24 comprises programmable processors, which are programmed in software to carry out the functions described herein. The software may be downloaded to any of the processors in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.



FIG. 2 is a block diagram that schematically illustrates an example use-case of filesystem emulation in system 20 of FIG. 1 above, in accordance with an embodiment of the present invention. In this embodiment, the bus storage protocol being emulated by DPU 24 is the Virtio-fs file system. Virtio-fs is a shared file system designed to serve Virtual Machines (VMs), and is described, for example, in Section 5.11 of “Virtual I/O Device (VIRTIO) Version 1.1,” Committee Specification 01, Apr. 11, 2019.


In the example of FIG. 2, processing circuitry 56 of DPU 24 runs a virtio-fs SNAP controller 80 and a file system driver 84. Processing circuitry 56 exposes a dedicated virtio-fs PCIe device 88 to host 28. Driver 34 may be configured to interface with various types of networked, clustered or parallel file systems 32. Non-limiting examples of file systems include Lustre, GPFS, BeeGPFS, NFS (over TCP or over RDMA), or proprietary solutions offered by vendors such as Weka and Vastdata. Alternatively, any other suitable type of file system may be supported.


In an embodiment, Virtio-fs driver 88 presents to host 28 a local directory to mount, which is mapped to a directory in DPU 24. The protocol between DPU 24 and Virtio-fs driver 88 is defined in the virtio-fs specification, cited above, and is based on FUSE commands delivered over virtio-queues. In the more general case, virtio-fs SNAP controller 80 and file system driver 84 are tightly coupled to one another, and the virtio-fs folder presented to the host originates directly from the network without a local DPU folder representing it.



FIG. 3 is a flow chart that schematically illustrates a method for emulating a storage protocol in a peripheral device, in the present example DPU 24, in accordance with an embodiment of the present invention. The method begins with DPU 24 exposing to host 28 a dedicated PCIe device (e.g., device 62 of FIG. 1, or device 88 of FIG. 2) that presents a bus storage protocol, at an exposing step 90. At a transaction sending step 94, host 28 issues I/O transactions to the dedicated PCIe device.


At an I/O execution step 98, processing circuitry 56 of DEC 24 executes the I/O transactions for the host, in the appropriate storage system, in accordance with the network storage protocol. At a completion step 102, when the I/O transaction is completed, processing circuitry 56 of DPU 24 sends a completion notification to host 28.


Zero-Copy Completion of I/O Transactions

As can be appreciated, completing an I/O transaction by CPU 24 involves transfer of data between the memory of host 28 and a memory of the storage system managed by FS 32. When completing a write command, for example, processing circuitry 56 of DPU 24 transfers data from the memory of host 28 to the memory of the storage system. When completing a read command, processing circuitry 56 transfers data in the opposite direction, from the memory of the storage system to the memory of host 28.


In some embodiments, processing circuitry 56 performs these data transfers in a “zero-copy” manner. In the present context, the term “zero-copy” means that the data is transferred directly between the memory of host 28 to the memory of the storage system, without intermediate storage in DPU 24. Zero-copy completion of I/O transactions significantly reduces the overall transaction latency, and increases the achievable throughput.


Typically, the data transfer is performed using Remote Direct Memory Access (RDMA). In some embodiments, processing circuitry 56 performs zero-copy data transfer in two stages. In the first stage, processing circuitry 56 determines the appropriate address or addresses in the storage system for completing the I/O transaction (the address or addresses to which the data is to be written in case of a write command, or to be read from in case of a read command). Only then, in the second stage, processing circuitry 56 transfers the data between the appropriate addresses in the memory of host 28 and in the memory of the storage system.


In some embodiments, processing circuitry 56 performs zero-copy data transfer by accessing the memory of host 28 directly, using the host's own address space. Techniques of this sort are disclosed in U.S. patent application Ser. No. 17/189,303, entitled “Cross Address-Space Bridging,” filed Mar. 2, 2021, whose disclosure is incorporated herein by reference. In an embodiment of this sort, processing circuitry 56 of DPU 24 creates an RDMA MKEY that describes a memory of host 28. In this manner, an RDMA operation can be performed directly between the memory of host 28 and the memory of the storage system (a network entity), eliminating the need for an extra copy to the DPU memory.


In alternative embodiments, processing circuitry 56 may perform zero-copy data transfer in any other suitable way.


Doorbells and Other DPU Hardware Mechanisms

In various embodiments, DPU 24 comprises various hardware or hardware-software mechanisms that enhance the flexibility of receiving and handling I/O transactions issued by host 28, and also reduces latency. Such mechanisms may comprise, for example, queues and corresponding doorbells, hardware registers, interrupts and the like. Several examples are given below.


Typically, host CPU 40 issues an I/O transaction by posting one or more work requests on a queue that can be read by processing circuitry 56 of DPU 24. The host and DPU may interact via multiple queues in parallel, e.g., a queue per host core, per application, per thread, per QoS class, per user or per tenant. In order to reduce latency, in some embodiments the host and the DPU use a doorbell mechanism, in which host processor 40 (i) signals to processing circuitry 56 that one or more work requests have been posted, and (ii) indicates the queue from which the DPU should read the work requests. Typically, the doorbell triggers hardware 60 in processing circuitry 52 to read the work requests from the specified queue and pass the work requests to CPUs 56 for processing.


In an example embodiment, hardware 60 is configured to regard one or more addresses on the PCIe Base Address Register (BAR) as doorbells. The BAR is exposed to the host via dedicated PCIe device 62. In this embodiment, host CPU 40 issues a doorbell by writing to one of these addresses. Such a write triggers hardware 60 in the DPU, which in turn reads any pending work requests from the specified queue.


Various techniques can be used for specifying the identity of the queue to be read. In one embodiment, a single BAR address is assigned to serve as a doorbell, and the host writes the appropriate queue identifier to this address. In another embodiment, each queue is assigned a respective different BAR address; any write to one of these BAR addresses is interpreted by hardware 60 as a doorbell for the corresponding queue. The value written to the address can be interpreted as the producer index of the queue. Alternatively, any other suitable mechanism or convention can be used.


In some embodiments, the DPU software, running on CPUs 56, is configured to issue a Message Signaled interrupt (MSI or MSI-X) to the host upon completing an I/O transaction. The interrupt triggers host CPU 40, and therefore reduces latency.


In some embodiments, CPUs 56 (in DPU 24) and host processor 40 (in host 28) are configured to exchange information and/or report events to one another by writing and reading registers defined on the PCIe bus BAR. These registers are exposed to the host via dedicated PCIe device 62. In some cases CPUs 56 (in DPU 24) write to a register and host processor 40 (in host 28) reads the register. In other cases host processor 40 (in host 28) writes to a register and CPUs 56 (in DPU 24) read the register. In some embodiments more complex register mechanisms can be defined. For example, writing to one register can affect the meaning of a subsequent write to another register.


In some embodiments, in exposing dedicated PCIe device 62, processing circuitry 52 is configured to emulate a “hot-plug” indication to host 28. The hot-plug indication notifies the host that a storage device has connected to PCIe bus 36.


Additional Embodiments and Variations

In some embodiments, as part of emulating the storage protocol to the host, processing circuitry 52 in DPU 24 is configured to emulate various FS services to host 28. Any suitable FS service can be emulated, such as, for example, directory services and statistics collection.


Additionally or alternatively, as part of emulating the storage protocol to the host, processing circuitry 52 in DPU 24 is configured to perform one or more storage virtualization and data manipulation operations. Storage virtualization operations that may be performed by DPU 24 comprise, for example, cryptographic operations such as encryption, decryption, signing and authentication, deduplication, mirroring, isolation for security, Quality of Service (QoS), directory service, locking, compression, Artificial Intelligence (AI) operations, among others. In some embodiments, such operations can be carried out, at least in part, by hardware 60 in processing circuitry 52, and/or accelerated by a Graphics Processing Unit (GPU) coupled to DPU 24.


It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.

Claims
  • 1. A peripheral device, comprising: a host interface, configured to communicate with a host over a peripheral bus; andprocessing circuitry, configured to: expose on the peripheral bus a peripheral-bus device that communicates with the host using a bus storage protocol;receive, using the exposed peripheral-bus device, Input/Output (I/O) transactions that are issued by the host; andcomplete the I/O transactions for the host in accordance with a network storage protocol, by running at least part of a host-side protocol stack of the network storage protocol,wherein, in running at least part of the host-side protocol stack, the processing circuitry is configured to isolate the host from control-plane operations of the network storage protocol.
  • 2. The peripheral device according to claim 1, wherein the processing circuitry is configured to complete at least some, or at least part, of the I/O transactions for the host in a local storage.
  • 3. The peripheral device according to claim 1, further comprising a network port configured to communicate over a network, wherein the processing circuitry is configured to complete at least some, or at least part, of the I/O transactions for the host by communicating over the network with a storage system that operates in accordance with the network storage protocol.
  • 4. The peripheral device according to claim 3, wherein, in completing an I/O transaction over the network, the processing circuitry is configured to transfer data directly between a memory of the host and the storage system using zero-copy transfer.
  • 5. The peripheral device according to claim 4, wherein the processing circuitry is configured to determine one or more addresses for the data in the storage system, and then to transfer the data directly between the one or more addresses and the memory of the host, without intermediate storage of the data in the peripheral device.
  • 6. The peripheral device according to claim 1, wherein at least one of the bus storage protocol and the network storage protocol is a block storage protocol.
  • 7. The peripheral device according to claim 1, wherein at least one of the bus storage protocol and the network storage protocol is a File-System (FS) protocol.
  • 8. The peripheral device according to claim 1, wherein at least one of the bus storage protocol and the network storage protocol is an object storage protocol.
  • 9. The peripheral device according to claim 1, wherein at least one of the bus storage protocol and the network storage protocol is a Key-Value (KV) protocol.
  • 10. The peripheral device according to claim 1, wherein, in exposing the peripheral-bus device, the processing circuitry is configured to emulate a hot-plug indication, notifying the host that a storage device has connected to the peripheral bus.
  • 11. The peripheral device according to claim 1, wherein the processing circuitry is configured to receive from the host a doorbell indicative of a queue on which the host posted one or more work requests pertaining to an I/O transaction, and to read and execute the one or more work requests so as to complete the I/O transaction.
  • 12. The peripheral device according to claim 11, wherein the processing circuitry comprises hardware that is configured to receive the doorbell and to read the one or more work requests from the queue in response to the doorbell.
  • 13. The peripheral device according to claim 1, wherein the processing circuitry is configured to issue a Message Signaled Interrupt to the host upon completing an I/O transaction.
  • 14. The peripheral device according to claim 1, wherein the processing circuitry is configured to communicate with the host via one or more registers exposed on the peripheral bus.
  • 15. The peripheral device according to claim 1, wherein the processing circuitry is further configured to perform one or more of storage virtualization and data manipulation operations.
  • 16. A method, comprising: in a peripheral device that communicates with a host over a peripheral bus, exposing on the peripheral bus a peripheral-bus device that communicates with the host using a bus storage protocol;receiving in the peripheral device, using the exposed peripheral-bus device, Input/Output (I/O) transactions that are issued by the host; andcompleting the I/O transactions for the host, by the peripheral device, in accordance with a network storage protocol, by running at least part of a host-side protocol stack of the network storage protocol,wherein running at least part of the host-side protocol stack comprises isolating the host from control-plane operations of the network storage protocol.
  • 17. The method according to claim 16, wherein completing the I/O transactions comprises completing at least some, or at least part, of the I/O transactions for the host in a local storage.
  • 18. The method according to claim 16, wherein completing the I/O transactions comprises completing at least some, or at least part, of the I/O transactions for the host by communicating over a network with a storage system that operates in accordance with the network storage protocol.
  • 19. The method according to claim 18, wherein completing an I/O transaction over the network comprises transferring data directly between a memory of the host and the storage system using zero-copy transfer.
  • 20. The method according to claim 19, wherein transferring the data comprises determining one or more addresses for the data in the storage system, and then transferring the data directly between the one or more addresses and the memory of the host, without intermediate storage of the data in the peripheral device.
  • 21. The method according to claim 16, wherein at least one of the bus storage protocol and the network storage protocol is a block storage protocol.
  • 22. The method according to claim 16, wherein at least one of the bus storage protocol and the network storage protocol is a File-System (FS) protocol.
  • 23. The method according to claim 16, wherein at least one of the bus storage protocol and the network storage protocol is an object storage protocol.
  • 24. The method according to claim 16, wherein at least one of the bus storage protocol and the network storage protocol is a Key-Value (KV) protocol.
  • 25. The method according to claim 16, wherein exposing the peripheral-bus device comprises emulating a hot-plug indication, notifying the host that a storage device has connected to the peripheral bus.
  • 26. The method according to claim 16, wherein receiving the I/O transactions comprises receiving from the host a doorbell indicative of a queue on which the host posted one or more work requests pertaining to an I/O transaction, and wherein completing the I/O transactions comprises reading and executing the one or more work requests.
  • 27. The method according to claim 26, wherein receiving the doorbell comprises receiving the doorbell, and read the one or more work requests from the queue in response to the doorbell, using hardware.
  • 28. The method according to claim 16, wherein completing the I/O transactions comprises issuing a Message Signaled Interrupt to the host upon completing an I/O transaction.
  • 29. The method according to claim 16, wherein exposing the peripheral-bus device comprises communicating with the host via one or more registers exposed on the peripheral bus.
  • 30. The method according to claim 16, wherein running at least part of the host-side protocol stack comprises performing one or more of storage virtualization and data manipulation operations.
US Referenced Citations (190)
Number Name Date Kind
5003465 Chisholm et al. Mar 1991 A
5463772 Thompson et al. Oct 1995 A
5615404 Knoll et al. Mar 1997 A
5768612 Nelson Jun 1998 A
5864876 Rossum et al. Jan 1999 A
5893166 Frank et al. Apr 1999 A
5954802 Griffith Sep 1999 A
6070219 McAlpine et al. May 2000 A
6226680 Boucher et al. May 2001 B1
6321276 Forin Nov 2001 B1
6581130 Brinkmann et al. Jun 2003 B1
6701405 Adusumilli et al. Mar 2004 B1
6766467 Neal et al. Jul 2004 B1
6789143 Craddock et al. Sep 2004 B2
6901496 Mukund et al. May 2005 B1
6981027 Gallo et al. Dec 2005 B1
7171484 Krause et al. Jan 2007 B1
7225277 Johns et al. May 2007 B2
7263103 Kagan et al. Aug 2007 B2
7299266 Boyd et al. Nov 2007 B2
7395364 Higuchi et al. Jul 2008 B2
7464198 Martinez et al. Dec 2008 B2
7475398 Nunoe Jan 2009 B2
7548999 Haertel et al. Jun 2009 B2
7577773 Gandhi et al. Aug 2009 B1
7657659 Lambeth et al. Feb 2010 B1
7720064 Rohde May 2010 B1
7752417 Manczak et al. Jul 2010 B2
7809923 Hummel et al. Oct 2010 B2
7921178 Haviv Apr 2011 B2
7921237 Holland et al. Apr 2011 B1
7945752 Miller et al. May 2011 B1
8001592 Hatakeyama Aug 2011 B2
8006297 Johnson et al. Aug 2011 B2
8010763 Armstrong et al. Aug 2011 B2
8051212 Kagan et al. Nov 2011 B2
8103785 Crowley et al. Jan 2012 B2
8255475 Kagan et al. Aug 2012 B2
8260980 Weber et al. Sep 2012 B2
8346919 Eiriksson et al. Jan 2013 B1
8447904 Riddoch May 2013 B2
8504780 Mine et al. Aug 2013 B2
8645663 Kagan et al. Feb 2014 B2
8745276 Bloch et al. Jun 2014 B2
8751701 Shahar et al. Jun 2014 B2
8824492 Wang et al. Sep 2014 B2
8949486 Kagan et al. Feb 2015 B1
9038073 Kohlenz et al. May 2015 B2
9092426 Bathija et al. Jul 2015 B1
9298723 Vincent Mar 2016 B1
9678818 Raikin et al. Jun 2017 B2
9696942 Kagan et al. Jul 2017 B2
9727503 Kagan et al. Aug 2017 B2
9830082 Srinivasan et al. Nov 2017 B1
9904568 Vincent et al. Feb 2018 B2
10078613 Ramey Sep 2018 B1
10120832 Raindel et al. Nov 2018 B2
10135739 Raindel et al. Nov 2018 B2
10152441 Liss et al. Dec 2018 B2
10162793 Bshara et al. Dec 2018 B1
10210125 Burstein Feb 2019 B2
10218645 Raindel et al. Feb 2019 B2
10423774 Zelenov et al. Apr 2019 B1
10382350 Bohrer et al. Aug 2019 B2
10657077 Ganor et al. May 2020 B2
10671309 Glynn Jun 2020 B1
10684973 Connor et al. Jun 2020 B2
10715451 Raindel et al. Jul 2020 B2
10824469 Hirshberg et al. Nov 2020 B2
10841243 Levi et al. Nov 2020 B2
10999364 Itigin et al. May 2021 B1
11003607 Ganor et al. May 2021 B2
11086713 Sapuntzakis et al. Aug 2021 B1
20020152327 Kagan et al. Oct 2002 A1
20030023846 Krishna et al. Jan 2003 A1
20030046530 Poznanovic Mar 2003 A1
20030120836 Gordon Jun 2003 A1
20040039940 Cox et al. Feb 2004 A1
20040057434 Poon et al. Mar 2004 A1
20040158710 Buer et al. Aug 2004 A1
20040221128 Beecroft et al. Nov 2004 A1
20040230979 Beecroft et al. Nov 2004 A1
20050102497 Buer May 2005 A1
20050198412 Pedersen et al. Sep 2005 A1
20050216552 Fineberg et al. Sep 2005 A1
20060095754 Hyder et al. May 2006 A1
20060104308 Pinkerton et al. May 2006 A1
20060259661 Feng et al. Nov 2006 A1
20070011429 Sangili et al. Jan 2007 A1
20070061492 Van Riel Mar 2007 A1
20070223472 Tachibana et al. Sep 2007 A1
20070226450 Engbersen et al. Sep 2007 A1
20070283124 Menczak et al. Dec 2007 A1
20070297453 Niinomi Dec 2007 A1
20080005387 Mutaguchi Jan 2008 A1
20080147822 Benhase et al. Jun 2008 A1
20080147904 Freimuth et al. Jun 2008 A1
20080168479 Purtell et al. Jul 2008 A1
20080313364 Flynn et al. Dec 2008 A1
20090086736 Foong et al. Apr 2009 A1
20090106771 Benner et al. Apr 2009 A1
20090204650 Wong et al. Aug 2009 A1
20090319775 Buer et al. Dec 2009 A1
20090328170 Williams et al. Dec 2009 A1
20100030975 Murray et al. Feb 2010 A1
20100095053 Bruce et al. Apr 2010 A1
20100095085 Hummel et al. Apr 2010 A1
20100211834 Asnaashari et al. Aug 2010 A1
20100217916 Gao et al. Aug 2010 A1
20100228962 Simon et al. Sep 2010 A1
20110023027 Kegel et al. Jan 2011 A1
20110119673 Bloch et al. May 2011 A1
20110213854 Haviv Sep 2011 A1
20110246597 Swanson et al. Oct 2011 A1
20120314709 Post et al. Dec 2012 A1
20130067193 Kagan et al. Mar 2013 A1
20130080651 Pope et al. Mar 2013 A1
20130103777 Kagan et al. Apr 2013 A1
20130125125 Karino et al. May 2013 A1
20130142205 Munoz Jun 2013 A1
20130159568 Shahar Jun 2013 A1
20130263247 Jungck et al. Oct 2013 A1
20130276133 Hodges et al. Oct 2013 A1
20130311746 Raindel et al. Nov 2013 A1
20130325998 Hormuth et al. Dec 2013 A1
20130329557 Petry Dec 2013 A1
20130347110 Dalal Dec 2013 A1
20140089450 Raindel et al. Mar 2014 A1
20140089451 Eran et al. Mar 2014 A1
20140089631 King Mar 2014 A1
20140122828 Kagan et al. May 2014 A1
20140129741 Shahar et al. May 2014 A1
20140156894 Tsirkin Jun 2014 A1
20140181365 Fanning et al. Jun 2014 A1
20140185616 Bloch et al. Jul 2014 A1
20140254593 Mital et al. Sep 2014 A1
20140282050 Quinn et al. Sep 2014 A1
20140282561 Holt et al. Sep 2014 A1
20150006663 Huang Jan 2015 A1
20150012735 Tamir et al. Jan 2015 A1
20150032835 Sharp et al. Jan 2015 A1
20150081947 Vucinic et al. Mar 2015 A1
20150100962 Morita et al. Apr 2015 A1
20150288624 Raindel et al. Oct 2015 A1
20150319243 Hussain Nov 2015 A1
20150347185 Holt et al. Dec 2015 A1
20150355938 Jokinen et al. Dec 2015 A1
20160065659 Bloch et al. Mar 2016 A1
20160085718 Huang Mar 2016 A1
20160132329 Gupte et al. May 2016 A1
20160226822 Zhang et al. Aug 2016 A1
20160342547 Liss et al. Nov 2016 A1
20160350151 Zou et al. Dec 2016 A1
20160378529 Wen Dec 2016 A1
20170075855 Sajeepa et al. Mar 2017 A1
20170180273 Daly et al. Jun 2017 A1
20170187629 Shalev et al. Jun 2017 A1
20170237672 Dalal Aug 2017 A1
20170264622 Cooper et al. Sep 2017 A1
20170286157 Hasting et al. Oct 2017 A1
20170371835 Ranadive et al. Dec 2017 A1
20180004954 Liguori et al. Jan 2018 A1
20180067893 Raindel et al. Mar 2018 A1
20180109471 Chang et al. Apr 2018 A1
20180114013 Sood et al. Apr 2018 A1
20180167364 Dong et al. Jun 2018 A1
20180210751 Pepus et al. Jul 2018 A1
20180219770 Wu et al. Aug 2018 A1
20180219772 Koster et al. Aug 2018 A1
20180246768 Palermo et al. Aug 2018 A1
20180262468 Kumar et al. Sep 2018 A1
20180285288 Bernat et al. Oct 2018 A1
20180329828 Apfelbaum et al. Nov 2018 A1
20190012350 Sindhu et al. Jan 2019 A1
20190026157 Suzuki et al. Jan 2019 A1
20190116127 Pismenny et al. Apr 2019 A1
20190163364 Gibb et al. May 2019 A1
20190173846 Patterson et al. Jun 2019 A1
20190190892 Menachem et al. Jun 2019 A1
20190199690 Klein Jun 2019 A1
20190250938 Claes et al. Aug 2019 A1
20200012604 Agarwal Jan 2020 A1
20200026656 Liao et al. Jan 2020 A1
20200065269 Balasubramani et al. Feb 2020 A1
20200259803 Menachem et al. Aug 2020 A1
20200314181 Eran et al. Oct 2020 A1
20200401440 Sankaran et al. Dec 2020 A1
20210111996 Pismenny et al. Apr 2021 A1
20220075747 Shuler et al. Mar 2022 A1
20220100687 Sahin et al. Mar 2022 A1
Foreign Referenced Citations (3)
Number Date Country
1657878 May 2006 EP
2463782 Jun 2012 EP
2010062679 Jun 2010 WO
Non-Patent Literature Citations (40)
Entry
Bar-Ilan et al, U.S. Appl. No. 17/234,189, filed Apr. 19, 2021.
U.S. Appl. No. 17/372,466 Office Action dated Feb. 15, 2023.
Shirey., “Internet Security Glossary, Version 2”, Request for Comments 4949, pp. 1-365, Aug. 2007.
Information Sciences Institute, “Transmission Control Protocol; DARPA Internet Program Protocol Specification”, Request for Comments 793, pp. 1-90, Sep. 1981.
InfiniBand TM Architecture Specification vol. 1, Release 1.3, pp. 1-1842, Mar. 3, 2015.
Stevens., “TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms”, Request for Comments 2001, pp. 1-6, Jan. 1997.
Netronome Systems, Inc., “Open vSwitch Offload and Acceleration with Agilio® CX SmartNICs”, White Paper, pp. 1-7, Mar. 2017.
PCI Express® Base Specification ,Revision 3.0, pp. 1-860, Nov. 10, 2010.
Dierks et al., “The Transport Layer Security (TLS) Protocol Version 1.2”, Request for Comments: 5246 , pp. 1-104, Aug. 2008.
Turner et al., “Prohibiting Secure Sockets Layer (SSL) Version 2.0”, Request for Comments: 6176, pp. 1-4, Mar. 2011.
Rescorla et al., “The Transport Layer Security (TLS) Protocol Version 1.3”, Request for Comments: 8446, pp. 1-160, Aug. 2018.
Comer., “Packet Classification: A Faster, More General Alternative to Demultiplexing”, The Internet Protocol Journal, vol. 15, No. 4, pp. 12-22, Dec. 2012.
Salowey et al., “AES Galois Counter Mode (GCM) Cipher Suites for TLS”, Request for Comments: 5288, pp. 1-8, Aug. 2008.
Burstein, “Enabling Remote Persistent Memory”, SNIA-PM Summit, pp. 1-24, Jan. 24, 2019.
Chung et al., “Serving DNNs in Real Time at Datacenter Scale with Project Brainwave”, IEEE Micro Pre-Print, pp. 1-11, Mar. 22, 2018.
Talpey, “Remote Persistent Memory—With Nothing but Net”, SNIA—Storage developer conference , pp. 1-30, year 2017.
Microsoft, “Project Brainwave”, pp. 1-5, year 2019.
NVM Express Inc., “NVM ExpressTM Base Specification”, Revision 1.4, p. 1-403, Jun. 10, 2019.
Pismenny et al., “Autonomous NIC Offloads”, submitted for evaluation of the 26th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '21), p. 1-18, Dec. 13, 2020.
Pismenny et al., U.S. Appl. No. 17/204,968, filed Mar. 18, 2021.
U.S. Appl. No. 16/827,912 Office Action dated Jun. 1, 2021.
Lebeane et al., “Extended Task queuing: Active Messages for Heterogeneous Systems”, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC'16), pp. 933-944, Nov. 2016.
NVM Express Inc., “NVM Express over Fabrics,” Revision 1.0, pp. 1-49, Jun. 5, 2016.
“Linux kernel enable the IOMMU—input/output memory management unit support”, pp. 1-2, Oct. 15, 2007 downloaded from http://www.cyberciti.biz/tips/howto-turn-on-linux-software-iommu-support.html.
Hummel M., “IO Memory Management Hardware Goes Mainstream”, AMD Fellow, Computation Products Group, Microsoft WinHEC, pp. 1-7, 2006.
NVM Express, Revision 1.0e, pp. 1-127, Jan. 23, 2014.
InfiniBand Trade Association, “InfiniBandTM Architecture Specification”, vol. 1, Release 1.2.1, pp. 1-1727, Nov. 2007.
Shah et al., “Direct Data Placement over Reliable Transports”, IETF Network Working Group, RFC 5041, pp. 1-38, Oct. 2007.
Culley et al., “Marker PDU Aligned Framing for TCP Specification”, IETF Network Working Group, RFC 5044, pp. 1-75, Oct. 2007.
“MPI: A Message-Passing Interface Standard”, Version 2.2, Message Passing Interface Forum, pp. 1-64, Sep. 4, 2009.
Welsh et al., “Incorporating Memory Management into User-Level Network Interfaces”, Department of Computer Science, Cornell University, Technical Report TR97-1620, pp. 1-10, Feb. 13, 1997.
Tsirkin et al., “Virtual I/O Device (Virtio) Version 1.1”, Committee Specification 01, OASIS, Section 5.11, pp. 156-160, Apr. 11, 2019.
Burstein et al., U.S. Appl. No. 17/189,303, filed Mar. 2, 2021.
Mellanox Technologies, “Understanding On Demand Paging (ODP),” Knowledge Article, pp. 1-6, Feb. 20, 2019 downloaded from https://community.mellanox.com/s/article/understanding-on-demand-paging--odp-x.
Rosenbaum et al., U.S. Appl. No. 17/338,131, filed Jun. 3, 2021.
Microchip Technolohy Incorporated, “Switchtec PAX Gen 4 Advanced Fabric PCIe Switch Family—PM42100, PM42068, PM42052, PM42036, PM42028,” Product Brochure, pp. 1-2, year 2021.
Regula, “Using Non-Transparent Bridging in PCI Express Systems,” PLX Technology, Inc., pp. 1-31, Jun. 2004.
U.S. Appl. No. 17/372,466 Office Action dated Nov. 2, 2022.
U.S. Appl. No. 17/527,197 Office Action dated Sep. 28, 2023.
U.S. Appl. No. 17/527,197 Notice of allowance dated Dec. 13, 2023.
Related Publications (1)
Number Date Country
20220309019 A1 Sep 2022 US