Patent Application
20200279060
In computing, disaggregated storage may refer to hard disk drives, virtual drives, or any drives that store information external to a computer. Disaggregated storage may provide the convenience of expanding the amount of data one computer can store and access without having to buy a new computer with larger local storage. Disaggregated storage may be cabled to the computer, either directly cabled or cabled through storage fabric switches. Although storage data at rest within a drive may be protected by encryption within the drive, disaggregated storage exposes the data in flight over a fabric to snooping attack.
The present disclosure may be understood from the following detailed description when read with the accompanying Figures. In accordance with the standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.
Some examples of the present application are described with respect to the following figures.
Storage over fabric enables one or more computers to access one or more storage devices attached to one or more storage enclosures, and or one or more other computers using a fabric. The term, fabric, refers to, at least in part, the communication network that may be used between the one or more computers and the one or more storage devices. The communication network may use communication and transport protocols for data that may include Ethernet, Fibre Channel, lnfiniband℠, Gen-Z, and the like. Storage over fabric scales the storage accessible to a computer through disaggregation to one or more storage enclosures. The one or more storage enclosures, also referred to herein as a storage module. The one or more storage modules may include a disaggregated array of independent storage from the fabric to comprise a redundant array of independent disks. The one or more storage modules may include one or more storage devices. The one or more storage devices may include one or more memory devices, one or more drives, and or an array of independent drives. The one or more memory devices may include a circuit board of integrated circuits for computer memory. The one or more memory devices may be redundant to or may provide a redundant memory backup for at least a portion of the memory of the one or more drives and or the array of independent drives. The one or more drives and or the array of independent drives may be redundant to or may provide a redundant memory backup for at least a portion of the memory of the one or more memory devices. The drives may be rotating disk drives, solid state drives, redundant arrays of independent disks (RAID), virtual drives, and the like. The one or more memory devices may include one or more flash drives, one or more single in-line memory modules (SIMM), one or more dual in-line memory modules (DIMM), and or the like.
The Ethernet communication and transport protocol for data may operate within a physical layer and a data link layer on an open systems interconnection network protocol model. The Ethernet communication and transport protocol may include two units of transmission, a packet and a frame. The frame may include the payload of data being transmitted as well as the physical media access control (MAC) addresses of both the sender and receiver, virtual local area network (VLAN) tagging, quality of service information, and error correction information. Each packet may include a frame and additional information to establish a connection and mark where the frame starts. The Fibre Channel communication and transport protocol may include data link layer switching technologies where hardware may handle the entire protocol in a Fibre Channel fabric. The Infiniband communication and transport protocol may include a switch-based serial point-to-point interconnect architecture where data may be transmitted in packets that form a message. The Infiniband communication and transport protocol may include remote direct memory access support, simultaneous peer-to-peer communication, and end-to-end flow control. The Gen-Z communication and transport protocol may be an open-systems interconnect that may provide memory semantic access to data and devices via direct-attached, switched, or fabric topologies. The Gen-Z communication and transport protocol may enable any type and mix of dynamic random-access memory (DRAM) and non-volatile memory to be directly accessed by applications or through block-semantic communications.
However, the use of networks means that malicious users may be able to snoop on the data in flight. The term, in flight, refers to the data in the active state of passing over the network between the computer and the disaggregated storage modules. Currently, disaggregated storage modules do not protect the data in flight over any fabric. This may expose the data to snooping attacks.
Further, some disaggregated storage modules do not protect the data at rest. The term, at rest, refers to the data in the static state of storage on the disaggregated storage module drives. Because the data at rest is not protected, the data on the storage modules may be read offline if the drives are physically removed.
In some cases, the disaggregated storage modules may provide encryption of the data at rest. However, in such cases, the data in flight may still be transmitted without encryption and therefore be exposed to snooping. In addition, the data may pass unencrypted through a memory of the disaggregated storage module, and therefore be subject to potential theft if the memory is physically removed. Further, if the disaggregated storage module uses the same encryption key for all the data at rest on the drives in the storage modules, all the data at rest may be exposed to offline snooping if the drives of the storage modules are physically removed and the single encryption key is stolen.
Additionally, storage modules may provide disaggregated storage for multiple computers, compute nodes, and the like. As such, if the storage module provides encryption for the data in flight but uses a single encryption key for all the data, then theft of the single encryption key may expose the data in flight of all the compute nodes to snooping. However, if the storage module uses a different encryption key for the data in flight for every compute node, the additional processing overhead on the storage module may detrimentally impact the throughput and latency of storage and access.
Accordingly, examples of the present disclosure may provide encryption for data in flight over fabric, and at rest on disaggregated storage. In addition, examples may provide storage performance scalability of disaggregated storage modules by distributing the encryption to the computers that are storing their data on, and accessing their data from, the disaggregated storage. Herein, these computers are referred to as initiators. Providing encryption at the initiators provides an improvement in performance over encryption as a service on the disaggregated storage modules. In examples, data may be encrypted and decrypted at the initiator's connection to the fabric. Further, this approach may be applied to any system with fabric-connected storage including, but not limited to, non-volatile memory express (NVMe) external storage and Gen-Z persistent memory.
The compute nodes 102A may include a fabric network interface card (NIC) 108. The compute nodes 102A may include one or more fabric network interface cards (NICs). Each fabric network interface card 108 may be a network communication apparatus capable of performing computer network communications. Each fabric network interface card 108 may include an encryption capability. The encryption capability may encrypt one or more blocks of data for transmission to the storage modules 106. Each fabric network interface card 108 may include a decryption capability. The decryption capability may decrypt one or more blocks of data received by the fabric network interface card 108 from the storage modules 106 through the fabrics 104A. The encryption and or the decryption capability may include the necessary hardware and software components to encrypt and or decrypt data. The compute nodes 102A and or each fabric network interface card 108 may include one or more encryption keys for encrypting and or decrypting the one or more blocks of data. Each compute node 102A and or fabric network interface card 108 may include an encryption accelerator that may encrypt data that is being sent to the storage modules 106 for storage. Additionally, each compute node 102A and or fabric network interface card 108 may include a decryption accelerator that decrypts data that is retrieved from the storage modules 106. One or more fabric network interface cards 108 may include a firewall for security, a layer ⅔ switch for traffic steering, performance acceleration capabilities, and network visibility that may include remote NIC or network management.
Each fabric network interface card 108 may encrypt and decrypt one or more blocks of data to create one or more encrypted blocks of data. The one or more blocks of data may include one or more files, portions of files, updates to files, and or any number of data packets. The length of the one or more blocks of data may be any length from one data packet to a continuous stream of data packets over some period of time.
A key management entity, not shown, may generate the one or more encryption keys, manage the one or more encryption keys for encryption and or decryption, and may store each encryption key on compute nodes 102A and or one or more fabric network interface cards 108. In examples, a network or server management station may act as the key management entity. The network management station may be a server that may run a network management application. Network devices may communicate with the network management server to relay management and control information. The network management server may also enable network data analysis and reporting.
The network management station may send commands to the one or more fabric network interface cards 108 via a baseboard management controller, not shown, to control the one or more fabric network interface cards 108. The baseboard management controller may connect to the one or more fabric network interface cards 108 via an inter IC or I2C bus, not shown. The baseboard management controller may act as a passthrough to an 12C bus that connects to a management CPU that may be resident on the one or more fabric network interface cards 108.
Each of the one or more encryption keys may be sent to, retrieved from, or erased from the compute nodes 102A and or one or more fabric network interface cards 108 by the key management entity for encryption and decryption purposes. Metadata associated with the one or more encryption keys and any associated stored encrypted data may be managed by the key management entity and may be stored on the compute nodes 102A and or one or more fabric network interface cards 108 and or elsewhere. For example, one or more associated IP addresses of the storage modules 106 and the namespaces to access any stored encrypted data on the storage modules 106 along with any redundant arrays of independent disks (RAID) requirements may be sent by the key management entity to the compute nodes 102A and or one or more fabric network interface cards 108 for encryption and decryption purposes.
The encryption capability may encrypt the one or more blocks of data. The one or more blocks of data may be delivered to the compute nodes 102A already encrypted by another encryption capability (not shown) and then encrypted by software or hardware within the fabric network interface card 108. The encryption capability may be resident within the compute nodes 102A or within the fabrics 104A. For example, if a CPU within the compute nodes 102A executes an encryption/decryption algorithm in software, hardware, or combinations thereof, the algorithm may use the one or more encryption keys to encrypt each data block within the one or more blocks of data before writing the one or more blocks of encrypted data to the storage modules 106 over the fabrics 104A. If the fabric network interface card 108 has a resident capability to execute the encryption/decryption algorithm, in software, hardware, or combinations thereof, the algorithm may use the one or more encryption keys to encrypt each data block within the one or more blocks of data before writing the one or more blocks of encrypted data to the storage modules 106 over the fabrics 104A. Similarly, for reading the one or more blocks of encrypted data from the storage modules 106, either the CPU within the compute nodes 102A or the fabric network interface card 108 may use the appropriate encryption key to decrypt the one or more blocks of encrypted data before passing the one or more unencrypted data blocks to an operating system or one or more applications.
Metadata may be associated with the one or more encrypted blocks of data and the associated encryption key used to encrypt the data for use during decryption. The metadata associated with the encryption key may be associated with the metadata associated with the one or more blocks of encrypted data. The metadata may be stored on the compute node 102A for later use during retrieval and decryption of any amount of encrypted data stored on the storage module 106. The decryption capability may decrypt the one or more blocks of encrypted data after retrieval from the storage module 106. The metadata may be used by the compute node 102A to determine which encryption key to utilize during the decryption process.
The fabrics 104A may be a computer communications network that enables the compute nodes 102A to directly access the storage modules 106. In this way, the compute nodes 102A may perform reads and writes to the storage modules 106 without making calls to intervening software layers, such as an operating system.
The storage modules 106 may be nodes that provide data storage and retrieval capabilities over the fabrics 104A. Example storage modules 106 may include non-volatile memory express (NVMe) external storage, Gen-Z persistent memory, and the like. The storage modules 106 may include one or more storage fabric interfaces 110, storage controllers 112A, and drives 114A-1 to 114A-3 (also referred to collectively as drives 114A or individually and generally as a drive 114A). The storage fabric interface 110 may be network communications apparatus capable of performing computer network communications over the fabrics 104A. Accordingly, the storage fabric interface 110 may receive requests from the compute nodes 102A to write encrypted data to storage and read encrypted data from storage. When receiving requests to write encrypted data to storage, the storage fabric interface 110 may partition the encrypted data sent by the compute nodes 102A and provide the encrypted data to the storage controller 112A to write each partition to different drives 114A, recording metadata about each partition for later partition retrieval. The drives 114A-1 to 114A-3 may be storage devices, such as one or more memory devices, hard disk drives, solid state drives, RAID, virtual drives, and the like.
Because the data may be written across multiple drives 114A, the physical removal of a single drive 114A does not give access to all data of the compute nodes 102A. Further, because the data stored on the drives 114A-1 to 114A-3 may be encrypted, the data at rest on the drives 114A may not be read even if the drives 114A are physically removed.
The system 100A may provide an additional level of security to hypertext transfer protocol secure (HTTPS). HTTPS may provide secure communication over a computer network using Transport Layer Security. In HTTPS, individual data packets may be encrypted. Some of these data packets may include the data payload. Other data packets may be relevant to the communication protocol. In examples of the system 100A, the data payload, being stored and retrieved on and from the drives 114A, may itself be encrypted using an encryption key specific to the compute nodes 102A and or the one or more fabric network interface cards 108. Additionally, the whole data packet carrying the encrypted data payload may be further encrypted according to the HTTPS protocol.
The system 100A may be implemented in various configurations, depending on whether single or multiple components describe in greater detail with respect to
The features of
The storage modules 206-1 to 206-n may be one or more redundant array of independent disks (RAIDs). A RAID may be a data storage technology that combines physical disk drive devices into logical units in order to provide data redundancy and low latency. The compute nodes 202-1 to 202-n may include central processing units (CPUs) 214-1 to 214-n, memories 216-1 to 216-n, and fabric network interface cards 208-1 to 208-n. The storage modules 206-1 to 206-n may include embedded storage fabric interface 210-1 to 210-n, CPUs 218-1 to 218-n, memories 220-1 to 220-n, and drives 212-1 to 212-n. The CPUs 214, 218 may be general-purpose computer processors that execute programmed instructions. The memories 216, 220 may be memory devices, such as dual in-line memory modules (DIMMs) that provide random access memory. The memories 216, 220 may include a disaggregated array of independent storage from the fabric to include a redundant array of independent disks. The fabric network interface cards 208-1 to 208-n may be similar to the fabric network interface cards 108 described with respect to
Referring back to
Securing the data in flight at the compute node 202 may be transparent and compatible to all application programs running on that compute node 202. Further, if the encryption/decryption is handled by an accelerator such as a Smart IO device, then the security may be transparent and compatible with any operating system or hypervisor, with only a driver for the Smart IO device.
In examples, the system 200 may provide separate encryption keys for each compute node 202. As such, the data in flight from each compute node 202 to the storage modules 206 may be uniquely encrypted. Thus, even if a single encryption key is stolen, only the compute node 202 to which the encryption key is assigned is compromised. The security of the remaining compute nodes 202 may remain protected against snooping on the fabric 204.
In some examples, the system 200 may provide multiple encryption keys for each compute node 202. In this way, multiple blocks of data in flight from a compute node 202 to the storage module 206 may be uniquely encrypted. In this way, the security of data in flight may be increased. For example, if one of the storage modules 206-1 through 206-n is compromised, only the stream of data assigned to the compromised storage module may be vulnerable to snooping. If one of the encryption keys is compromised, only the stream of data assigned to the compromised encryption key may be vulnerable to snooping.
In response to the request to store the data, the storage module 206-1 may stripe the encrypted data across several of the drives 212. Striping data may involve partitioning data into blocks and writing each block to a different one of the drives 212. More specifically, the embedded storage fabric interface 210-1 may partition the encrypted data into multiple blocks. Further, the storage fabric interface 210-1 may assign drives 212 randomly for writing each block of the partitioned data. For example, the received data may be partitioned into two blocks. Further, the first block may be assigned to drive 2 for storage, and the second block may be assigned to drive 1. Accordingly, each block may be temporarily written to the memory 220-1. Additionally, the CPU 218-1 may write each block to the assigned drives 212. The data in the storage modules 206-1 may be protected from an attack involving the removal of the memory 220-1 because the data remains encrypted throughout its processing in the storage module 206-1.
In some examples, the storage modules 206 may include redundant controllers to dual-port the drives 212. Dual-porting the drives 212 may provide multiple independent data paths to shared storage, which improves the availability of data.
In some examples, the system 200 may add fabric isolation for the data in flight, such as, fibre channel zoning or Ethernet virtual local area networks (VLANs). Fibre channel zoning may involve the partitioning of the fabric 204 into reduced size subsets.
Advantageously, distribution of the encrypted data across multiple drives 212 in each storage module 206 means that an attacker may be prevented from accessing meaningful data by stealing one drive 212. Rather, the attacker may need more than one drive 212, potentially all the drives 212, in addition to the encryption keys from all the compute nodes, and the location of the data on the drives 212 to recover the data from a single compute node.
Advantageously, no single device in the system 200 may be used by itself to steal data. The compute nodes 202 may have the encryption keys, but the data is on the drives 212 in separate storage module(s) 206. Further, the storage modules 206 may contain all the drives 212, but not the encryption keys. Additionally, a stolen drive 212 may not contain all the data for any compute node 202 if the storage for the compute node 202 is striped across several drives 212. Further, storing partial data stripes for multiple compute nodes 202 on one of the drives 212 may further impede attempts by malicious users to extract the data.
At block 404, the fabric network interface card 108 may encrypt the data to be stored using an encryption accelerator. In examples, the compute nodes 102A may include an encryption key for storing data over the fabrics 104A. In some examples, the compute nodes 102A may include multiple encryption keys, one for each stream of data sent over the fabrics 104A. Accordingly, the encryption accelerator of the compute nodes 102A may use different encryption keys to encrypt each stream of data.
At block 406, the fabric network interface card 108 may send the encrypted data to a storage module, such as the storage modules 106 over the fabrics 104A. By sending the data encrypted, the fabric network interface card 108 may protect the data in flight from a malicious user snooping on the fabrics 104A.
At block 408, the storage fabric interface 110 may store the encrypted data on a memory device, such as drive 114A-1 or 114A-2. At block 408, the storage fabric interface 110 may store a first portion of the encrypted data on a first memory device, such as drive 114A-1. Additionally, the storage fabric interface 110 may store a second portion of the encrypted data on a second memory device, such as the drive 114A-2. In examples, the storage fabric interface 110 may partition the encrypted data received from the compute nodes 102A into the multiple partitions. Further, the storage fabric interface 110 may randomly assign each of the partitions to one of the drives 114A. Additionally, the encrypted data may be protected by the encryption key. The encryption key may be stored on the fabric network interface card 108 or on the compute node 102A.
It is to be understood that the process flow diagram of
The tangible, non-transitory computer-readable medium 502 may be accessed by a processor 504 over a computer bus 506. The processor 504 may be a central processing unit that is to execute an operating system in the system 500. A region 508 of the tangible, non-transitory computer-readable medium 502 may store computer-executable instructions that receive a request to store data using a storage module that is connected with a compute node over a fabric. The compute node may include one or more encryption keys. The compute node may include a first network communication apparatus including an encryption capability. The storage module may include a second network communication apparatus. A region 510 of the tangible, non-transitory computer-readable medium may store computer-executable instructions that encrypt the data using a first encryption key and may use an encryption accelerator to encrypt the data. A region 512 of the tangible, non-transitory computer-readable medium may store computer-executable instructions that may send the encrypted data from the compute node to the storage module over the fabric. A region 514 of the tangible, non-transitory computer-readable medium may store computer-executable instructions that may store a first portion of the encrypted data on a first memory device of the storage module and may store a second portion of the encrypted data on a second memory device of the storage module. A region 514 of the tangible, non-transitory computer-readable medium may store computer-executable instructions that may store a first portion of the encrypted data on a first plurality of memory devices of the storage module and may store a second portion of the encrypted data on a second plurality of memory devices of the storage module. In examples, the second network communication apparatus may parse or generate the first portion of the encrypted data and the second portion of the encrypted data. The second network communication apparatus may specify that the first portion of the encrypted data be stored on the first memory device and that the second portion of the encrypted data be stored on the second memory device.
Although shown as contiguous blocks, the software components may be stored in any order or configuration. For example, if the tangible, non-transitory computer-readable medium 502 is a hard drive, the software components may be stored in non-contiguous, or even overlapping, sectors.
The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the disclosure. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the systems and methods described herein. The foregoing descriptions of specific examples are presented for purposes of illustration and description. They are not intended to be exhaustive of or to limit this disclosure to the precise forms described. Obviously, many modifications and variations are possible in view of the above teachings. The examples are shown and described in order to best explain the principles of this disclosure and practical applications, to thereby enable others skilled in the art to best utilize this disclosure and various examples with various modifications as are suited to the particular use contemplated. It is intended that the scope of this disclosure be defined by the claims and their equivalents below.
