Patent Application
20240256510
Distributed file systems today usually target a specific workload. For example, most such file systems assume a small number of very large files that are frequently read with little sharing. Consequently, existing distributed file systems have a rigid design which does not allow for dynamic adjustments to make fundamental trade-offs, for example, read performance vs. write performance, performance vs. scalability, etc. In addition, none of the existing distributed file systems have been designed for disaggregated clusters and, consequently, do not offer the best resource allocation strategies.
Embodiments provide a distributed file system that is built on top of a tree structure and is deployed across a plurality of host computer systems. A method for operating the distributed file system includes the steps of: forming a tree structure, the tree structure having a plurality of nodes including a root node at an uppermost level, internal nodes below the root node, and leaf nodes containing data items, each of the root node and the internal nodes including pivot keys and pointers to at least one child node or at least one leaf node, wherein each of the host computer systems maintains at least one of the nodes and non-leaf nodes are allocated buffers according to a workload of the distributed file system; performing a write operation on a file in the distributed file system by inserting first data specified in the write operation as write data into one of the nodes of the tree structure having a buffer; and performing a read operation on a file in the distributed file system by traversing the tree structure down to a leaf node that stores second data, collecting updates to the second data, which are stored in buffers of the nodes that are traversed, applying the collected updates to the second data, and returning the updated second data as read data.
Further embodiments include a non-transitory computer-readable storage medium comprising instructions that cause a computer system to carry out the above method, as well as a computer system configured to carry out the above method.
In the embodiments, a distributed file system is built on top of a tree data structure that is a modified version of a Bε-tree by mapping file system operation to one or more operations on the modified Bε-tree (hereinafter referred to as the hybrid Bε-tree). In the hybrid Bε-tree of the embodiments, the position of dividing lines between upper-level nodes which do not have buffers and lower-level nodes which have buffers are dynamically adjustable. Adjusting the position of these dividing lines alters the trade-off between scalability, the performance of read operations, and the performance of write operations, thereby allowing the distributed file system that is built on top of this hybrid Bε-tree to adapt to more diverse workloads.
Each of operating systems 110, 118, 126 has a corresponding file system (file systems 111, 119, 127, respectively), which includes a file system driver and data structures maintained by the file system driver. The file system controls how data is stored in a storage device of its corresponding hardware platform, e.g., storage device 115 of hardware platform 114, storage device 123 of hardware platform 122, and storage device of hardware platform 130. Examples of the storage device include a hard disk drive or a solid state drive. In the embodiments, the file system drivers cooperate with each other such that a request for data in distributed file system 120 occurring in one host computer system may be fetched from a local storage device or from another host computer system.
In the embodiments, distributed file system 120 is built on top of a hybrid Bε-tree by mapping every file system operation to one or more operations on the hybrid Bε-tree.
In tree 220 depicted in
In the embodiments, nodes may be locked using leases, and lease state 312 indicates whether a host computer system has acquired a lease and the type of lease that has been acquired. The type of lease may be read-shared or write-exclusive. Lease duration 314 indicates the duration of the lease, in particular the expiration date/time for the lease. Lease owner 316 identifies the host computer system (with the host ID thereof) that has acquired the lease. When a host computer system attempts to acquire a read-shared lease to a particular node and the lease is not available because another host computer has a write-exclusive lease to the node, the host computer system retries at a later time that is guided by the lease duration. When a host computer system attempts to acquire a write-exclusive lease to a particular node and the lease is not available because another host computer has a write-exclusive lease to the node or one or more other host computers have a read-shared lease to the node, the host computer system retries at a later time that is guided by the lease durations. If the lease is available, the host computer system acquires the lease by writing its host ID into the data field for lease owner 316.
In the embodiments, leases are used to lock a node when a structural update to the tree occurs, such as creating a new child node or splitting a node. Leases may also be used to control contention for concurrent operations on the nodes due to multiple host computer systems having independent access to the nodes. In other embodiments, concurrency control can be implemented with atomic operations on the nodes.
Some nodes also include a buffer 306. Buffer 306 represents a location in storage for buffering writes. As will be described below, writes are stored in buffer 306 as key-value pairs, where the key is associated with a target of the write operation (e.g., a file location or a location within a file) and the value is a message that encodes updates to data stored at the target. In the embodiments, upper-level nodes do not have a buffer. Also, leaf nodes do not have pivot keys and child pointers. In addition, each node of the hybrid Bε-tree resides in and is maintained by one of the host computer systems across which distributed file system 120 is implemented.
Tree operations of the hybrid Bε-tree involve a key, which logically represents a file or directory path, and the key maps to a location in distributed file system 120, e.g., an address of a file block and an offset. A query operation involves a traversal of the hybrid Bε-tree to find a node that matches a key that is submitted with the query. A point query operation (PointQuery in
An upsert operation is a special form of an insert operation. The insert operation is performed on a message that is in the form of a key-value pair, where the key is used to find a node in which the message is to be inserted. In the embodiments, a write operation performed on a location in distributed file system 120 translates into an insert operation, where the key maps to the location in distributed file system 120 and the value encodes updates to data stored at the target location. With an upsert operation, an upsert message is inserted into the node. The upsert message contains (k, (f, Δ)) where k is the key, f is a callback function, and Δ is auxiliary data specifying the update to be performed. An upsert operation can be used to implement a file system operation known as read-modify-write. In the embodiments, an update to an entire file block translates to an insert operation whereas a partial block modification translates into an upsert operation.
Some file operations of distributed file system 120 that must be atomic may involve multiple Bε-tree operations. For example, file renames and file creates each involve a range query and upsert operations. To enable this, Bε-tree exposes a transaction API to distributed file system 120.
In step 503, the local host acquires a read-shared lease to the leaf node containing the value associated with the key. If the lease is not available, the local host waits until the expiration of the lease before trying again. If the lease is available, the local host acquires the lease, e.g., by writing its host ID into data field for lease owner 316 of the leaf node.
The processing of messages associated with the key begins in step 504. If there are no such messages (step 504, No), the local host in step 516 releases the lease on the leaf node, and in step 518 returns the value associated with the key, which is stored in the leaf node. If one of the messages is a tombstone message, which is a message to delete a value associated with a key (step 506, Yes), the operation returns ‘Not Found’ and deletes the value in the leaf node along with the tombstone message and any other collected messages that are stored in their respective buffers in step 520.
If there is no tombstone message in the collected messages (step 506, No), the local hosts updates the value associated with the key that is stored in the leaf node by applying the collected messages to the value (step 512). Then, the local host in step 516 releases the lease on the leaf node, and in step 518 returns the updated value.
When the node with a buffer is found in step 652, the local host checks the condition of the node in step 654. If the buffer in the node is not full (i.e., has available space to absorb the message), it acquires a write-exclusive lease to the node in step 656, inserts the message to the buffer of the node in step 658, and releases the lease to the node in step 659. The operation ends after step 659.
If the buffer in the node is full, step 660 is carried out for a non-leaf node and step 666 is carried out next for a leaf node. In step 660, the local host selects a child node having available buffer space into which messages currently stored in the full buffer of the node can be moved, and acquires write-exclusive leases to the node and to the selected child node. Then, the local host moves the messages to the selected child node in step 662. In step 664, the leases acquired in step 660 are released. After step 664, the flow returns to step 654 for the local host to check the condition of the node and proceeds to execute steps 656, 658, and 659 described above if the buffer in the node is no longer full.
If the node is a leaf node and its buffer is full, then the leaf node is split in step 666, creating a new leaf node. The new leaf node is randomly assigned to one of the host computer systems, and pivot keys and child pointers in the parent node are modified accordingly. In step 668, the local host acquires a write-exclusive lease to both leaf nodes, and in step 670, moves one or more messages to the new leaf node. In step 672, the leases acquired in step 668 are released. After step 672, the flow returns to step 654 for the local host to check the condition of the node and proceeds to execute steps 656, 658, and 659 described above if the buffer in the node is no longer full.
Certain embodiments as described above involve a hardware abstraction layer on top of a host computer. The hardware abstraction layer allows multiple contexts to share the hardware resource. These contexts are isolated from each other in one embodiment, each having at least a user application program running therein. The hardware abstraction layer thus provides benefits of resource isolation and allocation among the contexts. In the foregoing embodiments, virtual machines are used as an example for the contexts and hypervisors as an example for the hardware abstraction layer. As described above, each virtual machine includes a guest operating system in which at least one application program runs. It should be noted that these embodiments may also apply to other examples of contexts, such as containers not including a guest operating system, referred to herein as “OS-less containers” (see, e.g., www.docker.com). OS-less containers implement operating system-level virtualization, wherein an abstraction layer is provided on top of the kernel of an operating system on a host computer. The abstraction layer supports multiple OS-less containers, each including an application program and its dependencies. Each OS-less container runs as an isolated process in userspace on the host operating system and shares the kernel with other containers. The OS-less container relies on the kernel's functionality to make use of resource isolation (CPU, memory, block I/O, network, etc.) and separate namespaces and to completely isolate the application program's view of the operating environments. By using OS-less containers, resources can be isolated, services restricted, and processes provisioned to have a private view of the operating system with their own process ID space, file system structure, and network interfaces. Multiple containers can share the same kernel, but each container can be constrained only to use a defined amount of resources such as CPU, memory, and I/O.
Certain embodiments may be implemented in a host computer without a hardware abstraction layer or an OS-less container. For example, certain embodiments may be implemented in a host computer running a Linux® or Windows® operating system.
The various embodiments described herein may be practiced with other computer system configurations, including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like.
One or more embodiments of the present invention may be implemented as one or more computer programs or as one or more computer program modules embodied in one or more computer-readable media. The term computer-readable medium refers to any data storage device that can store data which can thereafter be input to a computer system. Computer-readable media may be based on any existing or subsequently developed technology for embodying computer programs in a manner that enables them to be read by a computer. Examples of a computer-readable medium include a hard drive, network-attached storage (NAS), read-only memory, random-access memory (e.g., a flash memory device), a CD (Compact Discs)—CD-ROM, a CDR, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, and other optical and non-optical data storage devices. The computer-readable medium can also be distributed over a network-coupled computer system so that the computer-readable code is stored and executed in a distributed fashion.
Although one or more embodiments of the present invention have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be made within the scope of the claims. Accordingly, the described embodiments are to be considered as illustrative and not restrictive, and the scope of the claims is not to be limited to details given herein but may be modified within the scope and equivalents of the claims. In the claims, elements and/or steps do not imply any particular order of operation unless explicitly stated in the claims.
Plural instances may be provided for components, operations, or structures described herein as a single instance. Finally, boundaries between various components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention(s). In general, structures and functionality presented as separate components in exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements may fall within the scope of the appended claim(s).
