Patent Grant
9852150
This disclosure generally relates to techniques for providing flexible and extensible network storage systems. More specifically, this disclosure relates to techniques for storing and accessing data in a distributed filesystem.
Enterprise data centers typically include large numbers of storage devices that can be accessed using high-speed networks. However, the management overhead for a large number of storage devices can become substantial. For instance, maintaining data consistency, redundancy, and storage system performance in the presence of hard drive failures can involve substantial time, effort, and expertise.
A number of “cloud-based storage” vendors attempt to simplify storage management by providing large-scale remote network storage solutions. Such vendors can leverage economies of scale to provide extensive data storage capacity that can be leased and accessed by clients. Clients can leverage such storage solutions to offload storage management overhead and to quickly and easily increase their data storage capacity on an as-needed basis. However, cloud-based storage involves another set of inherent risks and overheads. For instance, storing data remotely (“in the cloud”) often increases access latency, and multiple clients simultaneously accessing a shared data set in a cloud-based storage system may suffer from data consistency problems. Furthermore, network failures and/or outages in cloud-based storage systems can prevent clients from accessing their data for significant periods of time.
Hence, what is needed are techniques for providing network-based storage capabilities without the above-described problems of existing techniques.
The disclosed embodiments disclose techniques that facilitate of avoiding client timeouts in a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem in a local storage pool. During operation, a cloud controller receives from a client system a request for a data block in a target file that is stored in the distributed filesystem. Although the cloud controller is already caching the requested data block, the cloud controller delays transmission of the cached data block for a delay interval; injecting this additional delay gives the cloud controller more time to access one or more uncached data blocks for the target file from a cloud storage system, thereby ensuring that subsequent requests of such data blocks do not exceed a timeout interval on the client system.
In some embodiments, a cloud controller: (1) determines the set of data blocks for the target file that are currently cached in its local storage pool; (2) determines that the requested data block is currently being cached locally; (3) requests one or more uncached data blocks for the target file from the cloud storage system; (4) calculates a delay interval for the requested data block; and (5) sends the requested data block to the client system after the delay interval.
In some embodiments, the cloud controller: (1) determines the number of data blocks for the target file that are not locally cached; (2) determines an expected arrival rate for an uncached data block; and (3) sets the delay interval to ensure that, upon receiving a subsequent request for the uncached data block from the client system, the cloud controller has sufficient time to receive the uncached data block from the cloud storage system and forward the uncached data block to the client system before the timeout interval is exceeded on the client system.
In some embodiments, the cloud controller tracks a round-trip delay that measures the amount of time needed to receive previously requested data blocks from the cloud storage system. The cloud controller then dynamically adjusts the delay interval for one or more subsequent requests for data blocks based on this tracked round-trip delay.
In some embodiments, a cloud controller uses metadata for the distributed filesystem to: (1) determine a cloud file that includes an uncached data block; and (2) to determine the location of the uncached data block within the cloud file, thereby determining the portions of the cloud file that need to be transferred to the cloud controller. The cloud controller then determines the delay interval based on the amount of data from the cloud file that needs to be transferred to the cloud controller to access the uncached data block. Note that in some scenarios the amount of cloud file data that needs to be transferred may substantially exceed the size of the uncached data block.
In some embodiments, calculating the delay interval involves using conservative estimates for the round-trip delay for an initial set of uncached data blocks to ensure that changes in network characteristics and cloud controller load that delay the transmission of the uncached data block do not result in a timeout on the client system. In some scenarios, this calculation may involve adding an additional safety margin to the delay interval.
In some embodiments, dynamically adjusting a delay interval may involve considering: an estimated network bandwidth available to the cloud controller; the number of uncached data blocks for a target file that the cloud controller has received from the cloud storage system; the number of uncached data blocks for the target file that remain to be downloaded to the cloud controller; and the rate at which the cloud controller is receiving requests from the client system for data blocks of the target file.
In some embodiments, the cloud controller tracks file and data block access patterns, and uses these tracked access patterns to determine the set of data blocks that will be cached locally.
In some embodiments, a cloud controller transfers a first set of cached data blocks for a target file to the client system without delay to reduce client-perceived file access latency. If the client system requests more data blocks for the target file, the cloud controller may delay the transfer of a second set of cached data blocks for the target file for a delay interval to ensure that additional uncached data blocks for the target file can be retrieved by the cloud controller in a manner that does not exceed a timeout interval on the client system.
The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
The data structures and code described in this detailed description are typically stored on a non-transitory computer-readable storage medium, which may be any device or non-transitory medium that can store code and/or data for use by a computer system. The non-transitory computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing code and/or data now known or later developed.
The methods and processes described in the detailed description section can be embodied as code and/or data, which can be stored in a non-transitory computer-readable storage medium as described above. When a computer system reads and executes the code and/or data stored on the non-transitory computer-readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the non-transitory computer-readable storage medium.
Furthermore, the methods and processes described below can be included in hardware modules. For example, the hardware modules can include, but are not limited to, application-specific integrated circuit (ASIC) chips, a full-custom implementation as part of an integrated circuit (or another type of hardware implementation on an integrated circuit), field-programmable gate arrays (FPGAs), a dedicated or shared processor that executes a particular software module or a piece of code at a particular time, and/or other programmable-logic devices now known or later developed. When the hardware modules are activated, the hardware modules perform the methods and processes included within the hardware modules.
Evolution of Network-Based Storage Systems
The proliferation of the Internet and large data sets has motivated a range of specialized data storage techniques. For instance, network-attached storage (NAS) devices often serve as centralized storage devices that provide large amounts of storage space for a set of heterogeneous clients in an enterprise. Such devices are typically tuned to provide a desired level of performance, redundancy (e.g., using a redundant array of independent disks (RAID)), and high availability. For example, while typical filesystems may take a substantial amount of time to recover from a crash (as the system has to process logs and/or journals to correctly rebuild modified data that was queued or in the process of being written at the time of the crash), NAS devices often incorporate transactional copy-on-write filesystems, which sacrifice some read performance in exchange for faster crash recovery. In a transactional copy-on-write filesystem, a file is not modified in place; instead, the system uses a delta encoding to append modifications (“deltas”) to the previous file data. Such encodings increase the overhead associated with read operations, because the system incurs additional computation and access time to read and process deltas stored at the end of a file. However, this encoding also ensures that files are “data-consistent” (e.g., reliably incorruptible and consistent across crashes and reboots), thereby allowing NAS devices to recover quickly from failures. Such characteristics and capabilities have made NAS devices popular in enterprise environments.
Unfortunately, storage scalability can become problematic when data needs outscale the capabilities of a single NAS device; providing redundancy across multiple separate NAS devices (as illustrated in
Note that there is a significant distinction between filesystem-level operations and block-level (e.g., disk-level) operations. A filesystem typically serves as an intermediary between an operating system and one or more block-level devices. More specifically, a filesystem typically attempts to efficiently manage one or more block-level devices to provide more sophisticated storage services to an operating system. For instance, filesystems often manage disk blocks and metadata to provide structure (e.g., files and directories) and some notion of access rights and data consistency (e.g., via file lock operations) for an underlying block storage mechanism. Hence, filesystem-level operations provide a higher level of abstraction (e.g., a filename and an ordering associated with an underlying set of disk blocks) for the block storage mechanism.
Typically, a filesystem and an associated block storage device both operate in the context of the same computing device, and the block storage device has been specially initialized (e.g., formatted) to support the filesystem. Upon receiving a request for a filesystem operation (e.g., from an operating system and/or application), the filesystem determines and initiates a set of block-level operations needed to service the request. Hence, there is a notion of “filesystem-level information” (e.g., the level of information managed by the filesystem and received in a request for a file operation) and a separate notion of “block-level information” that is used when the filesystem operates upon the underlying block storage device. In the example of
A number of “cloud-based storage” vendors (also sometimes referred to as “cloud storage providers”) attempt to simplify storage management by providing large-scale network storage solutions. Such vendors can leverage economies of scale to provide data centers with extensive data storage capacity that can then be rented and accessed by clients, thereby allowing clients to offload storage management overhead and easily increase their data storage on an as-needed basis. However, cloud-based storage also includes another set of inherent risks and overheads. Storing data remotely (“in the cloud”) often increases access latency, and network failures and/or outages in cloud-based storage systems can prevent clients from accessing their data for substantial time intervals. Furthermore, multiple clients simultaneously accessing a shared data set in a cloud-based storage system may suffer from data consistency issues.
Consider a scenario where one remote client attempts to write a set of data to a cloud storage system, and a second remote client attempts to read the data that is being written. In some systems, a reader may not be able to see the existence of newly written file data until the entire write operation has completed (e.g., after the first remote client has closed the file). In other (non-data-consistent) arrangements, the reader may see and access the file, but because writes are stateless and potentially out-of-order (e.g., as in the Network File System (NFS) protocol), does not know which file sections have already been written, and hence may access a mix of valid data and garbage.
Embodiments of the present invention combine aspects of NAS capabilities and cloud-based storage capabilities to provide a high-capacity, high-reliability storage system that ensures that data can be accessed in a data-consistent manner.
Providing Data Consistency in a Cloud Storage System
In some embodiments, a set of caching storage devices (referred to as “cloud controllers”) collectively cache, manage, and ensure data consistency for a set of data that is stored in a network storage system (e.g., a cloud-based storage system, which is also referred to as a cloud storage system). More specifically, one or more cloud controllers work together (e.g., as a federation) to manage a distributed filesystem with a global address space. Each cloud controller maintains (e.g., stores and updates) metadata that describes the file and directory layout of the distributed filesystem and the location of the data blocks in the cloud storage system. Each cloud controller can also cache a subset of the data that is stored in the cloud storage system. A cloud controller that writes (or modifies) data ensures that: (1) data changes are reflected in the cloud storage system; and (2) other cloud controllers in the system are informed of file and metadata changes.
Note that while the cloud storage system stores the data for the distributed filesystem, the cloud storage capabilities may be provided by an external vendor. An enterprise storing sensitive data in the distributed filesystem may not want this vendor to be able to access such data, and hence, the cloud storage system may be configured to store the distributed filesystem's data in the form of encrypted storage volumes (referred to as “cloud files” or “drive files”). This configuration enhances data security, but also prevents the cloud storage system from actively assisting in ensuring data consistency and performing other operations that require knowledge of the data and data layout. More specifically, in some embodiments the cloud controllers fully manage the filesystem and manage data consistency, with the cloud storage system providing purely storage capabilities.
Note that using a transactional filesystem in each cloud controller does involve some additional overhead. As described above, the transactional filesystem tracks modifications using delta encoding (instead of the more typical read/copy/modify operations used in many non-data-consistent filesystems). For instance, consider a 1 KB modification to an existing 3 KB file in a filesystem that supports 4 KB blocks. Using a traditional approach, the filesystem might read out the original 4 KB block, modify the block to reflect the updates, and then write the modified file back to the same block. In contrast, in a transactional filesystem, the original block is left unchanged, and the filesystem writes out the modifications and additional data to another empty 4 KB block. The metadata for the transactional filesystem is extended to support the notion of partial blocks and deltas (e.g., including one pointer that points to 3 KB of data in one block and another pointer that points to another block that contains 1 KB of additional data and a set of changes that should be applied to the initial 3 KB of data).
In some embodiments, using a transactional filesystem (e.g., transactional filesystem 308 in
In some embodiments, cloud controllers generate separate metadata snapshots and file data snapshots. Metadata is typically much smaller than file data, and is needed to access file data. Furthermore, each cloud controller is typically configured to maintain (and update) the full set of metadata, but only caches file data that is needed by local clients. Hence, uploading (or sending) a metadata snapshot separately means that the updated metadata will be more quickly available to other peer cloud controllers. Each of these peer cloud controllers can then determine (e.g., based on client data usage and needs) whether to access the related file data associated with the updated metadata. Note that a cloud controller may still upload both metadata updates and file data updates to the cloud storage system, but may split them into different sets of cloud files (or both include the metadata with the file data as well as generate another separate, duplicative update that includes only metadata) so that other cloud controllers can access the two separately. In such an organization, a cloud controller might then send a message to other cloud controllers specifying the location of the stored metadata snapshot. Alternatively, cloud controllers may also be configured to send metadata snapshots directly to a set of peer cloud controllers.
Consider an example of a cloud controller receiving a request from a client to store a 10 GB file, in an environment where the network link between the cloud controller and a cloud storage system supports a transfer speed of 1 GB/minute and the cloud controller is configured to send a metadata snapshot every minute. Upon determining the scope of the file operation, the cloud controller can already allocate a set of corresponding disk blocks and cloud files, and generate a set of corresponding metadata that indicates the respective disk addresses and CVAs for the file's data blocks. The cloud controller then uploads the file data to the cloud storage system over a time interval (e.g., roughly ten minutes), and sends out metadata snapshots that indicate the existence and location of the data blocks. The cloud controller may convey a range of information about the data being uploaded to other cloud controllers depending on the level of transparency and availability desired for modified data. For instance, in some embodiments, the file remains accessible by clients via the originating cloud controller throughout the upload process. However, other cloud controllers that have received the corresponding metadata and seek to access modified data that has not yet been received by the cloud storage system may receive an indication that the data is not yet available, and that their access attempts should be re-tried at a later time (or after a specified time interval). Alternatively, in some instances, when a set of data has not yet been uploaded to the cloud storage system, a client (and/or cloud controller) that hence cannot yet access this data via the cloud storage system may be configured to gain access to the desired data by directly interacting with the cloud controller hosting the desired data. Such alternative access techniques may depend on the capabilities and topography of the network connecting the cloud controllers and cloud storage system.
In some embodiments, the originating cloud controller may propagate additional intermediate metadata that informs other cloud controllers as portions of the modified data become available in the cloud storage system. For instance, metadata snapshots may indicate files that are in the process of being uploaded, and include a field that indicates whether a given data block has been successfully stored in the cloud storage system. The cloud controller updates (and propagates) this metadata as it receives acknowledgments of receipt from the cloud storage system, thereby indicating that some of the data being uploaded is now already available in the cloud storage system. For example, immediately after first storing the 10 GB file locally, the cloud controller may have already reserved 10 GB of space in cloud files at a given set of CVA addresses (e.g., in the cloud storage system), but have not yet transferred any file data. A snapshot sent at this point includes metadata that indicates the existence of the file, but also indicates that none of the data is available in the cloud storage system yet. After one minute, the cloud controller sends out another snapshot containing metadata that reflects the set of data that has already been transferred to (and been acknowledged as received by) the cloud storage system.
In some embodiments, each cloud controller maintains a set of structures that track snapshots and changes in metadata, and updates its local metadata to reflect updates from the rest of the distributed system. For instance, a cloud controller receiving the first snapshot from the above example may note the creation of a 10 GB file (as described in the above example), but then also determine that none of the associated data blocks is available yet. After receiving and processing the second snapshot, the receiving cloud controller determines the presence and location of the first GB of stored data that is now available. At this point, the receiving cloud controller may, if desired, use the received metadata to already download and access the available file data from the cloud storage system on behalf of a client. If, however, a client requests additional parts of the file that have not yet been stored in the cloud storage system (as indicated by the metadata in the most recent snapshot), the cloud controller can signal that the desired data is not yet available, and delay the access. More of the file data becomes available over time, as indicated by the subsequent snapshots.
Note that cloud controllers can use the detailed information received in snapshots to provide a range of data access and data consistency capabilities. More specifically, each cloud controller receives ongoing updates that identify valid data, and indicate how to find and access such data. If data is written to the cloud storage system out-of-order, this is reflected in the received snapshot(s), and the cloud controller (and/or a requesting client) can use such received snapshot information to determine how to proceed.
In some embodiments, cloud controllers may use stored snapshot data to provide access to different versions of a file. For instance, in the preceding example, a cloud controller may allow a client to already access the uploaded (and acknowledged) portions of a new file before the file has been completely uploaded to the cloud storage system. Similarly, the cloud controller may allow the client to access modified file data as it becomes available in the cloud storage system. Alternatively, in other scenarios, when an existing file is being modified, a cloud controller may be configured to present a previous version of the file to clients until the complete set of data for the modified version is available in the cloud storage system. In some embodiments, cloud controllers may maintain records of past snapshots to allow file accesses to be rolled back across multiple different versions, thereby allowing clients to view historical versions of files and/or the changes made to files over time.
In general, the disclosed techniques leverage transactional filesystem techniques and snapshots to ensure that only valid data can be accessed. While these techniques involve some additional complexity, they also provide an assurance of data consistency for a distributed filesystem that leverages cloud storage. The following sections describe additional aspects of storing and accessing data in the disclosed distributed filesystem.
Generating Snapshots and Cloud Files
A number of factors affect the performance of accessing data from a cloud storage system. In a typical computer data is stored locally on a disk, and a number of hardware and operating system mechanisms attempt to minimize the latency of reads and writes. For instance, processors and operating systems strive to load frequently used data into memory and multiple levels of hardware caches, thereby reducing the latency associated with reading data from disk. Accessing data stored on a cloud storage system involves an additional set of latencies. For instance, in addition to normal disk latency, accessing a cloud storage system may involve additional latency due to network latency, network protocol handshaking, network transfer times, and delays associated with encryption or decryption. One of the challenges of a distributed filesystem is minimizing such latencies as much as possible.
One factor that can significantly affect the latency of data access in the described distributed filesystem is cloud file size. Overly small cloud files can result in higher network negotiation and transfer overhead. Conversely, overly large cloud files can result in large transfer delays; for instance, a cloud controller that needs only a small piece of data from a large cloud file that is serially encrypted may need to wait for the entire file to be downloaded and decrypted before it can access the desired data. Determining a reasonable cloud file size that maximizes throughput to and from the cloud storage system may depend on factors such as network link size and latency (e.g., transfer speeds), local filesystem block sizes (e.g., making the cloud file size a multiple of a local block size), and CVA pointer sizes or boundaries. Another trade-off involves determining whether to use fixed-sized or variable-sized cloud files. Variable-sized cloud files allow some level of customization to match network and application characteristics, but also involve additional complexity to manage the different sizes. Hence, in some embodiments the system reduces management overhead by using a single fixed cloud file size (e.g., 32 MB) throughout the cloud controllers and cloud storage system. Note, however, that the contents of each cloud file may vary based on the set of data currently being generated or modified. For instance, data blocks being stored for large files (e.g., larger than 32 MB in the case of 32 MB cloud files) may be split across two or more cloud files. Alternatively, if the current load involves storing data for multiple small files or making a large number of small file modifications, a corresponding cloud file may contain multiple user files and deltas. Note also that, in some embodiments, data and meta-data are always separated into different cloud files. In general, cloud controllers may use a range of techniques to stripe chunks of data across cloud files in an attempt to optimize subsequent accesses from such cloud files.
Note that cloud files are also written to in an incremental, transactional fashion, to preserve data consistency. More specifically, new and modified file data is written to a separate cloud file, as in a transactional filesystem, to ensure that the consistency of previous file versions is preserved. Thus, an initial set of data for a given file is written to one cloud file, and later additions or modifications to the file detected by a subsequent snapshot are written to a new, different cloud file.
The filesystem metadata for each disk block includes information that specifically identifies the location and enables the lookup of the disk block in a cloud file. For instance, the metadata may include one or more of the following: a CVA (cloud virtual address) that uniquely addresses the cloud file; the offset of the disk block in the cloud file; a physical and logical size for the disk block; the portions of the disk block that are valid; compression information; a checksum hash value or other checksum information; and information that indicates whether the disk block has already been successfully uploaded to the cloud storage system.
To ensure data consistency, cloud controllers need to ensure that each cloud controller assigns unique CVAs that create non-overlapping cloud files. More specifically, the cloud controllers need to collectively manage the global address space for the distributed filesystem. In some embodiments, each cloud controller is assigned a unique identifier, the collective set of cloud controllers are associated with a total amount of cloud storage space, and each cloud controller is pre-allocated a portion of the global address space. In such embodiments, a cloud controller can already allocate a cloud file in this pre-allocated address range at the time that it writes a new disk block, and store the CVA of the cloud file in the block's metadata. This organization ensures that there are no collisions in allocating cloud file addresses, and also ensures that even the first metadata snapshot for a new disk block already includes an accurate (future) location of the disk block in the cloud storage system. Note that the allocations of the global address space can be adjusted as needed over time if more data is created on a subset of the cloud controllers.
While most updates and file changes can propagate through the distributed filesystem via incremental snapshots, some data consistency issues can arise if multiple clients accessing two different cloud controllers attempt to simultaneously write the same file. Downloading and importing snapshot data may involve some latency, and thus such conflicting operations may lead to race conditions and errors. Hence, in some embodiments, each file is associated with a cloud controller that “owns” (e.g., actively manages) the file. For instance, the cloud controller from which a file was first written may by default be registered (in the file block metadata) as the owner (e.g., the owning cloud controller) of the file. A cloud controller attempting to write a file owned by another cloud controller first contacts the owner with a request to lock the file. The owner can determine whether to grant or deny the lock request. In some embodiments, even if this request is granted, all write operations may be required to go through the cloud controller that owns the file (e.g., new data is written to the local filesystem of the owning cloud controller). Note that while every cloud controller actively manages a set of files, a given cloud controller may not need to continue to cache every disk block of files that it owns; once such blocks have been written to the cloud storage system, they may subsequently be cleared from the cloud controller to make space for other needed data. However, the metadata for all of the files in the distributed system is typically maintained in every cloud controller. In some embodiments, the system may also include mechanisms for transferring ownership of files between cloud controllers (e.g., migrating file ownership to cloud controllers that are the primary modifiers of the file to reduce network latency).
Optimizing the Creation of Cloud Files
Note that a cloud controller may use a range of techniques to generate cloud files. For instance, one such technique may involve: (1) allocating one or more cloud-file-size memory buffers; (2) copying the file and metadata for the cloud file into a memory buffer; (3) encrypting the contents of the memory buffer; and (4) uploading the encrypted contents of the memory buffer to a cloud storage system as a cloud file. Note, however, that this technique involves allocating and using additional memory buffers, and potentially performing a large number of data copy operations.
In some embodiments, a cloud controller generates an additional set of filesystem overlay metadata that allows existing file data and metadata to be virtually linked together into a cloud file view. For instance, the system can construct such overlay metadata when writing new blocks (and corresponding metadata). Alternatively, the cloud controller may instead generate such overlay metadata while traversing the filesystem to find changed data to include in the next incremental snapshot. Either way, unlike the above memory-buffer technique, this overlay metadata facilitates minimizing the use of additional resources by creating cloud files “in place” (e.g., without allocating additional memory buffers or additional copy operations); instead, a set of pointers point to the original blocks in the transactional filesystem that contain the modified data and metadata. Note that while such additional overlay metadata may involve some additional space and computational complexity, these additional needs are typically small compared to the space and copy overhead associated with other approaches.
When creating a snapshot, a cloud controller can access the overlay metadata to read, encrypt, and upload the cloud file to the cloud storage system. For instance, the overlay metadata may facilitate accessing the virtual cloud file via a special filesystem directory that presents a view of the disparate data blocks as a single, consolidated cloud file that can be read and transferred. In many scenarios the cloud controller primarily maintains overlay metadata for data that has not yet been written out to a cloud file; in some embodiments, once the data has been uploaded to the cloud storage system, the cloud controller clears the overlay metadata and begins generating new overlay metadata to track changes destined for a subsequent set of cloud files. In other embodiments, a cloud controller may maintain snapshot information and overlay cloud files locally for a longer time interval (e.g., until space constraints on the cloud controller prompt the removal of infrequently used data).
In some embodiments, a cloud controller attempts to optimize the placement of data into cloud files to reduce future access overhead. For instance, the cloud controller may strive to, when possible, store all blocks for a file in the same cloud file (e.g., assuming the size of the file and/or file modifications are smaller than the size of a cloud file). Toward this end, the cloud controller may place data into multiple cloud files in parallel, and avoid storing data for multiple files in the same cloud file unless the complete set of data for some or all of the files will fit. A cloud controller may also perform additional file grouping based on user configuration and/or automatic analysis of file access trends. For example, users may be provided with a way to configure a policy that reflects anticipated file access patterns, groupings, and/or priorities (e.g., a user policy that indicates files with a certain extension are likely to be accessed together, and thus should be grouped together).
Note that some cloud files may be partially empty. For instance, a cloud controller that is generating a snapshot based on a time interval or a cloud controller that is placing data into multiple cloud files in parallel to optimize future read operations may not have enough data to fill a complete cloud file. In such scenarios, the cloud controller may simply write out the available data, and leave the wasted space to be reclaimed using a future reclamation operation. For example, in some embodiments a cloud controller may be configured to: (1) download file data spread across two or more cloud files; (2) reassemble the desired data into a new cloud file; (3) upload the new cloud file to the cloud storage system; and (4) distribute a metadata snapshot that updates the access information for the affected files. In some scenarios, such optimizations may be performed by the cloud controller that owns the files, and involve locking the files during the cloud file defragmentation process. In scenarios where locking files for a long interval is undesirable, such operations may involve: (1) maintaining the previous cloud files the final metadata snapshot has been generated; (2) determining that the file data has not changed during the download, reassembly, and upload operations; and (3) briefly locking the original cloud files and metadata sections until the system can confirm that the updated metadata has been successfully propagated to all of the cloud controllers in the system. If the file data being compacted has changed during the interval, the changes can be discarded and compaction can be re-tried at a later time. In some embodiments, compaction operations may be limited to older (e.g., less likely to be modified) cloud files, to reduce the likelihood of collisions. Alternatively, such operations may involve determining timeframes in which particular cloud files are less likely to be used, and performing compaction operations at those times.
Accessing Cloud Files and Managing Local Disk Layout
The previous sections disclose techniques for generating snapshots and uploading data to cloud files. Using such techniques, cloud controllers can treat the cloud storage system as an object store. Other cloud controllers receiving metadata updates can then access data from cloud files as needed. Furthermore, a cloud controller that has uploaded data can, if needed, flush data that has been uploaded from its local filesystem (e.g., “clear its cache”) to make space for other data that is more likely to be needed immediately. Note, however, that a cloud controller flushing data still keeps the accompanying metadata, so that the flushed data can be found and reloaded from the cloud storage system if needed again.
In some embodiments, a cloud controller downloads a cloud file into a dedicated memory buffer, and operates directly upon this memory buffer to access the desired data. The cloud file format is easy to operate upon in memory, and the downloaded data can be accessed very quickly from memory. However, storing such blocks in memory also constrains memory use, and (depending on the application) client data demands may involve reloading and caching more data than can be stored in memory. Furthermore, operating upon downloaded cloud files purely in memory may require a cloud file to be re-downloaded if the cloud controller is power cycled. Hence, in alternative embodiments, the cloud file is unpacked and re-integrated into the local transactional filesystem of a downloading cloud controller (e.g., into the highlighted subset of disk blocks 428 in
Note that a cloud controller can choose the target disk blocks that will receive the downloaded data blocks; for instance, in the context of
In some implementations, lookup structure 434 is implemented as a single-level hash table that is indexed using a hash value derived from the CVA and cloud file offset associated with a requested file data block. In such implementations, the hash lookup determines whether the requested file data block is cached, and if so, returns a pointer to the data block's location in PRC 430. If the block is not cached, the cloud file referenced by the CVA is downloaded from cloud storage system 302, the requested data block is loaded into PRC 430, and lookup structure 434 is updated to point to the now-cached data block.
Unfortunately, a single-level hash table implementation may sometimes become unwieldy and fragmented; for instance, hashing on a per-block basis (e.g., basing hash values on a combination of the CVA and offset) can lead to the hash table becoming sufficiently large that part of the hash table may be paged out of memory (e.g., written to disk), thus leading to a large number of disk reads which may negatively affect the performance of data accesses. Hence, in some implementations, lookup structure 434 is structured as a multi-level hash table and/or lookup structure (as illustrated in
Note that the use of a separate lookup structure 434 adds a level of indirection that can facilitate updating the system when disk blocks are flushed from PRC 430. More specifically, when a disk block is being flushed from PRC 430, cloud controller 420 can use the disk block's CVA and offset to look up and remove the pointer to the data block from the lookup structure 434. In an alternative embodiment that did not use a lookup structure (e.g., instead having the block pointer in a block record point directly to a cached data block), cloud controller 420 might need to traverse filesystem metadata 424 to find and update a target block record or maintain a reverse mapping that facilitates looking up the block records for data blocks 432 stored in PRC 430.
While
In some embodiments, the block allocation policy used in a cloud controller's transactional filesystem is altered to prioritize a selected set of disk sectors toward either data or metadata. More specifically, by dynamically weighting some disk regions toward metadata, the filesystem can create dedicated, metadata areas on the disk that are distinct from their respective data blocks, and no longer interleaved on a per-file basis. While distinct, these metadata areas can still be allocated in close-enough proximity to the data blocks that they reference that both can be read without substantially degrading performance. When data is subsequently flushed, all of the disk blocks holding data are cleared, and new data and metadata can be written into the disk region; new metadata is written into the disk blocks weighted toward metadata, while the new data blocks can be stored into the nearby (flushed) disk regions. Because metadata is typically much smaller than the actual file data (e.g., in many scenarios metadata is on the order of 0.1% of the size of the file data that it manages), this arrangement facilitates avoiding fragmentation across a large number of write/flush cycles.
Note that the amount of metadata in a distributed filesystem may, depending on the client and application load, grow significantly over time. Hence, cloud controllers may need to periodically adjust the number and weighting of disk regions containing metadata (and/or a number of allocated metadata regions), so that more blocks that store metadata are allocated in proximity to a set of blocks used to store file data. Note also that in some embodiments the filesystem may attempt to optimize read accesses by trying to store (where possible) disk blocks that have been loaded from cloud files in proximity to their metadata.
In some embodiments, metadata and data may be divided across multiple storage volumes (e.g., multiple storage devices or redundant arrays of independent disks (RAIDs)). Storing metadata and its associated cached data on different storage volumes facilitates reducing seek latencies by allowing metadata and cached data to be accessed in parallel from their respective storage volumes. Note that such organizations may still involve splitting (and balancing) metadata and cached data across the different volumes; for instance, the metadata and cached data may be split such that each volume stores a subset of the metadata and a subset of the cached data, but no individual volume stores cached data referenced by its stored metadata. Such an arrangement allows metadata to be accessed in parallel from all of the volumes when high throughput is needed, while also still allowing associated sets of metadata and cached data to be accessed in parallel. Note also that each individual module may still be partitioned into metadata and data regions, to avoid the same fragmentation issues described above.
In some embodiments, a cloud controller may arrange data cached in local storage to optimize performance. For instance, the level of indirection described in
In some embodiments, a cloud controller may be configured to perform defragmentation operations for cached data blocks based on a range of factors. For instance, a cloud controller may track usage statistics for file data blocks on an ongoing basis, and then use such tracking information to ensure that cached data blocks that are frequently accessed together are grouped together in the local cache. However, because tracking (and defragmentation in general) can involve fairly high overhead, a cloud controller may alternatively be configured to track and configure specific file types that are known to exhibit fragmentation, or to emphasize optimizing frequently accessed files whose tracking data indicates a certain common access pattern (e.g., a contiguous access pattern). In some embodiments, a locality policy may be used to specify: (1) specific file types to be considered and/or emphasized for defragmentation; (2) specific access patterns to detect and optimize for; and (3) a frequency and/or time interval for performing fragmentation checks and/or operations.
Pre-Fetching Cloud Files
As mentioned previously, cloud files are also written to in an incremental, transactional fashion. For instance, files that are written and/or modified across multiple snapshots may have data stored in different cloud files. Unfortunately, accessing data split across multiple cloud files can increase access latency substantially. Consider a client that sends a request to a cloud controller to access a data block in a file. The cloud controller inspects its current set of metadata, determines that the data block is not currently cached, downloads the corresponding cloud file containing the data block, and presents the data block to the client. The client may then request additional data blocks from the same file. In some scenarios, all of these blocks will have been stored in the same cloud file, and are now available in the cloud controller. If, however, some of these file blocks are in another cloud file, the client will have to endure additional network, download, and processing latencies as one or more additional cloud files are accessed. While an initial delay may be acceptable, recurring access delays may cause a noticeable delay and substantial user frustration. In some embodiments, pre-fetching techniques facilitate reducing the access latency experienced by a user accessing files from a distributed filesystem.
In some embodiments, a cloud controller predictively pre-fetches additional cloud files in an attempt to reduce access latency. For instance, upon receiving a request to access a given data block for a file, a cloud controller may analyze the metadata for the file and then predictively pre-fetch other cloud files that contain other nearby data blocks (or even all other data blocks for the file, depending on the file size). Alternatively (and/or additionally), the cloud controller may also pre-fetch data for other associated files that are likely to be accessed in conjunction with the original file. For instance, when a user browses a given file directory, the cloud controller may pre-fetch the first data block for every file in that directory and the contents of each subdirectory to anticipate the user's next actions. Alternatively, the cloud controller may pre-fetch files that are known to be associated with a requested file (e.g., an icon file that contains an icon representation for a selected file, along with any other support files for the requested file). In such situations, the cloud controller can traverse its stored set of metadata to look up the physical locations (e.g., the CVAs and offsets) for cloud files that should be pre-fetched from the cloud storage system.
Note that NAS workloads are very user-based, and hence can be quite random as a large number of users access a fairly random set of files; the amount and range of pre-fetching performed may vary based on factors such as the available network bandwidth, cloud controller load, and types of accessing clients or applications. For instance, a basic optimization may involve pre-fetching data for other files in the same directory as the current file being accessed. More sophisticated pre-fetching techniques may involve receiving application (or user) feedback that specifies file associations and/or files that are likely to be needed soon. Cloud controllers may also be configured to track access patterns over time to determine files and file sections that have temporal locality. For example, consider a large (e.g., terabyte-sized) set of seismic data that is typically accessed sequentially. Cloud controller tracking may: (1) detect the sequential nature of the data; (2) make note of this behavior in the file's metadata; and then (3) perform selective pre-fetching upon successive accesses to ensure that each subsequent cloud file is pre-fetched, thereby providing latency-free file access for a client. Note that some of these techniques may also facilitate initially grouping commonly accessed files and/or data blocks into the same cloud file so that they can be accessed more efficiently at a later time. Pre-fetching configurations may be specified and customized for a range of different granularities (e.g., globally, per-site, or for individual cloud controllers) using locality policies (as described in a following section).
While the previous paragraphs describe techniques for identifying and pre-fetching additional cloud files, pre-fetching and caching decisions may also occur at the granularity of an individual cloud file. As described previously, a cloud file may contain data blocks associated with a large number of files. However, not all of these data blocks may be related to and/or useful for a given user request; furthermore, caching all of the data blocks for every cloud file accessed by a cloud controller can quickly fill the cloud controller's cache with potentially un-useful data. Hence, in some embodiments, cloud controllers may be configured to analyze local filesystem metadata and metadata from a cloud file being downloaded to determine the portions of the cloud file that should be downloaded and/or cached.
In some embodiments, a cloud controller performs “opportunistic caching” by using block metadata stored in a cloud file to determine other data blocks in the cloud file that should be downloaded and cached. For instance, the beginning of each cloud file may include a set of metadata for each data block that is stored in the cloud file. This metadata block specifies, but is not limited to, one or more of the following: a unique filename and/or number that identifies the file in the distributed filesystem that its corresponding data block in the cloud file is associated with; a compression algorithm used to compress the data block; the logical size of the data block; the physical size of the data block; a checksum for the data block; the checksum algorithm used to calculate the checksum; and the type of checksum. The cloud controller can analyze distributed filesystem metadata and metadata for the cloud file to determine which data blocks may be associated with the current block request, and hence should be cached.
Upon receiving the metadata from the cloud file, the cloud controller can determine the unique filenames and/or file identifiers that are stored in the metadata. However, the cloud controller may not be able to perform a reverse mapping to determine the location (e.g., path) of this file data in the distributed filesystem; for instance, distributed filesystem metadata may be stored in a manner that optimizes using file data structures to determine the CVA for a given file but does not include a reverse lookup mechanism that maps file identifiers back to a location in the distributed filesystem. If no such reverse mapping is available, scanning the metadata of the distributed filesystem to determine the location of a given file's data block may involve substantial computation and delay. However, the cloud controller does know the location of the requested file data block in the distributed filesystem, and thus can determine the file identifiers for files and directories in proximity to the requested file without substantial additional effort. The cloud controller can compare the set of file identifiers for the data blocks in the cloud file with the file identifiers for these proximate files and directories to determine which data blocks from the cloud file should be opportunistically cached; unrelated data blocks are not cached. As previously noted, related files have temporal and special locality, and thus are likely to be stored in the same cloud file and accessed again in a substantially similar timeframe; hence, such opportunistic caching techniques are likely to cache blocks that will be used in the near future, thereby improving access performance.
In some embodiments, a cloud controller downloading a cloud file may determine from the metadata in the cloud file that a large number of data blocks at the end of the cloud file are not likely to be needed, and request “transport termination” for the remaining unneeded portion of the cloud file, thereby saving network bandwidth. In some implementations where the cloud file is not encrypted, the cloud controller may be able to analyze the metadata from the cloud file to determine sections of the cloud file that are not needed, and instruct the cloud storage system to only transfer needed portions of the cloud file. In alternative implementations where cloud files are encrypted and compressed from beginning to end, additional portions of the cloud file may need to be transferred. More specifically, the metadata for the blocks of the cloud file can be stored at the beginning of the file data, and are analyzed upon receipt and decryption. Because of the serial encryption and compression, all data up to and including a given target data block will need to be downloaded, decrypted, and decompressed to allow the target data block to be accessed. If, however, data after the location of the target data block (or the location of some other beneficial data block later in the cloud file) is not determined to be of likely benefit, the cloud controller can instruct the cloud storage system to stop sending any data after that specified location. In some embodiments, the cloud controller can make this determination after receiving and analyzing the cloud file metadata, thereby ensuring that no extraneous data is sent from the cloud storage system. Such techniques can be particularly beneficial in situations where cloud files contain data blocks for a large number of unrelated files (especially if the target data is near the beginning of the cloud file) and in situations where metadata and/or application hints can indicate that only a limited set of data blocks from the cloud file will be accessed.
Receiving and Servicing Client Requests
Client systems typically use network protocols (such as the Network File System (NFS) and the Common Internet File System (CIFS) protocols) to access network-based storage systems. CIFS (also sometimes referred to as Server Message Block (SMB)) is a complex application-layer network protocol that includes many application-specific capabilities that blur the typical separation between filesystems and applications. When a user accesses a file on a client system using software (e.g., Microsoft Office) that supports CIFS functionality, the client system may send CIFS requests to the network storage system to inform that the file has been opened and to store specific data. The CIFS server receiving the request for the file operation recognizes the file type, and can provide application-specific support. For instance, the CIFS server may maintain a network connection, and after determining that only that client is accessing the file, instruct the client to continue to cache file data indefinitely. Later, upon detecting that another client is attempting to open the file, the CIFS server can instruct the first client to flush the cached file data, and provide collaboration capabilities to both clients. Such functionality can enhance the user experience, but can complicate the interface between clients and storage systems; crafting a high-performance implementation of the CIFS protocol can involve substantial effort and expense.
In addition to complexity, CIFS also suffers from a range of other drawbacks and/or limitations. CIFS does not allow requests to be proxied; all requests need to be addressed directly to the hosting server, which locks the underlying data. CIFS is also a “chatty” protocol (e.g., CIFS generates substantial bi-directional status and update traffic). In a distributed environment, these characteristics can result in substantial load and latency issues.
Embodiments of the present invention combine cloud controllers with NAS capabilities and cloud-based storage to provide a high-capacity, high-reliability storage system that can be accessed from multiple front-ends via an application-layer network protocol (e.g., CIFS).
In some embodiments, a customized filesystem device driver in an operating system decouples filesystem functionality from an underlying block storage mechanism, thereby allowing filesystem-level information to be forwarded to another filesystem and/or data management mechanism. For instance, in some embodiments a customized filesystem device driver in an operating system may forward such filesystem-level request information to a range of network storage devices and/or distributed architectures that can provide enhanced data storage capabilities to client computing devices.
Operating systems often support several standard filesystems (e.g., the DOS filesystem, the New Technology File System (NTFS), and CDROM filesystems), but sometimes also include an open interface that facilitates accessing special devices and providing third-party interoperability (e.g., to support USB flash drives and filesystems associated with other operating systems, and to allow the development and support of future filesystems). In some embodiments, such interfaces can be used to create a filesystem device driver that emulates a local filesystem and storage device to the local operating system, but actually instead forwards filesystem-level request information to (and receives responses from) a non-block-level storage management system and/or filesystem.
In some embodiments, the disclosed forwarding techniques facilitate leveraging an existing implementation of a request server to reduce implementation overhead and complexity. For instance, as described above, implementing CIFS involves substantial effort and expense. Hence, some embodiments may provide CIFS capabilities by executing an operating system (e.g., Microsoft Windows Server 2008 Core, which includes embedded, high-performance CIFS server functionality with low memory usage) that can manage CIFS requests, but then use a customized filesystem device driver to extract and forward filesystem-level information that was received in CIFS requests. Note that this is distinct from techniques that direct file operation requests to a filesystem (such as NTFS) that then directly performs corresponding block-level operations upon a block storage device; forwarding filesystem-level information facilitates adding additional layers of capabilities and services prior to (or in place of) performing lower-level storage operations.
In some embodiments, an NAS device serves as a cloud controller for the cloud storage system. In this role, the NAS device presents a standard request interface (e.g., CIFS) to clients, and uses local storage capabilities to cache the working data set that is being accessed in its local computing environment. The high-capacity cloud storage system stores the full data set for an enterprise, and serves as a backing store for the NAS device. This architecture combines the performance and interface of a local NAS device with the capacity of a cloud storage system, while reducing management complexity.
In some embodiments, filesystem-level information can be forwarded by executing an operating system with a desired server capability (e.g., Microsoft Windows Server 2008 Core) in a virtual machine. Note that, as above, this “guest” operating system does not execute in a stand-alone configuration (e.g., with an underlying NTFS filesystem and disk drive), but instead can be configured to forward filesystem-level information (e.g., CIFS requests) to (and receive responses from) an underlying storage management system that provides data-consistent capabilities. For instance, a customized filesystem device driver in the guest operating system can forward request information to (and receive responses from) a storage management system in the host operating system. Note that such forwarding behavior in the guest operating system is distinct from typical storage operations for a guest operating system. Typically, upon receiving a request for a file operation, a guest operating system accesses an associated (virtual) filesystem and outputs a block-level storage request that is received and handled by the host operating system; hence, no filesystem-level information is sent to the host operating system. In contrast, in the described embodiments, the guest operating system forwards filesystem-level information, not block-level information, to the host operating system.
In some embodiments, the customized filesystem device driver extracts, tracks, and forwards client file interactions on a per-file and a per-directory basis. More specifically, semantic filesystem-level information included in the application-layer network protocol (e.g., CIFS) is forwarded by the filesystem device driver to a storage management system. This semantic information can include, but is not limited to: a file name; a file type; a requested file operation (e.g., a read, write, or update operation); a set of application information associated with the file; one or more users accessing the file; and security information for the file. Cloud controllers can use this information to determine whether a file and its associated information should be cached locally and/or forwarded to the cloud storage system (or other devices accessing the cloud storage system, as described below). For instance, the storage management system may know that certain files will be duplicated and/or shared shortly after being modified, and hence may ensure that such files are both cached locally and forwarded to the cloud storage system to facilitate the expected duplication operation.
In some embodiments, decoupling a filesystem from underlying block storage devices facilitates transparently changing (e.g., either increasing or decreasing) the amount of storage space accessible by clients. Operating systems typically assume that filesystem device drivers always manage fixed-size volumes; storage devices normally have a fixed size, so this usually is not an issue. However, one of the benefits of using cloud-based storage is the ability to easily increase data capacity on demand. For instance, for the above-described scenario where a cloud controller caches data for a cloud storage system, the amount of space available to clients can be increased by leasing additional space in the cloud (network) storage system and communicating the change to clients as needed (e.g., upon request) through the filesystem device driver. Hence, in such embodiments the customized filesystem device driver and the disclosed caching architecture substantially simplify adjusting data storage capabilities. In contrast, expanding traditional storage systems typically may involve shutting down a storage device, physically adding additional storage devices, and then reconfiguring the entire storage system to accommodate the added storage space.
In some embodiments, volume size can be changed transparently regardless of any pending client operations. In alternative embodiments, the presence of some active filesystem connections and/or operations may require some or all connected clients to be disconnected during volume size changes (e.g., preventing some clients from actively accessing files), which may be infeasible or inconvenient. Hence, in some embodiments, the filesystem device driver may be configured to claim an initial fixed size that substantially overstates the expected amount of storage, to prevent future resizing logistics. The allocated portion of the cloud storage system may initially provide only a small subset of this claimed storage size, but then subsequently be dynamically expanded as needed. In some embodiments, such size configurations may be accompanied by user quotas to prevent storage space wastage.
Note that a filesystem device driver provides a higher level of abstraction than techniques that attempt to analyze disk-block-level traffic (e.g., in the disk-level remote storage system illustrated in
In some embodiments, the described techniques provide filesystem-level proxy functionality. In many situations, proxies are implemented at the abstraction level of networking protocols, but this becomes more difficult as network protocols become more complicated (e.g., by adding application-specific information, as in CIFS). Hence, instead of re-implementing a complicated network protocol, some of the disclosed embodiments create a proxy at another, simpler layer by using a customized filesystem device driver that extracts and “tunnels” (e.g., forwards) filesystem-level information to another storage management system. Note that a filesystem-level proxy can also provide additional benefits. For instance, by emulating a local filesystem and storage device, the disclosed techniques can also overcome restrictions imposed by certain resource-intensive applications (e.g., certain databases, email server products, and/or data protection managers) to only use local storage devices.
Using Multiple Cloud Controllers to Access Cloud-Based Storage Via CIFS
The previous section described some possible architectures for cloud controllers. As described previously, two or more cloud controllers may work together to collectively manage and access a shared set of files that are stored in a cloud storage system.
Note that the described architecture can overcome limitations in an application-layer network protocol. As described above, CIFS does not allow requests to be proxied, and requires that a single hosting server manage the underlying filesystem. However, the disclosed filesystem forwarding mechanisms provide a level of abstraction where each guest operating system assumes it is the sole manager of the shared data. The underlying storage management systems in the host operating systems can pass application-level request information to one another to ensure that consistency is maintained. Thus, the disclosed techniques leverage existing CIFS implementations (via the guest operating system), but also extend CIFS capabilities to allow multiple CIFS servers to share access to underlying data. Note that the storage management systems may not need to understand all of the application-level information being received via CIFS requests; in some instances, the cloud controllers may simply forward such information to each other. However, for some data access scenarios (e.g., multiple clients accessing a shared file via different gateways), the storage management systems in each cloud controller may need to track and actively manage file operations to ensure that data consistency is maintained for the underlying data. Note also that receiving and filtering CIFS requests at the cloud controllers can reduce the amount of protocol chatter that travels over wide-area network links. For instance, a storage management system in a local cloud controller may be able to resolve some subset of CIFS requests, thereby eliminating the need to forward associated request information to the other components of the distributed storage system and reducing request latency.
In some embodiments, an additional network storage system may be used to provide another level of redundancy (and ensure high availability). For instance, in
Disaster Recovery and Business Continuity
Some embodiments of the disclosed system architecture support the rapid recovery and/or replacement of cloud controllers. For instance, a new (or recovering) cloud controller immediately begins reading and processing the available metadata snapshots (e.g., in the case of a recovering cloud controller, the set of incremental metadata snapshots written to the cloud storage system since the cloud controller went offline). Because metadata is relatively small in comparison to the actual file data, this process can be performed more quickly than transferring a full data set. A cloud controller with relatively up-to-date metadata can immediately begin processing client requests; while the cloud controller may not yet have the requested data (or any data at all) cached, once it has retrieved a sufficient set of metadata it can already identify and access the cloud files containing requested file data.
In some embodiments, one or more additional cloud controllers serve as backups for an existing set of cloud controllers. These additional backup cloud controllers can take over the load of damaged and/or failing cloud controllers to provide disaster recovery and business continuity.
Note that in
In some embodiments, a backup cloud controller may also be configured to predictively download some of the data created and/or cached by the other cloud controllers of the system, in an attempt to download data that is in the working set of other cloud controllers. While a backup cloud controller that is provisioned similarly to the other cloud controllers can only store a partial subset of the full caches of the combined set of other cloud controllers, caching the full set of metadata and at least some of the current data actively being operated on in the system can be of benefit if the backup cloud controller needs to take over the role of a failed cloud controller.
For instance, a backup cloud controller that is configured to serve as a “hot backup” may continuously load the (most-recently-created) data that was most recently uploaded to the cloud storage system. Due to temporal locality, recently created data is very likely to be accessed again, so in some scenarios caching such data in a backup cloud controller may be an optimal caching choice. For example, consider a scenario in which a set of globally distributed cloud controllers collectively manage data stored in a cloud storage system. Due to geographic constraints, only a subset of the cloud controllers may be active (e.g., actively servicing client requests) at a time, thereby allowing a backup cloud controller using the above-described technique to cache a larger portion of each active cloud controller's active data set. In the most extreme scenario, if only one cloud controller is active, the backup cloud controller may mirror a substantial percentage of the working set of that one active cloud controller. If an active cloud controller fails, the backup cloud controller's cache is already loaded with much of the recent data. Alternatively, if an inactive cloud controller were to fail, the backup cloud controller has more time and leeway (due to the failing cloud controller's current inactivity) to transparently load a data set that more closely matches the working set of the failed cloud controller.
Alternatively, if constantly downloading the most-recently-uploaded data to the backup cloud controller incurs too high a bandwidth cost, the backup cloud controller may instead be configured to periodically update the local cache (e.g., hourly). Other options include tracking the most-commonly-accessed set of files in each cloud controller (and sharing this tracking information with the backup cloud controller, which then caches that specific data) and/or allowing system users and/or administrators to select the set of data that is cached on the backup cloud controller (e.g., using locality policies, as discussed in a following section).
In some embodiments, a backup cloud controller may be located at a location that is distinct from any other cloud controller, to ensure that a local disaster or issue does not also affect the backup cloud controller. However, in alternative embodiments, and in embodiments that include multiple backup cloud controllers (which might also serve in a load-balancing capacity when needed), some or all of the one or more backup cloud controllers may be co-located with active (non-backup) cloud controllers. Note that in scenarios that include multiple backup cloud controllers, individual backup cloud controllers may be associated with (and cache data for) a distinct subset of the active (non-backup) cloud controllers.
In some embodiments, backup cloud controllers (and/or other cloud controllers or other system entities) may be configured to automatically detect the failure of a cloud controller and perform a set of network and/or client reconfiguration options that automatically reroute requests from orphaned clients to a backup cloud controller. In scenarios where multiple backup cloud controllers are available, backup cloud controllers may also be configured to periodically track the network latency and bandwidth between each backup cloud controller and non-backup cloud controller; such information can assist in determining which backup cloud controller can provide better service to the orphaned clients. In alternative embodiments, rerouting requests from orphaned clients to a backup cloud controller may involve manual intervention by a system administrator (e.g., explicitly changing a distributed filesystem mapping in a local network to route client requests to the backup cloud controller).
Note that transitioning from a failed cloud controller to a backup cloud controller may not be completely transparent from the perspective of orphaned clients of the failed cloud controller. For example, a client may have been transmitting data to be written to the failing cloud controllers, or the failing cloud controller may have been preparing (or even already been transmitting) data and/or metadata updates to the cloud storage system. Data in transit to or from a failed cloud controller should typically be considered lost; a primary consideration for the distributed storage system is that no data corruption occurs at the remaining cloud controllers or the cloud storage system.
In some embodiments, depending on the desired failure characteristics, the distributed storage system may have different modes of operation. For instance, in some modes of operation cloud controllers may perform strict ordering of updates; e.g., always upload an incremental data snapshot before its associated incremental metadata snapshot, to ensure that the global system metadata isn't updated until the uploading cloud controller has confirmed that both the data and metadata have been successfully received by the cloud storage system. For example, the cloud controller may calculate checksums (e.g., MD5 checksums) prior to uploading each snapshot, and confirm that these checksums match checksums generated by the cloud storage system upon receiving and successfully storing the snapshots. In the worst case, such strict ordering may result in the loss of some storage space in the cloud storage system (e.g., if the cloud controller fails after uploading snapshots but prior to sending notification of the metadata and/or data updates to other cloud controllers); this storage space can be recovered during cloud-controller recovery or via administrator actions.
In alternative embodiments, incremental metadata snapshots are uploaded first (to propagate the awareness of new data throughout the system as quickly as possible), with data snapshots following. In such embodiments, a cloud-controller failure that occurs between the upload of the metadata and data snapshots may result in metadata references to unavailable cloud files. In such situations, data corruption issues may need to be resolved by administrator actions or by inspecting log data to revert the metadata uploaded by the failing cloud controller.
Multiple Cloud Storage Providers
Cloud storage providers may be differentiated (and selected) based on a range of characteristics including, but not limited to:
In some embodiments, multiple cloud storage providers may also be used to provide high availability and disaster recovery. For example, data may be mirrored across multiple cloud storage providers to ensure that data is still available to clients even if there is an outage in one of the cloud storage providers. Alternatively, multiple different cloud storage providers that provide different tiers of performance (and have different cost structures) may also be combined to balance performance and overall system cost.
In some embodiments of a distributed filesystem with data mirrored across multiple cloud storage systems, a cloud controller may be configured to immediately write a cloud file to a first cloud storage provider (thereby allowing the data to be propagated to other cloud controllers), but then delay the transfer of the cloud file to the mirror to a time when network bandwidth is cheaper. In such embodiments, the cloud controller may be specially configured to ensure that the cached local copy of the data in the cloud file is not flushed until after it has been mirrored to the second cloud storage provider.
In some embodiments, an FSID field can be used to support multiple cloud storage providers. For instance, part of the FSID field may form a bit vector that indicates one or more cloud storage providers that are currently storing an associated cloud file. In another example, a cloud controller may use a value stored in a CVA's FSID field to perform a lookup in a table of cloud service provider credentials. This table may include a list of cloud storage providers that are currently storing the cloud file, as well as “cloud account” information (e.g., information identifying a specific user account at a cloud storage provider and credentials that are needed to access that user account). Note that in addition to accessing different cloud storage providers, a cloud controller may also be configured to access different cloud accounts at the same cloud storage provider (e.g., different user accounts with different configurations and/or levels of service at the same cloud storage provider). FSIDs may also be used to support other filesystem-level features, such as storage quotas.
In some embodiments, data in the distributed filesystem may be split across multiple different cloud storage providers based on factors such as access frequency, age, and cost. For instance, new data may initially be written to a higher-cost cloud storage provider that instantly replicates the stored data across multiple POPs; this wide initial distribution allows other cloud controllers requesting the new data (and metadata) to download it quickly. At some subsequent time, data that is no longer frequently accessed may be migrated into a cheaper lower-tier cloud storage provider (e.g., a cloud storage provider with higher latency and lower cost) and deleted from the first cloud storage provider. Such moves may be performed asynchronously and as background operations to ensure that users accessing cloud controllers and data are not adversely affected. For example, data may be migrated to the lower-tier cloud storage provider at a time of day when the load of the distributed filesystem and the cost of network bandwidth are both lower.
In some embodiments, multiple factors are considered prior to migrating data between cloud storage providers. For instance, in some scenarios deciding whether to migrate a given cloud file may involve considering: the cost of storage at both the source and target cloud storage providers; a variable network bandwidth cost and/or the network bandwidth cost for the transfer; the access frequency and/or history for the contents of the cloud file; the potential performance impact of moving the cloud file to a lower tier; and the load of one or more cloud controllers. In some scenarios, cloud controllers actively monitor the cloud files and/or data files that they “own” (e.g., created) to determine how frequently they are accessed, and then use this information to determine candidates for migration. For example, a cloud controller may track the most recent access (e.g., the last read time) for individual blocks in its local persistent read cache (and/or in a persistent read cache that is distributed across multiple cloud controllers). After the last block for a cloud file is evicted from the read cache (e.g., due to not being used recently), the cloud controller may initiate a counter; if no blocks from the cloud file are used before the counter reaches zero, the cloud file becomes a candidate to be moved to a lower tier. Alternatively, the cloud storage system may be configured to track how often each given cloud file is accessed; the cloud controller that created a drive file may also check this access log to determine data that is no longer frequently used. Note that the above scenarios keep a cloud file in the higher tier cloud storage system if any of its blocks are still being actively used. In other scenarios, such decisions may be more complex (e.g., migration choices may also be affected by user-defined locality policies and/or cost-performance trade-offs).
In some embodiments, migrating a cloud file to a different cloud storage provider and deleting the copy from the previous cloud storage provider involves some additional logistical operations and/or policies to ensure that cloud controllers can still access the cloud file as needed. For instance, in one system metadata may not be updated to reflect migrations, and cloud controllers are configured to incrementally check cloud storage providers in priority order (using the same CVA as an identifier) until the target cloud file is found. In other scenarios, the metadata for files may be updated to indicate that the cloud file has moved. Such indications may range from changing a bit field that is associated with a CVA (but not included in the actual CVA itself) to actually creating a new CVA that correctly references the cloud file on the new cloud storage provider. In the latter scenario, updating the metadata may be a multi-step operation. For example, a cloud controller may: (1) determine that a given cloud file should be migrated; (2) determine a new CVA for the cloud file at the new cloud storage provider; (3) upload the cloud file to the new cloud storage provider using the new CVA as the identifier; (4) upon receiving confirmation of receipt from the new cloud storage provider, update the metadata for all of the file blocks in the migrated cloud file to point to the new CVA (ideally without otherwise modifying the access history or other characteristics for the file blocks); (5) distribute an incremental metadata snapshot to the other cloud controllers; and (6) after enough time has passed to ensure that the other cloud controllers have updated their metadata, delete the cloud file from the old cloud storage provider.
Archival Cloud Storage Providers
In some embodiments, a distributed filesystem can also leverage an “archival” cloud storage system. Archival cloud storage systems typically provide storage for a very low cost, but are equipped with a limited set of storage capabilities that are geared toward storing data that is infrequently accessed. For instance, archival cloud storage systems may be architected such that new data can be received and written immediately, but read requests for stored data sometimes involve potentially substantial delays (e.g., sometimes on the order of multiple hours, or longer). Such relaxed service constraints allow the cloud storage provider to optimize its data center to substantially reduce costs (e.g., by receiving data in a rotating set of active servers and powering down all other servers and storage devices completely, thereby dramatically reducing power and cooling costs); storage costs for archival cloud storage systems may be one or more orders of magnitude cheaper than non-archival storage systems. While such access constraints may not be appropriate for data that is being actively accessed by users, such storage capabilities can provide low-cost storage options for certain sets of archival data in a distributed filesystem.
In some embodiments, data writes can be mirrored to an archival storage system for disaster recovery. In such embodiments, writes can be mirrored as described previously (for multiple tiers), but the archival storage system would typically only be read if a primary (non-archival) cloud storage system were to be irretrievably lost (e.g., if the primary cloud storage provider were to go out of business or suffer from a substantial national disaster). In such arrangements, data might be sequentially read out of the archival storage system (e.g., in order of priority) and then immediately written into another non-archival cloud storage system.
In some embodiments, the distributed filesystem collects and writes a set of archival data that is being retired from active use to an archival cloud storage system. This archived data will typically no longer be directly accessible by cloud controllers, but instead would need to be recovered by an administrator of the distributed filesystem.
As described earlier, cloud controllers notify each other whenever new data is created in the distributed filesystem. More specifically, cloud controllers periodically generate snapshots that reference every block, including both data and metadata, that is valid and stored in the cloud storage system at the time that the snapshot is taken. These snapshots are stored using data structures that allow each cloud controller to determine the set of new data and metadata that have been created in a recent time interval. Each cloud controller compares its two most recent snapshots (e.g., snapshot A and a later snapshot B) to determine the incremental difference between the two snapshots. For instance, data blocks that were created after snapshot A but before snapshot B (and are still referenced by snapshot B) are considered part of this incremental difference. Furthermore, any data blocks that were freed (e.g., deleted) between the snapshots are also considered part of this incremental difference. This incremental difference can be logically separated into incremental differences in metadata (e.g., new metadata created to reference newly created file data blocks) and incremental differences in data (e.g., the actual newly created file data blocks). In this disclosure, incremental differences in metadata are referred to as incremental metadata snapshots, and incremental differences in data are referred to as incremental data snapshots.
Each cloud controller stores both sets of incremental differences (e.g., the incremental metadata snapshot and the incremental data snapshot) in a cloud storage system. After uploading an incremental metadata snapshot to the cloud storage system, each cloud controller notifies the other cloud controllers. Each cloud controller then downloads these incremental metadata snapshots from the cloud storage system, updates its local metadata based on the downloaded metadata, and then deletes the downloaded incremental metadata snapshot. Other cloud controllers can then use the new metadata to access the new data stored in the incremental data snapshot (in the cloud storage system). In some scenarios, a cloud controller may also receive metadata (and data) directly from other cloud controllers. As mentioned previously, owning cloud controllers maintain data consistency by arbitrating multiple attempts to simultaneously write a given file. Cloud controllers can request updates directly from other cloud controllers if the data in the cloud storage system (or in a requesting cloud controller) is not up-to-date. In this scenario, the cloud controller storing the most recent copy of a file will determine the incremental difference between what the requesting controller has and the up-to-date version, and send the incremental difference directly to the requesting cloud controller.
Stored snapshots can be used for a range of features, including, but not limited to: “rolling-back” the filesystem to an earlier date; recovering accidentally deleted files; and archiving data from the distributed filesystem. The data structures used to store snapshots facilitate storing periodic snapshots compactly. For instance, a cloud controller may store hourly snapshots that capture the full state of the system at each hour interval; instead of storing each snapshot as a separate entity, however, these data structures may store incremental differences in a manner that is more compact but still allows each hourly snapshot to be accessed as needed. For example, in this scenario the incremental difference would reference the full set of changed data and metadata for all cloud controllers over the specified time interval (as opposed to the incremental metadata and incremental data snapshots, which only specify the set of changed data and metadata for one cloud controller). When using such an organization, accessing a desired snapshot for a previous time may involve creating a copy of an even earlier snapshot, and then applying one or more incremental differences for snapshots subsequent to the earlier snapshot to generate the desired snapshot.
A cloud controller may also periodically create “full” snapshots (e.g., complete, distinct copies of the metadata for the system at a given time that can be accessed without having to apply incremental differences). In summary, in this disclosure:
During an archival operation, a cloud controller analyzes file metadata changes for a time interval between two snapshots (e.g., snapshots 1600 and 1602) to determine files that existed and were deleted in the time interval 1604 but are still actively stored in cloud files in a cloud storage system. For example, cloud controller 1606 may be able to determine from snapshots 1600 and 1602 (and/or the intervening and surrounding snapshots) that a given file that existed (and/or was created) in snapshot 1600 and deleted in snapshot 1602 is still stored in a cloud file in a non-archival cloud storage system. If this deletion occurred sufficiently far in the past (e.g., more than 30 days ago), the cloud controller determines that the active lifespan of this file is over, and that any data for the file that is stored in cloud files no longer need to be kept in active storage.
The archival operation illustrated in
Note that in some embodiments a cloud file typically is only archived if all of the blocks of the cloud file have been deleted; if any of the blocks have not been deleted, the cloud file remains in non-archival cloud storage. In alternative embodiments, individual blocks in a cloud file may be archived, and cloud files can be shrunk to conserve cloud storage space. For instance, if a subset of the blocks in a cloud file are moved to an archival cloud storage system, the portion of the cloud file remaining in the primary cloud storage system may need to be reconstructed to preserve the remaining blocks. For example, filling the archived blocks of a cloud file with zeros and then compressing the cloud file effectively reduces the amount of storage space needed for the empty blocks without changing block indexing and accesses for the cloud file. In a specific (small) example, for an exemplary cloud file containing blocks “ABC”, a cloud controller archiving block B could generate a modified cloud file “A0C” for the primary cloud storage system and “0B0” for the archival cloud storage system (where ‘0’ represents a block filled with zeros). If block ‘C’ were later archived, the blocks could be updated to become “A00” and “0BC”, respectively.
In some embodiments, a storage administrator manages and/or initiates archival operations. Alternatively, such operations may be scheduled at regular time intervals. Note that some archival techniques (e.g., the snapshot-comparison technique) can be initiated on a single cloud controller that has access to the collected snapshot data. Furthermore, archival operations may be initiated as low-priority background jobs (in comparison with client accesses) and/or on a cloud controller during a timeframe that the cloud controller typically has low load. Alternatively, such operations may be initiated on a backup cloud controller if available, especially if backup cloud controllers are configured to not receive client requests. Archival transfers can also be timed to occur when network bandwidth is cheaper.
In some embodiments, a distributed filesystem may include additional tables (or other structures) that facilitate accessing archived data; note that reading data from an archival cloud storage system may involve special access techniques and/or administrator assistance. After ensuring that a cloud file has successfully been copied to an archival storage system, a cloud controller can delete that cloud file from the originating (non-archival) cloud storage system. At this point, the cloud controller may generate an incremental metadata snapshot to indicate that the metadata stored in previous snapshots can no longer be used to access the deleted data that was previously stored in the (now deleted) cloud files. However, a different set of tracking structures will need to be updated to reflect that the newly archived data in the archival cloud storage system can still be accessed if needed. For instance, archived data may be tracked in a separate set of metadata, and indexed by date such that an administrator can selectively recover and access archived data within a desired timeframe (e.g., data that is approximately one year old, data that existed during a certain timeframe, or data that was deleted between two specified dates) as needed.
Note that recovering archived data may also involve recovering archived snapshots. Because only a subset of snapshots are kept over time, a cloud controller performing an archival operation may also write an old snapshot of the distributed filesystem to the archival cloud storage provider; these old snapshots can be re-populated into the cloud controllers at a later point if needed to access cloud file data that is restored from archives. If the archived snapshot is written together with the deleted data that it references, a recovery operation may recover both simultaneously. For example, the index of archived data may indicate the deletion timeframe, original lifespan, data size, format, and identifier for each archival record, thereby facilitating archival recovery.
Note also that, in some embodiments, archival operations can also be opportunities to defragment and/or reorganize data. As described previously, cloud file size is often chosen to balance download throughput and latency; for data being archived, these are no longer issues, and hence cloud file constraints may be eliminated. For instance, file data distributed across multiple cloud files may be re-written into contiguous files (that can be recovered more easily in a single access) regardless of size during archiving. In such implementations, an index of archived files may simply store filenames, relevant metadata (e.g., creation and deletion dates, originating cloud controller, size, etc.), and a reference identifier to access the data from the archival cloud storage system.
In some embodiments, cloud controllers can also serve as intermediaries to an archival cloud storage provider for backup purposes. For instance, cloud controllers may implement a virtual tape library (VTL) interface that allows clients to perform backups as if the cloud controller were a tape backup device, with the data instead being written to the archival cloud storage provider. These techniques allow existing client backup infrastructure to be used transparently while adding additional capabilities (e.g., offsite storage in the archival cloud storage provider, redundant simultaneous writes to multiple archival cloud storage providers, etc.). Note that the high read latency of archival cloud storage providers is comparable to that of typical tape backup systems, and hence not a drawback in such usage scenarios.
Accessing Cloud Controllers as Peer Caches
Accessing cloud files involves some overhead—a cloud controller seeking to download a target file that is not locally cached needs to: (1) contact a cloud storage system hosting one or more cloud files that contain the target file's data blocks; (2) download some or all of the target cloud files; and (3) if the cloud files are encrypted, decrypt the portions of the target cloud files that precede and include the target data blocks. Delaying factors may include the distance and network latency between the requesting cloud controller and the cloud storage system, the load and delay in the cloud storage system, the size of cloud files, and the location of the target block in the cloud file. Furthermore, accessing the cloud storage system may be associated with an access cost and/or network bandwidth cost. Hence, while a requesting cloud controller will typically eventually get the file from the cloud storage system, accessing the data blocks for the target file may involve considerable delay. If another (peer) cloud controller in the distributed filesystem is already caching desired data blocks, requesting and transferring cached data blocks from a peer cloud controller potentially facilitates accessing the desired file data more quickly (and cheaply).
In some embodiments, cloud controllers can query one or more of the other cloud controllers of the distributed filesystem to locate and access a needed data block that is already being cached by a peer cloud controller. Each individual cloud controller strives to determine the working set of its clients and locally cache a set of data that reduces the average client access latency. However, cloud controllers have limited storage capacity, and hence can only cache a limited amount of data. Inter-cloud-controller cache-sharing techniques can effectively increase the size of a cached working set, thereby potentially reducing average file access latencies as well as the number of requests that are sent to the cloud storage system. Furthermore, such techniques transfer only the needed data blocks; in contrast, accessing cloud files from the cloud storage system to access the same set of data blocks potentially transfers a larger amount of data (due to the size and organization of cloud files).
However, attempts to leverage the caches of other cloud controllers can also introduce additional overhead and trade-offs. As described previously, a cloud controller writes new data received from a client to a cloud storage system. The cloud controller will typically continue to cache this data locally while it is actively used by its clients; data that is no longer actively used is eventually pushed out of the local cache by other frequently and/or recently accessed data, but can be re-loaded if needed from the cloud storage provider. Thus, while it is possible that other cloud controllers may cache data that a requesting cloud controller needs, determining which cloud controllers are likely to be currently caching a needed file block is difficult. One option is to query all of the cloud controllers in the distributed filesystem. Unfortunately, while broadcasting requests to all cloud controllers increases the probability of finding a cached block, processing and/or responding to such requests may substantially increase each cloud controller's load. As a result, broadcast techniques become increasingly inefficient as the number of cloud controllers in the distributed filesystem grows. Furthermore, cloud controllers that are distributed across a large internal network or wide-area network are typically not be in the same broadcast domain; hence, broadcasts will require a large number of point-to-point messages. Such messages can increase network load, potentially to the point of slowing down accesses to the cloud storage system. Directory-based techniques that track the location and status of cached data blocks provide an alternative to broadcast-based techniques, but also require storage space and substantial message update traffic to keep the cache directories up-to-date.
In some embodiments, cloud controllers are configured to query a limited subset of peer cloud controllers in the distributed system for cached data blocks. For instance, a cloud controller may be configured to only send such “peer cache requests” to: one or more co-located cloud controllers; the cloud controller that owns (e.g., created) the cloud file containing a needed data block; a backup cloud controller; one or more cloud controllers that are determined to be in close network proximity to the requesting cloud controller; and/or one or more cloud controllers that are specified using a locality policy.
In some embodiments, a cloud controller is configured to query the caches of one or more peer cloud controllers for a data block in parallel with sending a request to the cloud storage system for the cloud file containing the data block. If a target peer cloud controller is indeed caching the data block, it can send the cached data block to the requesting cloud controller, which can then cancel and/or interrupt the transfer of the cloud file from the cloud storage system (e.g., depending on whether the cloud storage system is already transferring the cloud file or is still processing the request). If the peer cloud controller indicates that it is not caching the data block (or does not respond at all), the requesting cloud controller still receives the cloud file from the cloud storage system, and caches the requested (and now received) data block. Performing these two operations simultaneously: (1) ensures that peer cache requests do not increase the worst-case latency for a data block access; (2) adds only a small number of additional network messages and cloud controller look-ups (that are proportional to the number of peer cloud controllers that are queried); and (3) in the best case facilitates transferring needed data blocks to a requesting cloud controller more quickly than would be possible from the cloud storage system, thereby allowing the requesting cloud controller to quickly present the requested data blocks to a requesting client and cancel the parallel request to the cloud storage system before part (or sometimes even any) of the cloud file has been transferred.
In a second scenario for
In a third scenario for
In some embodiments, cloud controllers may determine a target cloud controller for a peer cache request based on a range of criteria. The preceding exemplary scenarios determined targets for peer cache requests based on ownership, proximity, and/or a specific role (e.g., the role of the backup cloud controller), but the disclosed techniques are in no way limited by these examples, and peer cache requests may be routed or partitioned based on other factors. For instance, cloud controllers may periodically calculate and share both the network latency and bandwidth between cloud controllers as well as cloud controller load (e.g., how many clients are connected to a given cloud controller, and/or how many current requests are currently queued for disk I/O), and use such information when determining the appropriateness and targets for peer cache requests. For example, a requesting cloud controller may decide not to send a peer cache request if a target cloud controller's load is above a specified threshold, or only send high-priority requests in such instances.
In some embodiments, cloud controllers may be configured to operate in different modes during different times of the day. For instance, a cloud controller may be configured to cache data that is being actively accessed by a set of local clients during a particular time period (e.g., daytime working hours). However, when these clients are idle (e.g., during the night), this cloud controller may be configured to replace some or all of this cached data with data that is being accessed in a different time zone. More specifically, instead of being idle, this cloud controller can be configured to load a different set of data and serve this data to other active cloud controllers via peer cache requests. Such configurations may provide substantial improvements to the efficiency and average response time of the other cloud controllers. For example, depending on the situation, retrieving a small set of data blocks from a distant cloud controller may still be faster than downloading an entire cloud file from a closer cloud storage system. Such configurations may be specified and customized for individual cloud controllers using locality policies (as described in following sections).
Note that accessing other cloud controllers as peer caches does not involve having to maintain or check coherency for data blocks. The characteristics of the transactional filesystem ensure that each data block is unique and read-only (e.g., new data is written to new blocks and new cloud files). Thus, requesting cloud controllers only need to be concerned with retrieving needed data blocks as quickly as possible, and do not need to perform checks to determine whether another cloud controller and/or client have modified the contents of the requested data block. Note also that peer cache requests are typically constant-time requests that are sent directly to target cloud controllers, and not multi-hop queries.
Deduplication
A given data block may be stored multiple times in a filesystem; for instance, users may back up or otherwise keep multiple copies of the same file, or may send copies of a file to other users in their organization. Over time, file duplication can lead to an increasing number of duplicate data blocks and a substantial amount of wasted storage space. Data deduplication techniques involve calculating and tracking hash values for previously written data blocks, and comparing the hash values for newly written data blocks against these previous hash values to determine whether new data blocks have already been previously stored in a filesystem (and, if so, referencing the existing data block instead of writing a new, additional data block).
The hash keys 2908 generated by hash function 2906 are used as indices into a hash table of block entries 2910 (which is sometimes also referred to as a deduplication, or “dedup,” table). Each block entry can include metadata identifying the storage location for the data block and a reference count for the data block (e.g., the number of times that the block is currently referenced in the filesystem metadata). During operation, a request to store a new block of data prompts the filesystem to calculate a hash key 2908 for the data block and then use this hash key 2908 as an index into hash table 2910 to determine whether the data block has already been written previously (e.g., determine whether a block entry already exists in hash table 2910 for that specific hash key). If no block entry exists for the hash key, the filesystem: (1) writes the data block to storage; (2) updates the filesystem metadata for the data block to point to the storage location; (3) creates a new block entry for the data block (that points to the storage location and stores an initial reference count of one for the newly written data block); and (4) updates hash table 2910 so that the index for the hash key points to the new block entry. Alternatively, if a block entry already exists for the hash key, the filesystem: (1) accesses the block entry to retrieve the storage location for the previously written data block; (2) updates the filesystem metadata for the current data block to point to the storage location (e.g., updating the metadata with additional references to an existing storage location instead of writing a duplicate data block to another storage location); and (3) increments the reference count in the block entry to reflect the additional reference to the previously written data block. For the example illustrated in
In general, deduplication techniques increase the effective write performance of a storage device by reducing the total number of writes that need to be made to the storage device. More specifically, the ability to detect and prevent duplicate data from being written effectively increases write throughput while also reducing the amount of storage space that is actually needed. However, deduplication techniques do typically involve some additional overhead. For instance, if the number of data blocks is large, memory constraints may lead to deduplication structures being stored lower in the memory hierarchy (e.g., on a hard drive). In such situations, the architecture illustrated in
In some embodiments, deduplication techniques can be extended to encode time information in deduplication structures. For instance, the data structures used to store block entries can be re-architected so that the information for data blocks is temporally grouped in a manner that reduces the number of disk reads needed during subsequent deduplication operations.
While creating and populating TDS 2922 involves some additional overhead, subsequent duplicate writes benefit from lower overhead. For example, consider a subsequent set of duplicate writes for the same three data blocks 2900-2904. Upon determining that an entry exists in hash table 2920 for the hash key for data block 2900, the filesystem receives a reference to TDS 2922. The filesystem can then: (1) use lookup structures 2926 to find and access block entry 2912 to retrieve the storage location for the previously written data block; (2) update the filesystem metadata for the current data block to point to the storage location; and (3) increment the reference count in block entry 2912. However, for the second and third blocks, the hash key lookups in hash table 2920 both return references to the same TDS, TDS 2922, which was already read from disk and loaded into memory during the preceding lookup for the first data block. Thus, while the duplicate write of the first data block (2900) still involves two disk reads (to access hash table 2920 and TDS 2922), the (duplicate) writes for the second and third data blocks (2902-2904) only involve at most one further disk read each (to access hash table 2920), thereby reducing the total number of disk reads needed and substantially improving deduplication performance. Consider the duplicate write of a file with 100 data blocks in a scenario in which each TDS is the size of a disk block and holds 100 block entries. In this scenario, a deduplication operation in the deduplication architecture of
In some embodiments, temporal deduplication structures can be optimized to reduce the number of disk reads even further. For instance, the block entries of TDS 2922 can be enhanced so that each block entry includes the hash value for its associated data block and a “next pointer” that points to the block entry following it in TDS 2922 (e.g., block entry pointers 2928 in
In some embodiments, multiple sets of block entry pointers may be used to form a doubly-linked list of block entries within and between TDSs. Maintaining doubly-linked lists sometimes involves higher overhead, but can also provide performance benefits, for instance in the face of unusual write patterns (e.g., when a client for some reason performs a duplicate write for a set of blocks in the reverse order that the blocks were initially written and ordered in a TDS). Alternative embodiments may also involve loading the contents of TDSs into a memory cache of block entries, thereby speeding up accesses and efficiently supporting atypical access patterns.
Note that the concept of temporal deduplication applies not only to individual files, but also to groups of files. Many automated computing tasks are defined processes (e.g., automated scripts) that execute in a very regular, predictable manner. For instance, applications are typically written to execute in the same way across multiple iterations, e.g., compilation jobs typically create files and data in the same order, and directory copy or backup commands will typically traverse a directory hierarchy that is being operated upon in the same order for every invocation. The above-described temporal deduplication techniques leverage the notion that things written with temporal locality are likely to be re-written again in the same temporal order; the disclosed temporal deduplication structures can be used to efficiently detect and handle long, contiguous sets of duplicate data blocks that are being written by an automated process.
In some embodiments, temporal deduplication structures can also be optimized to accommodate the subsequent addition of data blocks to a previously created file. For instance, a filesystem may be configured to reserve some block entries in each TDS that can then be used at a later time (e.g., to add a few additional data blocks to a file, and have the block entries associated with those new data blocks show up in the same TDS as the rest of the block entries for the file), in contrast with a purely temporal approach that would add the block entries for subsequent data blocks being appended to a previously written file to another, separate TDS that is currently being populated (and, hence, potentially splitting the block entries for the appended file across multiple non-contiguous TDSs, and requiring more TDSs to be read in subsequent deduplication operations that involve the entire appended file). Note, however, that such additional capabilities may interfere with some of the advantages of the above-described temporal deduplication structures; for instance, all of the block entries for a given file may still be in the same TDS (e.g., after an additional block entry has been added to the TDS), but an efficient stream of contiguous temporal block entry lookups may have been disrupted as a result. Hence, the decision of whether to reserve some block entries in a TDS for subsequent additions may be limited to certain situations (e.g., files and/or filesystem regions that are more likely to be accessed in isolation) and/or be selectively specified using a locality policy.
Global Deduplication for a Distributed Filesystem
Deduplication techniques can be applied across a range of scopes. For instance, the above-described deduplication techniques can be performed on (individual) single- or multi-user workstations and/or servers to conserve storage space and increase user-perceived write performance. However, deduplication techniques can be even more effective and beneficial when leveraged across a distributed filesystem that encompasses a large number of users and data files. More specifically, the types of regular operations that typically cause file duplication (e.g., duplicate compilation jobs, recurring backup jobs, the sharing of files between users, etc.) quite often increase as the size of the filesystem and user base grow, leading to substantial storage space and bandwidth savings for a distributed filesystem.
In some embodiments, cloud controllers use data deduplication techniques to reduce the amount of duplicate data that is stored in a distributed filesystem. For instance, supporting deduplication techniques across the distributed filesystem may involve leveraging incremental metadata snapshots to update deduplication information in the cloud controllers that manage the data in the distributed filesystem. Each cloud controller then monitors the writes made by its clients and performs deduplication operations to avoid storing duplicate copies of data blocks in the cloud storage system. Note that the disclosed deduplication techniques are distributed across the cloud controllers; attempts to perform centralized deduplication operations for a large distributed filesystem on a single server typically limit the scalability and performance of deduplication operations.
Other cloud controllers can make use of the metadata updates propagated via incremental metadata snapshots to update their own local deduplication tables. For instance, metadata updates may specify a set of data blocks that are being referenced, created, and/or deleted, and include the hash values and locations (e.g., CVA address and offset) for each new or newly referenced data block. Hence, in the context of
The regular exchange of incremental metadata snapshots between the cloud controllers of the distributed filesystem ensures that deduplication information is shared and that each cloud controller's local deduplication table remains up to date, thereby allowing duplicate writes to be detected and avoided. Note that temporal deduplication information can also be shared between the cloud controllers. Consider, for example, a cloud controller that receives from a client a write request for a file with ten data blocks that have not previously been written to the distributed filesystem. The cloud controller: (1) takes hash values for these ten data blocks, and determines that the blocks are new; (2) allocates new CVAs and offsets for the new data blocks; (3) updates a TDS in its local deduplication table, creating temporally related block entries for the new data blocks; (4) updates the local metadata hierarchy appropriately; and (5) sends out incremental data and metadata snapshots. Other cloud controllers receiving the incremental metadata snapshot detect the creation of the ten new blocks, and write the information for these ten blocks into their own deduplication tables in a temporal manner (e.g., updating TDSs in their local deduplication tables in a manner that creates temporally related block entries for the new data blocks). Hence, if a client of one of these other cloud controllers attempts to write the same file, the deduplication operation in that cloud controller benefits from the same temporal lookup aspects described previously (e.g., a reduced number of disk reads in the deduplication operation due to the temporal similarities).
Note, however, that some challenges can arise for temporal deduplication techniques in a distributed filesystem environment. For instance, consider a cloud controller that is receiving incremental metadata snapshots (from another cloud controller) while simultaneously locally writing a set of new data blocks in response to a client request. Unfortunately, such scenarios can lead to situations in which sets of block entries from the remote deduplication information and from local writes (e.g., two sets of unrelated write operations) are written into the same TDS in an interleaved manner. As described above, temporal deduplication techniques benefit from grouping block entries for temporally related data blocks together in TDSs; however, while these two sets of writes occur in the same approximate temporal timeframe, they originate on different cloud controllers, and are most likely not related. In general, interleaving two unrelated sets of block entries in a TDS typically reduces the temporal nature and efficiency of the deduplication table.
Another problem may arise when block references are not sent between cloud controllers in the order in which they were written. For instance, incremental metadata snapshots may send block references between cloud controllers in a “filesystem:file:block” format, instead of the order in which the blocks were actually written (and the order in which their respective block entries were written to a TDS in the originating cloud controller). Such reordering of write information can affect the ordering of a temporal deduplication table; for example, a second cloud controller may end up with a significantly different block entry order in its local deduplication table. Consider a client subsequently writing the same file to this second cloud controller—the deduplication operation will still determine that all of the data blocks for the file have been previously written in the distributed filesystem, but the deduplication operation may be highly inefficient. Because of the difference in the write order of the block entries, the cloud controller may not be able to take benefit from temporal locality in the deduplication table, and instead may have to perform individual hash lookups for each data block in the file. For example, in the worst case, the original write order for the initial file may have been “data block 1, 2, 3, 4, . . . , N−2, N−1, N,” but the temporal deduplication table may have been populated in reverse order (e.g., “data block N, N−1, N−2, . . . , 2, 1”) or some other non-temporal order, in which case a subsequent write of the same file received by the second cloud controller might result in a backwards walk through and/or random accesses into the temporal deduplication table, which are typically less efficient. Note, however, that the worst-case behavior for temporal deduplication is still no worse than non-temporal deduplication techniques (e.g., the deduplication techniques described in
In some embodiments, cloud controllers reorder write information to improve the temporal locality of block entries for writes performed both locally and on remote cloud controllers. For instance, write information can be sorted based on one or more secondary qualifiers (e.g., the data's source cloud controller, the data's source filesystem, a snapshot identifier, a file ID, and/or a block offset) to separate write information received directly from local clients and from remote clients (via other cloud controllers' incremental metadata updates) back into individual segments that are “temporally accurate” and facilitate efficient temporal deduplication. For example, one specific scenario may involve sorting write operations by CVA and offset; cloud controllers may use the CCID in CVAs to differentiate which cloud controller is the source for each given write, and may then use other portions of the CVA (e.g., the FSID, SSID, FileID and/or offset) to determine the exact order in which data blocks were written on each cloud controller and/or block entries should be written into a TDS. Each cloud controller can then use such determinations to ensure that any desired aspects of the original write order (e.g., in the originating cloud controller) are maintained for corresponding block entries in each local deduplication table, thereby avoiding interference between remote and local writes and ensuring improved temporal deduplication performance for local writes.
As mentioned above, a range of sorting (or any other form of reordering) techniques may be used to ensure high temporal locality on both local and remote filesystems and to facilitate efficient global deduplication efforts across all of the cloud controllers of the distributed filesystem. In some embodiments, one or more parameters for a secondary sort operation may be specified using a locality policy. For instance, a locality policy may be used to tune temporal deduplication behavior on a given cloud controller to match the specific needs of the cloud controller's expected clients and client load. For example, a cloud controller expected to receive a large amount of new client data may be configured to use a specific sort order. Consider as a specific example a cloud controller that is configured to gather data from multiple other cloud controllers and then perform backups of the full data set. In such a scenario, if the host cloud controller is configured to traverse and write this backup data on a per-cloud-controller basis, a sensible approach might be to ensure that the block entries in the temporal deduplication table are also grouped together similarly; a locality policy can be used to ensure that this is indeed the case.
In some embodiments, a locality policy may also be used to specify that the block entries for one or more files should be written in a different order than the data blocks for the files. Files are quite frequently accessed (e.g., copied, or re-written) in the same manner that they were initially written, but in some instances it may be known that a certain set (or type) of files will have a specific unusual future re-write pattern; in such scenarios, ordering the block entries for these files to anticipate these future write patterns may improve the performance of subsequent deduplication operations for those files.
In some embodiments, a cloud controller may be configured to maintain multiple simultaneous sets of TDSs that are all referenced by the same top-level hash table. For instance, a cloud controller may maintain one set of TDSs that store block entries for locally written data, while a second set of TDSs store block entries for remote data written by other cloud controllers. The appropriate TDS is determined based on the initial hash value for a given deduplication operation. Separating the stream of local and remote writes completely ensures that block entries for local and remote writes are never interleaved. Note that the block entries for remote writes are automatically grouped at the snapshot level already (e.g., if incremental metadata snapshots from different cloud controllers are processed serially), and thus are less likely to become interleaved.
The above sections describe how temporal deduplication techniques can be managed and synchronized across cloud controllers in a manner that facilitates maintaining temporal accuracy and locality, thereby providing a range of benefits for a distributed filesystem. These benefits can be broken down into benefits of deduplication “at rest” (e.g., specific storage space savings gained from reducing duplicate stores) as well as benefits of deduplication “in transit” (e.g., specific network bandwidth savings gained from avoiding network transfers of duplicate data). However, as also mentioned above, deduplication techniques also involve a certain level of overhead (e.g., extra reads and writes for the deduplication tables as well as additional complexity and compute overhead). The actual value of deduplication efforts in a distributed filesystem depends on the amount of duplicate data being written (e.g., the “dedup ratio,” or ratio between the average number of references per stored data block). For instance, a 10:1 at-rest dedup ratio indicates that there are on average ten references for each actual stored data block, e.g., deduplication efforts have on average saved nine additional copies of each data block from being written. A 10:1 in-transit dedup ratio indicates that on average ten metadata references are sent for every copy of a data block that is actually transferred between a cloud controller and a cloud storage system; because metadata is much smaller than actual data, deduplication efforts can save substantial network bandwidth.
Performing temporal deduplication for a distributed filesystem can provide substantial at-rest and in-transit benefits. While certain minimum dedup ratios are needed to break even (e.g., justify the additional deduplication overheads), a system that profits from both at-rest and in-transit benefits begins to reap those benefits at lower dedup ratios than systems that only provide one or the other of the two benefits. Deduplication benefits also tend to scale with the number of users and amount of data in the system; distributed filesystems tend to be large and encompass a larger number of users, and hence tend to have higher natural levels of duplication that in turn result in higher dedup ratios. Hence, tuning deduplication efforts to optimize a distributed filesystem can result in definite performance improvements and reduced system costs. For example, effectively reducing network load can have direct economic benefits, because upgrading network links often involves considerable delay and cost, and may not even be feasible in some situations.
Locality Policies
In some embodiments, a set of locality policies specify how data in the distributed filesystem should be managed. Such locality policies may define and/or control how data is managed at different levels of scope. For instance, locality policies may be defined at a global level (e.g., for the entire distributed filesystem) as well as at a range of finer granularities (e.g., on a per-cloud-controller, per-filesystem, per-user, per-file-type, or per-directory basis). The ability to define arbitrary locality policies that span a range of granularities allows the behavior of the distributed filesystem to be customized to meet the needs of an organization, specific users, and/or specific applications or data sets.
In some embodiments, a locality policy can specify that one or more files should be “pinned” in the cache(s) of one or more cloud controllers. For instance, a locality policy may specify that a specific set of files that are being actively collaborated on by users throughout an organization should be pinned into the cache of every cloud controller. For example, this configuration may be achieved by specifying a locality policy that identifies one or more directories whose contents (and sub-contents) should be pinned in all cloud controllers; users can explicitly ensure that a file will be immediately distributed to and actively cached in all of the cloud controllers by storing it in such directories. In another similar example, the home directory of an important user (e.g., a chief executive officer) who frequently travels throughout an organization may be pinned so that that person can access their data with the fastest possible access times no matter where they are currently located. Alternatively, a project directory for a single site in an organization may be pinned in only the cloud controller for that site, to ensure that the data is quickly available locally (e.g., even if there is a network failure/partition between the cloud controller and the cloud storage system storing the cloud file containing that data). In another example of per-cloud-controller pinning, a locality policy may specify that certain blocks that are frequently read by specific applications (e.g., internet browsers or local file browsers) be pinned for users accessing the cloud controller to ensure that common, frequently used user applications execute as quickly as possible. Note that in such situations part of the cache is fixed (e.g., may not follow normal cache replacement policies), while the remaining portions of the cache may be managed using traditional cache policies (e.g., “best effort” cache replacement policies).
In some embodiments, a locality policy can specify storage behavior for different classes and/or types of files. For instance, a locality policy may specify that certain files should be written together (e.g., putting a project file and all of its supporting files in the same cloud file), or that certain file types should not be written into the same cloud file. For example, image and video files typically consume a substantial amount of space, and are often only viewed for a brief initial time window. Hence, a locality policy may specify that image and video files (e.g., files with “.jpg” and “.avi” extensions) should be considered low-priority (e.g., in comparison with other project file types), and should immediately be stored in a second-level (e.g., cheaper) cloud storage system instead of in a costlier high-speed, highly replicated cloud storage system that is used for high-priority files. Conversely, other file types may be identified as being high-priority, and as a result be pinned in cloud controllers and/or stored (and kept) in the first-level cloud storage system even if they are infrequently accessed. Note that while an initial user accessing low-priority files may experience a slightly higher latency as the files are loaded from the second-level cloud storage system, subsequent users accessing these files fairly soon thereafter from the same location (e.g., before they are flushed out of the cloud controller's cache) will be able to access the files quickly from the cache of the cloud controller. Note also that filling drive files with files of the same type can facilitate performing uniform operations on certain classes of files at a later time (e.g., archiving all of the image files in a portion of the filesystem hierarchy) without having to split cloud files or duplicate portions of cloud files.
In some embodiments, a locality policy may include time-based specifications and/or time-outs. For instance, a certain project directory may be specified to only be pinned for the lifetime of a project, or a training video may be pinned for an initial period (e.g., 15 days), and then subject to normal cache replacement policies. For example, an alternative locality policy for image files may specify that image files should be kept in a first-level cloud storage system for 30 days (or 30 days after the last access), and then moved to a second-level cloud storage system.
In some embodiments, locality policies can specify the synchronization and management of metadata and data. For example:
Note that some locality policies can be modified at any time without incurring substantial overhead. For instance, decisions to no longer pin certain portions of the distributed filesystem and/or to pin a different set of data may simply result in the formerly pinned data now being subject to normal cache replacement policies and the new data being downloaded into the cloud controller. Some policies, however, are primarily applied at the time new data is written (e.g., data policies that group certain types of files into common cloud files); changing the organization of cloud files that were written based on a previous locality policy to reflect a new locality policy may involve walking the metadata of the distributed filesystem and rewriting previously stored data into new cloud files that reflect the new policy.
Deleting Files from a Distributed Filesystem
A distributed filesystem based upon cloud storage systems can efficiently store and access extremely large data sets. At some point, however, some of this stored data will no longer be needed, and hence will need to be deleted from the distributed filesystem. Deletion techniques may involve substantial complexity, computation, and potential delay, and hence need to be carefully managed to ensure that deletions do not negatively impact perceived user file access performance.
Consider as an example an organization that regularly backs up the data on all of its client machines to the distributed filesystem. For instance, all of the clients of the distributed system may be configured to periodically (e.g., on a daily, weekly and/or monthly basis) write backups of their local data to the distributed filesystem. All of this backup data may be collected into a single “tarball” (e.g., a single tape archive file that encompasses the full collection of backed up files while preserving important file system information, such as user permissions, dates, and directory structures). These tarballs may have only a limited lifespan—for instance, a tarball for a given backup date may only be preserved for a given timeframe (e.g., a month, or 60 days), and then be deleted to conserve storage space and/or make way for future backups. Depending, however, on the number of clients and the amount of data on each client, the collected data size may be substantial; for instance, performing backups may involve creating and deleting files that consume on the order of terabytes (or larger) of data in the distributed system, and hence may involve considerable time and effort.
In some embodiments, deletion from the distributed filesystem involves a series of steps that first hide a deleted file from users (to provide a user perspective of instant response) and then perform a number of background operations that traverse the file's metadata and then actually delete the data from the distributed filesystem over time in a manner that does not affect the performance of other simultaneous data accesses in the distributed filesystem.
The set of hash values in the list of deduplication updates 2714 indicate data blocks in the deduplication tables 2716 that need to be decremented (e.g., to reflect the fact that File X no longer refers to these data blocks). These updates are applied to deduplication tables 2716 (operation 3); during this process, cloud controller 2700 determines whether any of these data blocks are no longer referenced in the distributed filesystem (e.g., the reference count for a given data block is decremented to zero). If this is the case, cloud controller 2700 further determines from cloud file status-tracking structures whether these changes result in any cloud files whose data blocks have all been deleted; if so, these cloud files are added to a list of cloud files that are queued for deletion 2718 (operation 4). The cloud controller can then subsequently send requests to cloud storage system 302 to actually delete (or archive) these cloud files (operation 5); only at this point, after the cloud storage system has executed and confirmed the deletion request, has the data actually been deleted from the distributed filesystem. Note that each cloud controller maintains and updates its own internal deduplication table. For example, the process by which the other cloud controllers of the distributed filesystem update their deduplication tables in response to the above-described file deletion on cloud controller 2700 may comprise performing the following on each of the other cloud controllers: (1) receiving an incremental metadata snapshot uploaded by cloud controller 2700 that indicates that the metadata for File X should be removed; (2) using the metadata for File X (e.g., the deduplication hash values for the blocks of File X) to properly update (e.g., decrement references in) the cloud controller's local deduplication table; and (3) updating the locally stored metadata to reflect the removal of File X.
In some embodiments, operations 2-5 as described for
In some embodiments, a cloud controller may also intentionally delay some of the steps of the deletion process for a time interval to ensure that the deletion was intended. For instance, upon receiving user file delete request 2712, cloud controller 2700 may initially delete the view of the file from active filesystem metadata 2704 (operation 1), but then delay performing the subsequent deletion operations for a specified time interval to ensure that the deletion request was not accidental. This delay (and other deletion parameters) may be configured using a locality policy. For example, a locality policy influencing deletions might specify: whether deletions should be delayed, and if so, for how long of a time interval; whether other cloud controllers, administrators, and/or clients should be notified of pending deletes; and/or whether deletion requests for certain (or all) files should to be moderated (e.g., approved) by an administrator and/or file owner before being executed. A locality policy may also be used to group backup files into a distinct set of cloud files (e.g., not mixed with other data) to simplify the deletion of the backup files at a later time.
Note that the point at which cached data blocks for a file that is being deleted should be flushed from cloud controller caches may vary. If deduplication is enabled, a data block referenced by the file being deleted may also still be referenced (and actively used) by other files, and hence should not be flushed from the cloud controller's cache. If, however, the cloud controller determines that a data block is no longer referenced, it can check whether the block is still cached and, if so, delete it from the cache. Alternatively, instead of being explicitly flushed, the cache controller may instead leave the data block in the cache and let it be flushed out eventually (due to disuse) via normal cache replacement policies. Note also that deduplication techniques can help to reduce the overhead of deletion operations by reducing the amount of redundant data that is stored and will later need to be deleted; deletion operations that delete unique data involve somewhat more computation and overhead (e.g., additional operations 4 and 5 in
In some embodiments, a distributed filesystem can be tuned to balance a range of performance and cost goals. For a scenario in which performance is the only factor and cost is no issue, the data for the distributed filesystem might be stored in the cloud storage system that provides the highest availability, redundancy, and performance, and each client site may include multiple peer cloud controllers as well as backup cloud controllers. In a more typical scenario, however, performance may need to be balanced by cost factors.
For instance, the distributed filesystem may leverage multiple cloud service providers (including archival cloud service providers) to minimize the storage costs for infrequently accessed data while maintaining high performance for frequently accessed data. For example, cloud controllers may be configured to track and consider a range of parameters (e.g., the most commonly used data, the most frequently transferred data, access frequencies, typical access sizes, the costs of storing data in different cloud storage providers, the network cost of leveraging distributed cloud controllers as peer caches, etc.) and attempt to automatically rebalance and optimize system behavior to maximize performance for a given fixed cost.
Supporting and Recovering Virtual Machines in a Distributed Filesystem
Some organizations may allocate one or more distinct cloud controllers (e.g., separate, dedicated compute servers) at every site in the organization, to provide access to the shared, distributed filesystem. Some scenarios, however, may motivate combining multiple services into a single hardware device. For instance, a small, remote site may not have sufficient rack space for another computing device or may not be large enough to justify the cost of multiple independent servers. However, such sites might still benefit from the capability to leverage the storage, archive, and backup capabilities of an organization-wide distributed filesystem and cloud storage.
In some embodiments, one or more distributed filesystem services may be executed in a virtual machine on another computing device. For instance, an application with cloud controller functionality may be executed in a virtual machine to provide access to the distributed filesystem at a site that otherwise might not be able to host (or otherwise justify) a dedicated cloud controller device. This cloud controller application (sometimes also referred to as a cloud controller instance) can provide all of the previously described cloud controller functionality to other processes executing on the same host server as well as to external clients.
In some embodiments, one or more services are executed in virtual machines in a manner that leverages the distributed filesystem. More specifically, services and applications can be executed in virtual machines in a manner that ensures that their executables, runtime structures, and/or application data are all stored in the distributed filesystem. This arrangement allows such services and applications to be automatically backed up to (and, if needed, restored from) the distributed filesystem. Note that such capabilities are not limited to situations in which cloud controller functionality is executed in a virtual machine; any stand-alone cloud controller can also export a mount point for the distributed filesystem that can then be used to facilitate the execution of virtual machines in the context of the distributed filesystem.
Executing multiple applications in virtual machines in the context of the distributed filesystem may involve substantial additional network overhead for some remote sites with limited network bandwidth. For instance, executing such applications may involve: (1) accessing a large application executable file from the distributed filesystem (e.g., loading the application executable from the cloud storage system); (2) accessing application data needed by the application (e.g., from the cloud storage system); and (3) storing a large amount of temporary, swap, and application data that is created while executing the application to the cloud storage system. In some scenarios, an application executable may be cached by a host server (or pinned in a cloud controller) to reduce an initial startup delay for an application. Furthermore, while a cloud controller and/or virtual machine may be configured to store runtime environment data to the distributed filesystem (and cloud storage system), such runtime environment data may be delayed, metered, and/or marked as low-priority to ensure that storing such data in the distributed filesystem does not interfere with other higher-priority data accesses and/or network transfers. Storing all of the data associated with the virtual machine facilitates restoring the exact execution state on the same or another host server in case of failure. In alternative scenarios where only application data is written to the distributed filesystem, an application may still be recoverable after a failure, but perhaps only to the point of the application's last written data output and/or internal snapshot, as opposed to an exact application state that could be conveyed using the full set of stored data. Such configuration and storage decisions may be influenced by the expected data usage, data access patterns, duration of execution, and runtime requirements of an application.
Note that some applications may access both structured and unstructured data. For instance, a given application may access both structured data (e.g., an SQL database) as well as unstructured raw files. In some instances, executing a given application may also involve executing other additional support applications (e.g., a database application) in the same or in another separate virtual machine; the disclosed techniques can scale to any number of applications and virtual machines, depending on the capabilities of one or more host servers. These multiple applications can all share the same distributed filesystem view provided by a cloud controller (and/or cloud controller instance), and hence can interact normally, as if they were operating upon locally-stored data, while the actual structured and unstructured data being operated upon are stored in the distributed filesystem.
In some embodiments, executing applications in a manner that leverages a distributed filesystem facilitates recovering from a range of failures. For instance, failures may include, but are not limited to: (1) the failure of a host server; (2) the crash of a cloud controller and/or cloud controller instance; and/or (3) the crash of an individual virtual machine or application. Some organizations may select a highly redundant hardware platform to serve as a host server to avoid failure, but even such devices fail sometimes. However, if all of the virtual machine and application data that is written and modified is also stored in a cloud storage system (via the distributed filesystem), the entire state of an executing VM and application can be recovered from the distributed filesystem and executed via another host server and/or cloud controller. In a scenario in which a cloud controller or cloud controller instance crashes, as long as the persistent storage managed by the cloud controller remains intact (e.g., a cloud controller's local storage, or a storage pool allocated to a cloud controller instance), the cloud controller (or cloud controller instance) can be quickly restarted and use the locally cached data to immediately continue providing data services. In a worst case scenario in which the cached data is lost, the cloud controller (or cloud controller instance) can reload requested metadata and data from the cloud storage system. In a scenario in which an individual application and/or its host virtual machine crash, its data is likely to still be cached (and/or pinned, perhaps, via a locality policy) in its associated cloud controller (or cloud controller instance), or can be re-loaded from the cloud storage system, thereby allowing the application to be quickly restarted to its previous state. Hence, leveraging the distributed filesystem ensures that disaster recovery solutions are available for all of the services being run on a host server (e.g., in a remote office).
In some scenarios, data management for applications and virtual machines may be structured and/or adjusted to reduce overhead and facilitate different recovery options. For instance, because a large number of applications may be executed in virtual machines via a number of different cloud controllers, data from these applications and virtual machines may benefit from being logically separated (and/or indexed) in the distributed filesystem on a per-VM and/or per-cloud-controller basis (e.g., with each cloud controller, and each VM per cloud controller, being associated with different, distinct directories in the distributed filesystem). Distinguishing each instance of an application and/or VM facilitates recovering and re-executing a given application on the same or another host server after a failure. Note that data may also be logically separated for each individual application and/or VM. For example, data associated with a given application and virtual machine may also be logically separated into different sub-directories (and/or directory hierarchies) in the distributed filesystem that store the application's executable, the temporary and swap data (e.g., runtime environment data) for each executing application instance, and application data stored by each executing application instance.
In some embodiments, the size of the storage pool allocated to a cloud controller instance may vary depending on factors that include, but are not limited to: the amount of local storage in the host server; the anticipated data needs for the site and/or host server; the set of files (and application executables) from the distributed filesystem that will be pinned in the storage pool; and/or the set of virtual machines that will be executed on the host server. For instance, in some scenarios where the host server is primarily dedicated to executing applications that will execute in virtual machines whose data will be backed up via the distributed filesystem, most of the host server's local storage may be allocated to the cloud controller instance to ensure that as much space as possible is available to cache application and VM data.
Applications executed in virtual machines may be used to provide a range of important compute services and/or information resources to a site, and hence supporting cloud controllers may sometimes be configured to perform additional operations to improve data access performance. In some embodiments, a cloud controller (or cloud controller instance) may be configured to optimize (e.g., reduce the latency of) the data accesses associated with virtual machines as much as possible. For instance, a cloud controller may be configured to request and cache data that is expected to be accessed by a virtual machine in a local storage pool before the virtual machine is even executed (e.g., via locality policies that pre-populate the local storage pool based on expected data use, an expected time of use, and/or other factors).
Another optimization involves adjusting the write behavior for scenarios in which multiple entities attempt to perform modifications to the same portion of the distributed filesystem. As described previously, a cloud controller attempting to write a file “owned” (e.g., created) by another cloud controller typically first contacts the owning cloud controller with a request to modify the file. More specifically, the cloud controller that owns a file may be considered to hold a “write lock” which identifies the cloud controller as the arbitrator for further writes for the file. However, contacting another cloud controller to perform a write operation may involve additional latency. Hence, optimizing write behavior to improve the performance of virtual machines may also involve identifying and transferring the write locks for contentious portions of the distributed filesystem hierarchy to a local cloud controller (or cloud controller instance) to reduce write latencies. A further optimization may involve ensuring that such write locks cannot be requested by and moved to another cloud controller (and/or cloud controller instance) for a given time interval and/or while a given virtual machine is executing. Such optimizations may be specified by techniques including, but not limited to, locality policies and other predictive mechanisms.
Note that, in some scenarios, a host server may execute two or more cloud controller instances that provide access to multiple, distinct distributed filesystems. In such scenarios, the host server may allocate separate storage pools for the different cloud controller instances, and each cloud controller instance would present a different export mount point for its respective distributed filesystem. The host server may also simultaneously write data related to applications executing in virtual machines to multiple distributed filesystems.
Avoiding Client Timeouts in a Distributed Filesystem
Some client systems may be configured to “time out” a data request (e.g., disconnect and abort the request) if a target server does not return the requested data block within a specified time interval. Such timeouts can lead to performance issues in a distributed system. For instance,
Note that while client timeout intervals can typically be configured or adjusted to provide more time, adjusting the timeout interval for all of the clients in an enterprise may involve considerable overhead and/or otherwise be infeasible. For instance, some clients (e.g., clients using the CIFS protocol) may be initially configured to disconnect from a server and report an error if they have not received their requested data within 60 seconds. If a given site has a slow network link (or intermittently has a high network load that would delay data transfer from a cloud storage system), all of the clients at the site might be reconfigured with a longer timeout interval. However, depending on the number (and mobility) of clients, such reconfiguration operations may involve substantial effort, and may also cause delays in realizing and identifying actual failures (e.g., due to the lengthened timeout intervals). Another option involves adjusting the behavior of the cloud controller to consider client-perceived delay and client timeouts.
In some embodiments, a cloud controller may adjust the rate at which requested data blocks are delivered to one or more clients to ensure that clients to not experience timeouts. For instance, a cloud controller may determine how many of the data blocks for a requested file are currently being cached and, depending on factors such as the available network bandwidth between the cloud controller and a cloud storage system containing the uncached data blocks, control the rate at which cached data blocks are served to a client to minimize the probability of timeouts for the client. For example, a cloud controller may pre-request uncached file data blocks from the cloud storage system while simultaneously restricting the rate at which cached file data blocks are sent to the client; these combined actions ensure that there is sufficient time for cloud files containing subsequent needed data blocks to be transferred from the cloud storage system and processed. Delaying the delivery of some cached blocks can facilitate the operation of the distributed filesystem by avoiding and/or reducing the number of timeout-related errors in clients.
In some embodiments, a cloud controller tracks typical file access patterns, and determines a set of data blocks that will be cached for each file. For instance, some frequently-accessed files may be cached in their entirety during a peak-access time interval. However, due to space constraints, a cloud controller may cache only limited portions of less-frequently accessed files. For example, a cloud controller may determine that, for a certain type or class of file, clients frequently access only the first and last data blocks of a file (or initially access these blocks prior to sequentially accessing all the other blocks of the file). In such scenarios, the cloud controller may adopt a policy of caching only the first and last block of such files (e.g., the first and last 64 KB of all files, as allowed by storage space constraints in the cloud controller); the cloud controller can then slightly delay the transmission of these cached blocks to a requesting client to create a time buffer during which subsequent file data blocks can be predictively pre-fetched from the cloud storage system.
In some embodiments, techniques for choosing a delay interval may involve either using fixed delay intervals or adaptively adjusting delay intervals based on characteristics of the distributed system. For instance, in some embodiments, a cloud controller may consider and/or calculate one or more of the following when determining a delay interval for a requested data block:
In some embodiments, a cloud controller adaptively decreases and/or increases the delay interval for data blocks sent to a client as needed. For instance, as described above, a cloud controller receiving a request from a client determines how much of the file's data is cached locally. Upon detecting that not all of the file's data blocks are cached locally, the cloud controller starts out with an initial delay interval (e.g., 20 seconds per data block) and measures how long it takes to retrieve the remaining uncached data blocks of the file. More specifically, while delaying the initial responses for cached blocks, the cloud controller starts a separate thread that reads data blocks (e.g., 128 kB blocks) from a cloud storage system hosting the data blocks. This thread tracks the received blocks to determine how long each read is taking, and uses this information to compute the amount of bandwidth between the cloud controller currently and the cloud storage system. After making an initial estimate of how long it will take to retrieve the rest of the uncached data blocks (based on the initial set of received data blocks), the cloud controller can begin to adjust the delay interval to match the expected arrival rate of the data blocks. For example, upon determining how long it will take to retrieve the rest of the data blocks for the file at the current bandwidth, and assuming that the arrival rate is lower than the client timeout interval, the cloud controller can progressively reduce the delay interval; when all of the file data is cached on the cloud controller, the delay interval can be reduced to zero. Note that in some implementations, the delay interval may be decreased in fixed steps (e.g., the cloud controller may smoothly reduce the delay interval by halving the delay interval after every set of five successful data block reads). However, in some scenarios, if the time taken to retrieve blocks increases, the cloud controller may need to increase the delay interval in a similar manner to ensure that the client does not trigger request timeouts. Note also that in some implementations the set of I/O credits issued to a client may need to be reduced so that the client doesn't issue parallel requests (thereby partially bypassing the flow control being provided by the cloud controller as well as increasing the traffic congestion between the cloud controller and cloud storage system). The described techniques facilitate a wide range of data block access rates—if the arrival rate for the file's data blocks is slow, the cloud controller can keep the delay interval high to avoid client timeouts; alternatively, if the arrival rate is fast, the cloud controller can quickly reduce the delay interval to provide data to the client more quickly (while continuing to track the arrival rate to ensure that a reduced delay interval continues to be the appropriate).
In some embodiments, a cloud controller may be configured to adjust caching behavior in an attempt to balance both client performance and avoid client timeouts. For instance, in scenarios where a cloud controller is provisioned with sufficient local storage, the cloud controller may be configured to cache a larger set of initial data blocks for each file (or a specified subset of files). If such scenarios, the cloud controller can determine from an initial set of accesses whether a file is likely to be accessed in its entirety, and respond appropriately. For instance, if enough data blocks for the file are cached, a cloud controller can immediately send an initial set of requested data blocks for a file to the client, thereby allowing the client to being operating upon them without delay. In some situations, the client may only need these first few blocks, and hence the operation completes without any additional delay. If the client is indeed accessing the entire file, the cloud controller, upon receiving requests for additional data blocks of the file, can delay the transmission of the remaining cached blocks while retrieving the subsequent set of uncached blocks from a cloud storage system. This configuration reduces the delay for requests that only touch a small subset of a file's data (e.g., when a user is probing a directory) while also reducing the likelihood of timeouts for files that are accessed in their entirety. Alternatively, if the cloud controller and the cloud storage system are provisioned with substantial network bandwidth, the cloud controller may assume that every file request will lead to a complete sequential file access, and begin downloading any additional uncached data blocks for a file from the time the first cached data block for the file is accessed.
In some embodiments, an additional technique for avoiding client timeouts (that can be used in conjunction with the above-described techniques) involves adjusting the size of the cloud files that are uploaded to (and downloaded from) cloud storage systems. For cloud controllers that are located in environments with relatively low network bandwidth, using smaller cloud files allows faster download and decryption of cloud files, hence involving less delay and reducing the likelihood of a client timeout. In some implementations, cloud controllers may initially evaluate their environments and then communicate to make a concerted determination of a cloud file size based on their respective network connections and bandwidth (e.g., based on the network bandwidth of the cloud controller hosted by the slowest site). Some implementations may even support adjusting cloud file size at a later point (e.g., to accommodate one or more cloud controllers in low-bandwidth environments that are added to the distributed filesystem at a later point), although such operations may involve considerable reconfiguration overhead. Note that because each cloud file include a number of metadata fields, reducing the size of the cloud file does increase the amount of storage overhead, thereby increasing the number of total bytes that need to be uploaded and downloaded for each actual block of file data (thereby effectively reducing the maximum upload and download data rates for data files). However, in some environments, these higher storage overheads may be justified by the faster download and decryption times for smaller cloud files (while can result in lower-latency lookups for individual file blocks).
Cloud Commands for a Distributed Filesystem
As described previously, some embodiments of a distributed filesystem facilitate leveraging cloud storage capabilities while providing clients with an abstraction that all of the data in the distributed filesystem is locally available and accessible via local cloud controllers. More specifically, caching data for the distributed filesystem on each cloud controller provides transparent access to large data sets on an on-demand basis. However, in some scenarios, providing such abstractions to clients of a distributed filesystem can negatively impact performance.
For instance, consider a typical file copy operation that is initiated by a client that transparently accesses a file stored in a distributed filesystem, as illustrated in
In some embodiments, aspects of the implementation and/or distributed nature of a distributed filesystem are exposed to one or more end users to improve the responsiveness, reliability, and/or performance of operations for the distributed filesystem. More specifically, a set of cloud-storage-aware directives (also referred to as “cloud commands”) can be used to selectively perform system-aware operations that can provide substantial performance benefits and/or additional functionality over a transparent distributed filesystem.
In some scenarios, a user request to copy a file may indicate that one or both files will be accessed and/or modified soon. Hence, in some instances, a cloud controller may optionally be configured to begin pre-fetching part or all of a file that has been copied via a cloud-aware copy operation. For instance, in the context of
In some embodiments, cloud commands can be initiated using a range of invocation mechanisms. For instance, in some embodiments, client applications can be configured to communicate directly with a cloud controller to invoke a cloud command (e.g., using CIFS private filesystem controls (FSCTLS)—private filesystem controls provide a mechanism for extending existing fileystem protocols). In alternative embodiments, cloud commands may be initiated using an enhanced filesystem interface and abstraction. More specifically, cloud controllers may use a filesystem abstraction to present clients with additional files and directories that do not actually represent stored data, but instead represent cloud commands that can be executed to perform cloud-aware operations.
In some embodiments, a cloud controller may be configured to manage the items in the cloud-command sub-hierarchy using filesystem mechanisms and abstractions. For instance, in the context of
In some embodiments, a set of cloud commands may be made available to privileged users of a distributed filesystem to enable (or improve the performance of) a set of operations that would otherwise be infeasible. For instance, a system administrator or IT (information technology) staff member that is conversant with the organization of the distributed filesystem may write a script that can be executed by authorized users to invoke one or more cloud commands to complete a specific task. For example, one exemplary operation of this type might involve a large copy (e.g., or terabytes of data) that might saturate a corporate network for several days if not executed using a cloud-aware copy command. More specifically, consider a common distribution operation in a software development build environment. Application developers may initiate a “release” operation that copies a large number of executable binary files and supporting libraries to an export directory in a workflow hand-off to one or more other (local and/or remote) teams that then perform additional operations for the files (e.g., application testing, language localization, etc.). Using a script that leverages cloud commands (e.g., cloud-aware copy commands) allows the files to be copied in a fast and seamless manner regardless of whether they are already cached by the cloud controller. The use of cloud commands enables the copy to complete quickly, thereby allowing the metadata for the copied files to propagate to the other cloud controllers quickly. Locality policies can be used to selectively pre-load (or not pre-load) the data blocks for the copied files at other sites (e.g., at each given site's cloud controller) based on the anticipated immediacy of use, with each receiving team then accessing the files normally via the distributed filesystem. As described previously; any cloud controller already caching data blocks for the “released” files (potentially due to previously-accessed data blocks from other file data blocks containing the same content) can already serve these same data blocks to clients accessing the new export directory, or can load uncached data blocks from the cloud storage system as needed.
In some embodiments, cloud commands can be used to perform and/or enhance a set of distributed filesystem operations that include, but are not limited to:
In some embodiments, cloud commands enable users to initiate a snapshot for some or all of a distributed filesystem at a given moment in time. For instance, a user completing some unit of work upon a set of files may want to ensure that a snapshot of those modified files be created immediately instead of at a system-chosen time (e.g., so that the state of those files and/or the entire distributed filesystem at that specific point can be revisited if needed in the future). Note that the computing and storage overhead for a snapshot operation may vary depending on the extent of the distributed filesystem being snapshotted. As described previously, global snapshots for a distributed filesystem can be achieved by initiating a snapshot operation in each cloud controller and then triggering a set of update events that share the resulting snapshot state between the cloud controllers (via incremental metadata snapshots), thereby effectively capturing the entire state of the distributed filesystem at that moment. However, such global snapshot operations may involve considerable overhead, and frequent user-initiated global snapshots may result in the exchange or capture of substantial amounts of data (e.g., temporary data) that might otherwise might not be snapshotted or transferred throughout the system. The time interval for periodic system snapshots may be tuned to ensure that global data and metadata for temporary data is typically deleted before being snapshotted.
In some embodiments, to ensure that user-initiated snapshots do not contribute to declines in distributed filesystem performance, an administrator for a distributed filesystem may limit the scope of snapshots that can be initiated by a given user (via cloud commands). For example, an administrator may configure a set of permissions for each user to specify whether a user may initiate snapshots, and specify whether snapshots initiated by a given user are: limited to that user's portion of the filesystem; limited to a specified set of files; or unlimited (e.g., can snapshot the entire distributed filesystem). In some scenarios, a user may also be allowed to specify the scope of the snapshot when initiating a snapshot cloud command. In another exemplary scenario, an administrator may limit the frequency with which a user may initiate snapshots or specify a minimum time interval between user-initiated snapshots.
Note that techniques for using cloud commands to initiate snapshots and/or grant permissions to initiate snapshots may be implemented using a range of techniques. For instance, a system administrator may use command-line cloud commands such as:
Note also that exposing the notion of snapshots to users may involve providing some additional information and/or capabilities associated with snapshots to users and clients of the distributed filesystem. For instance, in some embodiments cloud controllers may expose information and/or different versions of snapshotted files to clients and users using a set of user-visible snapshot directories. In some implementations, such snapshot directories may be organized as separate file hierarchies (for each given snapshot) that can be traversed by clients (or users) that have permission to access that snapshot; more specifically, these file hierarchies can be traversed to access file state at the time of the given snapshot (and may involve using additional cloud commands to restore a previous snapshotted version of a file). Alternatively, in other implementations, every filesystem directory may include a hidden snapshot directory (e.g., a directory named “.snapshot”) that facilitates accessing the different versions available for files in each given filesystem directory. For example, in the context of
In some embodiments, a filesystem interface for the distributed filesystem can also be used to specify locality policies. More specifically, cloud commands can be used to implement, complement, and/or expand upon the capabilities of locality policies. For instance, command-line cloud commands may be used to identify files and/or directories to be: pinned on one or more cloud controllers; mirrored to one or more additional cloud controllers; immediately loaded to any cloud controller that is accessed by a given user and/or client; and/or subject to a range of other behaviors.
In some embodiments, cloud commands can be used to enhance the operation of virtual machines executing in the distributed filesystem environment. During normal operation, executing a virtual machine involves copying the virtual machine's executable code, executing the executable code, and performing a number of initialization operations. In contrast, cloud commands can facilitate quickly copying and/or cloning existing virtual machines. For example, a system administrator for the distributed filesystem may perform the initial execution and initialization for a virtual machine, and then perform a snapshot for the state and the data of the virtual machine (e.g., using a cloud command) after the initialization operations have completed. Users can then subsequently use another cloud command (e.g., /cloudcmd/vm_copy) to clone this existing virtual machine, thereby bypassing the overhead of instantiating a new virtual machine. Note that these techniques may incorporate aspects of the above-described cloud-aware copy command.
As described previously, virtual machines that are being executed from the distributed filesystem may be configured to write runtime and/or operating data into the distributed filesystem. In this context, cloning a virtual machine may involve performing such cloud-aware copies to copy all of the data for the snapshotted virtual machine to a new directory, and then having the new clone of the virtual machine begin execution in this new directory. If the host cloud controller is already caching the data blocks for the snapshotted virtual machine, the new clone of the virtual machine will already have access to all of its needed data, and can begin executing immediately (e.g., the copy-on-write properties for the distributed filesystem mean that each cloned virtual machine will initially have the same characteristics and data blocks as its “parent” virtual machine, and that only modified portions of the cloned virtual machine will need to be subsequently written to the cloud storage system). Hence, cloud commands can be used to substantially improve the start-up times of virtual machines. For instance, users may use cloud commands in a scripting manner to programmatically create (e.g., clone) a large number of virtual machines without requiring any additional data storage.
In some implementations, a parent virtual machine (or multiple different parent virtual machines) may be created in a separate, dedicated filesystem or sub-hierarchy of a filesystem, thereby logically separating the virtual machine data from other unrelated data and facilitating subsequent cloning operations for one or more virtual machines. Note that, as described previously for backing-up virtual machines, some sub-portions of the virtual machine data (e.g., temporary files) may not be needed for the cloned virtual machines; the vm_copy command and/or scripts may be configured to exclude such unneeded data, or a system administrator may configure the parent virtual machines to store this unneeded data in a different portion of the distributed filesystem (e.g., in a location distinct from the dedicated storage areas that will be cloned).
In some embodiments, cloud commands can be used to enhance the operation of database applications executing in the distributed filesystem environment. A database “dump” operation (e.g., backup) typically involves writing out all of the data contents of the tables in the database to one or more files that are structured for backup purposes; note that these files are structured differently from how an active database typically stores data on a disk storage device. Creating such backups involves bringing the database into a quiescent state (e.g., pausing all incoming queries or other requests and flushing all modified data in memory to persistent storage) to ensure that the database data is in a consistent state, and then writing out the full set of (consistent) database data to perform the actual dump operation. In contrast, for a database application executing in the context of a distributed filesystem, all of the database data blocks that are written to the distributed filesystem can be preserved in the cloud storage system persistently, thereby allowing subsequent database applications to be restored to a previous state as needed (e.g., after the database application has crashed) without needing to perform a dump operation. However, the database application does still need to synchronize all in-memory data to the distributed filesystem so that all of the database data is consistently stored in a set of data blocks at a given time. A cloud command (e.g., /cloudcmd/db_sync) may be used to trigger the database application to perform such a synchronization operation. For example, a system administrator may use this synchronization cloud command followed by a snapshot cloud command to put the database into a consistent state and then ensure that an explicit snapshot is made of that consistent state; all updated data blocks written by the database application are then propagated to the cloud storage system using the previously described mechanisms, thereby effectively ensuring that the data used by the active database application is mirrored (and backed up) without incurring the additional overhead and database down time associated with performing an explicit dump operation.
In some embodiments, cloud commands can be used to both archive data that is not currently needed in the (active, non-archived) distributed filesystem to an archival cloud storage system as well as to retrieve and access archived data that has been previously moved to an archival cloud storage system. As described previously, data blocks may be moved to such an archival cloud storage system after not being accessed for some specified time interval. Alternatively, a user may also use a cloud command to identify specific files that can already be archived, for instance by performing the following exemplary command-line cloud commands:
While previous sections describe having an administrator of the distributed filesystem recover archived files and/or data blocks, in some embodiments cloud commands may also offer an alternative for recovering archived data. For example, cloud controllers may be configured to preserve the metadata for archived file data, continue to present archived files to (authorized) users, and enable (authorized) users to initiate the recovery of archived files via cloud commands. The following section describes techniques for restoring archived data in more detail.
In some implementations, permissions and authentication for a distributed filesystem are provided using standard authentication techniques (e.g., an Active Directory service, an NT LAN Manager (NTML), the Kerberos protocol, etc.). Cloud commands for the distributed filesystem can be implemented to leverage such existing authentication techniques as well as existing filesystem abstractions. More specifically, users attempting to access cloud command functionality can do so via existing filesystem mechanisms (e.g., initiating cloud commands by invoking special files or scripts that appear in the CLOUDCMD branch of the distributed filesystem, as described above) and can be authenticated using their existing user names and credentials. These capabilities allow system administrators to delegate cloud command permissions using existing filesystem commands and permissions, thereby allowing trusted users to perform some management activities and potentially reducing the load upon IT staff. Note that in some scenarios users may also be granted the permission to delegate privileges to other users. For instance, a system administrator may grant a trusted user both the permission to invoke snapshots as well as the permission to grant the permission to invoke snapshots to other users. This trusted user can then grant the permission to invoke snapshots to a third user without requiring further interaction or permission from the system administrator.
In some embodiments, presenting cloud commands using a filesystem abstraction facilitates providing additional capabilities for the distributed filesystem without requiring additional infrastructure or support in a client of the distributed filesystem. For instance, using the filesystem abstraction to initiate cloud commands does not require a client to access any additional ports or connections in a cloud controller, nor perform additional authentication operations. Because all commands flow through the filesystem, no additional firewall rules are needed between clients and cloud controllers (e.g., in scenarios where the two entities are separated by one or more firewalls). Furthermore, using the filesystem abstraction to invoke cloud commands is distinct from alternative techniques that use client plug-ins to perform management operations for the distributed filesystem. Vendors associated with the distributed filesystem no longer need to implement additional security features or create application plug-ins to provide additional functionality; invoking cloud commands via the filesystem abstraction allows the set of cloud commands to be changed and/or expanded without having to modify clients, client plug-ins, or the client/cloud-controller interface.
Restoring Archived Data for a Distributed Filesystem
As mentioned in the previous section, cloud commands can be used to retrieve and access archived data that has been moved from the (active) distributed filesystem to an archival cloud storage system. In some embodiments, the distributed filesystem may be configured to expose the availability of archived files (as well as other filesystem status information) to clients and users using either separate file hierarchies with archived data or hidden per-directory archive directories (e.g., by including in every existing directory a sub-directory named “.archive” or “.policy” that provides additional archival and/or status information for the parent directory), thereby exposing the notion of archived data in a manner substantially similar to the way snapshots were exposed in the preceding section. More specifically, a cloud controller presenting a view of the distributed filesystem to a client could present archived data files via such mechanisms, and allow (authorized) users accessing such archive directories to initiate requests to restore such archived files. For example, users may be able to enable or disable the view of archived files via a cloud command menu. Upon receiving a request from a user to access an archived file (e.g., via a command-line interface, a file browser, or some other mechanism), the cloud controller receiving the request may present the user with a set of options for restoring the file. Note that such accesses enter the domain of cloud commands in that at this point the file request is no longer returning actual file data (at least initially), but instead involves additional filesystem operations that expose the underlying nature of a distributed filesystem (e.g., archival cloud storage systems) and/or give the user a set of choices before providing the requested file data.
In some embodiments, cloud command interfaces also facilitate exposing additional information about the operation of and the current status of files in the distributed filesystem; this information can then be used to make informed decisions for restoring archived files or other operations. Filesystem protocols typically provide a standardized set of information for files (e.g., the name of a file, the file's size, and the last time the file was accessed), but the set of file metadata that is shared between clients and servers is often limited. Furthermore, the process of adding new information into a standard filesystem protocol is typically quite complex, thereby making extending such protocols to convey additional information related to the status of a distributed filesystem challenging. As an alternative, the distributed filesystem can leverage cloud command interfaces and status directories (e.g., .archive or .policy directories) to share information about files that cannot otherwise be conveyed using a normal filesystem protocol. For instance, information in a .policy directory may specify (but is not limited to):
Note that such status information may be updated using a range of update techniques. For example, keeping such status information up-to-date on every cloud controller for every data file in the distributed filesystem may involve substantial overhead (especially if the underlying files and file information change frequently). Instead, the system may be configured to populate such status information on demand, thereby only determining, retrieving, and/or populating the status information for a specific target file only when a user accesses the placeholder for that file in an associated .policy directory.
In some embodiments, the set of archival operations that are presented to users may vary based on policies and economics associated with the distributed filesystem and/or one or more cloud storage providers. For instance, system administrators may provide a specific set of archive and restore options based on a cloud storage provider's billing model. Such an arrangement gives users some control of the set of data that is kept in the distributed filesystem (thereby reducing the number of user archive and restore requests submitted to the system administrators), while also allowing system administrators to control the costs of restore operations.
In some implementations, accessing archived files data via cloud commands may involve choosing between a set of economic and performance choices. For instance, an archival cloud storage system may provide a range of price options and recovery times for a given set of archived data blocks. For example, accessing the requested data immediately may cost a certain amount, receiving the requested data within a week may cost a second (presumably lower) amount, and receiving the requested data for free may also be possible but incur an even longer time delay. Upon receiving a request from a user for an archived file, a cloud controller may determine the set of access options, and communicate these options back to the requesting client for resolution. In some implementations, this decision process may involve: (1) a user initiating a cloud command requesting access to an archived file (e.g., via a command-line /cloudcmd/ar_restore command or a file browser that selects an archived file); (2) the cloud controller determining a set of access options and costs; (3) the cloud controller returning the set of access options and cost choices to the user by initially writing the set of access choices as the contents of the archived file (or to a status file in the cloud-command sub-hierarchy); (4) the accessing user receiving and viewing these file contents, and choosing the desired access choice (e.g., by deleting the unwanted options from the file and then performing a file write); (5) the cloud controller receiving the write for the archived file, determining that this write is for an archived file being restored, and initiating the recovery operation from the archival cloud storage system according to the selected access choice; and (6) after the unarchiving operation has completed and the (non-archival) cloud storage system and the cloud controller have received the previously archived data blocks, having the distributed filesystem optionally notify the user of the availability of the now-unarchived file (e.g., by updating the status file or performing some other notification action). Note that this exemplary implementation is substantially similar to the cloud-aware copy command in that the user immediately receives an immediate response (with a set of access choices) for an archived file containing data that is not immediately retrievable, in contrast with an alternative approach that instead immediately initiates an unarchiving process (which may take multiple days to complete), thereby potentially providing a negative user experience. However, the described decision process is exemplary, and the process of unarchiving an archived file using cloud commands (and presenting a set of access options and receiving a user selection) may be implemented using a range of other techniques and interactions.
The preceding example describes allowing users to choose among a range of restore options. In many scenarios, however, system administrators may allocate (or be allocated) a fixed “restore budget,” and a large set of users may not be trusted to manage this budget fairly. For instance, consider an archival cloud storage provider that bills based on the maximum restore rate used at any one moment in a given month. In such a scenario, system administrators may be granted a fixed restore budget (e.g., $1000/month) that limits that maximum restore rate for all user accesses over that time period. Given this limitation, the system may be configured to offer only a single restore option to users attempting to restore archived files and track the set of outstanding restore requests to provide users appropriate time estimates for such retrieval requests. In situations where a file is needed more quickly, the distributed filesystem may configured to (either with or without system administrator intervention) either increase the restore budget (to make outstanding restore requests clear more quickly) or adjust the schedule and/or priority of outstanding restore operations to effectively boost the restore of a higher-priority file.
As described above, authorized users can perform restore operations for individual files. In some embodiments, users may also be provided abstractions that allow them to restore larger portions of a filesystem sub-hierarchy in a single operation. For instance, as illustrated in
In some embodiments, users can be granted the ability to also recover archived snapshot data. As described previously, the distributed filesystem may be configured to take, distribute (to other cloud controllers), and preserve both user- and system-initiated snapshots of modified data. The size of the snapshot data preserved in the distributed filesystem may become substantial over time, and hence the cloud controllers managing the distributed filesystem may be configured to selectively archive some snapshot data in an archival cloud storage system. Users, however, may still be enabled to selectively unarchive such data snapshots to retrieve historical versions of a file. For instance, as illustrated in
Note that the filesystem view presented to users may be configurable to reduce user confusion. More specifically, preserving and displaying information for deleted and archived files and directories may sometimes overwhelm users who only want to see the current set of available files. In some implementations, users may be able to select a set of filesystem views that hide archived and deleted files and directories. In such implementations, for instance, a user may be able to choose whether files that are archived (or deleted) are still displayed in their given directories, or are only displayed in their associated .policy directories. For example, a client may be configured to present a view that displays archived files and allows a typical set of normal file operations (e.g., list the most recent access times and other information) but raises an error and/or status notification whenever a user attempts to access the contents of an archived file.
In many organizations, archived data is typically written once to an archival storage system. Such data may frequently never be read again, but may still need to be preserved (e.g., due to a legal obligation). In such situations, rapid access to the archived data may not be critical, and the restore cost model for an archival cloud storage system may not be an issue. In other organizations, however, archived data may need to sometimes be accessed more quickly and/or may need to be accessed as quickly as possible within a specified set of cost bounds. In such environments, the billing model for an archival cloud storage system may become an issue. Every cloud storage provider may provide different archival storage plans, and the billing models for these plans may change dynamically. Some vendors may provide an API (application programming interface) that allows services such as a distributed filesystem to programmatically check and detect changes to the billing model and parameters (e.g., per-service costs) that are associated with a storage service.
In some embodiments, a distributed filesystem may dynamically track and consider billing models for one or more archival cloud storage systems and adapt archiving and restore behavior accordingly. For instance, the distributed filesystem may be configured to detect updates to the billing models of archival storage providers, and to consider the storage costs incurred by such billing models when making storage decisions. For example, exemplary optimizations made by a distributed filesystem based on billing model parameters may include:
Consider a scenario in which the billing model for an archival cloud storage system calculates the “free” restore rate as a function (e.g., a percentage) of the total amount of archived data being stored. For instance, in one exemplary archival cloud storage system, every gigabyte of data stored may cost a certain amount (e.g., $0.01/gigabyte), and may be billed for a minimum number of months (e.g., a minimum of three months); in this billing model, the customer still pays some ongoing storage costs for data that is written and then immediately deleted. In addition, the amount of data that can be restored by a customer (and the restore rate) may be a function of the total amount of data that is being stored by the customer; for example, if the billing model specifies that 0.1% of the data archived can be restored per month for free, every terabyte of data stored would allow a customer to restore one gigabyte of data free per month at a given transfer rate. As an additional option, if the customer pays $1000/month, the billing model might multiply this restore rate by ten (e.g., the customer can restore 10 gigabytes per month at ten times the original transfer rate). In this scenario, the billing model discourages the deletion of data—deleting half of the archived data effectively halves the amount of data that can be restored and the restore rate, while not necessarily decreasing storage costs (due to the minimum three-months-of-billing requirement). Thus, the distributed filesystem may determine that executing a deletion request might reduce the restore rate without reducing storage costs. Based on such calculations, the distributed filesystem may choose to delay the deletion of data to a later time (e.g., determining that because the cost of storing data for that interval will already be incurred, preserving the deleted data for the full three-month interval allows for a higher restore rate without incurring additional cost).
In another exemplary billing model (called “peak billing”), monthly restore costs for an archival cloud storage system are billed at the maximum restore rate used at any time in the month. More specifically, the cloud storage provider calculates the maximum restore rate (e.g., the maximum data transfer rate) at any time in the month, and charges that rate for the entire month. In this billing model, if even a very small amount of data is downloaded at a high rate at one point in the month, a customer may be billed at a very high rate for the entire month. In such a scenario, the cloud controllers managing the distributed filesystem may need to carefully track all restore operations to ensure that an allocated restore budget is not exceeded. Consider, for instance, a distributed filesystem that includes five terabytes of archived data and is managed by ten cloud controllers. The combined data managed by these cloud controllers enables a larger effective overall restore capacity for the distributed filesystem, but requires the cloud controllers to track and coordinate restore efforts over an ongoing time interval (e.g., perhaps exchanging restore information in conjunction with regular snapshot updates) to ensure that their combined peak restore rate does not exceed a desired (or specified) average rate. Note that such economies of scale can improve perceived user restore performance in comparison with multiple smaller, disjoint filesystems; for example, if only a single user of the distributed filesystem is currently restoring archived data (e.g., other cloud controllers in different timezones are currently idle), that user benefits from a higher effective restore bandwidth than might otherwise be available. Furthermore, the distributed filesystem may also be configured to consider user and/or system hints (e.g., a user indicating that a file being queued to be restored will not actually be needed until the following morning) to schedule restore operations more effectively across the set of cloud controllers.
In general, the cloud controllers managing the distributed filesystem can be configured to consider dynamic billing model parameters and other collected information to maximize restore operation throughput while also minimizing cost. In many scenarios, the distributed filesystem can optimize restore operations in a manner that benefits, but is transparent to, the end users. In particular, it is quite desirable to ensure that the managers of the distributed filesystem are not surprised by unexpected high costs due to restore operations.
In summary, embodiments of the present invention facilitate storing and accessing data in a distributed filesystem. A set of distributed cloud controllers manage data stored in a cloud-based storage system to provide a high-capacity, high-reliability storage system that ensures data consistency. These cloud controllers cache the set of data that is being used by their respective clients, store updates in cloud files on the cloud storage system, and forward updates to each other via incremental snapshots. The data capacity of the system can be easily extended as needed by leasing additional space for the cloud storage system. Hence, the disclosed embodiments present an abstraction of one global, extensible filesystem while preserving the abstraction of high-speed local data access.
Computing Environment
In some embodiments of the present invention, techniques for managing and/or accessing a distributed filesystem can be incorporated into a wide range of computing devices in a computing environment. For example,
Clients 910-912 can include any node on a network that includes computational capability and includes a mechanism for communicating across the network. Additionally, clients 910-912 may comprise a tier in an n-tier application architecture, wherein clients 910-912 perform as servers (servicing requests from lower tiers or users), and wherein clients 910-912 perform as clients (forwarding the requests to a higher tier).
Similarly, servers 930-950 can generally include any node on a network including a mechanism for servicing requests from a client for computational and/or data storage resources. Servers 930-950 can participate in an advanced computing cluster, or can act as stand-alone servers. For instance, computing environment 900 can include a large number of compute nodes that are organized into a computing cluster and/or server farm. In one embodiment of the present invention, server 940 is an online “hot spare” of server 950.
Users 920 and 921 can include: an individual; a group of individuals; an organization; a group of organizations; a computing system; a group of computing systems; or any other entity that can interact with computing environment 900.
Network 960 can include any type of wired or wireless communication channel capable of coupling together computing nodes. This includes, but is not limited to, a local area network, a wide area network, or a combination of networks. In one embodiment of the present invention, network 960 includes the Internet. In some embodiments of the present invention, network 960 includes phone and cellular phone networks.
Database 970 can include any type of system for storing data in non-volatile storage. This includes, but is not limited to, systems based upon magnetic, optical, or magneto-optical storage devices, as well as storage devices based on flash memory and/or battery-backed up memory. Note that database 970 can be coupled: to a server (such as server 950), to a client, or directly to a network. In some embodiments of the present invention, database 970 is used to store information that may later be stored in unused bits of a memory pointer. Alternatively, other entities in computing environment 900 (e.g., servers 930-950) may also store such data.
Devices 980 can include any type of electronic device that can be coupled to a client, such as client 912. This includes, but is not limited to, cell phones, personal digital assistants (PDAs), smartphones, personal music players (such as MP3 players), gaming systems, digital cameras, portable storage media, or any other device that can be coupled to the client. Note that, in some embodiments of the present invention, devices 980 can be coupled directly to network 960 and can function in the same manner as clients 910-912.
Appliance 990 can include any type of appliance that can be coupled to network 960. This includes, but is not limited to, routers, switches, load balancers, network accelerators, and specialty processors. Appliance 990 may act as a gateway, a proxy, or a translator between server 940 and network 960.
Cloud-based storage system 995 can include any type of networked storage devices (e.g., a federation of homogeneous or heterogeneous storage devices) that together provide data storage capabilities to one or more clients.
Note that different embodiments of the present invention may use different system configurations, and are not limited to the system configuration illustrated in computing environment 900. In general, any device that includes computational and storage capabilities may incorporate elements of the present invention.
In some embodiments, computing device 1000 uses receiving mechanism 1006, storage management mechanism 1008, and storage mechanism 1004 to manage data in a distributed filesystem. For instance, storage mechanism 1004 can store metadata for a distributed filesystem, and computing device 1000 can use receiving mechanism 1006 to receive a request to access a data block for a file. Program instructions executing on processor 1002 can traverse the stored metadata to identify a metadata entry that is associated with the data block. Storage management mechanism 1008 can use this metadata entry to download a cloud file containing the data block from a cloud storage system.
In some embodiments of the present invention, some or all aspects of receiving mechanism 1006, storage management mechanism 1008, and/or a filesystem device driver can be implemented as dedicated hardware modules in computing device 1000. These hardware modules can include, but are not limited to, processor chips, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), memory chips, and other programmable-logic devices now known or later developed.
Processor 1002 can include one or more specialized circuits for performing the operations of the mechanisms. Alternatively, some or all of the operations of receiving mechanism 1006, storage management mechanism 1008, and/or a filesystem device driver may be performed using general-purpose circuits in processor 1002 that are configured using processor instructions. Thus, while
In these embodiments, when the external hardware modules are activated, the hardware modules perform the methods and processes included within the hardware modules. For example, in some embodiments of the present invention, the hardware module includes one or more dedicated circuits for performing the operations described above. As another example, in some embodiments of the present invention, the hardware module is a general-purpose computational circuit (e.g., a microprocessor or an ASIC), and when the hardware module is activated, the hardware module executes program code (e.g., BIOS, firmware, etc.) that configures the general-purpose circuits to perform the operations described above.
The foregoing descriptions of various embodiments have been presented only for purposes of illustration and description. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.
This application is a continuation-in-part application of pending U.S. patent applications: (1) Ser. No. 12/772,806 filed on 3 May 2010 by inventors Randy Yen-pang Chou and Ravi Mulam, entitled “Cloud Commands”; and (2) Ser. No. 13/782,729 filed on 1 Mar. 2013 by inventors Randy Yen-pang Chou, John Richard Taylor and Andrew P. Davis, entitled “Facilitating the Recovery of a Virtual Machine Using a Distributed Filesystem”. U.S. patent application Ser. No. 13/782,729 is a continuation-in-part application of pending U.S. patent application Ser. No. 13/769,185 filed on 15 Feb. 2013 by inventors John Richard Taylor, Randy Yen-pang Chou, and Andrew P. Davis, entitled “Customizing Data Management for a Distributed Filesystem”. U.S. patent application Ser. No. 13/769,185 is a continuation-in-part application of pending U.S. patent application Ser. No. 13/725,738 filed on 21 Dec. 2012 by inventors John Richard Taylor, Randy Yen-pang Chou, and Andrew P. Davis, entitled “Distributing Data for a Distributed Filesystem Across Multiple Cloud Storage Systems”. U.S. patent application Ser. No. 13/725,738 is a continuation-in-part application of each of the following U.S. patent applications: (1) Ser. No. 12/772,927 filed on 3 May 2010 by inventors Randy Yen-pang Chou, Ravi Mulam, and Steve Jung, entitled, “Efficient Cloud Network Attached Storage”, now U.S. Pat. No. 8,341,363, issued 25 Dec. 2012; (2) Ser. No. 13/225,194 filed on 2 Sep. 2011 by inventor Randy Yen-pang Chou, entitled “Forwarding Filesystem-Level Information to a Storage Management System”, now U.S. Pat. No. 8,356,016, issued 15 Jan. 2013; and (3) Ser. No. 13/295,844 filed on 14 Nov. 2011 by inventors John Richard Taylor, Randy Yen-pang Chou, and Andrew P. Davis, entitled “Pre-Fetching Data for a Distributed Filesystem”. This application hereby claims priority under 35 U.S.C. §120 to the above-listed U.S. patent applications. The contents of U.S. patent application Ser. Nos. 12/772,806, 13/782,729, 13/769,185, 13/725,738, 12/772,927, 13/225,194, and 13/295,844 are incorporated by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6115357 | Packer | Sep 2000 | A |
| 6466978 | Mukherjee et al. | Oct 2002 | B1 |
| 6697846 | Soltis | Feb 2004 | B1 |
| 7069307 | Lee et al. | Jun 2006 | B1 |
| 7143439 | Cooper | Nov 2006 | B2 |
| 7610285 | Zoellner et al. | Oct 2009 | B1 |
| 7624134 | Stakutis et al. | Nov 2009 | B2 |
| 7657550 | Prahlad et al. | Feb 2010 | B2 |
| 7660829 | Steere et al. | Feb 2010 | B2 |
| 7844582 | Arbilla et al. | Nov 2010 | B1 |
| 7865873 | Zoellner et al. | Jan 2011 | B1 |
| 8296398 | Lacapra et al. | Oct 2012 | B2 |
| 8370312 | Sawhney | Feb 2013 | B1 |
| 8510848 | Zhao | Aug 2013 | B1 |
| 8612596 | Li | Dec 2013 | B1 |
| 20030217119 | Raman | Nov 2003 | A1 |
| 20040243644 | Steere et al. | Dec 2004 | A1 |
| 20050102300 | Madsen | May 2005 | A1 |
| 20060224852 | Kottomtharayil et al. | Oct 2006 | A1 |
| 20070088702 | Fridella et al. | Apr 2007 | A1 |
| 20070094659 | Singh | Apr 2007 | A1 |
| 20070106858 | Galipeau et al. | May 2007 | A1 |
| 20070198613 | Prahlad et al. | Aug 2007 | A1 |
| 20070288525 | Stakutis et al. | Dec 2007 | A1 |
| 20080098043 | Galipeau et al. | Apr 2008 | A1 |
| 20080162608 | Torii et al. | Jul 2008 | A1 |
| 20090055507 | Oeda | Feb 2009 | A1 |
| 20100077142 | Fienblit | Mar 2010 | A1 |
| 20100100698 | Yang et al. | Apr 2010 | A1 |
| 20100114832 | Lillbridge et al. | May 2010 | A1 |
| 20100325377 | Lango et al. | Dec 2010 | A1 |
| 20100333116 | Prahlad et al. | Dec 2010 | A1 |
| 20110066668 | Guarraci | Mar 2011 | A1 |
| 20110276713 | Brand | Nov 2011 | A1 |
| 20120011176 | Aizman | Jan 2012 | A1 |
| 20120089781 | Ranade et al. | Apr 2012 | A1 |
| 20130204849 | Chacko | Aug 2013 | A1 |
| 20140115232 | Goss | Apr 2014 | A1 |
| 20140149695 | Zaslavsky | May 2014 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20130339407 A1 | Dec 2013 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12772806 | May 2010 | US |
| Child | 13971621 | US | |
| Parent | 13782729 | Mar 2013 | US |
| Child | 12772806 | US | |
| Parent | 13769185 | Feb 2013 | US |
| Child | 13782729 | US | |
| Parent | 13725738 | Dec 2012 | US |
| Child | 13769185 | US | |
| Parent | 12772927 | May 2010 | US |
| Child | 13725738 | US | |
| Parent | 13225194 | Sep 2011 | US |
| Child | 12772927 | US | |
| Parent | 13295844 | Nov 2011 | US |
| Child | 13225194 | US |
