Patent Grant
12153690
Examples described herein relate generally to virtualized environments. Examples of systems and techniques are described which facilitate consistent access control lists across file servers for local users. Examples of the use of both a global ID and a local user system identification (SID) are described.
Traditionally, when data is created on a first file server and then migrated at the block level to a second file server, that replication often results in access control list inconsistencies. This is because when the ACLs from the first file server are replicated to the second file server, the destination file system cannot understand or interpret the first file server's data structures.
Certain details are set forth herein to provide an understanding of described embodiments of technology. However, other examples may be practiced without various of these particular details. In some instances, well-known computing system components, virtualization operations, and/or software operations have not been shown in detail in order to avoid unnecessarily obscuring the described embodiments. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here.
Traditionally, when data is created on a first file server and then migrated at the block level to a second file server, that replication often results in access control list inconsistencies. This is because when the ACLs from the first file server are replicated to the second file server, the destination file system cannot understand or interpret the first file server's data structures (and/or security identifies, SIDs, etc.). For example, local users (e.g., users who are not in an Active Directory for the file server) may be given a local user system identification (SID) by the first file server, which identifies the first file server (e.g., has a prefix associated with the first file server). The replicated ACLs become invalid upon migration to the second file server because the second file server cannot interpret the SID for the local user particular to the first file server, which is stamped in the ACL.
Example embodiments described herein are generally directed towards file server access controls, and more specifically towards a mechanism to create consistent access control lists for local users across different file servers in a distributed file server virtual machine (FSVM) environment. In examples described herein, a local user may be provided an SID (e.g., local user system identification (SID); local user security identifier (SID)) by a file server, which is specific to the file server. The file server may communicate with the user using the local user SID. Accordingly, the local user SID may be referred to as an external SID for the user. The local user SID may be specific to the file server which created the local user SID—e.g., some portion of the local user SID (such as a domain prefix portion, suffix portion, or other portion of the local user SID) may be created based on the identity of the file server which created the SID. Typically, the SID would be stored in (e.g., stamped in) any access control list storing access control information for the user. However, if the data is moved to another file server (e.g., at the block level, etc.), the other file server (e.g., the receiving file server, the destination file server, etc.) may be unable to read or interpret the local user SID to give the user access to the appropriate data at the appropriate levels. In some examples, this because the destination file server may not be able to interpret the local user SID specific to the source file server.
Accordingly, in examples described herein, the file server may assign a global ID to a local user. The global ID may be assigned based on a hash function applied to and/or using one or more user attributes of the user. In this manner, the global ID may be specific to the user, but not specific to the file server. The file server may utilize this global ID to determine access controls for the user, and the global ID may be written to (e.g., stamped in) the ACL. This way, when the ACL is moved to another file server (e.g., the receiving file server, the destination file server, etc.), it contains the global ID for the local user. When the local user requests access to data from the destination file server, the local user may be registered with the destination file server. The destination file server may assign (and/or map) a local user SID to the local user, which may be specific to the destination file server. However, the destination file server may additionally and/or alternatively assign a global ID to the local user. The global ID being assigned based on the same hash function of user attributes used by the source file server.
Accordingly, the same global ID will be generated and assigned by the destination file server to the same local user. The destination file server will use the global ID to determine access controls for the local user (e.g., to access the ACL). Accordingly, the correct and consistent access controls may be applied for the local user at the destination file server. As should be appreciated, while a virtualized environment is described in some examples herein, including examples of file servers comprising clusters of computing nodes configured to provide a distributed file system, the systems and methods described herein may be used in other environments (e.g., non-virtualized environments, etc.).
The components shown in
Examples of systems described herein may accordingly include one or more virtualized file servers, such as virtualized file server 106, virtualized file server 114, and virtualized file server 122 in
A virtualized file server may include a cluster of virtual machines and/or other virtualized entities (e.g., containers), which may be referred to as file server virtual machines (FSVMs). In some examples, each of the file server virtual machines of a cluster may be implemented on different computing nodes forming a computing node cluster. For example, the FSVM 108, FSVM 110, and FSVM 112 of virtualized file server 106 may each be implemented on separate computing nodes of a computing node cluster used by the virtualized file server 106. Similarly, the FSVM 116, FSVM 118, and FSVM 120 may each be implemented on separate computing nodes of a computing node cluster used by the virtualized file server 114. Similarly, the FSVM 124, FSVM 126, and FSVM 128 may each be implemented on separate computing nodes of a computing nodes cluster. In some examples, a cluster of FSVMs may be implemented on a cloud computing system.
The FSVMs may operate to provide a file system on the storage resources of the virtualized file server. The file system may have a single namespace and may store data in accordance with filenames and/or directories. The FSVMs may accordingly support one or more file system protocols, such as NFS and/or SMB. A virtualized file server (such as virtualized file server 106, virtualized file server 114, and/or virtualized file server 122) may translate file system protocol requests for one or more files and/or directories (e.g., a file path) into one or more storage requests to access the data corresponding to the file, directory, and/or file path. Any of a variety of components of the virtualized file server may be used to perform the translation (e.g., one or more FSVMs, one or more hypervisors, and/or one or more storage controllers). The translation may be performed using a map (e.g., a shard map) relating the location of the data to the file name, directory, and/or file path.
Virtualized file servers described herein may include virtualized storage. For example, the virtualized file server 106 may include virtualized storage 130. The virtualized file server 114 may include virtualized storage 132. The virtualized file server 122 may include virtualized storage 134. The virtualized storage may generally include any number or kind of storage devices—for example, network attached storage, local storage of one or more computing nodes forming the virtualized file server, and/or cloud storage. Storage devices may be implemented using, for example one or more memories, hard disk drives, solid state drives. The virtualized storage for a particular virtualized file server may be referred to as a storage pool. The virtualized storage may store one or more shares. Generally, the virtualized storage may refer to a storage pool which may include any of a variety of storage devices. In some examples, the virtualized file server(s), may be implemented in a hyperconverged architecture. For example, the storage pool may include local storage devices of the computing nodes used to host the virtualized file server. For example, virtualized storage 130 may include a storage pool. One or more shares of a file system provided by the virtualized file server 106 may be distributed across storage device of the storage pool, including local storage devices of one or more computing nodes on which the FSVM 108, FSVM 110, and/or FSVM 112 reside. In some examples, each file server virtual machine (FSVM) may manage (e.g., host) a corresponding share or a portion of the share. A map may store associations between shares and files, directories, and/or file paths.
Virtualized file servers described herein may include metadata. For example, virtualized file server 106 may include metadata 138. The virtualized file server 114 may include metadata 140. The virtualized file server 122 may include metadata 142. The metadata may be stored, for example, in the virtualized storage and/or other storage location accessible to the virtualized file server. The metadata may in some examples be distributed across the storage pool of a virtualized file server. In some examples, the metadata may be stored in a database accessible to and/or hosted by the virtualized file server. Metadata stored by a virtualized file server may include, for example, authentication information for the virtualized file server and/or virtual machines in the virtualized file server, authorization information for the virtualized file server and/or virtual machines in the virtualized file server, configuration information for the virtualized file server and/or virtual machines in the virtualized file server, end point information (e.g., supported API calls and/or endpoints), a number of shares stored in the virtualized storage of the virtualized file server, a protocol supported by each share and/or FSVM (e.g., NFS and/or SMB), identities of the shares stored in the virtualized storage of the virtualized file server, a number of file server virtual machines (FSVMs) present in the virtualized file server, a number of files and/or directories hosted by the virtualized file server, compute resources available and/or used at the virtualized file server, storage resources available and/or used at the virtualized file server, or other metadata regarding the virtualized file server. The metadata may be maintained by the virtualized file server, for example, the metadata may be updated as the number of shares, FSVMs, storage resources and/or compute resources change.
Examples described herein may include a file server manager, such as file server manager 102 of
File server managers, such as file server manager 102 of
Examples of file server managers described herein may provide one or more user interfaces, such as user interface 104 of
Examples of file server managers described herein may store registration information, such as registration information 144 of
Examples of file server managers described herein may include metadata, such as metadata 136. The metadata may be synchronized to the metadata of multiple virtualized file servers in communication with the file server manager. For example, the metadata 136 may be synchronized with metadata 138, metadata 140, and metadata 142. For example, the metadata 136 at any given time may include metadata 138, metadata 140, and metadata 142. Synchronization may be maintained over time—the metadata of multiple virtualized file servers may periodically (e.g., at regular and/or irregular intervals) synchronize with the metadata store of the file server manager. In this manner, the file server manager 102 may maintain an updated storage of metadata associated with each of virtualized file server 106, virtualized file server 114, and virtualized file server 122. The metadata may be accessed by the file server manager and used to manage, communicate with, and/or otherwise interact with the virtualized file servers.
While the metadata 136 and registration information 144 are depicted separately in
During operation, a file server manager described herein may register, such as by receiving a registration for, one or more virtualized file servers. For example, a virtualized file server (e.g., using an FSVM, a hypervisor, and/or another component of the virtualized file server), may transmit a registration (e.g., registration information) to the file server manager. In some examples, the file server manager may request such a registration by transmitting a request to register to the virtualized file server. In some examples, such as when the file server manager is hosted on a cluster and/or within a same system as the virtualized file server, an automatic registration may occur. For example, the registration process may include determining (e.g., from one or more IP addresses used), that a virtualized file server is hosted on a same domain as a file server manager. In other examples, virtualized file servers which hare not hosted on a same domain as a file server manger may nonetheless register with the file server manager In the example of
The file server manager may synchronize metadata of registered file servers such that up to date metadata of the registered file server may be accessible to the file server manager. For example, the metadata 136 may synchronize with metadata 138, metadata 140, and metadata 142 of
During operation, file server managers described herein, such as file server manager 102 of
The file server manager may format the received management request for the virtualization environment (e.g., virtualization platform) used to host the requested virtualized file server. For example, the file server manager may access the registration information 144 to identify a virtualization environment for a virtualized file server identified in the management request. The management request may then be formatted in a manner used by the virtualized environment. In some examples, the formatted management request may be implemented as an API call, with the API call specific to the virtualization environment of the target virtualized file server. In this manner, clients or other users providing management requests to the file server manager may not require knowledge of the virtualized environment hosting the virtualized file server. The file server manager may format the request in the manner used to communicate with the appropriate virtualization environment. This may provide flexibility in system design and usage, as multiple virtualization environments may be used, and virtualized file servers may in some examples be relocated from one virtualized environment to another without a need to update management requests being provided to the file server manager. Instead, an updated identification of the virtualized environment may be stored in registration information 144 and/or metadata 136.
During operation, the file server manager may utilize information from the registration to implement the management request. For example, access credentials provided during registration may be used to access one or more FSVMs and/or other components of the virtualized file server (e.g., hypervisor, other virtual machine(s) and/or container(s)) and implement the management request. In some examples, the management request may be provided to a particular FSVM. In some examples, the management request may be provided to an FSVM of the virtualized file server that is designated as a leader, and the leader FSVM may communicate the management request to an appropriate FSVM of the virtualized file server.
In some examples, file server managers described herein, such as file server manager 102 of
The virtualized file server 210 includes node 236 and node 254, which may each be a computing node. While only two nodes are shown in
As should be appreciated, the nodes 228 and 246 of the virtualized file server 202 may form a cluster to provide a virtualized file server. The nodes 236 and 254 of virtualized file server 210 may form a cluster to provide a virtualized file server. The virtualized file server 202 may comprise (and/or include) metadata 222. In some examples, metadata 222 may comprise metadata for data stored in the storage pool 218. In some examples, the virtualized file server 210 may comprise (and/or include) metadata 224. In some examples, the metadata 224 may comprise metadata for data stored in the storage pool 220. In some examples, the metadata stored in metadata 222 and/or metadata 224 itself may be stored in (e.g., distributed across) devices in the respective storage pool. In some examples, the metadata 222 may include ACL 226. In some examples, the metadata 224 may include ACL 234. In some examples, ACL 226 may include a local user system identification (SID) and/or global ID. In some examples, metadata 224 may include a local user system identification (SID) and/or global ID. In some examples, the local user SID stored in metadata 222 may be specific to virtualized file server 202. In some examples, the local user SID stored in metadata 224 may be specific to virtualized file server 210. As should be appreciated, additional and/or alternative data and/or metadata used to implement systems and methods described herein may be stored in metadata 222 and/or metadata 224. As should further be appreciated, other additional and/or alternative data and/or metadata may be stored in metadata 222 and/or metadata 224. In some examples, and as should be appreciated, metadata, such as metadata 222 and/or metadata 224 may be associated with one or more files. In some examples, each file may have separate metadata (e.g., a separate ACL) with a different (and/or same) set of SIDs (e.g., local and/or global), as described herein. In some examples, while metadata 222 and metadata 224 are illustrated in
During operation, a local user may be registered with virtualized file server 202. A local user may generally refer to a user who does not have an entry in an Active Directory for the file server, and therefore may not be identified through consulting the Active Directory provided by an operating system of the file server. The local user may, for example, be a user or process that is utilizing and/or accessing the user VM 230 and/or user VM 248. Accordingly, the virtualized file server 202 may assign a local user system identification (e.g., a local user SID) to the local user. The local user SID may be based on an identification of the virtualized file server 202, which is registering the local user. For example, the local user SID may include a prefix, suffix, or other portion, which identifies the virtualized file server 202, or is an indicia representative of virtualized file server 202. In one non-limiting example, a local user SID may be given as SID-1-2-xxx-yyy, where “yyy” is a portion unique for each user, but ‘xxx’ is a portion specific to the file server. In some examples, the “yyy” portion may be referred to as an RID. The FSVM 206, FSVM 208, hypervisor 232, hypervisor 252, and/or other components of the virtualized file server 202 and/or node 228 and node 246 may communicate with the local user using the local user SID.
The virtualized file server 202 may additionally and/or alternatively assign a global ID to the local user. In some examples, the global ID may be based on user attributes associated with the user. In some examples, the user attributes may be provided by the user when registering with the virtualized file server 202 and/or may be accessed by the virtualized file server 202 responsive to a request to register the local user. User attributes which may be used to generate the global ID include, but are not limited to, first name, last name, user name, address, city, phone number, quota, department, company, age, or combinations thereof. The virtualized file server 202 may utilize hash function 242 and/or hash function 250 to generate the global ID for the local user. In some examples, the FSVM 206 may utilize the hash function 242 to generate the global ID and/or FSVM 208 may utilize hash function 205 to generate the global ID. However, as should be appreciated, other components of the virtualized file server 202 may generate the global ID using the hash function 242 and/or the hash function 250 in other examples, such as the hypervisor 232, hypervisor 252, and/or the user VM 230 itself and/or the VM 248 itself. In some examples, a daemon or other container may be provided to generate the global D. The global ID may be stored in ACL 226 and associated with the access controls for the local user. The virtualized file server 202 may communicate with the ACL 226 regarding the local user using the global D.
In this manner, and as further described at least in
In some examples, data from virtualized file server 202 (e.g., from storage pool 218) may be migrated, duplicated, and/or moved to virtualized file server 210. In some examples, the migration, duplication, or other movement of data may occur as a result of a disaster, a planned migration, or other reason. In some examples, the movement of data may be controlled by virtualized file server 202, by a file server manager described herein (e.g., such as file server manager 102 of
In some examples, when the same local user contacts virtualized file server 210 to access data, the local user may be registered with the virtualized file server 210. The virtualized file server 210 may assign the local user a local user SID specific for the virtualized file server 210. As described herein, the local user SID may contain a prefix, suffix, or other portion that identifies or is specific to the virtualized file server 210. Note that the local user SID assigned by virtualized file server 210 will be different from the local user SID assigned to the local user by virtualized file server 202. For example, recall that the local user may be assigned a local user SID, such as SID-1-2-xxx-yyy, from virtualized file server 202, e.g., at registration. In some examples, virtualized file server 210 may assign that same local user local user SID, such as SID-1-2-aaa-yyy, e.g., at registration. Recall that in some examples, the “xxx” and “aaa” portions may be specific to the virtualized file server, and the “yyy” portion may be specific to the local user.
Additionally and/or alternatively to the local user SID, the virtualized file server 210 may also assign the local user a global ID using hash function 244 and/or has function 258. In some examples, hash function 244 and the hash function 258, are the same hash functions as hash function 242 and hash function 250. Accordingly, the virtualized file server 210 will generate the same global ID for the same local user based on local user attributes, and not based on file server identification. In some examples, the virtualized file server 210 may communicate with the local user using the local user SID, but may communicate with the storage and determine access control for the local user based on global ID. Accordingly, the virtualized file server 210 will access the ACL 234 based on the global ID to determine access control for the local user, and the determined access control will accordingly be the same and accurate as was established by the virtualized file server 202.
Operationally, and in some examples, each file server (e.g., virtualized file server 202, virtualized file server 210, etc.) may store a mapping between the local user SID of local users registered with the file server and the global IDs of those local users. The mapping may be stored in a database hosted by and/or accessible to the file server, which may be a distributed database. For example, the virtualized file server 202 may have a database of local user SID that map to (e.g., ↔) global ID mappings. In some examples, and as described herein at least in
For example, a mapping may be stored between RID (e.g., component of the local user SID, such as the “yyy” component in the above-described example) and/or the entire local user SID and a user name. Another mapping may be stored between the user name and user information (e.g., a UID and/or external SID). Another mapping may be stored between external SID and a user name. Another mapping may be stored between a user ID and a user name. Another mapping may be stored between SID and/or RID and global ID. As should be appreciated, any and/or all of these mappings may be stored, as well as additional and/or alternative mappings may be stored and are contemplated to be within the scope of this disclosure. In some examples, the global ID (e.g., the global ID that was translated from the local user SID) may be matched to the ACL to determine access to the file for the user at the second file server.
In one non-limiting example, a first file server may generate a first local user SID and a global ID for user A, a user of the first file server. The first local user SID may be specific to the first file server, while the global ID may be generated via a hash function using attributes of user A. In some examples, the virtualized file server may generate the global ID based on user A's name, though other user attributes may be used. In some examples, the local user SID and/or the global ID for user A may be used during access and authentication requests. The local user SID and/or the global ID for user A may be stamped in (e.g., stored) in an ACL that in some examples may include and/or comprise control and/or authentication information for user A. In some examples, data from the first file server may be migrated and/or replicated, etc. from the first file server to a second file server. Continuing with this example, upon registering with the second file server, user A may in some examples be assigned a second local user SID specific to the second file server, as well as a global ID specific to user A based on user attributes using a hash function. In this example, upon receiving an access request from user A to access the data (e.g., files, etc.) on the second file server and migrated from the first file server, the second file server will map the second local user SID for user A to the global ID for user A. Using this mapping, the second file server may in some examples determine access to the data for user A. in some examples, the determination may be based on a matching, such as matching the global ID to the ACL to determine access to the file for the user.
As should be appreciated, in some examples, virtualized file server 202 may comprise a cluster of nodes, such as nodes 228 and 246. In some examples, each node in the cluster may be a computing node. In some examples, virtualized file server 210 may comprise a cluster of nodes, such as nodes 236 and 254. In some examples, each node in the cluster may also be a computing node. In some examples, the cluster of nodes in virtualized file server 202 may be configured to provide a distributed file system. In some examples, the cluster of nodes in virtualized file server 210 may also be configured to provide the same and/or a different distributed file system. In some examples, the file servers described herein may comprise nodes, where each node of the cluster of computing nodes comprises one or more file server virtual machines (FSVMs). In some examples, the FSVMs in the cluster of computing nodes may be configured to present a single namespace of storage items distributed across a storage pool. In some examples, the storage pool may include local storage devices of the cluster of nodes (e.g., of the cluster of computing nodes).
Turning now to
Recall that, in some examples, a user may wish to access a file (e.g., a file migrated from a source file server) at a destination file server. Here, the user may interact with the destination file server using a local user ID specific to the destination file server, but the destination file server may determine access using the user's global ID specific to the user. As such, and in some examples, an external SID (e.g., a local user SID) may be received in block 302. In some examples, the local user SID may be provided by, e.g., one or more user VMs, other virtualized file servers, file managers, admin systems, or other components associated with a request to a virtualized file server (e.g., to an FSVM or other component of a virtualized file server).
In some examples, to determine access to the requested file, the destination file server must convert (e.g., translate, map, etc.), the received local user SID to a global ID. In some examples, and at block 304, the virtualized file server may look up a name entry associated with the received local user SID (e.g., associated with the RID portion of the SID). The virtualized file server may access a mapping between SID and/or RID and user name. However, as should be appreciated, the virtualized file server may additionally and/or alternatively access a mapping between the local user SID and/or RID and a first name, last name, user name, address, city, phone number, quota, department, company, age, or combinations thereof.
In block 306, the virtualized file server may look up user information associated with the user name determined in block 304. For example, the virtualized file server (e.g., one or more FSVMs or other component of the virtualized file server, such as FSVM 214 of node 236 and/or FSVM 216 of node 254, both of virtualized file server 210) may access a mapping between user information and user name. In some examples, the mapping may be stored in a database communicatively coupled to the file server, storage pool 220, and/or in metadata, such as metadata 224 of
In block 308, the virtualized file server may return a global ID (e.g., a local user SID) associated with that user information. In this manner, a global ID (internal SID) may be returned responsive to receipt of a local user's local user SID. The global ID may be used to access one or more ACLs and determine access control information and/or authentication for a user. The internal ACLs may be used to store the ACL on disk. In some examples, the process of
In some examples, and as described herein, a security token may be generated when the user logs in. In some examples, the security token may include and/or comprise the global ID of the user.
In some the examples, the global ID in the security token may be used to evaluate access rights of the user to one or more resources (e.g., files, directories, and the like). In some examples, the evaluation may be based on a comparison of the global ID to the ACEs stored in the ACL in the metadata of the file and/or directory.
In block 310, an internal SID (e.g., a global ID) is provided. For example, an internal SID may be provided by a virtualized file server when accessing an ACL and/or when returning accessed files or other data based on a global ID. For example, upon a user of a virtualized file server requesting access to a file, the virtualized file server may use the internal SID to determine access to that file. In some examples, the internal SID may be used to determine authentication and/or other access controls, and the like.
In block 312, a user name may be looked up (e.g., using stored mappings as described herein) based on the internal SID or global ID. In some examples, the mapping may be stored in a database or other storage, such as, for example, storage pool 218 and/or metadata 222 of
In block 314, user information (e.g., user attributes) may be looked up based on the user name determined in block 312. For example, a virtualized file server (e.g., one or more FSVMs or other components, such as virtualized file server 202 of
In block 316, an external SID may be provided by a virtualized file server based on the user information. For example, a virtualized file server (e.g., one or more FSVMs or other components, such as virtualized file server 202 of
In some examples, a client may provide an ACL with external SIDs, as illustrated in block 318. Recall, in some examples, users of a virtualized file server (e.g., registered users) may be assigned both a local user ID specific to the file server, and a global ID based on user attributes via a hash function. In some examples, an ACL with external SIDs (e.g., local user SIDs) may sometimes have been stored in metadata 222 and/or metadata 224 of
In some examples, and as illustrated in block 320, an ACL using internal SIDs may be provided based, in some examples, on a conversion of the local user SID to a global ID. For example, the virtualized file server may convert the external SIDs specific to the virtualized file server into global IDs (e.g., internal IDs) using the process described with reference to
Recall that, in some examples, a user may communicate and/or interact with a virtualized file server (e.g., a destination file server, a source file server, etc.) using the user's local user SID, while the virtualized file sever may determines access using the user's global ID. In some examples, once an access and/or authentication is determined, the virtualized file server may need to continue communication with the user. Accordingly, and in some examples, metadata 324 may be stored and may utilize internal SIDs (e.g., global SIDs). The metadata 324 may be used to implement and/or may be implemented by metadata 322 of
In some examples, the metadata 324 may be read in block 326 (e.g., an FSVM or other component of a virtualized file server may access the 324). The ACL with internal SIDs may be obtained in block 326. As described, the virtualized file server may determine access using the user's global ID (e.g., internal ID).
In some examples, and in block 328, the ACL with internal SIDs may be converted to an ACL with external SIDs by replacing the internal SIDs with external SIDs. For example, the process shown in
In some examples, and as described herein, a local user may be authenticated in block 330. For example, a virtualized file server and/or other virtualized environment may authenticate a local user. The authentication process may result in a token 332, which contains an external SID (e.g., an ID specific to the file server). In some examples, the external ID contained within the token 332 may present as SID-1-2-xxx-yyy, where the “xxx” portion is file server specific, and the “yyy” portion is user specific.
In block 334, the userID may be replaced, such as by looking up the global ID associated with the external SID in a process such as described with reference to
As should be appreciated, and as described herein, examples of systems and methods described herein may include a file server manager in communication with one or more virtualized file servers. In some examples, the file server manager may be and/or include the functionality of file server manager 102 of
The clustered virtualization environment 400 and/or VFS 432 may be used to implement one or more virtualization platforms and/or virtualized file servers described herein, such as the virtualized file server 106, virtualized file server 114, and/or virtualized file server 122 of
The architectures of
Each host machine 402, 416, 408 may run virtualization software, such as VMWARE ESX(I), MICROSOFT HYPER-V, or REDHAT KVM. The virtualization software includes hypervisors 442, 444, and 446 to create, manage, and destroy user VMs, as well as managing the interactions between the underlying hardware and user VMs. User VMs may run one or more applications that may operate as “clients” with respect to other elements within clustered virtualization environment 400. A hypervisor may connect to network 454. In particular embodiments, a host machine 402, 408, or 416 may be a physical hardware computing device; in particular embodiments, a host machine 402, 408, or 416 may be a virtual machine.
CVMs 436, 438, and 440 are used to manage storage and input/output (“I/O”) activities according to particular embodiments. These special VMs act as the storage controller in the currently described architecture. Multiple such storage controllers may coordinate within a cluster to form a unified storage controller system. CVMs may run as virtual machines on the various host machines, and work together to form a distributed system that manages all the storage resources, including local storage, network-attached storage 410, and cloud storage 406. The CVMs may connect to network 454 directly, or via a hypervisor. Since the CVMs run independent of hypervisors 442, 444, 446, this means that the current approach can be used and implemented within any virtual machine architecture, since the CVMs of particular embodiments can be used in conjunction with any hypervisor from any virtualization vendor. In some examples, CVMs may not be used and one or more hypervisors (e.g., hypervisors 442, 444, and/or 446) may perform the functions described with respect to the CVMs. In some examples, one or more CVMs may not be present, and the hypervisor or other component hosted on the computing nodes may provide the functions attributed to the CVM herein.
A host machine may be designated as a leader node within a cluster of host machines. For example, host machine 408, may be a leader node. A leader node may have a software component designated to perform operations of the leader. For example, CVM 438 on host machine 408 may be designated to perform such operations. A leader may be responsible for monitoring or handling requests from other host machines or software components on other host machines throughout the virtualized environment. If a leader fails, a new leader may be designated. In particular embodiments, a management module (e.g., in the form of an agent) may be running on the leader node and/or in communication with the leader node or virtual machines or containers on the leader node. For example, file server managers described herein may be in communication with the leader node in some examples.
Each CVM 436, 438, and 440 exports one or more block devices or NFS server targets that appear as disks to user VMs 414, 418, 422, 426, 430, and 434. These disks are virtual, since they are implemented by the software running inside CVMs 436, 438, and 440. Thus, to user VMs, CVMs appear to be exporting a clustered storage appliance that contains some disks. All user data (including the operating system) in the user VMs may reside on these virtual disks.
Significant performance advantages can be gained by allowing the virtualization system to access and utilize local storage 448, 450, and 452 as disclosed herein. This is because I/O performance is typically much faster when performing access to local storage as compared to performing access to network-attached storage 410 across a network 454. This faster performance for locally attached storage can be increased even further by using certain types of optimized local storage devices, such as SSDs. Further details regarding methods and mechanisms for implementing the virtualization environment illustrated in
As a user VM performs I/O operations (e.g., a read operation or a write operation), the I/O commands of the user VM may be sent to the hypervisor that shares the same server as the user VM. For example, the hypervisor may present to the virtual machines an emulated storage controller, receive an I/O command and facilitate the performance of the I/O command (e.g., via interfacing with storage that is the object of the command, or passing the command to a service that will perform the I/O command). An emulated storage controller may facilitate I/O operations between a user VM and a vDisk. A vDisk may present to a user VM as one or more discrete storage drives, but each vDisk may correspond to any part of one or more drives within storage pool 456. Additionally or alternatively, CVMs 436, 438, 440 may present an emulated storage controller either to the hypervisor or to user VMs to facilitate I/O operations. CVMs 436, 438, and 440 may be connected to storage within storage pool 456. CVM 436 may have the ability to perform I/O operations using local storage 448 within the same host machine 402, by connecting via network 454 to cloud storage 406 or network-attached storage 410, or by connecting via network 454 to local storage 450 or 452 within another host machine 408 or 416 (e.g., via connecting to another CVM 438 or 440). In particular embodiments, any suitable computing system may be used to implement a host machine.
In particular embodiments, the VFS 432 may include a set of File Server Virtual Machines (FSVMs) 404, 412, and 420 that execute on host machines 402, 408, and 416 and process storage item access operations requested by user VMs executing on the host machines 402, 408, and 416. The FSVMs 404, 412, and 420 may communicate with storage controllers provided by CVMs 436, 444, 440 and/or hypervisors executing on the host machines 402, 408, 416 to store and retrieve files, folders, SMB shares, or other storage items on local storage 448, 450, 452 associated with, e.g., local to, the host machines 402, 408, 416. The FSVMs 404, 412, 420 may store and retrieve block-level data on the host machines 402, 408, 416, e.g., on the local storage 448, 450, 452 of the host machines 402, 408, 416. The block-level data may include block-level representations of the storage items (e.g., files). The network protocol used for communication between user VMs, FSVMs, and CVMs via the network 454 may be Internet Small Computer Systems Interface (iSCSI), Server Message Block (SMB), Network Filesystem (NFS), pNFS (Parallel NFS), or another appropriate protocol.
For the purposes of VFS 432, host machine 416 may be designated as a leader node within a cluster of host machines. In this case, FSVM 420 on host machine 416 may be designated to perform such operations. A leader may be responsible for monitoring or handling requests from FSVMs on other host machines throughout the virtualized environment. If FSVM 420 fails, a new leader may be designated for VFS 432.
In particular embodiments, the user VMs may send data to the VFS 432 (e.g., to the FSVMs) using write requests, and may receive data from it using read requests. The read and write requests, and their associated parameters, data, and results, may be sent between a user VM and one or more file server VMs (FSVMs) located on the same host machine as the user VM or on different host machines from the user VM. The read and write requests may be sent between host machines 402, 408, 416 via network 454, e.g., using a network communication protocol such as iSCSI, CIFS, SMB, TCP, IP, or the like. When a read or write request is sent between two VMs located on the same one of the host machines 402, 408, 416 (e.g., between the user VM 414 and the FSVM 404 located on the host machine 402), the request may be sent using local communication within the host machine 402 instead of via the network 454. As described above, such local communication may be substantially faster than communication via the network 454. The local communication may be performed by, e.g., writing to and reading from shared memory accessible by the user VM 414 and the FSVM 404, sending and receiving data via a local “loopback” network interface, local stream communication, or the like.
In particular embodiments, the storage items stored by the VFS 432, such as files and folders, may be distributed amongst multiple FSVMs 404, 412, 420. In particular embodiments, when storage access requests are received from the user VMs, the VFS 432 identifies FSVMs 404, 412, 420 at which requested storage items, e.g., folders, files, or portions thereof, are stored, and directs the user VMs to the locations of the storage items. The FSVMs 404, 412, 420 may maintain a storage map, such as a sharding map, that maps names or identifiers of storage items to their corresponding locations. The storage map may be a distributed data structure of which copies are maintained at each FSVM 404, 412, 420 and accessed using distributed locks or other storage item access operations. Alternatively, the storage map may be maintained by an FSVM at a leader node such as the FSVM 420, and the other FSVMs 404 and 412 may send requests to query and update the storage map to the leader FSVM 420. Other implementations of the storage map are possible using appropriate techniques to provide asynchronous data access to a shared resource by multiple readers and writers. The storage map may map names or identifiers of storage items in the form of text strings or numeric identifiers, such as folder names, files names, and/or identifiers of portions of folders or files (e.g., numeric start offset positions and counts in bytes or other units) to locations of the files, folders, or portions thereof. Locations may be represented as names of FSVMs, e.g., “FSVM-1”, as network addresses of host machines on which FSVMs are located (e.g., “ip-addr1” or 128.1.1.10), or as other types of location identifiers.
When a user application executing in a user VM 414 on one of the host machines 402 initiates a storage access operation, such as reading or writing data, the user VM 414 may send the storage access operation in a request to one of the FSVMs 404, 412, 420 on one of the host machines 402, 408, 416. A FSVM 412 executing on a host machine 408 that receives a storage access request may use the storage map to determine whether the requested file or folder is located on the FSVM 412. If the requested file or folder is located on the FSVM 412, the FSVM 412 executes the requested storage access operation. Otherwise, the FSVM 412 responds to the request with an indication that the data is not on the FSVM 412, and may redirect the requesting user VM 414 to the FSVM on which the storage map indicates the file or folder is located. The client may cache the address of the FSVM on which the file or folder is located, so that it may send subsequent requests for the file or folder directly to that FSVM.
As an example and not by way of limitation, the location of a file or a folder may be pinned to a particular FSVM 404 by sending a file service operation that creates the file or folder to a CVM 436 and/or hypervisor 442 associated with (e.g., located on the same host machine as) the FSVM 404. The CVM 436 subsequently processes file service commands for that file for the FSVM 404 and sends corresponding storage access operations to storage devices associated with the file. The CVM 436 may associate local storage 448 with the file if there is sufficient free space on local storage 448. Alternatively, the CVM 436 may associate a storage device located on another host machine 402, e.g., in local storage 450, with the file under certain conditions, e.g., if there is insufficient free space on the local storage 448, or if storage access operations between the CVM 436 and the file are expected to be infrequent. Files and folders, or portions thereof, may also be stored on other storage devices, such as the network-attached storage (NAS) network-attached storage 410 or the cloud storage 406 of the storage pool 456.
In particular embodiments, a name service 424, such as that specified by the Domain Name System (DNS) Internet protocol, may communicate with the host machines 402, 408, 416 via the network 454 and may store a database of domain name (e.g., host name) to IP address mappings. The domain names may correspond to FSVMs, e.g., fsvm1.domain.com or ip-addr1.domain.com for an FSVM named FSVM-1. The name service 424 may be queried by the user VMs to determine the IP address of a particular host machine 402, 408, 416 given a name of the host machine, e.g., to determine the IP address of the host name ip-addr1 for the host machine 402. The name service 424 may be located on a separate server computer system or on one or more of the host machines 402, 408, 416. The names and IP addresses of the host machines of the VFS 432, e.g., the host machines 402, 408, 416, may be stored in the name service 424 so that the user VMs may determine the IP address of each of the host machines 402, 408, 416, or FSVMs 404, 412, 420. The name of each VFS instance, e.g., each file system such as FS1, FS2, or the like, may be stored in the name service 424 in association with a set of one or more names that contains the name(s) of the host machines 402, 408, 416 or FSVMs 404, 412, 420 of the VFS instance VFS 432. The FSVMs 404, 412, 420 may be associated with the host names ip-addr1, ip-addr2, and ip-addr3, respectively. For example, the file server instance name FS1.domain.com may be associated with the host names ip-addr1, ip-addr2, and ip-addr3 in the name service 424, so that a query of the name service 424 for the server instance name “FS1” or “FS1.domain.com” returns the names ip-addr1, ip-addr2, and ip-addr3. As another example, the file server instance name FS1.domain.com may be associated with the host names fsvm-1, fsvm-2, and fsvm-3. Further, the name service 424 may return the names in a different order for each name lookup request, e.g., using round-robin ordering, so that the sequence of names (or addresses) returned by the name service for a file server instance name is a different permutation for each query until all the permutations have been returned in response to requests, at which point the permutation cycle starts again, e.g., with the first permutation. In this way, storage access requests from user VMs may be balanced across the host machines, since the user VMs submit requests to the name service 424 for the address of the VFS instance for storage items for which the user VMs do not have a record or cache entry, as described below.
In particular embodiments, each FSVM may have two IP addresses: an external IP address and an internal IP address. The external IP addresses may be used by SMB/CIFS clients, such as user VMs, to connect to the FSVMs. The external IP addresses may be stored in the name service 424. The IP addresses ip-addr1, ip-addr2, and ip-addr3 described above are examples of external IP addresses. The internal IP addresses may be used for iSCSI communication to CVMs and/or hypervisors, e.g., between the FSVMs 404, 412, 420 and the CVMs 436, 444, 440 and/or hypervisors 442, 444, and/or 446. Other internal communications may be sent via the internal IP addresses as well, e.g., file server configuration information may be sent from the CVMs to the FSVMs using the internal IP addresses, and the CVMs may get file server statistics from the FSVMs via internal communication as needed.
Since the VFS 432 is provided by a distributed set of FSVMs 404, 412, 420, the user VMs that access particular requested storage items, such as files or folders, do not necessarily know the locations of the requested storage items when the request is received. A distributed file system protocol, e.g., MICROSOFT DFS or the like, is therefore used, in which a user VM 414 may request the addresses of FSVMs 404, 412, 420 from a name service 424 (e.g., DNS). The name service 424 may send one or more network addresses of FSVMs 404, 412, 420 to the user VM 414, in an order that changes for each subsequent request. These network addresses are not necessarily the addresses of the FSVM 412 on which the storage item requested by the user VM 414 is located, since the name service 424 does not necessarily have information about the mapping between storage items and FSVMs 404, 412, 420. Next, the user VM 414 may send an access request to one of the network addresses provided by the name service, e.g., the address of FSVM 412. The FSVM 412 may receive the access request and determine whether the storage item identified by the request is located on the FSVM 412. If so, the FSVM 412 may process the request and send the results to the requesting user VM 414. However, if the identified storage item is located on a different FSVM 420, then the FSVM 412 may redirect the user VM 414 to the FSVM 420 on which the requested storage item is located by sending a “redirect” response referencing FSVM 420 to the user VM 414. The user VM 414 may then send the access request to FSVM 420, which may perform the requested operation for the identified storage item.
A particular virtualized file server, such as VFS 432, including the items it stores, e.g., files and folders, may be referred to herein as a VFS “instance” and/or a file system and may have an associated name, e.g., FS1, as described above. Although a VFS instance may have multiple FSVMs distributed across different host machines, with different files being stored on FSVMs, the VFS instance may present a single name space to its clients such as the user VMs. The single name space may include, for example, a set of named “shares” and each share may have an associated folder hierarchy in which files are stored. Storage items such as files and folders may have associated names and metadata such as permissions, access control information, size quota limits, file types, files sizes, and so on. As another example, the name space may be a single folder hierarchy, e.g., a single root directory that contains files and other folders. User VMs may access the data stored on a distributed VFS instance via storage access operations, such as operations to list folders and files in a specified folder, create a new file or folder, open an existing file for reading or writing, and read data from or write data to a file, as well as storage item manipulation operations to rename, delete, copy, or get details, such as metadata, of files or folders. Note that folders may also be referred to herein as “directories.”
In particular embodiments, storage items such as files and folders in a file server namespace may be accessed by clients such as user VMs by name, e.g., “\Folder-1\File-1” and “\Folder-2\File-2” for two different files named File-1 and File-2 in the folders Folder-1 and Folder-2, respectively (where Folder-1 and Folder-2 are sub-folders of the root folder). Names that identify files in the namespace using folder names and file names may be referred to as “path names.” Client systems may access the storage items stored on the VFS instance by specifying the file names or path names, e.g., the path name “\Folder-1\File-1”, in storage access operations. If the storage items are stored on a share (e.g., a shared drive), then the share name may be used to access the storage items, e.g., via the path name “\\Share-1\Folder-1\File-1” to access File-1 in folder Folder-1 on a share named Share-1.
In particular embodiments, although the VFS instance may store different folders, files, or portions thereof at different locations, e.g., on different FSVMs, the use of different FSVMs or other elements of storage pool 456 to store the folders and files may be hidden from the accessing clients. The share name is not necessarily a name of a location such as an FSVM or host machine. For example, the name Share-1 does not identify a particular FSVM on which storage items of the share are located. The share Share-1 may have portions of storage items stored on three host machines, but a user may simply access Share-1, e.g., by mapping Share-1 to a client computer, to gain access to the storage items on Share-1 as if they were located on the client computer. Names of storage items, such as file names and folder names, are similarly location-independent. Thus, although storage items, such as files and their containing folders and shares, may be stored at different locations, such as different host machines, the files may be accessed in a location-transparent manner by clients (such as the user VMs). Thus, users at client systems need not specify or know the locations of each storage item being accessed. The VFS may automatically map the file names, folder names, or full path names to the locations at which the storage items are stored. As an example and not by way of limitation, a storage item's location may be specified by the name, address, or identity of the FSVM that provides access to the storage item on the host machine on which the storage item is located. A storage item such as a file may be divided into multiple parts that may be located on different FSVMs, in which case access requests for a particular portion of the file may be automatically mapped to the location of the portion of the file based on the portion of the file being accessed (e.g., the offset from the beginning of the file and the number of bytes being accessed).
In particular embodiments, VFS 432 determines the location, e.g., FSVM, at which to store a storage item when the storage item is created. For example, a FSVM 404 may attempt to create a file or folder using a CVM 436 on the same host machine 402 as the user VM 418 that requested creation of the file, so that the CVM 436 that controls access operations to the file folder is co-located with the user VM 418. In this way, since the user VM 418 is known to be associated with the file or folder and is thus likely to access the file again, e.g., in the near future or on behalf of the same user, access operations may use local communication or short-distance communication to improve performance, e.g., by reducing access times or increasing access throughput. If there is a local CVM on the same host machine as the FSVM, the FSVM may identify it and use it by default. If there is no local CVM on the same host machine as the FSVM, a delay may be incurred for communication between the FSVM and a CVM on a different host machine. Further, the VFS 432 may also attempt to store the file on a storage device that is local to the CVM being used to create the file, such as local storage, so that storage access operations between the CVM and local storage may use local or short-distance communication.
In particular embodiments, if a CVM is unable to store the storage item in local storage of a host machine on which an FSVM resides, e.g., because local storage does not have sufficient available free space, then the file may be stored in local storage of a different host machine. In this case, the stored file is not physically local to the host machine, but storage access operations for the file are performed by the locally-associated CVM and FSVM, and the CVM may communicate with local storage on the remote host machine using a network file sharing protocol, e.g., iSCSI, SAMBA, or the like.
In particular embodiments, if a virtual machine, such as a user VM 414, CVM 436, or FSVM 404, moves from a host machine 402 to a destination host machine 408, e.g., because of resource availability changes, and data items such as files or folders associated with the VM are not locally accessible on the destination host machine 408, then data migration may be performed for the data items associated with the moved VM to migrate them to the new host machine 408, so that they are local to the moved VM on the new host machine 408. FSVMs may detect removal and addition of CVMs (as may occur, for example, when a CVM fails or is shut down) via the iSCSI protocol or other technique, such as heartbeat messages. As another example, a FSVM may determine that a particular file's location is to be changed, e.g., because a disk on which the file is stored is becoming full, because changing the file's location is likely to reduce network communication delays and therefore improve performance, or for other reasons. Upon determining that a file is to be moved, VFS 432 may change the location of the file by, for example, copying the file from its existing location(s), such as local storage 448 of a host machine 402, to its new location(s), such as local storage 450 of host machine 408 (and to or from other host machines, such as local storage 452 of host machine 416 if appropriate), and deleting the file from its existing location(s). Write operations on the file may be blocked or queued while the file is being copied, so that the copy is consistent. The VFS 432 may also redirect storage access requests for the file from an FSVM at the file's existing location to a FSVM at the file's new location.
In particular embodiments, VFS 432 includes at least three File Server Virtual Machines (FSVMs) 404, 412, 420 located on three respective host machines 402, 408, 416. To provide high-availability, there may be a maximum of one FSVM for a particular VFS instance VFS 432 per host machine in a cluster. If two FSVMs are detected on a single host machine, then one of the FSVMs may be moved to another host machine automatically, or the user (e.g., system administrator and/or file server manager) may be notified to move the FSVM to another host machine. The user and/or file server manager may move a FSVM to another host machine using an administrative interface that provides commands for starting, stopping, and moving FSVMs between host machines.
In particular embodiments, two FSVMs of different VFS instances may reside on the same host machine. If the host machine fails, the FSVMs on the host machine become unavailable, at least until the host machine recovers. Thus, if there is at most one FSVM for each VFS instance on each host machine, then at most one of the FSVMs may be lost per VFS per failed host machine. As an example, if more than one FSVM for a particular VFS instance were to reside on a host machine, and the VFS instance includes three host machines and three FSVMs, then loss of one host machine would result in loss of two-thirds of the FSVMs for the VFS instance, which would be more disruptive and more difficult to recover from than loss of one-third of the FSVMs for the VFS instance.
In particular embodiments, users, such as system administrators or other users of the user VMs, may expand the cluster of FSVMs by adding additional FSVMs. Each FSVM may be associated with at least one network address, such as an IP (Internet Protocol) address of the host machine on which the FSVM resides. There may be multiple clusters, and all FSVMs of a particular VFS instance are ordinarily in the same cluster. The VFS instance may be a member of a MICROSOFT ACTIVE DIRECTORY domain, which may provide authentication and other services such as name service.
As an example, a network share may be presented to a user VM as one or more discrete virtual disks, but each virtual disk may correspond to any part of one or more virtual or physical disks within a storage pool. Additionally or alternatively, the FSVMs may present a VFS either to the hypervisor or to user VMs of a host machine to facilitate I/O operations. The FSVMs may access the local storage via Controller/Service VMs, other storage controllers, hypervisors, or other components of the host machine. As described herein, a CVM 436 may have the ability to perform I/O operations using local storage 448 within the same host machine 402 by connecting via the network 454 to cloud storage or NAS, or by connecting via the network 454 to 450, 452 within another host machine 408, 416 (e.g., by connecting to another CVM 438, 440).
In particular embodiments, each user VM may access one or more virtual disk images stored on one or more disks of the local storage, the cloud storage, and/or the NAS. The virtual disk images may contain data used by the user VMs, such as operating system images, application software, and user data, e.g., user home folders and user profile folders. For example,
In particular embodiments, since local communication is expected to be more efficient than remote communication, the FSVMs may store storage items, such as files or folders, e.g., the virtual disk images, as block-level data on local storage of the host machine on which the user VM that is expected to access the files is located. A user VM may be expected to access particular storage items if, for example, the storage items are associated with the user VM, such as by configuration information. For example, the virtual disk image 510 may be associated with the user VM 414 by configuration information of the user VM 414. Storage items may also be associated with a user VM via the identity of a user of the user VM. For example, files and folders owned by the same user ID as the user who is logged into the user VM 414 may be associated with the user VM 414. If the storage items expected to be accessed by a user VM 414 are not stored on the same host machine 402 as the user VM 414, e.g., because of insufficient available storage capacity in local storage 448 of the host machine 402, or because the storage items are expected to be accessed to a greater degree (e.g., more frequently or by more users) by a user VM 422 on a different host machine 408, then the user VM 414 may still communicate with a local CVM 436 to access the storage items located on the remote host machine 408, and the local CVM 436 may communicate with local storage 450 on the remote host machine 408 to access the storage items located on the remote host machine 408. If the user VM 414 on a host machine 402 does not or cannot use a local CVM 436 to access the storage items located on the remote host machine 408, e.g., because the local CVM 436 has crashed or the user VM 414 has been configured to use a remote CVM 438, then communication between the user VM 414 and local storage 450 on which the storage items are stored may be via a remote CVM 438 using the network 454, and the remote CVM 438 may access local storage 450 using local communication on host machine 408. As another example, a user VM 414 on a host machine 402 may access storage items located on a disk 506 of local storage 452 on another host machine 416 via a CVM 438 on an intermediary host machine 408 using network communication between the host machines 402 and 408 and between the host machines 408 and 416.
In particular embodiments, file systems FileSystem-1A 742 and FileSystem-2A 740 implement the structure of files and folders for portions of the FS1 and FS2 file server instances, respectively, that are located on (e.g., served by) FileServer-VM-1 720 on Host-1 704. Other file systems on other host machines may implement other portions of the FS1 and FS2 file server instances. The file systems 742 and 740 may implement the structure of at least a portion of a file server instance by translating file system operations, such as opening a file, writing data to or reading data from the file, deleting a file, and so on, to disk I/O operations such as seeking to a portion of the disk, reading or writing an index of file information, writing data to or reading data from blocks of the disk, allocating or de-allocating the blocks, and so on. The file systems 742, 740 may thus store their file system data, including the structure of the folder and file hierarchy, the names of the storage items (e.g., folders and files), and the contents of the storage items on one or more storage devices, such as local storage 728. The particular storage device or devices on which the file system data for each file system are stored may be specified by an associated file system pool (e.g., 748 and 750). For example, the storage device(s) on which data for FileSystem-1A 742 and FileSystem-2A, 740 are stored may be specified by respective file system pools FS1-Pool-1 748 and FS2-Pool-2 750. The storage devices for the pool may be selected from volume groups provided by CVM-1 724, such as volume group VG1 732 and volume group VG2 734. Each volume group 732, 734 may include a group of one or more available storage devices that are present in local storage 728 associated with (e.g., by iSCSI communication) the CVM-1 724. The CVM-1 724 may be associated with a local storage 728 on the same host machine 402 as the CVM-1 724, or with a local storage 730 on a different host machine 706. The CVM-1 724 may also be associated with other types of storage, such as cloud storage, networked storage or the like. Although the examples described herein include particular host machines, virtual machines, file servers, file server instances, file server pools, CVMs, volume groups, and associations there between, any number of host machines, virtual machines, file servers, file server instances, file server pools, CVMs, volume groups, and any associations there between are possible and contemplated.
In particular embodiments, the file system pool 748 may associate any storage device in one of the volume groups 732, 734 of storage devices that are available in local storage 728 with the file system FileSystem-1A 742. For example, the file system pool FS1-Pool-1 748 may specify that a disk device named hd1 in the volume group VG1 732 of local storage 728 is a storage device for FileSystem-1A 742 for file server FS1 on FSVM-1 720. A file system pool FS2-Pool-2 750 may specify a storage device FileSystem-2A 750 for file server FS2 on FSVM-1 720. The storage device for FileSystem-2A 740 may be, e.g., the disk device hd1, or a different device in one of the volume groups 732, 734, such as a disk device named hd2 in volume group VG2 734. Each of the file systems FileSystem-1A 742, FileSystem-2A 740 may be, e.g., an instance of the NTFS file system used by the WINDOWS operating system, of the UFS Unix file system, or the like. The term “file system” may also be used herein to refer to an instance of a type of file system, e.g., a particular structure of folders and files with particular names and content.
In one example, referring to
In particular embodiments, FileServer-VM-1 (abbreviated FSVM-1) 720 on Host-1 704 is a leader for a portion of file server instance FS1 and a portion of FS2, and is a backup for another portion of FS1 and another portion of FS2. The portion of FS1 for which FileServer-VM-1 720 is a leader corresponds to a storage pool labeled FS1-Pool-1 748. FileServer-VM-1 is also a leader for FS2-Pool-2 750, and is a backup (e.g., is prepared to become a leader upon request, such as in response to a failure of another FSVM) for FS1-Pool-3 752 and FS2-Pool-4 754 on Host-2 706. In particular embodiments, FileServer-VM-2 (abbreviated FSVM-2) 722 is a leader for a portion of file server instance FS1 and a portion of FS2, and is a backup for another portion of FS1 and another portion of FS2. The portion of FS1 for which FSVM-2 722 is a leader corresponds to a storage pool labeled FS1-Pool-3 752. FSVM-2 722 is also a leader for FS2-Pool-4 754, and is a backup for FS1-Pool-1 748 and FS2-Pool-2 750 on Host-1 704.
In particular embodiments, the file server instances FS1, FS2 provided by the FSVMs 720 and 722 may be accessed by user VMs 708, 710, 712 and 714 via a network file system protocol such as SMB, CIFS, NFS, or the like. Each FSVM 720 and 722 may provide what appears to client applications on user VMs 708, 710, 712 and 714 to be a single file system instance, e.g., a single namespace of shares, files and folders, for each file server instance. However, the shares, files, and folders in a file server instance such as FS1 may actually be distributed across multiple FSVMs 720 and 722. For example, different folders in the same file server instance may be associated with different corresponding FSVMs 720 and 722 and CVMs 724 and 726 on different host machines 704 and 706.
The example file server instance FS1 604 shown in
Continuing with the data request example, the associated CVM is CVM 724, which may in turn access the storage device associated with the requested data as specified in the request, e.g., to write specified data to the storage device or read requested data from a specified location on the storage device. In this example, the associated storage device is in local storage 728, and may be an HDD or SSD. CVM-1 724 may access the HDD or SSD via an appropriate protocol, e.g., iSCSI, SCSI, SATA, or the like. CVM 110a may send the results of accessing local storage 728, e.g., data that has been read, or the status of a data write operation, to CVM 724 via, e.g., SATA, which may in turn send the results to FSVM-1 720 via, e.g., iSCSI. FSVM-1 720 may then send the results to user VM via SMB through the Hypervisor 716.
Share-2 610 may be located on FSVM-2 722, on Host-2. Network file service protocol requests from user VMs to read or write data on Share-2 may be directed to FSVM-2 722 on Host-2 by other FSVMs. Alternatively, user VMs may send such requests directly to FSVM-2 722 on Host-2, which may process the requests using CVM-2 726 and local storage 730 on Host-2 as described above for FSVM-1 720 on Host-1.
A file server instance such as FS1 604 in
In particular embodiments, when the client 804 requests access to Folder-3, a VFS client component executing in the user VM may use a distributed file system protocol such as MICROSOFT DFS, or the like, to send the storage access request to one or more of the FSVMs of
In particular embodiments, at step 864, the client may send a request for a list of addresses of FSVMs to a name server 802. The name server 802 may be, e.g., a DNS server or other type of server, such as a MICROSOFT domain controller (not shown), that has a database of FSVM addresses. At step 848, the name server 802 may send a reply that contains a list of FSVM network addresses, e.g., ip-addr1, ip-addr2, and ip-addr3, which correspond to the FSVMs in this example. At step 866, the client 804 may send an access request to one of the network addresses, e.g., the first network address in the list (ip-addr1 in this example), requesting the contents of Folder-3 of Share-1. By selecting the first network address in the list, the particular FSVM to which the access request is sent may be varied, e.g., in a round-robin manner by enabling round-robin DNS (or the like) on the name server 802. The access request may be, e.g., an SMB connect request, an NFS open request, and/or appropriate request(s) to traverse the hierarchy of Share-1 to reach the desired folder or file, e.g., Folder-3 in this example.
At step 868, FileServer-VM-1 810 may process the request received at step 866 by searching a mapping or lookup table, such as a sharding map 822, for the desired folder or file. The map 822 maps stored objects, such as shares, folders, or files, to their corresponding locations, e.g., the names or addresses of FSVMs. The map 822 may have the same contents on each host machine, with the contents on different host machines being synchronized using a distributed data store as described below. For example, the map 822 may contain entries that map Share-1 and Folder-1 to the File Server FSVM-1 810, and Folder-3 to the File Server FSVM-3 812. An example map is shown in Table 1 below. While the example of
In particular embodiments, the map 822 or 824 may be accessible on each of the host machines. The maps may be copies of a distributed data structure that are maintained and accessed at each FSVM using a distributed data access coordinator 826 and 830. The distributed data access coordinator 826 and 830 may be implemented based on distributed locks or other storage item access operations. Alternatively, the distributed data access coordinator 826 and 830 may be implemented by maintaining a master copy of the maps 822 and 824 at a leader node such as the host machine 808, and using distributed locks to access the master copy from each FSVM 810 and 812. The distributed data access coordinator 826 and 830 may be implemented using distributed locking, leader election, or related features provided by a centralized coordination service for maintaining configuration information, naming, providing distributed synchronization, and/or providing group services (e.g., APACHE ZOOKEEPER or other distributed coordination software). Since the map 822 indicates that Folder-3 is located at FSVM-3 812 on Host-3 808, the lookup operation at step 868 determines that Folder-3 is not located at FSVM-1 on Host-1 806. Thus, at step 862 the FSVM-1 810 (or other component of the virtualized system) sends a response, e.g., a “Not Covered” DFS response, to the client 804 indicating that the requested folder is not located at FSVM-1. At step 860, the client 804 sends a request to FSVM-1 for a referral to the FSVM on which Folder-3 is located. FSVM-1 uses the map 822 to determine that Folder-3 is located at FSVM-3 on Host-3 808, and at step 858 returns a response, e.g., a “Redirect” DFS response, redirecting the client 804 to FSVM-3. The client 804 may then determine the network address for FSVM-3, which is ip-addr3 (e.g., a host name “ip-addr3.domain.name” or an IP address, 10.1.1.3). The client 804 may determine the network address for FSVM-3 by searching a cache stored in memory of the client 804, which may contain a mapping from FSVM-3 to ip-addr3 cached in a previous operation. If the cache does not contain a network address for FSVM-3, then at step 850 the client 804 may send a request to the name server 802 to resolve the name FSVM-3. The name server may respond with the resolved address, ip-addr3, at step 852. The client 804 may then store the association between FSVM-3 and ip-addr3 in the client's cache.
In particular embodiments, failure of FSVMs may be detected using the centralized coordination service. For example, using the centralized coordination service, each FSVM may create a lock on the host machine on which the FSVM is located using ephemeral nodes of the centralized coordination service (which are different from host machines but may correspond to host machines). Other FSVMs may volunteer for leadership of resources of remote FSVMs on other host machines, e.g., by requesting a lock on the other host machines. The locks requested by the other nodes are not granted unless communication to the leader host machine is lost, in which case the centralized coordination service deletes the ephemeral node and grants the lock to one of the volunteer host machines and, which becomes the new leader. For example, the volunteer host machines may be ordered by the time at which the centralized coordination service received their requests, and the lock may be granted to the first host machine on the ordered list. The first host machine on the list may thus be selected as the new leader. The FSVM on the new leader has ownership of the resources that were associated with the failed leader FSVM until the failed leader FSVM is restored, at which point the restored FSVM may reclaim the local resources of the host machine on which it is located.
At step 854, the client 804 may send an access request to FSVM-3 812 at ip-addr3 on Host-3 808 requesting the contents of Folder-3 of Share-1. At step 870, FSVM-3 812 queries FSVM-3's copy of the map 824 using FSVM-3's instance of the distributed data access coordinator 830. The map 824 indicates that Folder-3 is located on FSVM-3, so at step 872 FSVM-3 accesses the file system 832 to retrieve information about Folder-3 844 and its contents (e.g., a list of files in the folder, which includes File-2 846) that are stored on the local storage 820. FSVM-3 may access local storage 820 via CVM-3 816, which provides access to local storage 820 via a volume group 836 that contains one or more volumes stored on one or more storage devices in local storage 820. At step 856, FSVM-3 may then send the information about Folder-3 and its contents to the client 804. Optionally, FSVM-3 may retrieve the contents of File-2 and send them to the client 804, or the client 804 may send a subsequent request to retrieve File-2 as needed.
The computing node 900 includes one or more communications fabric(s) 902, which provide communications between one or more processor(s) 904, memory 906, local storage 908, communications unit 910, and/or I/O interface(s) 912. The communications fabric(s) 902 can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, the communications fabric(s) 902 can be implemented with one or more buses.
The memory 906 and the local storage 908 may be computer-readable storage media. In the example of
Various computer instructions, programs, files, images, etc. may be stored in local storage 908 and/or memory 906 for execution by one or more of the respective processor(s) 904 via one or more memories of memory 906. In some examples, local storage 908 includes a magnetic HDD 924. Alternatively, or in addition to a magnetic hard disk drive, local storage 908 can include the SSD 922, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage media that is capable of storing program instructions or digital information.
The media used by local storage 908 may also be removable. For example, a removable hard drive may be used for local storage 908. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of local storage 908.
Communications unit 910, in some examples, provides for communications with other data processing systems or devices. For example, communications unit 910 may include one or more network interface cards. Communications unit 910 may provide communications through the use of either or both physical and wireless communications links.
I/O interface(s) 912 may allow for input and output of data with other devices that may be connected to computing node 900. For example, I/O interface(s) 912 may provide a connection to external device(s) 918 such as a keyboard, a keypad, a touch screen, and/or some other suitable input device. External device(s) 918 can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention can be stored on such portable computer-readable storage media and can be loaded onto and/or encoded in memory 906 and/or local storage 908 via I/O interface(s) 912 in some examples. I/O interface(s) 912 may connect to a display 920. Display 920 may provide a mechanism to display data to a user and may be, for example, a computer monitor.
From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made while remaining with the scope of the claimed technology.
Examples described herein may refer to various components as “coupled” or signals as being “provided to” or “received from” certain components. It is to be understood that in some examples the components are directly coupled one to another, while in other examples the components are coupled with intervening components disposed between them. Similarly, signal may be provided directly to and/or received directly from the recited components without intervening components, but also may be provided to and/or received from the certain components through intervening components.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5276867 | Kenley et al. | Jan 1994 | A |
| 5664144 | Yanai et al. | Sep 1997 | A |
| 5870555 | Pruett et al. | Feb 1999 | A |
| 5873085 | Enoki et al. | Feb 1999 | A |
| 5924096 | Draper et al. | Jul 1999 | A |
| 6044367 | Wolff | Mar 2000 | A |
| 6055543 | Christensen et al. | Apr 2000 | A |
| 6085234 | Pitts et al. | Jul 2000 | A |
| 6101508 | Wolff | Aug 2000 | A |
| 6212531 | Blea et al. | Apr 2001 | B1 |
| 6289356 | Hitz et al. | Sep 2001 | B1 |
| 6341340 | Tsukerman et al. | Jan 2002 | B1 |
| 6442602 | Choudhry | Aug 2002 | B1 |
| 6539381 | Prasad et al. | Mar 2003 | B1 |
| 6963914 | Breitbart et al. | Nov 2005 | B1 |
| 6968345 | Muhlestein | Nov 2005 | B1 |
| 7120631 | Vahalia et al. | Oct 2006 | B1 |
| 7159056 | Goldick | Jan 2007 | B2 |
| 7162467 | Eshleman et al. | Jan 2007 | B2 |
| 7356679 | Le et al. | Apr 2008 | B1 |
| 7366738 | Yorke et al. | Apr 2008 | B2 |
| 7409511 | Edwards et al. | Aug 2008 | B2 |
| 7606868 | Le et al. | Oct 2009 | B1 |
| 7702843 | Chen et al. | Apr 2010 | B1 |
| 7707618 | Cox et al. | Apr 2010 | B1 |
| 7725671 | Prahlad et al. | May 2010 | B2 |
| 7752492 | Armangau et al. | Jul 2010 | B1 |
| 7774391 | Le et al. | Aug 2010 | B1 |
| 7805469 | Nagaralu et al. | Sep 2010 | B1 |
| 7805511 | Panicker et al. | Sep 2010 | B1 |
| 7840533 | Prahlad et al. | Nov 2010 | B2 |
| 7890529 | Srinivasan et al. | Feb 2011 | B1 |
| 7937453 | Hayden et al. | May 2011 | B1 |
| 8095810 | Matsuzawa et al. | Jan 2012 | B2 |
| 8095931 | Chen et al. | Jan 2012 | B1 |
| 8352482 | Hansen | Jan 2013 | B2 |
| 8352608 | Keagy et al. | Jan 2013 | B1 |
| 8359594 | Davidson et al. | Jan 2013 | B1 |
| 8365167 | Beaty et al. | Jan 2013 | B2 |
| 8407448 | Hayden et al. | Mar 2013 | B1 |
| 8447728 | Prahlad et al. | May 2013 | B2 |
| 8473462 | Banerjee | Jun 2013 | B1 |
| 8484163 | Yucel et al. | Jul 2013 | B1 |
| 8484356 | Douglis et al. | Jul 2013 | B1 |
| 8539076 | Nakano et al. | Sep 2013 | B2 |
| 8543790 | Chen et al. | Sep 2013 | B2 |
| 8549518 | Aron et al. | Oct 2013 | B1 |
| 8601473 | Aron et al. | Dec 2013 | B1 |
| 8635351 | Astete et al. | Jan 2014 | B2 |
| 8688660 | Sivasubramanian et al. | Apr 2014 | B1 |
| 8700573 | Enko | Apr 2014 | B2 |
| 8719522 | Chait et al. | May 2014 | B1 |
| 8725679 | Nair et al. | May 2014 | B2 |
| 8751515 | Xing et al. | Jun 2014 | B1 |
| 8762335 | Prahlad et al. | Jun 2014 | B2 |
| 8805951 | Faibish et al. | Aug 2014 | B1 |
| 8838923 | Prahlad et al. | Sep 2014 | B2 |
| 8850130 | Aron et al. | Sep 2014 | B1 |
| 8863124 | Aron | Oct 2014 | B1 |
| 8909753 | Nakamura et al. | Dec 2014 | B2 |
| 8914429 | Pitts | Dec 2014 | B2 |
| 8935563 | Rajaa et al. | Jan 2015 | B1 |
| 8949557 | Kamei et al. | Feb 2015 | B2 |
| 8966188 | Bardale | Feb 2015 | B1 |
| 8983952 | Zhang et al. | Mar 2015 | B1 |
| 8996783 | Huang et al. | Mar 2015 | B2 |
| 9009106 | Aron et al. | Apr 2015 | B1 |
| 9043567 | Modukuri et al. | May 2015 | B1 |
| 9069708 | Gill et al. | Jun 2015 | B2 |
| 9152628 | Stacey et al. | Oct 2015 | B1 |
| 9154535 | Harris | Oct 2015 | B1 |
| 9165003 | Tummala et al. | Oct 2015 | B1 |
| 9201698 | Ashok et al. | Dec 2015 | B2 |
| 9201704 | Chang et al. | Dec 2015 | B2 |
| 9201887 | Earl et al. | Dec 2015 | B1 |
| 9208210 | Erofeev | Dec 2015 | B2 |
| 9213513 | Hartz et al. | Dec 2015 | B2 |
| 9244674 | Waterman et al. | Jan 2016 | B2 |
| 9244969 | Love et al. | Jan 2016 | B1 |
| 9256475 | Aron et al. | Feb 2016 | B1 |
| 9256612 | Bhatt et al. | Feb 2016 | B1 |
| 9268586 | Voccio et al. | Feb 2016 | B2 |
| 9274817 | Fan et al. | Mar 2016 | B1 |
| 9286298 | Gillett, Jr. | Mar 2016 | B1 |
| 9292327 | Von Thenen et al. | Mar 2016 | B1 |
| 9336132 | Aron et al. | May 2016 | B1 |
| 9348702 | Hsu et al. | May 2016 | B2 |
| 9405566 | Chawla et al. | Aug 2016 | B2 |
| 9411628 | Bezbaruah et al. | Aug 2016 | B2 |
| 9448887 | Ben Dayan et al. | Sep 2016 | B1 |
| 9495478 | Hendrickson et al. | Nov 2016 | B2 |
| 9497257 | Love et al. | Nov 2016 | B1 |
| 9513946 | Sevigny et al. | Dec 2016 | B2 |
| 9535907 | Stringham | Jan 2017 | B1 |
| 9563555 | Flynn et al. | Feb 2017 | B2 |
| 9571561 | Jang | Feb 2017 | B2 |
| 9619257 | Aron et al. | Apr 2017 | B1 |
| 9639428 | Boda et al. | May 2017 | B1 |
| 9652265 | Narayanasamy et al. | May 2017 | B1 |
| 9658899 | Jenkins | May 2017 | B2 |
| 9690670 | Paulzagade et al. | Jun 2017 | B1 |
| 9733958 | Cui et al. | Aug 2017 | B2 |
| 9740436 | Fiebrich-kandler et al. | Aug 2017 | B2 |
| 9740472 | Sohi et al. | Aug 2017 | B1 |
| 9740723 | Prahlad et al. | Aug 2017 | B2 |
| 9747287 | Bhardwaj et al. | Aug 2017 | B1 |
| 9772866 | Aron et al. | Sep 2017 | B1 |
| 9846706 | Basov et al. | Dec 2017 | B1 |
| 9853978 | Tellvik et al. | Dec 2017 | B2 |
| 9870291 | Bezbaruah et al. | Jan 2018 | B2 |
| 9886215 | Ramachandran et al. | Feb 2018 | B1 |
| 9940154 | Ramani et al. | Apr 2018 | B2 |
| 9946573 | Mcdermott | Apr 2018 | B2 |
| 9952782 | Chandrasekaran et al. | Apr 2018 | B1 |
| 10009215 | Shorey | Jun 2018 | B1 |
| 10050862 | Nambiar et al. | Aug 2018 | B2 |
| 10083022 | Fukui et al. | Sep 2018 | B2 |
| 10084873 | Dornemann | Sep 2018 | B2 |
| 10095506 | Gopalapura Venkatesh et al. | Oct 2018 | B2 |
| 10101989 | Sinha et al. | Oct 2018 | B2 |
| 10114706 | Chougala et al. | Oct 2018 | B1 |
| 10127059 | Astete et al. | Nov 2018 | B2 |
| 10140115 | Fukui et al. | Nov 2018 | B2 |
| 10152233 | Xu et al. | Dec 2018 | B2 |
| 10210048 | Sancheti | Feb 2019 | B2 |
| 10248657 | Prahlad et al. | Apr 2019 | B2 |
| 10310953 | Mjayan et al. | Jun 2019 | B2 |
| 10311153 | Mason, Jr. et al. | Jun 2019 | B2 |
| 10394547 | Fukui et al. | Aug 2019 | B2 |
| 10419426 | Bakshan et al. | Sep 2019 | B2 |
| 10523592 | Byers et al. | Dec 2019 | B2 |
| 10530742 | Shah et al. | Jan 2020 | B2 |
| 10540164 | Bafna et al. | Jan 2020 | B2 |
| 10540165 | Bafna et al. | Jan 2020 | B2 |
| 10540166 | Arikatla et al. | Jan 2020 | B2 |
| 10541064 | Lee et al. | Jan 2020 | B2 |
| 10719305 | Sinha et al. | Jul 2020 | B2 |
| 10719306 | Deshmukh et al. | Jul 2020 | B2 |
| 10719307 | Kanada et al. | Jul 2020 | B2 |
| 10728090 | Deshmukh et al. | Jul 2020 | B2 |
| 10809998 | Gopalapura Venkatesh et al. | Oct 2020 | B2 |
| 10824455 | Arikatla et al. | Nov 2020 | B2 |
| 10831465 | Sharpe et al. | Nov 2020 | B2 |
| 10838708 | Sinha et al. | Nov 2020 | B2 |
| 10949192 | Gopalapura Venkatesh et al. | Mar 2021 | B2 |
| 11025626 | Todd et al. | Jun 2021 | B1 |
| 11086826 | Thummala et al. | Aug 2021 | B2 |
| 11281484 | Bafna et al. | Mar 2022 | B2 |
| 11288239 | Bafna | Mar 2022 | B2 |
| 11294777 | Venkatesh et al. | Apr 2022 | B2 |
| 11347601 | Nachiappan et al. | May 2022 | B1 |
| 20010047400 | Coates et al. | Nov 2001 | A1 |
| 20020019936 | Hitz et al. | Feb 2002 | A1 |
| 20020069196 | Betros et al. | Jun 2002 | A1 |
| 20020120763 | Miloushev et al. | Aug 2002 | A1 |
| 20030115218 | Bobbitt et al. | Jun 2003 | A1 |
| 20030163597 | Hellman et al. | Aug 2003 | A1 |
| 20030195942 | Muhlestein et al. | Oct 2003 | A1 |
| 20040054777 | Ackaouy et al. | Mar 2004 | A1 |
| 20040199734 | Rajamani et al. | Oct 2004 | A1 |
| 20040210591 | Hirschfeld et al. | Oct 2004 | A1 |
| 20040225742 | Loaiza et al. | Nov 2004 | A1 |
| 20040267832 | Wong et al. | Dec 2004 | A1 |
| 20050120160 | Plouffe et al. | Jun 2005 | A1 |
| 20050120180 | Schornbach et al. | Jun 2005 | A1 |
| 20050125503 | Iyengar et al. | Jun 2005 | A1 |
| 20050193221 | Yoneyama | Sep 2005 | A1 |
| 20050193245 | Hayden et al. | Sep 2005 | A1 |
| 20050210067 | Nakatani et al. | Sep 2005 | A1 |
| 20050226059 | Kavuri et al. | Oct 2005 | A1 |
| 20050228798 | Shepard et al. | Oct 2005 | A1 |
| 20060010227 | Atluri | Jan 2006 | A1 |
| 20060047685 | Dearing et al. | Mar 2006 | A1 |
| 20060080445 | Chang et al. | Apr 2006 | A1 |
| 20060167921 | Grebus et al. | Jul 2006 | A1 |
| 20060206901 | Chan | Sep 2006 | A1 |
| 20060224918 | Koike | Oct 2006 | A1 |
| 20060225065 | Chandhok et al. | Oct 2006 | A1 |
| 20060271510 | Harward et al. | Nov 2006 | A1 |
| 20060271931 | Harris et al. | Nov 2006 | A1 |
| 20070022129 | Bahar et al. | Jan 2007 | A1 |
| 20070038913 | Allen et al. | Feb 2007 | A1 |
| 20070100905 | Masters et al. | May 2007 | A1 |
| 20070171921 | Wookey et al. | Jul 2007 | A1 |
| 20070179991 | Burnett et al. | Aug 2007 | A1 |
| 20070276838 | Abushanab et al. | Nov 2007 | A1 |
| 20070300220 | Seliger et al. | Dec 2007 | A1 |
| 20080040483 | Nakatani et al. | Feb 2008 | A1 |
| 20080071997 | Loaiza et al. | Mar 2008 | A1 |
| 20080098194 | Hashimoto et al. | Apr 2008 | A1 |
| 20080104349 | Maruyama et al. | May 2008 | A1 |
| 20080104589 | Mccrory et al. | May 2008 | A1 |
| 20080114854 | Wong et al. | May 2008 | A1 |
| 20080133486 | Fitzgerald et al. | Jun 2008 | A1 |
| 20080134178 | Fitzgerald et al. | Jun 2008 | A1 |
| 20080189468 | Schmidt et al. | Aug 2008 | A1 |
| 20080201414 | Amir et al. | Aug 2008 | A1 |
| 20080201457 | London | Aug 2008 | A1 |
| 20080229142 | Anand et al. | Sep 2008 | A1 |
| 20080263113 | Krishnaiyer et al. | Oct 2008 | A1 |
| 20080270677 | Kolakowski | Oct 2008 | A1 |
| 20080307500 | Garnier | Dec 2008 | A1 |
| 20080320499 | Suit | Dec 2008 | A1 |
| 20080320583 | Sharma et al. | Dec 2008 | A1 |
| 20090006801 | Shultz et al. | Jan 2009 | A1 |
| 20090100248 | Kami | Apr 2009 | A1 |
| 20090150885 | Safari et al. | Jun 2009 | A1 |
| 20090158082 | Jain et al. | Jun 2009 | A1 |
| 20090171971 | Goddard et al. | Jul 2009 | A1 |
| 20090193272 | Matsuzawa et al. | Jul 2009 | A1 |
| 20090216975 | Halperin et al. | Aug 2009 | A1 |
| 20090248870 | Kamei et al. | Oct 2009 | A1 |
| 20090249470 | Litvin et al. | Oct 2009 | A1 |
| 20090271412 | Lacapra et al. | Oct 2009 | A1 |
| 20090287887 | Matsuki et al. | Nov 2009 | A1 |
| 20090288084 | Astete et al. | Nov 2009 | A1 |
| 20100023521 | Arcese et al. | Jan 2010 | A1 |
| 20100070725 | Prahlad et al. | Mar 2010 | A1 |
| 20100082716 | Agetsuma et al. | Apr 2010 | A1 |
| 20100082774 | Pitts | Apr 2010 | A1 |
| 20100095289 | Nguyen et al. | Apr 2010 | A1 |
| 20100138921 | Na et al. | Jun 2010 | A1 |
| 20100174745 | Ryan et al. | Jul 2010 | A1 |
| 20100214908 | Ralev | Aug 2010 | A1 |
| 20100241785 | Chen et al. | Sep 2010 | A1 |
| 20100250824 | Belay | Sep 2010 | A1 |
| 20100275205 | Nakajima | Oct 2010 | A1 |
| 20110004649 | Nord et al. | Jan 2011 | A1 |
| 20110022694 | Dalal et al. | Jan 2011 | A1 |
| 20110022695 | Dalal et al. | Jan 2011 | A1 |
| 20110022812 | Van et al. | Jan 2011 | A1 |
| 20110022883 | Hansen | Jan 2011 | A1 |
| 20110047340 | Olson et al. | Feb 2011 | A1 |
| 20110078318 | Desai et al. | Mar 2011 | A1 |
| 20110119668 | Calder et al. | May 2011 | A1 |
| 20110119763 | Wade et al. | May 2011 | A1 |
| 20110125835 | Soltis | May 2011 | A1 |
| 20110137879 | Dubey et al. | Jun 2011 | A1 |
| 20110161299 | Prahlad et al. | Jun 2011 | A1 |
| 20110179414 | Goggin et al. | Jul 2011 | A1 |
| 20110184993 | Chawla et al. | Jul 2011 | A1 |
| 20110185292 | Chawla et al. | Jul 2011 | A1 |
| 20110225574 | Khalidi et al. | Sep 2011 | A1 |
| 20110239213 | Aswani et al. | Sep 2011 | A1 |
| 20110251992 | Bethlehem et al. | Oct 2011 | A1 |
| 20110252208 | Ali et al. | Oct 2011 | A1 |
| 20110255538 | Srinivasan et al. | Oct 2011 | A1 |
| 20110265076 | Thorat et al. | Oct 2011 | A1 |
| 20110271279 | Pate | Nov 2011 | A1 |
| 20110276578 | Allalouf et al. | Nov 2011 | A1 |
| 20110276963 | Wu et al. | Nov 2011 | A1 |
| 20110283277 | Castillo et al. | Nov 2011 | A1 |
| 20110289561 | Ivanov et al. | Nov 2011 | A1 |
| 20110307729 | Matsuzawa et al. | Dec 2011 | A1 |
| 20110320690 | Petersen et al. | Dec 2011 | A1 |
| 20120017114 | Timashev et al. | Jan 2012 | A1 |
| 20120023495 | Machida | Jan 2012 | A1 |
| 20120030456 | Wu et al. | Feb 2012 | A1 |
| 20120054736 | Arcese et al. | Mar 2012 | A1 |
| 20120081395 | Adi et al. | Apr 2012 | A1 |
| 20120084381 | ABadi et al. | Apr 2012 | A1 |
| 20120166866 | Rao et al. | Jun 2012 | A1 |
| 20120209983 | Bronner et al. | Aug 2012 | A1 |
| 20120233463 | Holt et al. | Sep 2012 | A1 |
| 20120254445 | Kawamoto et al. | Oct 2012 | A1 |
| 20120254567 | Umbehocker | Oct 2012 | A1 |
| 20120266162 | Baron | Oct 2012 | A1 |
| 20120272237 | Baron | Oct 2012 | A1 |
| 20120290630 | Aizman et al. | Nov 2012 | A1 |
| 20120310881 | Shadmon | Dec 2012 | A1 |
| 20120310892 | Dam et al. | Dec 2012 | A1 |
| 20120324183 | Chiruvolu et al. | Dec 2012 | A1 |
| 20130046740 | Li et al. | Feb 2013 | A1 |
| 20130047160 | Conover | Feb 2013 | A1 |
| 20130055018 | Joshi et al. | Feb 2013 | A1 |
| 20130061110 | Zvibel | Mar 2013 | A1 |
| 20130061167 | Rhodes et al. | Mar 2013 | A1 |
| 20130066930 | Kamei et al. | Mar 2013 | A1 |
| 20130117744 | Klein et al. | May 2013 | A1 |
| 20130132674 | Sundrani | May 2013 | A1 |
| 20130144921 | Nakamura et al. | Jun 2013 | A1 |
| 20130151888 | Bhattiprolu et al. | Jun 2013 | A1 |
| 20130152085 | D'Amore et al. | Jun 2013 | A1 |
| 20130185716 | Yin et al. | Jul 2013 | A1 |
| 20130198738 | Reddin et al. | Aug 2013 | A1 |
| 20130212345 | Nakajima | Aug 2013 | A1 |
| 20130227379 | Gupta et al. | Aug 2013 | A1 |
| 20130227552 | Reddin et al. | Aug 2013 | A1 |
| 20130227566 | Higuchi et al. | Aug 2013 | A1 |
| 20130232491 | Radhakrishnan et al. | Sep 2013 | A1 |
| 20130246705 | Diare | Sep 2013 | A1 |
| 20130247036 | Fujiwara | Sep 2013 | A1 |
| 20130262396 | Kripalani et al. | Oct 2013 | A1 |
| 20130268491 | Chung et al. | Oct 2013 | A1 |
| 20130283267 | Cooper et al. | Oct 2013 | A1 |
| 20130297869 | Mills et al. | Nov 2013 | A1 |
| 20130318229 | Bakre et al. | Nov 2013 | A1 |
| 20140006708 | Huynh et al. | Jan 2014 | A1 |
| 20140013398 | Hotti | Jan 2014 | A1 |
| 20140025796 | Vibhor et al. | Jan 2014 | A1 |
| 20140059392 | Ren et al. | Feb 2014 | A1 |
| 20140095544 | Eshel et al. | Apr 2014 | A1 |
| 20140095555 | Kim et al. | Apr 2014 | A1 |
| 20140095816 | Hsu et al. | Apr 2014 | A1 |
| 20140108587 | Goldberg et al. | Apr 2014 | A1 |
| 20140115182 | Sabaa et al. | Apr 2014 | A1 |
| 20140122547 | Agetsuma | May 2014 | A1 |
| 20140123138 | Lee et al. | May 2014 | A1 |
| 20140146055 | Bala et al. | May 2014 | A1 |
| 20140149794 | Shetty et al. | May 2014 | A1 |
| 20140149983 | Bonilla et al. | May 2014 | A1 |
| 20140173199 | Gupta et al. | Jun 2014 | A1 |
| 20140181116 | Wang | Jun 2014 | A1 |
| 20140188808 | Wolf et al. | Jul 2014 | A1 |
| 20140189429 | Gill et al. | Jul 2014 | A1 |
| 20140189677 | Curzi et al. | Jul 2014 | A1 |
| 20140189685 | Kripalani | Jul 2014 | A1 |
| 20140189686 | Masters et al. | Jul 2014 | A1 |
| 20140196038 | Kottomtharayil et al. | Jul 2014 | A1 |
| 20140201725 | Tian et al. | Jul 2014 | A1 |
| 20140207824 | Brandwine et al. | Jul 2014 | A1 |
| 20140230024 | Uehara et al. | Aug 2014 | A1 |
| 20140237464 | Waterman et al. | Aug 2014 | A1 |
| 20140250300 | Runkis et al. | Sep 2014 | A1 |
| 20140279909 | Sudarsanam et al. | Sep 2014 | A1 |
| 20140298185 | Chen et al. | Oct 2014 | A1 |
| 20140310710 | Lubsey et al. | Oct 2014 | A1 |
| 20140359612 | D'Amato et al. | Dec 2014 | A1 |
| 20140365811 | Veiga et al. | Dec 2014 | A1 |
| 20150006707 | Malik et al. | Jan 2015 | A1 |
| 20150006788 | Liu et al. | Jan 2015 | A1 |
| 20150007180 | Sharp et al. | Jan 2015 | A1 |
| 20150026682 | Singh et al. | Jan 2015 | A1 |
| 20150032690 | Hoque et al. | Jan 2015 | A1 |
| 20150039735 | Zeyliger et al. | Feb 2015 | A1 |
| 20150039837 | Quan et al. | Feb 2015 | A1 |
| 20150081644 | Pitts | Mar 2015 | A1 |
| 20150095788 | Thiele et al. | Apr 2015 | A1 |
| 20150106802 | Ivanov et al. | Apr 2015 | A1 |
| 20150142745 | Tekade et al. | May 2015 | A1 |
| 20150142747 | Zou | May 2015 | A1 |
| 20150143164 | Veerta et al. | May 2015 | A1 |
| 20150172412 | Escriva et al. | Jun 2015 | A1 |
| 20150178019 | Hegdal et al. | Jun 2015 | A1 |
| 20150205639 | Matsumoto et al. | Jul 2015 | A1 |
| 20150213032 | Powell et al. | Jul 2015 | A1 |
| 20150215389 | Spencer | Jul 2015 | A1 |
| 20150220324 | Arcese et al. | Aug 2015 | A1 |
| 20150242291 | Chang et al. | Aug 2015 | A1 |
| 20150278046 | Zellermayer et al. | Oct 2015 | A1 |
| 20150278543 | Zhang et al. | Oct 2015 | A1 |
| 20150293830 | Bhide et al. | Oct 2015 | A1 |
| 20150293896 | Runkis et al. | Oct 2015 | A1 |
| 20150301903 | Mutha et al. | Oct 2015 | A1 |
| 20150309891 | Saika | Oct 2015 | A1 |
| 20150324217 | Shilmover et al. | Nov 2015 | A1 |
| 20150326531 | Cui et al. | Nov 2015 | A1 |
| 20150331757 | Durge et al. | Nov 2015 | A1 |
| 20150355862 | Hayes et al. | Dec 2015 | A1 |
| 20150378761 | Sevigny et al. | Dec 2015 | A1 |
| 20150378853 | Sevigny | Dec 2015 | A1 |
| 20160011898 | Lee et al. | Jan 2016 | A1 |
| 20160070492 | Cherubini et al. | Mar 2016 | A1 |
| 20160077988 | Tipton et al. | Mar 2016 | A1 |
| 20160078068 | Agrawal et al. | Mar 2016 | A1 |
| 20160085480 | Chiu et al. | Mar 2016 | A1 |
| 20160085574 | Dornemann et al. | Mar 2016 | A1 |
| 20160087861 | Kuan et al. | Mar 2016 | A1 |
| 20160110214 | Vincent et al. | Apr 2016 | A1 |
| 20160110267 | Earl et al. | Apr 2016 | A1 |
| 20160124665 | Jain et al. | May 2016 | A1 |
| 20160162371 | Prabhu et al. | Jun 2016 | A1 |
| 20160171241 | Yun | Jun 2016 | A1 |
| 20160179419 | Yamaguchi et al. | Jun 2016 | A1 |
| 20160188232 | Ramachandran et al. | Jun 2016 | A1 |
| 20160188407 | Bronnikov et al. | Jun 2016 | A1 |
| 20160202916 | Cui et al. | Jul 2016 | A1 |
| 20160203008 | Cui et al. | Jul 2016 | A1 |
| 20160216993 | Beckwith et al. | Jul 2016 | A1 |
| 20160224363 | Joy | Aug 2016 | A1 |
| 20160274926 | Narasimhamurthy et al. | Sep 2016 | A1 |
| 20160301766 | Ionescu et al. | Oct 2016 | A1 |
| 20160316003 | Snider et al. | Oct 2016 | A1 |
| 20160328226 | Arya et al. | Nov 2016 | A1 |
| 20160335108 | Ryu et al. | Nov 2016 | A1 |
| 20160335134 | Gupta et al. | Nov 2016 | A1 |
| 20160359697 | Scheib et al. | Dec 2016 | A1 |
| 20160378528 | Zamir | Dec 2016 | A1 |
| 20160378616 | Wigmore et al. | Dec 2016 | A1 |
| 20170004131 | Ben Dayan et al. | Jan 2017 | A1 |
| 20170005990 | Birger et al. | Jan 2017 | A1 |
| 20170012904 | Matzek et al. | Jan 2017 | A1 |
| 20170024152 | Bhagi et al. | Jan 2017 | A1 |
| 20170024224 | Bakke et al. | Jan 2017 | A1 |
| 20170039078 | Chen et al. | Feb 2017 | A1 |
| 20170039218 | Prahlad et al. | Feb 2017 | A1 |
| 20170048223 | Anantha Padmanaban et al. | Feb 2017 | A1 |
| 20170068469 | Shankar et al. | Mar 2017 | A1 |
| 20170075921 | Benton et al. | Mar 2017 | A1 |
| 20170090776 | Kowles | Mar 2017 | A1 |
| 20170091047 | Bangalore et al. | Mar 2017 | A1 |
| 20170109184 | Ramani et al. | Apr 2017 | A1 |
| 20170160983 | Fiske et al. | Jun 2017 | A1 |
| 20170177638 | Bhosale et al. | Jun 2017 | A1 |
| 20170193021 | Deng et al. | Jul 2017 | A1 |
| 20170206074 | Arcese et al. | Jul 2017 | A1 |
| 20170206207 | Bondurant et al. | Jul 2017 | A1 |
| 20170220661 | Cao et al. | Aug 2017 | A1 |
| 20170228300 | Thomas et al. | Aug 2017 | A1 |
| 20170235507 | Sinha et al. | Aug 2017 | A1 |
| 20170235562 | Bafna et al. | Aug 2017 | A1 |
| 20170235563 | Bafna et al. | Aug 2017 | A1 |
| 20170235589 | Gopalapura Venkatesh et al. | Aug 2017 | A1 |
| 20170235590 | Sinha et al. | Aug 2017 | A1 |
| 20170235591 | Kanada et al. | Aug 2017 | A1 |
| 20170235653 | Arikatla et al. | Aug 2017 | A1 |
| 20170235654 | Deshmukh et al. | Aug 2017 | A1 |
| 20170235751 | Gupta et al. | Aug 2017 | A1 |
| 20170235758 | Gopalapura Venkatesh et al. | Aug 2017 | A1 |
| 20170235760 | Sharpe et al. | Aug 2017 | A1 |
| 20170235761 | Bafna et al. | Aug 2017 | A1 |
| 20170235762 | Sharpe et al. | Aug 2017 | A1 |
| 20170235763 | Gopalapura Venkatesh et al. | Aug 2017 | A1 |
| 20170235764 | Sharpe et al. | Aug 2017 | A1 |
| 20170235950 | Gopalapura Venkatesh et al. | Aug 2017 | A1 |
| 20170242599 | Patnaik et al. | Aug 2017 | A1 |
| 20170262346 | Pradhan et al. | Sep 2017 | A1 |
| 20170264684 | Spillane et al. | Sep 2017 | A1 |
| 20170277556 | Ishii et al. | Sep 2017 | A1 |
| 20170277903 | Christodorescu et al. | Sep 2017 | A1 |
| 20170279674 | Zhu | Sep 2017 | A1 |
| 20170286228 | Redko et al. | Oct 2017 | A1 |
| 20170302589 | Leafe et al. | Oct 2017 | A1 |
| 20170302731 | Cui et al. | Oct 2017 | A1 |
| 20180004766 | Darling | Jan 2018 | A1 |
| 20180011766 | Lee et al. | Jan 2018 | A1 |
| 20180062993 | Wu et al. | Mar 2018 | A1 |
| 20180129426 | Aron et al. | May 2018 | A1 |
| 20180143845 | Chawla et al. | May 2018 | A1 |
| 20180143880 | Dornemann | May 2018 | A1 |
| 20180145960 | Bakshan et al. | May 2018 | A1 |
| 20180157521 | Arikatla et al. | Jun 2018 | A1 |
| 20180157522 | Bafna et al. | Jun 2018 | A1 |
| 20180157561 | Venkatesh et al. | Jun 2018 | A1 |
| 20180157677 | Bafna et al. | Jun 2018 | A1 |
| 20180157752 | Arikatla et al. | Jun 2018 | A1 |
| 20180157860 | Nair et al. | Jun 2018 | A1 |
| 20180159729 | Deshmukh et al. | Jun 2018 | A1 |
| 20180159826 | Yisan et al. | Jun 2018 | A1 |
| 20180173731 | Nazari et al. | Jun 2018 | A1 |
| 20180196719 | Glass | Jul 2018 | A1 |
| 20180205787 | Ben Dayan et al. | Jul 2018 | A1 |
| 20180278602 | Koushik et al. | Sep 2018 | A1 |
| 20180332105 | Huang et al. | Nov 2018 | A1 |
| 20180357251 | Kumarasamy et al. | Dec 2018 | A1 |
| 20190026101 | Gopalapura Venkatesh et al. | Jan 2019 | A1 |
| 20190034240 | Nabi et al. | Jan 2019 | A1 |
| 20190079747 | Sinha et al. | Mar 2019 | A1 |
| 20190129808 | Acharya et al. | May 2019 | A1 |
| 20190179918 | Singh et al. | Jun 2019 | A1 |
| 20190196718 | Pai et al. | Jun 2019 | A1 |
| 20190207925 | Anantha Padmanaban et al. | Jul 2019 | A1 |
| 20190228147 | Formato et al. | Jul 2019 | A1 |
| 20190243703 | Rooney et al. | Aug 2019 | A1 |
| 20190332683 | Thummala et al. | Oct 2019 | A1 |
| 20190339883 | Aron et al. | Nov 2019 | A1 |
| 20200007530 | Mohamad Abdul et al. | Jan 2020 | A1 |
| 20200034069 | Batra et al. | Jan 2020 | A1 |
| 20200036647 | Gupta et al. | Jan 2020 | A1 |
| 20200081704 | Bafna et al. | Mar 2020 | A1 |
| 20200081733 | Buck et al. | Mar 2020 | A1 |
| 20200125580 | Shao | Apr 2020 | A1 |
| 20200137157 | Joseph et al. | Apr 2020 | A1 |
| 20200193059 | Narayanswamy | Jun 2020 | A1 |
| 20200274869 | Tahenakos et al. | Aug 2020 | A1 |
| 20210141630 | Sharpe et al. | May 2021 | A1 |
| 20210157690 | Wexler et al. | May 2021 | A1 |
| 20210224233 | Bafna et al. | Jul 2021 | A1 |
| 20210247973 | Gupta et al. | Aug 2021 | A1 |
| 20210318938 | Benke et al. | Oct 2021 | A1 |
| 20210326358 | Seelemann et al. | Oct 2021 | A1 |
| 20210334178 | Yang et al. | Oct 2021 | A1 |
| 20210342237 | Polimera et al. | Nov 2021 | A1 |
| 20210344772 | Arikatla et al. | Nov 2021 | A1 |
| 20210349859 | Bafna et al. | Nov 2021 | A1 |
| 20210365257 | Gopalapura Venkatesh et al. | Nov 2021 | A1 |
| 20210373815 | Kumar et al. | Dec 2021 | A1 |
| 20210390080 | Tripathi et al. | Dec 2021 | A1 |
| 20210397587 | Thummala et al. | Dec 2021 | A1 |
| 20210406136 | Venkatesh et al. | Dec 2021 | A1 |
| 20220004377 | Sharpe et al. | Jan 2022 | A1 |
| 20220147342 | Sharpe et al. | May 2022 | A1 |
| 20220147495 | Sharpe et al. | May 2022 | A1 |
| 20220156107 | Bafna et al. | May 2022 | A1 |
| 20220283708 | Grunwald et al. | Sep 2022 | A1 |
| 20220292002 | Kumar et al. | Sep 2022 | A1 |
| 20230056217 | Rathi et al. | Feb 2023 | A1 |
| 20230056425 | Gopalapura Venkatesh et al. | Feb 2023 | A1 |
| 20230066137 | Vijjapurapu et al. | Mar 2023 | A1 |
| 20230068262 | Bafna et al. | Mar 2023 | A1 |
| 20230237022 | Thomas et al. | Jul 2023 | A1 |
| 20240045774 | Wang et al. | Feb 2024 | A1 |
| 20240070032 | Wang et al. | Feb 2024 | A1 |
| Number | Date | Country |
|---|---|---|
| 105100210 | Nov 2015 | CN |
| 110516005 | Nov 2019 | CN |
| 110519112 | Nov 2019 | CN |
| 110569269 | Dec 2019 | CN |
| 1062581 | Oct 2003 | EP |
| 1214663 | Jun 2006 | EP |
| 1979814 | Oct 2008 | EP |
| 2010050944 | May 2010 | WO |
| 2012126177 | Sep 2012 | WO |
| 2016018446 | Feb 2016 | WO |
| 2018014650 | Jan 2018 | WO |
| Entry |
|---|
| US 11,048,595 B2, 06/2021, Venkatesh et al. (withdrawn) |
| U.S. Appl. No. 17/581,418 titled “File Server Managers and Systems for Managing Virtualized File Servers” filed Jan. 21, 2022. |
| U.S. Appl. No. 17/877,769 titled “Self-Service Restore (SSR) Snapshot Replication With Share-Level File System Disaster Recovery on Virtualized File Servers” filed Jul. 29, 2022. |
| “Architecture for Continuous Replication”, Commvault, May 7, 2020, pp. 1-4. |
| “Disaster Recovery of Workloads on AWS: Recovery in the Cloud”, AWS, Feb. 12, 2021, pp. 1-33. |
| “End User Access”, Commvault, 2019, pp. 1-4. |
| “General Share Properties”, Oracle ZFS Storage Application Administration Guide, 2013, pp. 1-5. |
| “Introducing HPE Cloud vols. Backup Chalk Talk”, HPE Technology, 2020, pp. 1-5. |
| “Neverfail User Guide”, Neverfail HybriStor v2.3 | https://neverfail.com/, 2017, pp. 1-58. |
| “Oracle ZFS Storage Appliance: How To Optimize Replication Performance”, Oracle System Handbook—ISO 7.0, Aug. 1, 2018, pp. 1-3. |
| “Quantum User Essentials”, Quantum, 2013, pp. 1-2. |
| “Replication for File Systems”, Commvault, Jan. 4, 2022, pp. 1. |
| “Replication: Remote Replication Introduction”, Oracle Sun ZFS Storage 7000 System Administration Guide, Dec. 2012, pp. 1-6. |
| “Reversing the Direction of Replication”, Oracle ZFS Storage Appliance Administration Guide, 2014, pp. 1-2. |
| Jeffrey, Hemmes , et al., “Cacheable Decentralized Groups for Grid Resource Access Control”, 2006 7th IEEE/ACM International Conference on Grid Computing Department of Computer Science and Engineering, University of Notre Dame, Sep. 2006, pp. 192-199. |
| Nemnom, Charbel , “How To Enable Self-Service Restore in Azure File Sync”, charbelnemnom.com, Jan. 22, 2022, pp. 1-7. |
| U.S. Appl. No. 18/410,903 titled “Systems and Methods for Generating Consistent Global Identifiers Within Adistributed File Server Environment Including Examples of Global Identifiersacross Domains” filed Jan. 11, 2024. |
| “IBM Storwize V7000 Unified”, https://www.IBM.com/docs/en/flashsystem-v7000u/1.6.2?topic=maim-best-practices-authentication-id-mapping-configurations-while-using-asynchronous-replication-remote-caching, Mar. 31, 2021, pp. 1-3. |
| “ID mapping basic concepts”, https://www.ibm.com/docs/en/flashsystem-v7000u/1.6.2?topic=mapping-id-basic-concepts, Feb. 27, 2021, pp. 1-8. |
| U.S. Appl. No. 18/178,400 titled “Application Level to Share Level Replication Policy Transition for File Serverdisaster Recovery Systems” filed Mar. 3, 2023. |
| “Common Tasks—Remote Replication”, Arcserve https://support.storagecraft/com/s/article/common-tasks-remote-replication-ox?language=en_US, Oct. 20, 2021, pp. 1-11. |
| “DFS Namespaces and DFS Replication Overview”, Microsoft Learn, Aug. 31, 2016, pp. 9. |
| “Introduction to Oracle Data Guard”, Oracle https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/introduction-to-oracle-data-guard-concepts.html#GUID-5E73667D-4A56-445E-911 F-1 E99092DD8D7, pp. 1-20. |
| “Nutanix Files Guide”, Nutanix Files 3.7, Nov. 23, 2021, pp. 1-174. |
| “Nutanix-Files”, Nutanix, Inc. v7.0 https://download.nutanix.com/solutionsDocs/TN-2041-Nutanix-Files.pdf, Oct. 2021, pp. 1-89. |
| “Server-to-Server Storage Replication with Storage Replica”, Microsoft https://docs.microsoft.com/en-us/windows-server/storage/storage-replica/server-to-server-storage-replication,, Jul. 29, 2021, pp. 1-20. |
| “Storage Replica Overview”, Microsoft thttps://docs.microsoft.com/en-us/windows-server/storage/storage-replica/ storage-replica-overview, Jul. 29, 2021, pp. 1-11. |
| “Nutanix Files Guide”; Nutanix; Sep. 14, 2018; pp. all. |
| “Setting up and Using Acropolis File Services (AFS) on Nutanix AOS 5.0”; Virtual Dennis—Sharing Technical Tips Learned the Hard Way; Posted Dec. 30, 2016; pp. all. |
| U.S. Appl. No. 17/129,425, titled “Parallel Change File Tracking in a Distributed File Server Virtual Machine (FSVM) Architecture”, filed Dec. 21, 2020; pp. all pages of the application as filed. |
| U.S. Appl. No. 17/180,257 titled “Virtualized File Server User Views” filed Feb. 19, 2021, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/169,137 titled “Virtualized File Server Data Sharing” filed Feb. 5, 2021, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/364,453 titled “Virtualized Server Systems and Methods Including Domain Joining Techniques” filed Jun. 30, 2021, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/302,343 titled “Disaster Recovery for Distributed File Servers, Including Metadata Fixers” filed Apr. 30, 2021, pp. all pages of the application as filed. |
| Bas van Kaam “New in AOS 5.0: Nutanix Acropolis File Services”; basvankaam.com; Jan. 5, 2017; pp. all. |
| Dell: “High Availability and Data Protection With Dell EMC Isilon Scale-Out NAS”; Jul. 2019, Dell Inc., pp. all. |
| Jay Bounds “High-Availability (HA) Pair Controller Configuration Overview and Best Practices”; NetApp; Feb. 2016; pp. all. |
| Jorge Costa “High Availability Setup Using Veritas Cluster Server and NetApp Synchronous SnapMirror—One button Failover/Failback with SnapMirror Sync and Veritas Cluster Server”; NetApp Community; Nov. 18, 2010; pp. all. |
| NetApp “Preparing Storage Systems for Snapmirror Replication”; Apr. 2005, NetApp, Inc., pp. all. |
| NetApp; “Clustered Data Ontap 8.2 File Access Management Guide for CIFS”; Feb. 2014 (year 2014); pp. all. |
| U.S. Appl. No. 17/238,001 titled “Cloning Virtualized File Servers” filed Apr. 22, 2021, pp. all pages of the application as filed. |
| Ruth, Paul “Autonomic Live Adaption of Virtual Computational Environments in a Multi-Domain Infrastructure”; 2006 IEEE International Conference on Autonomic Computing, 2006, pp. 5-14. |
| U.S. Appl. No. 17/443,009, titled “Scope-Based Distributed Lock Infrastructure for Virtualized File Server” filed Jul. 19, 2021, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/448,315 titled “Virtualized File Server” filed Sep. 21, 2021, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/580,555 titled “Virtualized File Server” filed Jan. 20, 2022, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/581,418 titled “File Server Managers and Systems for Managing Virtualized File Servers” filed Jan. 21, 2022, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/648,654 titled “Failover and Failback of Distributed File Servers” filed Jan. 21, 2022, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/648,661 titled “User Interfaces for Disaster Recovery of Distributed File Servers” filed Jan. 21, 2022, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/581,562 titled “Share-Based File Server Replication for Disaster Recovery” filed Jan. 21, 2022, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/648,796 titled “Virtualized Server Systems and Methods Including Scaling of File System Virtual Machines” filed Jan. 24, 2022, pp. all pages of the application as filed. |
| U.S. Appl. No. 15/829,602 entitled “Handling Permissions for Virtualized File Servers”, filed Dec. 1, 2017, pp. all pages of the application as filed. |
| Young-Woo Jung et al. “Standard-Based Vitrual Infrastructure Resource Management for Distributed and Heterogeneous Servers”; Feb. 15, 2009; ICACT; pp. all. |
| U.S. Appl. No. 15/966,943 titled “Virtualized Server Systems and Methods Including Domain Joining Techniques” filed Apr. 30, 2018, pp. all pages of the application as filed. |
| U.S. Appl. No. 16/687,327, titled “Virtualized File Server Rolling Upgrade”, filed Nov. 19, 2019, pp. all pages of the application as filed. |
| U.S. Appl. No. 17/091,758 titled “Virtualized File Server Distribution Across Clusters”, filed Nov. 6, 2020, pp. all pages of the application as filed. |
| U.S. Appl. No. 16/942,929 titled “Method Using Access Information in a Distributed File Server Virtual Machine (Fsvm) Architecture, Including Web Access”; filed Jul. 30, 2020, pp. all pages of the application as filed. |
| U.S. Appl. No. 16/944,323 titled “Actions Based on File Tagging in a Distributed File Server Virtual Machine (FSVM) Environment”, filed Jul. 31, 2020, pp. all pages of the application as filed. |
| Dell EMC; Dell EMC Isilon OneFS Operating System; Scale-out NAS to maximize the data capital and business value of your unstructured data; 2020, pp. all. |
| Dell EMC; White Paper; Dell EMC Isilon OneFS Operating System; Powering the Isilon Scale-Out Storage Platform; Dec. 2019, pp. all. |
| EMC Isilon OneFS Operating System; Powering scale-out storage for the new world of Big Data in the enterprise; www.EMC.com; captured Feb. 2020, pp. all. |
| Isilon OneFS, Version 8.0.1; Web Administration Guide; Published Oct. 2016, pp. all. |
| U.S. Appl. No. 15/833,255, entitled “Cloning Virtualized File Servers”, filed Dec. 6, 2017, pp. all pages of the application as filed. |
| U.S. Appl. No. 15/833,391, entitled “Virtualized Server Systems and Methods Including Scaling of File System Virtual Machines”, filed Dec. 6, 2017, pp. all pages of the application as filed. |
| U.S. Appl. No. 15/422,220, entitled “Virtualized File Server” filed Feb. 1, 2017, pp. all pages of the application as filed. |
| U.S. Appl. No. 15/829,340, entitled “Configuring Network Segmentation for a Virtualization Environment”, filed Dec. 1, 2017, pp. all pages of the application as filed. |
| U.S. Appl. No. 15/829,731, entitled “Transparent Referrals for Distributed File Servers”, filed Dec. 1, 2017, , pp. all pages of the application as filed. |
| U.S. Appl. No. 15/829,781, entitled “Virtualized Server Systems and Methods Including Load Balancing for Virtualized File Servers”, filed Dec. 1, 2017, , pp. all pages of the application as filed. |
| U.S. Appl. No. 15/832,310 entitled “Disaster Recovery for Distributed File Servers, Including Metadata Fixers”, filed Dec. 5, 2017, , pp. all pages of the application as filed. |
| U.S. Appl. No. 16/140,250 titled “Virtualized File Server Data Sharing” filed Sep. 24, 2018, , pp. all pages of the application as filed. |
| U.S. Appl. No. 16/160,618 titled “Virtualized File Server Backup To Cloud” filed Oct. 15, 2018, , pp. all pages of the application as filed. |
| VMware vSphere VMFS “Technical Overview and Best Practices”, a VMware Technical White Paper updated for VMware vSphere 5.1, Version 3.0; Nov. 27, 2012, pp. all. |
| “Administering VMware vSAN—VMware vSphere 7.0”, 2015-2020, pp. 1-114. |
| “Backup vSAN 7 File Share with Veeam Backup & Replication 10”, Sysadmin Stories, https://www.sysadminstories.com/2020/06/backup-vsan-7-file-share-with-veeam.html, Jun. 2, 2020, pp. 1-7. |
| “Characteristics of a vSAN Cluster”, May 31, 2019, pp. 1-2. |
| “Designing and Sizing Virtual SAN Fault Domains”, Administering VMware Virtual SAN; VMware vSphere 6.5; vSAN 6.6; https://docs.vmware.com/en/VMware-vSphere/6.5/virtual-san-66-administration-guide.pdf, captured Aug. 20, 2021, 2017, pp. 34. |
| “Enabling or disabling SMB automatic node referrals”, NetApp https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.cdot-famg-cifs%2FGUID-AC7E8515-3A4C-4BB5-A8C8-38B565C952E0.html, Captured Sep. 19, 2019, pp. all. |
| “Guaranteeing throughput with QoS”, NetApp https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.pow-perf-mon%2FGUID-77DF9BAF-4ED7-43F6-AECE-95DFB0680D2F.html, Captured Sep. 19, 2019, pp. all. |
| “How to troubleshoot the ‘Autolocation’ feature in Clustered Data ONTAP”, NetApp https://kb.netapp.com/app/answers/answer_view/a_id/1030857/loc/en_US#_highlight, Captured Sep. 19, 2019, pp. all. |
| “How to Troubleshoot the ‘Autolocation’ feature in Clustered Data ONTAP—Results”, NetApp https://kb.netapp.com/app/results/kw/autolocation/, Captured Sep. 19, 2019, pp. all. |
| “Hybrid Cloud Storage with Cloudian HyperStore and Amazon S3”, Cloudian Inc.; www.cloudian.com, 2014, pp. all. |
| “Improving client response time by providing SMB automatic node referrals with Auto Location”, NetApp https://library.netapp.com/ecmdocs/ECMP1196891/html/GUID-0A5772A4-A6D7-4A00-AC2A-928868C5B3B5.html, Captured Sep. 19, 2019, pp. all. |
| “Managing Workloads”, NetApp https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.pow-perf-mon%2FGUID-13D35FC5-AF37-4BBD-8A8E-B10B41451A16.html, captured Sep. 19, 2019, pp. all. |
| “Nutanix AFS—Introduction & Steps For Setting Up”, Retrieved from https://virtual building blocks. com/2 0 1 8/01 /03/nutanix-afs-introduction-steps-for-setting-up/ (Year: 2018), Jan. 3, 2018, 1-23. |
| “Path Failover and Virtual Machines”, vSphere Storage; Update 2; VMware vSphere 7.0; VMware ESXi 7.0; vCenter Server 7.0; https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-702-storage-guide.pdf, Jun. 25, 2021, pp. 238. |
| “Protect Your Data With Netapp Element Software”, Solution Brief; NetApp, 2018, pp. all. |
| “Tech TopX: AHV One Click Upgrade”, Screen captures from YouTube video clip entitled “Tech TopX: AHV One Click Upgrade,” 13 pages, uploaded on Dec. 8, 2015 by user “Nutanix University”. Retrieved from Internet: https://www.youtube.com/watch?v=3dALdzw6qZM, Dec. 8, 2015, pp. all. |
| “Understanding Multipathing and Failover”, vSphere Storage; VMware vSphere 7.0; VMware ESXi 7.0; vCenter Server 7.0 https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-702-storage-guide.pdf, Jun. 25, 2021, pp. 234-268. |
| “Virtual Disk Manager User's Guide: Virtual Disk Development Kit”, vmware.com, 2008, 1-12. |
| “VMware vCenter Server: Centrally Mananged Virtual Infrastructure Delivered with Confidence”, VMWare Datasheet; https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vCenter/vmware-vcenter-server-datasheet.pdf, captured Aug. 20, 2021, 2015, pp. 1-2. |
| “VMware VSAN 7.0 Release Notes”, VMware; https://docs.vmware.com/en/VMware-vSphere/7.0/m/vmware-vsan-70-release-notes.html, Mar. 8, 2021, pp. 1-12. |
| “VSAN 7.0 U2 Proof of Concept Guide”, VMwareStorage; https://images.core.vmware.com/sites/default/files/resource/vsan_70_u2_proof_of_concept_guide_noindex.pdf, printed May 18, 2021, Apr. 2021, pp. 1-267. |
| “VSAN File Services Tech Note | VMware”, Aug. 2021, pp. 1-7. |
| “VSAN Health Service—File Service—File Server Health (77165)”, VMware, Knowledge Base; https://kb.vmware.com/s/article/77165, May 15, 2021, pp. 1-5. |
| “VSAN Monitoring and Troubleshooting—VMware vSphere 7.0”, https://docs.vmware.com/, Sep. 2018, pp. 1-61. |
| “VSAN Performance Graphs in the vSphere Web Client (2144493)”, Nov. 9, 2020, pp. 1-42. |
| “VSan Planning and Deployment”, Update 2 VMWare vSphere 6.7; VMware vSAN 6.7; https://docs.vmware.com/en/VMware-vSphere/6.7/vsan-673-planning-deployment-guide.pdf, Aug. 20, 2019, pp. 1-85. |
| “VSan Stretched Cluster Guide”, VMwareStorage; https://images.core.vmware.com/sites/default/files/resource/vsan_stretched_cluster_guide_noindex.pdf, printed Jun. 24, 2021, Jun. 2020, pp. 1-62. |
| “VSphere Availability—VMware vSphere 6.7”, https://docs.vmware.com/, Jan. 11, 2019, pp. 1-105. |
| “VSphere Storage—VMware vSphere 6.7”, https://docs.vmware.com/, Jan. 4, 2021, pp. 1-382. |
| Bhardwaj, Rishi “The Wonderful World of Distributed Systems and the Art of Metadata Management”, Nutanix, Inc., https://www.nutanix.com/blog/the-wonderful-world-of-distributed-systems-and-metadata-management; captured Aug. 19, 2021, Sep. 24, 2015, pp. 1-8. |
| Birk, Ryan “How it Works: Understanding vSAN Architecture Components”, altaro.com, Feb. 28, 2018, pp. 1-10. |
| Cormac “Native File Services for vSAN 7”, CormacHogan.com, Mar. 11, 2020, pp. 1-23. |
| Feroce, Danilo “Leveraging VMware vSAM for Highly Available Management Clusters”, VMware, Inc., Version 2.9, VMware, Inc., Jan. 2018, 1-22. |
| Fojta, Tomas “Quotas and Quota Policies in VMware Cloud Director—Tom Fojta's Blog”, Nov. 6, 2020, pp. 1-4. |
| Fojta, Tomas “vSAN File Services with vCloud Director—Tom Fojta's Blog”, (wordpress.com) (“Fojta Blog”) captured Feb. 11, 2021, pp. 1-8. |
| Hogan, Cormac “New updates from Nutanix—NOS 3.0 and NX-3000”, https://cormachogan.com/2012/12/20/new-from-nutanix-nos-3-0-nx-3000/, Dec. 20, 2012, pp. 1-7. |
| Kemp, Erik “NetApp SolidFire SnapMirror Architecture and Configuration”, Technical Report, NetApp, Dec. 2017, pp. all. |
| Kleyman, Bill “How Cloud Computing Changes Storage Tiering”, https://www.datacenterknowledge.com, captured Jun. 4, 2019, Nov. 12, 2015, pp. all. |
| Leibovici, Andre “Nutanix One-Click Upgrade now takes care of Firmware and Hypervisor too!”, myvirtualcloud.net https://myvirtualcloud.net/nutanix-one-click-upgrade-now-takes-care-of-firmware-and-hypervisor-too/, Jul. 31, 2014, pp. 1-4. |
| Rajendran, Cedric “Working with vSAN Health Checks”, VMware vSan Virtual Blocks Blog; https://blogs.vmware.com/virtualblocks/2019/07/18/working-with-vsan-health-checks/, Jul. 18, 2019, pp. 1-6. |
| Seget, Vladan “VMware vSAN 7 now with native file services and quotas”, May 1, 2020. |
| Seget, Vladan “VMware vSphere 7.0 and vSAN storage improvements”, Apr. 1, 2020, pp. 1-12. |
| U.S. Appl. No. 17/585,403 titled “Virtualized File Server Smart Data Ingestion” filed Jan. 27, 2022, pp. all pages of the application as filed. |
| Sturniolo, Andy “VMware vSAN File Services and Veeam”, Veeam Blog, https://www.veeam.com/blog/veeam-backup-vsan-file-services.html, Jul. 22, 2020, 1-9. |
| Poitras, Steven. “The Nutanix Bible” (Oct. 15, 2013), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown); pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jan. 11, 2014), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown); pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jun. 20, 2014), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown); pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jan. 7, 2015), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown); pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jun. 9, 2015), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown); pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Sep. 4, 2015), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jan. 12, 2016), from https://nutanixbible.com/ ; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jun. 9, 2016), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jan. 3, 2017), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jun. 8, 2017), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jan. 3, 2018), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jun. 25, 2018), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jan. 8, 2019), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Jul. 25, 2019), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Sep. 17, 2019), from https://nutanixbible.com/; pp. all. |
| Cano, Ignacio et al. “Curator: Self-Managing Storage for Enterprise Clusters”; University of Washington; published Mar. 2017; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Mar. 2, 2020), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Sep. 1, 2020), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Mar. 2, 2021), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven. “The Nutanix Bible” (Sep. 3, 2021), from https://nutanixbible.com/; pp. all. |
| Poitras, Steven , “The Nutanix Bible”, https://nutanixbible.com/, Apr. 9, 2019, pp. all. |
| “Carbonite Availability for Windows User's Guide Version 8.1.0”, Carbonite, Inc. (United States), Feb. 27, 2018, pp. 1-391. |
| Number | Date | Country | |
|---|---|---|---|
| 20230237170 A1 | Jul 2023 | US |
