Patent Grant
9201677
Various types of computing environments share resources across multiple hosts or other systems. For example, virtualized systems and virtualized environments often support the sharing and load balancing of resources across multiple hosts or other systems. In this example, a single host can support multiple virtual machines that share common hardware components, storage systems, and the like. These virtual machines may also be referred to as “guest operating systems” as each host is capable of supporting multiple instances of one or more operating systems.
When sharing a data storage system across multiple hosts or multiple virtual machines, the computing environment must properly manage a high volume of data input/output (I/O) operations. The volume of I/O operations is commonly measured in IOPS (I/O Operations Per Second).
During their normal operation, virtual machines 110 initiate data I/O requests, such as data read requests and data write requests, associated with data storage system 108. Data storage system 108 includes multiple data storage drives 112 and/or other data storage mechanisms. The storage resources associated with data storage system 108 are shared among the multiple hosts 102-106 and the virtual machines 110 included in those hosts. Each host 102-106 includes a virtualization kernel 114 (also referred to as a “hypervisor”) that manages the virtual machines 110 as well as shared resources, such as data storage system 108.
As the number of virtual machines associated with a particular host increases, there is a corresponding increase in demand for shared resources, such as memory and I/O resources. An increase in I/O resource utilization includes an increased number of data I/O operations that cause a corresponding increase in data communicated between a host and a data storage system. In existing virtualized systems, the increased demand for shared resources such as I/O bandwidth often degrades the performance or application throughput of latency sensitive workload operations within a virtualized system. In these situations, one or more of the virtual machines experiences increased latency or decreased throughput, which may decrease the performance of the virtual machines. Thus, it is desirable to provide a computing environment that improves the handling of data I/O operations associated with multiple hosts or other systems.
It is also desired to provide approaches that leverage existing virtual machine system applications that depend on offsite resources, but still optimizes memory and I/O resources. According to embodiments of the invention, some of these approaches require a local system to cooperate with existing virtual system operating systems to appear to be operating as expected and without any different, circumventing or disruptive operations. As will be seen, the invention provides such approaches, each in an elegant manner.
Throughout the description, similar reference numbers may be used to identify similar elements.
The systems and methods described herein relate to the management of data input/output (I/O) operations in a computing environment. Although particular examples discussed herein relate to virtualized environments, the same systems and methods are applicable to any type of computing environment. In particular implementations, the described systems and methods intercept I/O operations in the virtualized environment to dynamically allocate resources, such as cache resources, across multiple virtual machines in the virtualized environment. This management of data I/O operations improves the performance of the virtual machines and reduces the number of I/O operations handled by the primary storage system. Additionally, the management of I/O operations is transparent to other components in the virtualized environment and can be implemented without modification to existing application software or existing data storage systems. Thus operating systems that currently exist will be oblivious to the operations of the embodiments described herein, which will cooperate with the basic operation characteristics of virtual operating systems and not disrupt them, while better optimizing the operations of virtual machines resident in hosts.
Specific systems and methods described herein utilize a cache memory constructed with various memory devices, such as flash memory devices or RAM (random access memory) that may or may not be block oriented. The systems and methods described herein do not differentiate between flash memory, RAM or other types of memory, and further envision new types of memory developed in the future that will utilize various embodiments described herein. The described systems and methods may utilize any type of memory device, regardless of the specific type of memory device shown in any figures or described herein. Particular systems and methods described herein may generally be referred to as an “I/O hypervisor” due to their management of I/O operations in a virtualized environment.
Each virtual machine 206-210 can implement a different operating system, such as Windows, Linux, and so forth. In a particular embodiment, host 200 is a computing device capable of hosting the multiple virtual machines 206-210 and supporting the applications executed by the virtual machines and the functions associated with those applications. Host 200 includes, for example, one or more processors, memory devices, communication devices, I/O interfaces, and related components. Although three virtual machines 206-210 are shown within host 200, a particular embodiment may include any number of virtual machines.
Virtualization kernel 204 manages the operation of virtual machines 206-210 as well as other components and services provided by host 200. For example, virtualization kernel 204 handles various I/O operations associated with a primary storage system 212 or other storage devices. Primary storage system 212 is shared among the multiple virtual machines 206-210, and may be shared by multiple hosts. In a particular embodiment, primary storage system 212 includes multiple disk drives or other storage devices, such as storage arrays.
Host 200 also includes a cache provisioner 214 and a cache 216 containing one or more memory devices, such as flash memory devices or RAM. A flash memory device is a non-volatile memory that can be repeatedly erased and reprogrammed. A cache memory constructed using flash memory may also be referred to as a solid state drive (SSD). Cache 216 is managed by cache provisioner 214 to dynamically provision capacity and IOPS to virtual machines 206-210. Cache provisioner 214 allows multiple virtual machines to share the same cache without risk of having two virtual machines access the same cache page. Additional details regarding the operation of cache provisioner 214 and cache 216 are discussed herein.
Each virtual machine 206-210 includes an I/O driver 218 and a cache management system 220—also referred to as a CFS (Cache File System). I/O driver 218 intercepts I/O operations generated by the associated virtual machine and directs the I/O operation to cache provisioner 214 for processing. I/O driver 218 is particularly effective at intercepting I/O operations due to its location within the virtual machine and its close proximity to the source of the data associated with the I/O operation. I/O driver 218 may also be referred to as a “device driver”. In a particular embodiment, the I/O drivers are included with an operating system. For example, each device comes with its own device driver. These device drivers have a generic component that is a part of the operating system and there is a device-specific component that is typically supplied by the device vendor. In a particular embodiment, the I/O drivers discussed herein are implemented on top of both these drivers. These I/O drivers are in the path of the device driver and intercept well known I/O APIs that are published by the operating system. This architecture is often referred to as a filter driver. In a particular implementation, this is referred to as a filter driver that sits above standard device drivers for I/O operations.
Cache management system 220 contained in each virtual machine interacts with cache provisioner 214 and other components to manage access to cache 216. For example cache management system 220 includes multiple cache tags that are used in associating an address in a virtual machine with a physical address in cache 216. Cache provisioner 214 manages the storage capacity of cache 216 by, for example, allocating cache space among the multiple virtual machines 206-210, as discussed herein. The allocation information associated with a particular virtual machine is communicated to the cache management system in that virtual machine. Additional details regarding the operation of I/O driver 218 and cache provisioner 214 as well as the use of cache tags are provided below.
In a particular embodiment, each virtual machine 206-210 represents a virtual desktop, such as a desktop environment associated with a particular user. In this embodiment, the user accesses the desktop environment via a terminal or other system. This desktop environment is commonly referred to as VDI (Virtual Desktop Infrastructure). Thus, a single host can replace hundreds or more individual desktop computing systems. In another embodiment, each virtual machine 206-210 represents a server application. In this embodiment, a single host can replace any number of individual software or application servers running multiple server applications.
Virtualized environment 300 also includes a primary storage system 316 that is shared among the multiple host systems 302-306 and the multiple virtual machines 308 in those host systems. In a particular embodiment, primary storage system 316 includes multiple disk drives or other storage devices.
Cache management system 220 also includes a steal candidate module 408 that identifies stored cache data that are candidates for removal from the cache. A cache page management module 410 manages various cache page data and related operations. A valid unit map module 412 identifies valid data stored in a cache and/or a primary storage system. A page size management module 414 performs various page size analysis and adjustment operations to enhance cache performance. Finally, an interface module 416 allows cache management system 220 to interact with other components, devices and systems.
Procedure 500 continues as the cache management system determines whether the requested data is available in the cache (block 508), such as cache 216 or 314. If the data is determined to be in the cache (block 510), the procedure branches to block 512, where the requested data is retrieved from the cache. If the data is not available in the cache, the procedure branches to block 514, where the requested data is retrieved from the primary storage system, such as primary storage system 212 or 314 discussed above. After retrieving the requested data from the primary storage system, the procedure determines whether to write the retrieved data to the cache (block 516) to improve the storage I/O performance of the virtual machine. This determination is based on various cache policies and other factors.
The cache management system discussed herein (also referred to as a “Cache File System or CFS”) treats the flash memory devices as a cache, but uses a file system model. The cache management system develops and maintains a working set for the cache. In general, the working set is the set of data that should be contained in the cache to support optimal performance of the host and its supported virtual machines.
As mentioned above, the cache is created using flash memory devices. These devices typically provide fast read operations, but slow write operations. These slow write operations can result in a significant delay when initially developing the working set for the cache. Additionally, flash devices can generally accept a limited number of write operations. After reaching the “write lifetime” of the flash device, portions of the flash device become unusable and the integrity of the device begins to deteriorate. These characteristics of flash devices are taken into consideration by the cache management system when managing the cache.
Cache chunks 602 are assigned to virtual machines based on the cache needs of each virtual machine. The number of chunks 602 assigned to a particular virtual machine can change over time as the cache needs of the virtual machine changes. The number of chunks 602 assigned to a specific virtual machine defines the cache capacity of that virtual machine. For example, if two 256 MB chunks are assigned to a specific virtual machine, that virtual machine's cache capacity is 512 MB. The assignment of chunks 602 to particular virtual machines is handled by the cache provisioner.
Cache tags are used in mapping storage I/O addresses in a virtual machine to actual cache pages 604 (e.g., physical addresses in the cache). The cache tags can cache data associated with any storage device assigned to a virtual machine. These cache tags perform translations between the addresses of blocks on the storage device (e.g., the primary storage system) and a cache address. Cache tags are organized linearly in RAM or other memory. This allows the address of the cache tag to be used to locate a physical cache page because of the algorithmic assumption that each cache tag has a linear 1:1 correspondence with a physical cache page.
As shown in
In a particular embodiment, a “thin provisioning” approach is used when allocating cache chunks to the virtual machines. In this embodiment, each virtual machine is allocated a particular number of cache chunks, as discussed above. However, the entire cache capacity is “published” to each of the virtual machines. For example, if the total cache size is 1 TB, each virtual machine reports that it has access to the entire 1 TB of storage space. However, the actual allocation of cache chunks may be considerably smaller (e.g., 256 MB or 512 MB)—based on the current needs of the virtual machine. The allocated cache chunks represent a specific range of addresses allocated to each virtual machine within the cache. The cache provisioner dynamically changes these cache chunk allocations as each virtual machine's working set requirements change. Regardless of the number of cache chunks actually allocated to a particular virtual machine, that virtual machine reports that it has access to the entire 1 TB cache.
If the decision is to write the retrieved data to the cache, the cache management system uses the memory address of the cache tag to determine a physical cache address associated with the data to be written. The data is then written to the cache using the physical cache address associated with the data.
If the requested data is in the cache (block 706), the cache management system uses the memory address of the cache tag to determine a physical cache address associated with the requested data (block 712). The requested data is then retrieved from the cache using the physical cache address associated with the requested data (block 714).
Storing the cache tag information within the associated virtual machine allows the virtual machine to easily determine where the data is stored physically in the cache without having to access a different system or process. Instead, the systems and methods described herein allow each virtual machine to quickly access cache tags, which increases the speed and efficiency of the I/O operations. Additionally, the virtual machine typically understands the data it is processing better than other systems. For example, the virtual machine understands the nature and context of the data it is processing. This understanding of the data enhances the development and management of an effective working set. Other systems that are external to the virtual machine may simply see the data as raw data without any context or other understanding. Thus, having the cache tag information stored locally in the virtual machine enhances the operation of the virtual machine and the I/O operations.
Next, the virtual machine writes the data associated with the data write operation to the cache using the physical cache address (block 808). The virtual machine also simultaneously writes the data associated with the data write operation to the primary storage system (block 810). The original data write operation is completed when the primary storage system acknowledges a completed write operation (block 812).
In a particular implementation, the cache discussed herein is a write-through cache. This type of cache writes data to both the primary storage system and the cache. A write completion is acknowledged after the write operation to the primary storage system is completed, regardless of whether a corresponding write operation to the cache has completed. In specific embodiments, cache write operations can be queued and completed as the cache speed allows. Thus, a cache with a slow write speed (or a queue of pending write operations) does not degrade performance of the overall system. Cache tags associated with incomplete or queued write operations are identified as “pending.” After the write operation completes, the associated cache tag is identified as “valid”. When the cache tag is identified as “pending,” any attempted read of the data associated with the cache tag results in a cache miss, causing retrieval of the requested data from the pending memory buffer associated with the I/O, or from the primary storage system.
As mentioned above, each cache tag stored in a virtual machine is associated with a particular cache page. Additionally, the systems and methods described herein are capable of dynamically allocating cache resources (e.g., cache chunks) to the virtual machines in a virtualized environment. Using the features of the present invention, the number of cache tags associated with a particular virtual machine can be increased beyond the number of cache pages actually associated with the virtual machine. This increase in cache tags allows the cache management system to determine whether increasing the number of cache pages assigned to the particular virtual machine will likely improve the cache hit rate for that virtual machine. In other words, the systems and procedures described herein assist in determining cache misses caused by limited cache storage capacity. Specifically, this allows us to determine cache capacity misses.
The procedure increases the number of cache tags associated with the virtual machine without increasing the cache size (block 908). For example, the procedure may increase the number of cache tags by an amount that corresponds to assigning an additional cache chunk to the virtual machine. However, the additional cache chunk is not actually assigned to the virtual machine at this point in the evaluation procedure. Next, procedure 900 monitors the cache hit rate using the increased number of cache tags (block 910). After monitoring the cache hit rate with the increased number of cache tags for a period of time, the procedure determines whether the cache hit rate has improved (block 912). If the cache hit rate has improved as a result of the additional cache tags, the procedure returns to block 908 to further increase the number of cache tags associated with the virtual machine.
The process of increasing the number of cache tags and monitoring the results continues until the increase in cache tags does not improve the cache hit rate. At this point, procedure 900 determines the minimum number of cache tags that provide improved cache performance (block 914). In an alternate embodiment, the procedure determines an optimal number of cache tags that provide optimal cache performance. The procedure then adjusts the cache size allocated to the virtual machine based on the number of cache tags that provide improved cache hit rate performance (block 916). Dynamic addition of cache chunks or capacity to a virtual machine is based on both the hit rate and other policy that handles cache resource provisioning to other virtual machines. The hit rate, IOPS improvements, and cache capacity are also adjusted using policy that can be controlled by the user or implemented algorithmically based on rules specified by the user.
In a particular embodiment, the number of cache tags added at block 908 is substantially the same as the number of the cache pages in a particular cache chunk. Thus, allocating additional cache resources to the virtual machine is performed by allocating a number of cache chunks that corresponds to the minimum number of cache tags that provide improved cache performance.
Each clock hand has a different time interval. In the example of
As shown in
After a second ten minute clock sweep without any access of the cache page, the Bit 1 of clock hand 1 is cleared, leaving a clock hand value of “00”. At this time, the cache page associated with this example is identified as a “steal” candidate; i.e., the cache page is a candidate for removal from the cache due to a lack of access of the cache page data. A separate table or other listing is maintained for cache pages in which both clock hands have been cleared. Cache pages with both clock hands cleared are top candidates for “steal” prior to cache pages with only one clock hand cleared.
As shown in
Finally, the cache tag data structure includes a valid unit map field, which is a dynamic field that identifies which units in a page are cached. An example of a unit within a cache page is a sector. For example, a particular page may have one or more sectors that are missing or no longer valid. The valid unit map identifies the status of all units associated with a particular cache page to prevent accessing data in units that is not valid.
As discussed above, cache chunks (and corresponding cache tags) are allocated dynamically among multiple virtual machines. The allocation is dynamic due to changes in data storage requirements, changes in applications executing on the virtual machines, and the like.
In a particular implementation of the systems and methods described herein, the cache supports multiple page sizes. Different applications executing in the virtual environment may require different page sizes to function properly. For example, some applications always perform 32K data I/O operations. For these applications, it is desirable to use a large cache page size, such as 16K or 32K, to minimize the number of data I/O operations necessary to handle the 32K of data. For example, if the cache page size is 4K and the application performs a 32K data I/O operation, eight cache pages must be accessed to read or write the 32K of data. Performing eight separate I/O operations to accommodate the 32K of data is a burden on system resources and dramatically increases the number of I/O operations that must be processed by the system. In contrast, if the cache page size is 16K, only two I/O operations are required to process the 32K of data. Thus, the larger cache page size reduces I/O operations and the corresponding burden on system resources.
Using larger cache page sizes also reduces the number of cache tags, thereby reducing the memory space required to store the cache tags. For example, in a one terabyte cache having 4K cache pages, 256M cache tags are necessary to provide a single cache tag for each cache page. In the same system using 16K cache pages, 64M cache tags are needed. Thus, the larger cache page size reduces the number of cache tags and the memory resources needed to store the cache tags.
Although larger cache page sizes can reduce I/O operations and reduce the number of cache tags, in certain situations a larger cache page size can result in underutilized cache resources. For example, if a system is using a 32K cache page size and an application performs a 4K I/O operation, only a small fraction of the 32K page is used (28K of the page is not needed). This situation results in significant unused cache resources. Therefore, the systems and methods described herein support multiple cache page sizes to improve utilization of system resources, such as I/O resources and cache storage resources.
Different applications have different data storage characteristics. Applications can be characterized as having “sparse address spaces” or “dense address spaces”. Sparse address spaces tend to have scattered data with significant gaps between different groupings of data. In contrast, dense address spaces tend to have data that is more compact with fewer (or smaller) gaps between different groupings of data. When selecting cache page sizes for a particular virtual environment, it is important to consider the data storage characteristics (e.g., sparse or dense address spaces) associated with applications executing in the virtual environment. There can be exceptions where a sparse address space may comprise groups of contiguous data where the groups are sparsely located. In such cases one can use large pages even though the address space is sparse.
In a particular embodiment, data associated with existing applications can be analyzed prior to implementing a system or method of the type described herein. This prior analysis allows the system to be “tuned” based on typical application data. After the systems and methods are implemented, the dynamic nature of the system adjusts cache page sizes, cache allocations, system resources, and other parameters based on changes in the operation of the application.
In a particular implementation, a cache is divided into multiple sections such that each section supports different page sizes. For example, a cache may be divided into four sections, two of which support 4K cache pages, one that supports 16K cache pages, and one that supports 32K cache pages. The cache pages in these different sections are allocated to different virtual machines and different applications based, for example, on the data storage characteristics of the applications.
In one embodiment, different hash tables are used for different cache page sizes. Each hash table has its own associated hash function that identifies a particular hash slot in the table based on an address provided to the hash function. When using multiple hash tables, such as a 4K hash table and a 16K hash table, the systems and methods perform a lookup operation for each hash table. Performing a lookup in both hash tables is necessary because a 4K address could be contained within a 16K entry in the 16K hash table. To enhance the lookup process, the systems and methods described herein apply one or more algorithms based on a percentage of cache hits associated with different cache page sizes, a success rate associated with different hash tables, and other factors.
In a particular implementation, an algorithm uses both the percentage of cache hits associated with cache page sizes and the success rate associated with different hash tables to search for data in a cache.
In other embodiments, the systems and methods use a single hash table associated with the smallest cache page size, such as 4K. Although the cache supports multiple cache page sizes, the hash table uses a 4K page size exclusively. This approach eliminates the need to perform a lookup in multiple hash tables associated with different cache page sizes. In this scheme a 16K page would require 4 hash table lookups and groups of cache tags are managed as one.
In certain situations, it is desirable to prevent one or more cache pages from being stolen or usurped by another virtual machine. This is accomplished in the systems and methods discussed herein by “pinning” the cache tags associated with the cache pages that are to be protected from being stolen. Cache tags are pinned by setting the state bit to “pinned state” in the cache tag.
Pinning cache tags is used in a variety of situations. For example, a system may “freeze” a group of cache tags associated with a virtual machine and move the cache tags to a persistent storage device to preserve the virtual machine's working set. Later, when the virtual machine “warms up,” the cache tags are retrieved from the persistent storage device, and actual data is read back from the primary or shared storage, thereby recreating the working set. This allows the virtual machine to resume operation immediately with a fully functioning working set, rather than taking a significant period of time recreating the working set.
Pinning cache tags are also useful to lock a range of addresses in the cache. For example, a user can pin specific data within the cache to prevent the data from being replaced or modified. The user may know that the specified data is critical to the operation of the virtual machine and wants to ensure that the data is always available in the cache.
In certain situations, a portion of data associated with a read operation is available in the cache, but a portion is not available (or not valid) in the cache. In these situations, the system must decide whether to retrieve all of the data from the primary storage system or retrieve a portion from the cache and the remainder from the primary storage system. The decisions involving what's available in the cache can result in more than 1 I/O to primary or shared storage (which is usually more efficient when doing sequential I/Os). Our algorithms have the ability to control the amount of fragmentation of I/Os to primary storage based on its I/O characteristics.
In a particular embodiment, a checksum is calculated for each cache page. When calculating the checksum, the system only performs the calculation on the valid data, based on a valid unit map (e.g., the valid data sectors). When a write operation is performed that increases the number of valid data sectors, the checksum is recalculated to include the new valid data sectors.
Computing device 1400 includes one or more processor(s) 1402, one or more memory device(s) 1404, one or more interface(s) 1406, one or more mass storage device(s) 1408, one or more Input/Output (I/O) device(s) 1410, and a display device 1428 all of which are coupled to a bus 1412. Processor(s) 1402 include one or more processors or controllers that execute instructions stored in memory device(s) 1404 and/or mass storage device(s) 1408. Processor(s) 1402 may also include various types of computer-readable media, such as cache memory.
Memory device(s) 1404 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM)) 1414 and/or nonvolatile memory (e.g., read-only memory (ROM)) 1416. Memory device(s) 1404 may also include rewritable ROM, such as Flash memory.
Mass storage device(s) 1408 include various computer readable media, such as magnetic tapes, magnetic disks, optical disks, solid state memory (e.g., Flash memory), and so forth. As shown in
I/O device(s) 1410 include various devices that allow data and/or other information to be input to or retrieved from computing device 1400. Example I/O device(s) 1410 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.
Display device 1428 includes any type of device capable of displaying information to one or more users of computing device 1400. Examples of display device 1428 include a monitor, display terminal, video projection device, and the like. Interface(s) 1406 include various interfaces that allow computing device 1400 to interact with other systems, devices, or computing environments. Example interface(s) 1406 include any number of different network interfaces 1420, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interfaces include a user interface 1418 and a peripheral device interface 1422.
Bus 1412 allows processor(s) 1402, memory device(s) 1404, interface(s) 1406, mass storage device(s) 1408, and I/O device(s) 1410 to communicate with one another, as well as other devices or components coupled to bus 1412. Bus 1412 represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.
For purposes of illustration, programs and other executable program components are shown herein as discrete blocks, although it is understood that such programs and components may reside at various times in different storage components of computing device 1400, and are executed by processor(s) 1402. Alternatively, the systems and procedures described herein can be implemented in hardware, or a combination of hardware, software, and/or firmware. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein.
In another embodiment, an issue of compatibility that occurs within virtual systems is addressed. In certain virtual systems, some of the processes make certain assumptions about the environment in order to properly operate.
In a single host, there typically will be multiple virtual machines operating in the host. Each virtual machine will have its own separate I/O drivers and also separate cache management module to manage local storage operations from the perspective of each particular virtual machine. Each virtual machine needs to share the local storage cache and each virtual machine will have its own unique demand for space on the local storage cache during its operation. Multiple virtual disks may be created on the local cache storage, and these can be exposed to the local virtual machines. During operation of the various virtual machines, the demand can vary among the different virtual machines. As a result, capacity in the local cache may not be efficiently utilized by the virtual machines, and cache capacity may be wasted.
In one example, a thin provisioned storage is provided, such as a thin provisioned cache, for dynamic allocation of storage space among multiple virtual machines within a host. Since virtual machines are dynamic in nature, their demand for storage space may vary. If they share actual storage space with other virtual machines, the use of the storage space by a group of virtual machines may conflict. For example, if one or more virtual machines experience a higher than normal I/O traffic rate, their operations may become bogged down, causing lags in output. Other machines may experience a lower than normal I/O traffic rate at the same time, leaving their allocated storage space unutilized. Thus, in some cases, the higher I/O virtual machines' use of actual storage space may be unnecessarily restricted by rigid or inefficient allocation schemes. Virtual machines may be transferred from one host to another, may become inactive or offline for some period of time, may power down or rest on a host that needs to power down, or its demand for storage space may change up or down during operation. Thus, it would be useful if the storage space allocated to the group of virtual machines could be dynamically allocated and re-balanced, where actual storage space allocated to any one machine can be apportioned more intelligently. As such, dynamic allocation of storage space could serve to reduce lag time for virtual machines that demand more space and I/O transfers by provisioning more space when other virtual machines associated with the same storage demand less space. The embodiment provides such solutions in an elegant manner.
In typical virtual machine environments, shared storage is utilized among multiple hosts that have equal access to the common storage space. The shared storage may be a clustered file system, a virtual machine file system (VMFS), where the system provides correctness and consistency among the various virtual machine hosts using file based locking and other methods.
One common feature in virtual machine systems is the ability to move a virtual machine from one host to another host. VMWare™ has a product called VMotion™ that enables virtual machines to move from one host to another, where the main storage of the moving virtual machine is maintained on storage that is shared among two or more hosts. The virtual machine may be a live operating virtual machine located on one host, and the desire is to be able to move the virtual machine from one host to another without interruption in the virtual machine during relocation. This is possible because the multiple hosts see and share the common data storage system. Thus, the virtual machine may move from one host to another without shutting down or rebooting the virtual machine, the move is transparent to the moving virtual machine.
When a virtual machine boots up and begins to run, it communicates with its available resources, such as storage devices, network devices, etc., similar to a physical machine. It may send out Small Computer System Interface (SCSI) inquiries to connected storage devices to determine what resources are available, and it discovers what storage is available to the virtual machine. The storage available to the virtual machine is virtual storage that is encapsulated in a file. The encapsulated file is the main storage space for the virtual machine. Thus, the storage for the virtual machine is now instantiated in a file and becomes a virtual hard drive. In prior art devices, this file is stored in the common data storage system shared among multiple hosts.
According to one embodiment, it is desired to store the virtual disk of the virtual machines hosted on a single host in local storage, such as the cache storage. In such a system, if a virtual machine existed that stores its main drive storage on the local cache storage located on the host, a virtual machine would not be able to move from one host to another host. Again, in prior art systems, the virtual disk of the virtual machines is located on storage that is shared among the hosts that are physically separate but commonly connected to the shared storage system.
A virtual disk's block number zero translates to offset zero in the file encapsulating the virtual disk. In response to the virtual machine sending out inquires to define its storage, the system replies that the virtual machine has a virtual storage. As that layer begins to receive reads and writes as SCSI (Small Computer System Interface) traffic it will convert this into file I/O and read and write to the shared file. Thus, a seed of a virtual disk is created on the shared storage that may be visible by the separate hosts. As a result, once the virtual machine moves from one host to another, the virtual machine may continue to operate because it can communicate with the original virtual disk associated with the virtual machine that was moved to the second host just as it did from the prior host. Therefore, in order to move a virtual machine from one host to another, there must be shared storage.
Once provisioned, each virtual machine expects to have access to predetermined and contiguous storage space for which it has the cache tags (discussed above). In one embodiment, a dynamic provisioning approach is provided to divide the cache storage into chunks that can be dynamically provisioned to the separate virtual machines according to their demand for space. According to one embodiment, a cache provisioner is encapsulated in a virtual logical unit number (VLUN) driver, provided to manage the chunks of storage data that is allocated to each virtual machine. A LUN is a misnomer of an acronym known in the art as a place where a machine can read and write a block of data, for example an array of storage disks or other storage devices. In a system, storage devices or arrays publish storage space as addressed that do not necessarily identify a particular storage device or individual storage disks. According to one embodiment, a VLUN disk is a virtual storage space allocated to a virtual machine. Since multiple virtual machines will typically be operating on a single host, the chunks of storage space that come available will likely be located in different physical areas of the cache storage. A VLUN device driver creates a VLUN disk that is assigned to the virtual machine.
In a virtual system, the virtual operating system runs processes and manages operations to allow different processes within the virtual system to properly operate and not conflict with other processes. In one example, an operating system of a virtual machine may operate with the assumption that each separate virtual machine operates with a fixed amount of storage space that typically does not change. Thus, the operating system of the virtual machine may react adversely or may not operate properly if there is sudden atypical change in storage space size for the virtual machine. Thus, it may be important for a virtual machine to appear to have a fixed allotment of storage space, such as cache storage space. According to one embodiment, this is achieved by allocating a limited amount of physical storage space to any one virtual machine as needed by the particular machine's needs. And, to avoid any potential conflict with the operating system that expects to detect a fixed storage space, a virtual amount of space is allocated to each virtual machine that is equal to a set amount of space that a virtual machine's operating system expects to detect. Thus, in operation, the operating system of the virtual machine will detect the set amount of virtual storage space that is allocated, and it will appear to the operating system that that amount of space is constant and consistent. However, in actual operation, the space allocated to a particular virtual machine may vary according to the machine's demand for storage space. And, the overall space will be traded among the virtual machines accessing storage cache to ensure that each virtual machine has no more cache storage than it actually needs, while the operating system is essentially fooled to thinking that each virtual machine has a fixed amount of cache storage space allocated to it.
Thus, the VLUN manager is configured to manage the dynamic allocation of the available chunks to the virtual machines that need them. The storage is thus physically managed in chunks by the VLUN driver that provides each virtual machine with the notion of contiguous chunks of storage space. The VLUN driver thus in a sense translates the allocation from the virtual space into the underlying physical chunks allocated to each virtual machine by the VLUN driver. As a result, the embodiment allows the system within the host to divide up the cache storage into chunks that it can allocate on the fly to the various virtual machines using virtualization of the storage space allocated to the individual virtual machines. In operation, the VLUN driver maintains mapping of the virtual space of each virtual machine to actual physical storage space located in the cache storage. This allows the VLUN to dynamically increase and decrease the size of the allocated storage space of each virtual machine.
Referring to
Map module 1522 may be configured within the VLUN to map the virtual space allotted to each virtual machine to physical space existing in the cache storage. Since the actual physical space allocated to a particular virtual machine may not be the same as the virtual cache storage space of the host, the two need to be reconciled so that the virtual machine can properly store and retrieve data (read and write data) stored in the physical cache storage.
For example, cache space allocated for Virtual Machine-11506 is illustrated diagrammatically as space 1524 in cache 1516. The virtual space allocated to this virtual machine in this example is two terabytes (2 TB), and the physical cache storage space that is actually allocated to this virtual machine in this example is four gigabytes (4 GB). As discussed above, in one embodiment, a virtual machine is configured to have an allocated physical storage space appear to its operating system as a fixed amount (2 TB in this example), but to have an actual physical storage allocation that is necessary for the particular virtual machine's operation (4 GB in this example). The actual physical storage space for any particular virtual machine may be more or less than that which appears to the operating system. Thus, the virtual barrier 1526 between a particular host's actual allocated physical space and virtual physical space may be different and may vary dynamically as groups of virtual machines that share common cache storage operate.
Furthermore, the chunks of storage space allocated to a particular virtual machine may be disbursed within the physical cache space 1516 in an incongruous manner, where the physical storage locations of data for one virtual machine may be interleaved with the storage locations of another virtual machine. This is a result of a configuration where chunks of physical cache storage space are allocated dynamically. The VLUN driver may usurp space from other machines that are not utilizing all of their allocated space and allocate the space to other virtual machines that need more allocated space in cache storage. Referring to
Further Referring to
The host system 1702 illustrated in
As discussed above, in operation, though the actual storage space that is allocated to a particular virtual machine is one value, another value is represented to the operating system so that the system as a whole operates in a stable manner. Thus, a virtual machine may have 4 GB of actual cache storage space allocated to it, but it may appear to the operating system by the virtual machine's representations that it has 2 TB of storage space allocated to it. Within the host there is a user space where the virtual machines reside, and there is a virtualization kernel 1720 where a VLUN (SCSI) driver 1722 resides and is configured to allocate the actual space that is allocated to each virtual machine in cache storage 1710. In order for the SCSI filter 1716 and CFS 1712 to properly operate and manage I/O operations, they both need to be informed of the actual storage space that is allocated to the virtual machine 1704 within the cache 1710, and they need to not be “fooled” that there is more space allocated to the virtual machine than has actually been provisioned to the virtual machine. There is a communication link 1724 that communicates separately from the I/O data traffic and between the VLUN driver 1722 and SCSI filter 1716 that informs CFS 1712 and I/O filter 1714 via the SCSI filter 1716 of the actual cache storage space allocated to the virtual machine 1704. Thus, asynchronous out of band messages may be sent between the VLUN driver 1722 and the SCSI filter 1716 to inform the Windows driver 1708 of actual space allocated to the virtual machine 1704 in the system. The information reaches CFS 1712 so that CFS manages the cache tags used to manage the data stored in the allocated cache storage space within cache storage 1710. Thus, the cache is a thin provisioned cache, where the operating system perceives the appearance of a large amount of space, such as 2 TB for example, but each virtual machine actually gets allocated the amount of storage space it actually needs, 4 GB for example. The communication path 1724 allows the ability to inform the Windows driver, particularly CFS 1712, of cache storage capacity changes when actual cache storage space that is allocated to the virtual machine 1704 changes.
Thus, in underlying operation, each virtual machine is actually allocated an amount of actual cache storage space that may vary over time as each virtual machine's storage needs change or in the event of power on/off events and also events where virtual machines move from one host to another, while the operating system perceives another set value that appears to not change from the perspective of the operating system. The VLUN driver 1722 manages this deception to the operating system together with the SCSI filter 1716. For example, assume for a moment that virtual machine 1704 had 4 GB of actual cache storage space located in cache storage 1710 allocated to it, and 2 TB of virtual storage space as perceived by the operating system.
During operation, a virtual machine's needs for cache storage may increase, and it thus needs more cache storage space allocated to it. According to one embodiment, the virtual machine may have its allocated cache storage space changed as its needs for cache storage changes. Referring to
Thus, a thin provisioned cache device is provided, where the limitation of a fixed disk capacity requirement in conventional virtual systems has been addressed. Thus, the operating system can essentially be deceived into thinking that a fixed amount of cache storage has been allocated so that applications in the operating system have no impact. And, the actual cache storage space allocated to any virtual machine may be resized on the fly without impacting other system operations. The result is an intelligent and optimized utilization of cache storage, where the available cache storage space is more efficiently utilized. Multiple virtual machines are dynamic in nature and their data flow and cache storage needs change dynamically. A virtual machine substantially reduces its demand for cache storage in different modes or circumstances. For example, it may power off or go into sleep mode, it may stall while moving from one host to another, and its needs will necessarily change when these operational changes occur. A virtual machine may alternatively increase its demand for cache storage in other modes or circumstances, such as when it wakes up from a sleep mode, arrives at a new host after moving, or simply experiences an upsurge in usage operations. This embodiment gives the host system the flexibility to dynamically change and optimizes the use of cache storage at the same time. Accordingly, the amount of cache storage designed in a host system can be minimized, substantially saving costs in a host system or device. The cache device, which is commonly implemented in expensive flash memory, is itself virtualized in this embodiment, and its operations are intelligently managed in a way that optimizes the use of its storage space, allocating cache storage to the various virtual machines according to their needs.
One fundamental precept of virtual systems is that shared storage must be secured among the different virtual machines. This is important because the different virtual machines may store confidential information in the various storage chunks in cache storage that possibly could be accessed by other virtual machines in the dynamic provisioning process. For example, a person's confidential financial and identity information may be stored by one virtual machine in one chunk of allocated cache data storage, and that machine's allocated cache storage may be resized as a result of low demand. The virtual machine may then give up allocated cache storage space to another machine in the dynamic allocation process, also giving the second virtual machine that acquires the data chunk having the person's confidential information stored in that chunk. This is thus a security risk, and the dynamic allocation process that has been designed to optimize the use of the cache storage may cause a problem when resizing cache space of particular machines and allocating cache storage chunks from one virtual machine to another. One embodiment of the invention addresses this security risk in an elegant manner without substantial impact to the work flow within the system while dynamically provisioning cache storage chunks.
Referring to
One way to address this issue is to erase any prior information from any chunk that is allocated from one virtual machine's cache space to another, ensuring that the data is deleted or otherwise rendered inaccessible to any subsequent virtual machine that gains access to the chunk. Though this is an attractive option that provides definiteness to securing information in reallocated chunks of cache storage, it has drawbacks. One primary drawback is that it requires I/O data transfers to zero out or otherwise delete the stored information, causing a burden on the system. In a dynamic allocation process, this would require all chunks of data storage that are to be transferred for use by a different dynamic machine to be deleted prior to the transfer by writing zeros into the chunks' space, which adds no value to the virtual system. One characteristic of cache storage devices that are made up of flash memory is that writing takes a long time to perform compared to reading. Thus, this may bog down the system during the dynamic provisioning process. Furthermore, the dynamic provisioning process causes latency in the system, delaying the operation of the virtual machine that is gaining the new chunk of data to be zeroed out.
One other approach is to use a new primitive operation that flash memory vendors refer to as TRIM. The problem is that not all vendors of flash memory support TRIM, and the contents of a data block that have been trimmed are undefined, and therefore TRIM cannot be trusted. Thus, there is no guarantee that the old data in the reallocated chunk has been deleted.
Yet another approach is to make sure that the new virtual machine acquiring the chunk having the old data has no access to it. This may be accomplished by keeping track of whether the prior virtual machine has touched or otherwise used the cache storage space in the chunk before the chunk was allocated to the current virtual machine. It may further be improved by tracking whether any prior machine has accessed the chunk. It can be determined whether a particular chunk has been written to or otherwise modified after being allocated to one or more previous virtual machines. A bit mask is used to protect from the reading of old data to a virtual machine that has acquired cache storage chunks that were utilized by other virtual machines. In one embodiment, each 4 kb sub portion of the chunk is monitored to determine whether there has been a read or a write in each 4 kb sub portion. This is determined at the time the subsequent virtual machine accesses the newly acquired chunk, and is performed only when necessary to prevent the reading of old data by the acquiring virtual machine. After the provisioning of the chunk to a machine, each sub portion of the chunk is tested prior to any read operation on the new chunk by the acquiring virtual machine. Referring to
In step 2004, a read is initiated. The process then proceeds to step 2006, where it is determined whether all pages were written to in the chunk at least once by the second virtual machine, and in particular whether this has occurred since the chunk was provisioned to the second virtual machine. If yes, then the need for this security test is obviated, and the reads to this chunk by the second virtual machine may resume in step 2008 without further testing. If all pages of the chunk in question have not been fully written over by the second virtual machine, then the reads resume in step 2012 with testing and continues on an as needed basis until each page is fully written over at least once, so long as the second virtual machine continues to be allocated the chunk in question and continues to initiate reads into the chunk's pages. In step 2006, it is determined whether there was a full write of the page that covers any possible old data. This determination may be done in various ways. In one embodiment, an indication of whether a particular page is partially written to or completely written over may be indicated by a bit that is on or off, logic 1 or logic 0 for example. This bit may be recorded in a table, such as table 1920. This may be in the form of a type of bit mask that can be stored in cache or other storage location. The indication of whether a page is entirely written over may be indicated by a single bit that is toggled to a binary “1” when a full write over occurs. When a partial write occurs to the page of a chunk, the VLUN driver converts the partial write to a full write with zeros filled in for the data space that is not covered by the partial write from CFS. Any other write to a page while the very first write is in progress will be returned with error. If a read occurs before a write, then a test would show that a full write has not occurred, and the binary bit should be “0.” As the chart 1920 shows, the table may start out as all logical “0,” indicating that the individual pages have not been fully written over since being allocated to the second virtual machine. As full write over occurs in each page, the full write indicator bits eventually become more populated across the array, eventually ending up over time with all logical “1” bits, indicating that each and every page has been written over by the second virtual machine at least once.
If there was a prior full page write, then the read is allowed in step 2008, otherwise the read is failed and not allowed in step 2010. This process is an intelligent and efficient process for preventing read before write security breaches, and substantially reduces the amount of I/O traffic and latencies. Using the bit indicator approach, the trade off is the use of some memory space for the indicator bits for each sub portion and the use of processor resources to perform the testing, but the valuable security and the minimization of unnecessary I/O traffic in the process makes this read before write testing process valuable and useful.
Referring to
The VLUN Manager 2118 is a user space daemon that configures the provisioning of the portions of the cache storage among the different virtual machines. This is an application that runs on the host to parcel out the cache storage device fairly and efficiently among the virtual machines residing on the host. When a management module 2120 is configured to establish allocations for the virtual machines. This sets the relative proportions allocated among the various virtual machines. Shares of storage space are then defined for each virtual machine, for example:
These shares are used as an abstract definition of proportions of cache storage that is allocated to particular machines without regard to actual size and space that is allocated to a particular virtual machine. This way, the definition and system works with any size cache storage space or device, whether it is flash memory or other type of memory, and the shares can be divided up and allocated among multiple virtual machines as they are added or subtracted from the system, as further cache memory is added, and as the system changes over time. The shares allow for the relative dynamic percentage for each virtual machine as the system changes. Thus, for each virtual machine (VM), the amount of cache storage it receives can be calculated as follows:
Capacity of VM-1 shares=(VM-1 shares/total active VM shares) Cache Capacity
where the total “active” VM shares are the shares allocated to powered-on virtual machines. Thus, for virtual machines that are not up and running, their shares are not accounted for in the capacity equation. Thus, for the example in
VM1 Capacity=(1000/4000)100 GB=25.0 GB
VM2 Capacity=(3000/4000)100 GB=75.0 GB
The different virtual machines may be powering on and off, vMotion'ing away and back to the host, so the capacity allocations can change over time. At the time Virtual Machine-3 (for example virtual machine 2122 having VLUN disk-32124 in
VM1 Capacity=(1000/6000)100 GB=16.7 GB
VM2 Capacity=(3000/6000)100 GB=50.0 GB
VM3 Capacity=(2000/6000)100 GB=33.3 GB
Thus, the current percentage may be calculated based on current allocations. In performing this transition of VM3 online and being allocated its percentage or shares of cache storage, VM3 must be allocated its percentage shares, and virtual machines VM1 and VM2 must relinquish storage space. This is accomplished by the methods discussed above in connection with
Additionally, to the extent virtual machines can be provisioned storage space according to the shares concept, IOPS capacity can also be allocated among the virtual machines. Thus, for each machine,
VM1 Capacity=(1000/6000)100 k IOPS
VM2 Capacity=(3000/6000)100 k IOPS
VM3 Capacity=(2000/6000)100 k IOPS
One feature that is desired in virtual systems is the ability to move virtual machines from one host to another without powering down or taking the virtual machine offline in the process. In conventional systems, since hosts are usually connected to shared storage, this process is well defined and seamless. However, in systems configured according to the various embodiments described above that utilize local cache storage rather than shared storage for certain virtual system operations, such as a thin provisioned cache, there are conflicts that result from exercising certain features common in virtual systems such as moving virtual machines from one host to another. In conventional virtual systems, a virtual machine may be moved from one host to another by utilizing shared storage. However, moving virtual machines from one host to another while utilizing the various embodiments described herein, problems would occur with the transfer, and critical data and virtual systems operations may be compromised. In some virtual systems, the move simply would not be allowed—such as by VMWare™ virtual system products for example. According to one embodiment, the issues related to the transfer of a virtual machine from one host to another without the aid of shared storage are addressed in an elegant manner, and transfers of virtual machines from one host to another is made seamless while also utilizing the various embodiments described herein. According to one embodiment, the virtual system may be configured to deceive the system into thinking that the local cache storage located in or communicating with the host is essentially a shared device.
In one embodiment, in establishing a virtual system and adding virtual machines on separate hosts without shared storage, copies of the VLUN disks may exist on two or more different host in anticipation that the virtual machines may be moved from one host to another. Referring to
In conventional virtual systems, shared storage 2230 stores the instances of the main storage drives 2232 of the virtual machines located among different hosts. This is typically a requirement in virtual systems to enable the transfer of virtual machines from one host to another. Given the embodiments described herein for optimizing cache transfers by utilizing the local cache devices of the hosts, this requirement is not met without further addressing this issue. According to one embodiment however, this requirement is addressed and virtual machine transfers from one host to another are accomplished by instantiating the main storage drive 2212-A in an active state in Host-l′s cache storage 2222 and also storing corresponding main storage drive 2212-B in a dormant state in Host-2′s cache storage 2228. In one example, a substantiation of VLUN disk-12212-A is created on VLUN disk-12212, having a serial number of “naa.200.cd123” in this example. An identical substantiation is made in on host-2 and has the same serial number, but it is dormant, where VM-1 does not actively use the copy 2212-B, but rather uses it as a type of holding place for VM-1 when and if it ever happens to transfer from Host-1 to Host-2.
In a system of multiple host computers that each have multiple virtual machines residing on them, multiple copies of VLUN disks can reside or be dynamically created among the different host computers to aid in transferring virtual machines from one host to another.
It has been observed that, in typical computing systems with peripheral and other system devices, such as virtual computing systems for example, SCSI operations serve as interfaces for devices within a system and can be utilized to fool the operating system into believing that the cache storage devices located in individual hosts are actually a unified shared device. When an operating system communicates to components within the system and discovers devices within the purview of the operating system, such as storage disks, VLUN disks, and other devices, it initiates queries when a device is found to learn the device's identity and relevant operating information. It questions who the manufacturer is, what the model number is, what the capacity is, and importantly for this embodiment, what the serial number is. The serial number is configured to be globally unique within the system. Thus, in a virtual system, the operating system queries discovered devices such as disks to identify them and to derive a serial number that will be used by the operating system to identify the storage device. For virtual machines, the operating system in conventional virtual systems identifies shared storage devices, and derives a unique serial number to identify it within the virtual system. Once the virtual machines are created, the conventional virtual systems identify each virtual machine as a shared storage device by using this unique serial number assigned to the shared storage.
According to the embodiments discussed herein however, cache storage devices are not shared among different hosts, but are local to the hosts and shared among virtual machines within the hosts. In operation, conventional virtual systems require that the virtual machines are assigned to shared storage in order to enable a transfer of a virtual machine from one host to another. According to one embodiment, fictitious shared storage is created and exported to the host as a Fibre channel or SAS device. Thus, the Fibre channel or SAS device is artificially recognized as a shared storage device with a unique serial number and is instantiated when a VLUN disk is created. VLUN disk devices are fictitious shared storage spaces that are associated with actual storage space in the local cache storage devices. Once created, these VLUN disks are treated as actual devices from the perspective of the operating system. The unique serial numbers for VLUN disks instantiated within the local cache devices, such as “naa.200.cd123”, are derived by the virtualization kernel from the serial number of the shared local storage, and each are unique and associated with a particular VLUN disk. Thus, when the VLUN disk is created, it is created with the unique serial number, and these are recognize by the operating system as legitimate entities, but are fictitious shared storage. This derived serial number is also used to create another VLUN disk in Host-2, such as VLUN disk-12212-B, so that a virtual machine such as VM-1 will have a corresponding VLUN disk in the other host to communicate to and continue its I/O data traffic after transferred to Host-2.
While running in Host-1 prior to moving, VM-1 through CFS 2234 believes it has some amount of data stored in the cache storage 2222, having chunks designated VM 1 in this illustration, and these chunks of data storage are not resident in Host-2 after the move.
Once a virtual machine moves from one host to another, the data is left resident on the prior host, Host-1 for example, and when the virtual machine arrives at the destination host, Host-2 in this example, the data is left behind. Again, this breaks the design assumption of conventional virtual systems that requires and assumes the existence of having the I/O data transfers associated with the virtual machine to be available to the virtual machine when it transfers, which is typically located on remote shared storage that is shared among different hosts. Having copies of the VLUN disks of the different virtual machines is an approach used to essentially deceive existing virtual systems into believing that each virtual machine is storing I/O data transfers in remote shared storage.
In conventional systems, the hypervisor stalls I/O transfers to the remote storage device prior to transferring from one host to another. The virtual machine is then transferred to another host, instantiated on that host, and operations resume. In this embodiment, however, there is not only remote shared storage for operations, but also local storage. After the transfer, in step 2310, VM-1 is associated with VLUN driver 2226 and cache 2228, which does not have the former cached data, and VM-1 has essentially zero capacity in the local cache 2228. Referring to
After arriving in Host-2 in step 2402, CFS will continue to send I/O transfers to the VLUN driver 2226 in step 2404. The VLUN driver will fail the I/O transfers in step 2406 with errors to the CFS that VM-1 is out of its range and has no capacity. The error code is interpreted and recognized by the SCSI filter 2238 within VM-1. The SCSI filter will fail the I/O transfer requests to the CFS 2234 and request CFS to invalidate the cache tags associated with the transfers. Thus, there is a small period of time after the transfer from Host-1 to Host-2 where there is no cache storage capacity available to VM-1 in Host-2. The small number of I/O transfers that are issued are failed, and the cache tags are invalidated. The CFS will then reissue the I/O transfers to shared storage 2230 in step 2408 until it acquires capacity in local cache storage 2228. VLUN Manager 2240 then recognizes the arrival of VM-1 in Host-2, and provisions cache storage capacity for VM-1 in step 2410 according to an allocation of shares as discussed above. In step 2412, the VLUN driver stalls CFS as discussed above to perform capacity allocation. In step 2414 VLUN instructs CFS to purge its data related to the I/O data transfers, because CFS essentially thinks that it has stored data in the cache storage and 4 GB in space, but that is all left behind in Host-l′s local cache storage 2222. This is different than the capacity allocation as discussed above, because the resize that occurs after a transfer from one host to another leaves the data behind, and the stored data in the allocated space is not the same data that CFS registers as the stored data, because it is left behind and does not exist in Host-2′s cache storage 2228. Thus, this is a unique resize and allocation of cache space. In step 2416, CFS will invalidate all cache tags, resize to new allocation of space and resume operation. Also, the allocation will utilize the bit mask processes to protect against VM-1 reading any old data that may have been written to the host by VM-3 or any other prior host. Thus, VM-1 would need to write to the chunks of allocated cache storage space before it can read, or the read requests will get a fail error. Thus, this embodiment allows the VLUN driver to essentially fool the virtual system to believe that shared storage exists, then allow a virtual machine to move from one host to another, then because VM-1 has a SCSI filter, it can talk to the VLUN driver to cover up the missteps, initially failing the I/Os to the VLUN device with an error, invalidate all past cache tags, allocating space to VM-1, and resuming operation of VM-1. Also the VLUN manager 2242 of 22B will allocate the space relinquished by VM-1 to virtual machines that are local to Host-1. Thus, virtual machines may be moved around to different hosts for non-destructive upgrades, balancing among hosts, failure recovery, and other operations that aid the virtual system.
As discussed herein, the invention may involve a number of functions to be performed by a computer processor, such as a microprocessor. The microprocessor may be a specialized or dedicated microprocessor that is configured to perform particular tasks according to the invention, by executing machine-readable software code that defines the particular tasks embodied by the invention. The microprocessor may also be configured to operate and communicate with other devices such as direct memory access modules, memory storage devices, Internet-related hardware, and other devices that relate to the transmission of data in accordance with the invention. The software code may be configured using software formats such as Java, C++, XML (Extensible Mark-up Language) and other languages that may be used to define functions that relate to operations of devices required to carry out the functional operations related to the invention. The code may be written in different forms and styles, many of which are known to those skilled in the art. Different code formats, code configurations, styles and forms of software programs and other means of configuring code to define the operations of a microprocessor in accordance with the invention will not depart from the spirit and scope of the invention.
Within the different types of devices, such as laptop or desktop computers, hand held devices with processors or processing logic, and also possibly computer servers or other devices that utilize the invention, there exist different types of memory devices for storing and retrieving information while performing functions according to the invention. Cache memory devices are often included in such computers for use by the central processing unit as a convenient storage location for information that is frequently stored and retrieved. Similarly, a persistent memory is also frequently used with such computers for maintaining information that is frequently retrieved by the central processing unit, but that is not often altered within the persistent memory, unlike the cache memory. Main memory is also usually included for storing and retrieving larger amounts of information such as data and software applications configured to perform functions according to the invention when executed by the central processing unit. These memory devices may be configured as random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, and other memory storage devices that may be accessed by a central processing unit to store and retrieve information. During data storage and retrieval operations, these memory devices are transformed to have different states, such as different electrical charges, different magnetic polarity, and the like. Thus, systems and methods configured according to the invention as described herein enable the physical transformation of these memory devices. Accordingly, the invention as described herein is directed to novel and useful systems and methods that, in one or more embodiments, are able to transform the memory device into a different state. The invention is not limited to any particular type of memory device, or any commonly used protocol for storing and retrieving information to and from these memory devices, respectively.
Embodiments of the systems and methods described herein facilitate the management of data input/output operations. Additionally, some embodiments may be used in conjunction with one or more conventional data management systems and methods, or conventional virtualized systems. For example, one embodiment may be used as an improvement of existing data management systems.
Although the components and modules illustrated herein are shown and described in a particular arrangement, the arrangement of components and modules may be altered to process data in a different manner. In other embodiments, one or more additional components or modules may be added to the described systems, and one or more components or modules may be removed from the described systems. Alternate embodiments may combine two or more of the described components or modules into a single component or module.
Finally, although specific embodiments of the invention have been described and illustrated, the invention is not to be limited to the specific forms or arrangements of parts so described and illustrated. The scope of the invention is to be defined by the claims appended hereto and their equivalents.
This application is a conversion of U.S. Provisional Patent Application No. 61/489,230 entitled “Managing Data Input/Output Operations”, filed on May 23, 2011.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4571674 | Hartung | Feb 1986 | A |
| 5043871 | Nishigaki et al. | Aug 1991 | A |
| 5193184 | Belsan et al. | Mar 1993 | A |
| 5261068 | Gaskins et al. | Nov 1993 | A |
| 5291496 | Andaleon et al. | Mar 1994 | A |
| 5307497 | Feigenbaum | Apr 1994 | A |
| 5313475 | Cromer et al. | May 1994 | A |
| 5325509 | Lautzenhaiser | Jun 1994 | A |
| 5392427 | Barrett et al. | Feb 1995 | A |
| 5404485 | Ban | Apr 1995 | A |
| 5438671 | Miles | Aug 1995 | A |
| 5469555 | Ghosh et al. | Nov 1995 | A |
| 5499354 | Aschoff et al. | Mar 1996 | A |
| 5504882 | Chai | Apr 1996 | A |
| 5535399 | Blitz et al. | Jul 1996 | A |
| 5551003 | Mattson et al. | Aug 1996 | A |
| 5553261 | Hasbun et al. | Sep 1996 | A |
| 5559988 | Durante et al. | Sep 1996 | A |
| 5586291 | Lasker et al. | Dec 1996 | A |
| 5594883 | Pricer | Jan 1997 | A |
| 5596736 | Kerns | Jan 1997 | A |
| 5598370 | Nijima et al. | Jan 1997 | A |
| 5603001 | Sukegawa et al. | Feb 1997 | A |
| 5651133 | Burkes | Jul 1997 | A |
| 5680579 | Young et al. | Oct 1997 | A |
| 5682497 | Robinson | Oct 1997 | A |
| 5682499 | Bakke et al. | Oct 1997 | A |
| 5701434 | Nakagawa | Dec 1997 | A |
| 5734861 | Cohn et al. | Mar 1998 | A |
| 5740367 | Spilo | Apr 1998 | A |
| 5745792 | Jost | Apr 1998 | A |
| 5754563 | White | May 1998 | A |
| 5757567 | Hetzler et al. | May 1998 | A |
| 5787486 | Chin et al. | Jul 1998 | A |
| 5802602 | Rahman et al. | Sep 1998 | A |
| 5809527 | Cooper et al. | Sep 1998 | A |
| 5809543 | Byers et al. | Sep 1998 | A |
| 5845313 | Estakhri et al. | Dec 1998 | A |
| 5845329 | Onishi et al. | Dec 1998 | A |
| 5860083 | Sukegawa | Jan 1999 | A |
| 5907856 | Estakhri et al. | May 1999 | A |
| 5924113 | Estakhri et al. | Jul 1999 | A |
| 5930815 | Estakhri et al. | Jul 1999 | A |
| 5957158 | Volz et al. | Sep 1999 | A |
| 5960462 | Solomon et al. | Sep 1999 | A |
| 5961660 | Capps, Jr. et al. | Oct 1999 | A |
| 6000019 | Dykstal et al. | Dec 1999 | A |
| 6014724 | Jennett | Jan 2000 | A |
| 6073232 | Kroeker et al. | Jun 2000 | A |
| 6075938 | Bugnion et al. | Jun 2000 | A |
| 6101601 | Matthews et al. | Aug 2000 | A |
| 6115703 | Bireley et al. | Sep 2000 | A |
| 6128695 | Estakhri et al. | Oct 2000 | A |
| 6141249 | Estakhri et al. | Oct 2000 | A |
| 6145051 | Estakhri et al. | Nov 2000 | A |
| 6170039 | Kishida | Jan 2001 | B1 |
| 6170047 | Dye | Jan 2001 | B1 |
| 6172906 | Estakhri et al. | Jan 2001 | B1 |
| 6173381 | Dye | Jan 2001 | B1 |
| 6185654 | Van Doren | Feb 2001 | B1 |
| 6209088 | Reneris | Mar 2001 | B1 |
| 6223308 | Estakhri et al. | Apr 2001 | B1 |
| 6230234 | Estakhri et al. | May 2001 | B1 |
| 6236593 | Hong et al. | May 2001 | B1 |
| 6240040 | Akaogi et al. | May 2001 | B1 |
| 6256642 | Krueger et al. | Jul 2001 | B1 |
| 6266785 | McDowell | Jul 2001 | B1 |
| 6279069 | Robinson et al. | Aug 2001 | B1 |
| 6289413 | Rogers et al. | Sep 2001 | B1 |
| 6330688 | Brown | Dec 2001 | B1 |
| 6336174 | Li et al. | Jan 2002 | B1 |
| 6356986 | Solomon et al. | Mar 2002 | B1 |
| 6370631 | Dye | Apr 2002 | B1 |
| 6385710 | Goldman et al. | May 2002 | B1 |
| 6393513 | Estakhri et al. | May 2002 | B2 |
| 6404647 | Minne | Jun 2002 | B1 |
| 6412080 | Fleming et al. | Jun 2002 | B1 |
| 6418478 | Ignatius et al. | Jul 2002 | B1 |
| 6507883 | Bello et al. | Jan 2003 | B1 |
| 6507911 | Langford | Jan 2003 | B1 |
| 6516380 | Kenchammana-Hoskote et al. | Feb 2003 | B2 |
| 6523102 | Dye et al. | Feb 2003 | B1 |
| 6564285 | Mills et al. | May 2003 | B1 |
| 6567889 | DeKoning et al. | May 2003 | B1 |
| 6587915 | Kim | Jul 2003 | B1 |
| 6587937 | Jensen et al. | Jul 2003 | B1 |
| 6601211 | Norman | Jul 2003 | B1 |
| 6625685 | Cho et al. | Sep 2003 | B1 |
| 6629112 | Shank | Sep 2003 | B1 |
| 6658438 | Moore et al. | Dec 2003 | B1 |
| 6671757 | Multer et al. | Dec 2003 | B1 |
| 6675349 | Chen | Jan 2004 | B1 |
| 6715027 | Kim et al. | Mar 2004 | B2 |
| 6715046 | Shoham et al. | Mar 2004 | B1 |
| 6728851 | Estakhri et al. | Apr 2004 | B1 |
| 6742082 | Lango et al. | May 2004 | B1 |
| 6751155 | Gorobets | Jun 2004 | B2 |
| 6754774 | Gruner et al. | Jun 2004 | B2 |
| 6757800 | Estakhri et al. | Jun 2004 | B1 |
| 6766413 | Newman | Jul 2004 | B2 |
| 6775185 | Fujisawa et al. | Aug 2004 | B2 |
| 6779088 | Benveniste et al. | Aug 2004 | B1 |
| 6779094 | Selkirk et al. | Aug 2004 | B2 |
| 6785776 | Arimilli et al. | Aug 2004 | B2 |
| 6785785 | Piccirillo et al. | Aug 2004 | B2 |
| 6801979 | Estakhri | Oct 2004 | B1 |
| 6804755 | Selkirk et al. | Oct 2004 | B2 |
| 6877076 | Cho et al. | Apr 2005 | B1 |
| 6880049 | Gruner et al. | Apr 2005 | B2 |
| 6883069 | Yoshida | Apr 2005 | B2 |
| 6883079 | Priborsky | Apr 2005 | B1 |
| 6910170 | Choi et al. | Jun 2005 | B2 |
| 6912537 | Selkirk et al. | Jun 2005 | B2 |
| 6912618 | Estakhri et al. | Jun 2005 | B2 |
| 6925533 | Lewis | Aug 2005 | B2 |
| 6938133 | Johnson et al. | Aug 2005 | B2 |
| 6957158 | Hancock et al. | Oct 2005 | B1 |
| 6959369 | Ashton et al. | Oct 2005 | B1 |
| 6977599 | Widmer | Dec 2005 | B2 |
| 6978342 | Estakhri et al. | Dec 2005 | B1 |
| 6981070 | Luk et al. | Dec 2005 | B1 |
| 6996676 | Megiddo | Feb 2006 | B2 |
| 7010652 | Piccirillo et al. | Mar 2006 | B2 |
| 7010662 | Aasheim et al. | Mar 2006 | B2 |
| 7013376 | Hooper, III | Mar 2006 | B2 |
| 7013379 | Testardi | Mar 2006 | B1 |
| 7035974 | Shang | Apr 2006 | B2 |
| 7036040 | Nicholson et al. | Apr 2006 | B2 |
| 7043599 | Ware et al. | May 2006 | B1 |
| 7047366 | Ezra | May 2006 | B1 |
| 7050337 | Iwase et al. | May 2006 | B2 |
| 7058769 | Danilak | Jun 2006 | B1 |
| 7069393 | Miyata et al. | Jun 2006 | B2 |
| 7073028 | Lango et al. | Jul 2006 | B2 |
| 7076560 | Lango et al. | Jul 2006 | B1 |
| 7076599 | Aasheim et al. | Jul 2006 | B2 |
| 7076723 | Saliba | Jul 2006 | B2 |
| 7082495 | DeWhitt et al. | Jul 2006 | B2 |
| 7082512 | Aasheim et al. | Jul 2006 | B2 |
| 7085879 | Aasheim et al. | Aug 2006 | B2 |
| 7089391 | Geiger et al. | Aug 2006 | B2 |
| 7093101 | Aasheim et al. | Aug 2006 | B2 |
| 7096321 | Modha | Aug 2006 | B2 |
| 7111140 | Estakhri et al. | Sep 2006 | B2 |
| 7130956 | Rao | Oct 2006 | B2 |
| 7130957 | Rao | Oct 2006 | B2 |
| 7143228 | Lida et al. | Nov 2006 | B2 |
| 7149947 | MacLellan et al. | Dec 2006 | B1 |
| 7155531 | Lango et al. | Dec 2006 | B1 |
| 7167953 | Megiddo et al. | Jan 2007 | B2 |
| 7171536 | Chang et al. | Jan 2007 | B2 |
| 7173852 | Gorobets et al. | Feb 2007 | B2 |
| 7178081 | Lee et al. | Feb 2007 | B2 |
| 7181572 | Walmsley | Feb 2007 | B2 |
| 7194577 | Johnson et al. | Mar 2007 | B2 |
| 7194740 | Frank et al. | Mar 2007 | B1 |
| 7197657 | Tobias | Mar 2007 | B1 |
| 7203815 | Haswell | Apr 2007 | B2 |
| 7215580 | Gorobets | May 2007 | B2 |
| 7219238 | Saito et al. | May 2007 | B2 |
| 7234082 | Lai et al. | Jun 2007 | B2 |
| 7243203 | Scheuerlein | Jul 2007 | B2 |
| 7246179 | Camara et al. | Jul 2007 | B2 |
| 7254686 | Islam | Aug 2007 | B2 |
| 7260820 | Waldspurger et al. | Aug 2007 | B1 |
| 7272606 | Borthakur et al. | Sep 2007 | B2 |
| 7275135 | Coulson | Sep 2007 | B2 |
| 7280536 | Testardi | Oct 2007 | B2 |
| 7293183 | Lee et al. | Nov 2007 | B2 |
| 7305520 | Voight et al. | Dec 2007 | B2 |
| 7337201 | Yellin et al. | Feb 2008 | B1 |
| 7340558 | Lee et al. | Mar 2008 | B2 |
| 7340566 | Voth et al. | Mar 2008 | B2 |
| 7356651 | Liu et al. | Apr 2008 | B2 |
| 7360015 | Matthews et al. | Apr 2008 | B2 |
| 7360037 | Higaki et al. | Apr 2008 | B2 |
| 7366808 | Kano et al. | Apr 2008 | B2 |
| 7392365 | Selkirk et al. | Jun 2008 | B2 |
| 7395384 | Sinclair et al. | Jul 2008 | B2 |
| 7398348 | Moore et al. | Jul 2008 | B2 |
| 7424593 | Estakhri et al. | Sep 2008 | B2 |
| 7437510 | Rosenbluth et al. | Oct 2008 | B2 |
| 7441090 | Estakhri et al. | Oct 2008 | B2 |
| 7447847 | Louie et al. | Nov 2008 | B2 |
| 7450420 | Sinclair et al. | Nov 2008 | B2 |
| 7464221 | Nakamura et al. | Dec 2008 | B2 |
| 7487235 | Andrews et al. | Feb 2009 | B2 |
| 7487320 | Bansai et al. | Feb 2009 | B2 |
| 7500000 | Groves et al. | Mar 2009 | B2 |
| 7516267 | Coulson et al. | Apr 2009 | B2 |
| 7526614 | van Riel | Apr 2009 | B2 |
| 7529905 | Sinclair | May 2009 | B2 |
| 7536491 | Kano et al. | May 2009 | B2 |
| 7549022 | Baker | Jun 2009 | B2 |
| 7552271 | Sinclair et al. | Jun 2009 | B2 |
| 7580287 | Aritome | Aug 2009 | B2 |
| 7603532 | Rajan et al. | Oct 2009 | B2 |
| 7610348 | Kisley et al. | Oct 2009 | B2 |
| 7617375 | Flemming et al. | Nov 2009 | B2 |
| 7640390 | Iwamura et al. | Dec 2009 | B2 |
| 7644239 | Ergan et al. | Jan 2010 | B2 |
| 7660911 | McDaniel | Feb 2010 | B2 |
| 7660941 | Lee et al. | Feb 2010 | B2 |
| 7664239 | Groff et al. | Feb 2010 | B2 |
| 7669019 | Fujibayashi et al. | Feb 2010 | B2 |
| 7673108 | Iyengar et al. | Mar 2010 | B2 |
| 7676628 | Compton et al. | Mar 2010 | B1 |
| 7685367 | Ruia et al. | Mar 2010 | B2 |
| 7694065 | Petev et al. | Apr 2010 | B2 |
| 7702873 | Griess et al. | Apr 2010 | B2 |
| 7721047 | Dunshea et al. | May 2010 | B2 |
| 7721059 | Mylly et al. | May 2010 | B2 |
| 7725628 | Phan et al. | May 2010 | B1 |
| 7801894 | Bone et al. | Sep 2010 | B1 |
| 7805449 | Bone et al. | Sep 2010 | B1 |
| 7831783 | Pandit et al. | Nov 2010 | B2 |
| 7831977 | Shultz et al. | Nov 2010 | B2 |
| 7853772 | Chang et al. | Dec 2010 | B2 |
| 7873782 | Terry et al. | Jan 2011 | B2 |
| 7873803 | Cheng | Jan 2011 | B2 |
| 7882305 | Moritoki | Feb 2011 | B2 |
| 7904647 | El-Batal et al. | Mar 2011 | B2 |
| 7913051 | Todd et al. | Mar 2011 | B1 |
| 7917803 | Stefanus et al. | Mar 2011 | B2 |
| 7941591 | Aviles | May 2011 | B2 |
| 7984230 | Nasu et al. | Jul 2011 | B2 |
| 8046526 | Yeh | Oct 2011 | B2 |
| 8055820 | Sebire | Nov 2011 | B2 |
| 8060683 | Schultz et al. | Nov 2011 | B2 |
| 8095764 | Bauer et al. | Jan 2012 | B1 |
| 8135900 | Kunimatsu et al. | Mar 2012 | B2 |
| 8135904 | Lasser et al. | Mar 2012 | B2 |
| 8151077 | Bauer et al. | Apr 2012 | B1 |
| 8151082 | Flynn et al. | Apr 2012 | B2 |
| 8171204 | Chow et al. | May 2012 | B2 |
| 8195929 | Banga et al. | Jun 2012 | B2 |
| 8214583 | Sinclair et al. | Jul 2012 | B2 |
| 8244935 | Leventhal et al. | Aug 2012 | B2 |
| 8479294 | Li et al. | Jul 2013 | B1 |
| 8549222 | Kleiman et al. | Oct 2013 | B1 |
| 20020069317 | Chow et al. | Jun 2002 | A1 |
| 20020069318 | Chow et al. | Jun 2002 | A1 |
| 20020103819 | Duvillier | Aug 2002 | A1 |
| 20020161855 | Manczak et al. | Oct 2002 | A1 |
| 20020181134 | Bunker et al. | Dec 2002 | A1 |
| 20020188711 | Meyer et al. | Dec 2002 | A1 |
| 20020194451 | Mukaida et al. | Dec 2002 | A1 |
| 20030061296 | Craddock et al. | Mar 2003 | A1 |
| 20030061550 | Ng et al. | Mar 2003 | A1 |
| 20030093741 | Argon et al. | May 2003 | A1 |
| 20030140051 | Fujiwara et al. | Jul 2003 | A1 |
| 20030145230 | Chiu et al. | Jul 2003 | A1 |
| 20030149753 | Lamb | Aug 2003 | A1 |
| 20030198084 | Fujisawa et al. | Oct 2003 | A1 |
| 20040002942 | Pudipeddi et al. | Jan 2004 | A1 |
| 20040003002 | Adelmann | Jan 2004 | A1 |
| 20040049564 | Ng et al. | Mar 2004 | A1 |
| 20040093463 | Shang | May 2004 | A1 |
| 20040117586 | Estakhri et al. | Jun 2004 | A1 |
| 20040148360 | Mehra et al. | Jul 2004 | A1 |
| 20040153694 | Nicholson et al. | Aug 2004 | A1 |
| 20040186946 | Lee | Sep 2004 | A1 |
| 20040205177 | Levy et al. | Oct 2004 | A1 |
| 20040225837 | Lewis | Nov 2004 | A1 |
| 20040268359 | Hanes | Dec 2004 | A1 |
| 20050002263 | Iwase et al. | Jan 2005 | A1 |
| 20050015539 | Horii et al. | Jan 2005 | A1 |
| 20050027951 | Piccirillo et al. | Feb 2005 | A1 |
| 20050055425 | Lango et al. | Mar 2005 | A1 |
| 20050055497 | Estakhri et al. | Mar 2005 | A1 |
| 20050076107 | Goud et al. | Apr 2005 | A1 |
| 20050120177 | Black | Jun 2005 | A1 |
| 20050132259 | Emmot et al. | Jun 2005 | A1 |
| 20050141313 | Gorobets et al. | Jun 2005 | A1 |
| 20050144361 | Gonzalez et al. | Jun 2005 | A1 |
| 20050144406 | Chong, Jr. | Jun 2005 | A1 |
| 20050149618 | Cheng | Jul 2005 | A1 |
| 20050149683 | Chong, Jr. et al. | Jul 2005 | A1 |
| 20050149819 | Hwang | Jul 2005 | A1 |
| 20050177672 | Rao | Aug 2005 | A1 |
| 20050177687 | Rao | Aug 2005 | A1 |
| 20050193166 | Johnson et al. | Sep 2005 | A1 |
| 20050216653 | Aasheim et al. | Sep 2005 | A1 |
| 20050229090 | Shen et al. | Oct 2005 | A1 |
| 20050240713 | Wu et al. | Oct 2005 | A1 |
| 20050246510 | Retnammana et al. | Nov 2005 | A1 |
| 20050257017 | Yagi | Nov 2005 | A1 |
| 20050257213 | Chu et al. | Nov 2005 | A1 |
| 20050273476 | Wertheimer et al. | Dec 2005 | A1 |
| 20050276092 | Hansen et al. | Dec 2005 | A1 |
| 20060004955 | Ware et al. | Jan 2006 | A1 |
| 20060020744 | Sinclair et al. | Jan 2006 | A1 |
| 20060026339 | Rostampour | Feb 2006 | A1 |
| 20060041731 | Jochemsen et al. | Feb 2006 | A1 |
| 20060053157 | Pitts | Mar 2006 | A1 |
| 20060059326 | Aasheim et al. | Mar 2006 | A1 |
| 20060075057 | Gildea et al. | Apr 2006 | A1 |
| 20060085626 | Roberson et al. | Apr 2006 | A1 |
| 20060090048 | Okumoto et al. | Apr 2006 | A1 |
| 20060106968 | Wooi Teoh | May 2006 | A1 |
| 20060117212 | Meyer et al. | Jun 2006 | A1 |
| 20060123197 | Dunshea et al. | Jun 2006 | A1 |
| 20060129778 | Clark et al. | Jun 2006 | A1 |
| 20060136657 | Rudelic et al. | Jun 2006 | A1 |
| 20060136685 | Griv et al. | Jun 2006 | A1 |
| 20060143396 | Cabot | Jun 2006 | A1 |
| 20060149893 | Barfuss et al. | Jul 2006 | A1 |
| 20060152981 | Ryu | Jul 2006 | A1 |
| 20060179263 | Song et al. | Aug 2006 | A1 |
| 20060184722 | Sinclair | Aug 2006 | A1 |
| 20060190552 | Henze et al. | Aug 2006 | A1 |
| 20060224849 | Islam et al. | Oct 2006 | A1 |
| 20060236061 | Koclanes | Oct 2006 | A1 |
| 20060248387 | Nicholson et al. | Nov 2006 | A1 |
| 20060265636 | Hummler | Nov 2006 | A1 |
| 20060271740 | Mark et al. | Nov 2006 | A1 |
| 20070006021 | Nicholson et al. | Jan 2007 | A1 |
| 20070016699 | Minami | Jan 2007 | A1 |
| 20070016754 | Testardi | Jan 2007 | A1 |
| 20070033325 | Sinclair | Feb 2007 | A1 |
| 20070033326 | Sinclair | Feb 2007 | A1 |
| 20070033327 | Sinclair | Feb 2007 | A1 |
| 20070033362 | Sinclair | Feb 2007 | A1 |
| 20070043900 | Yun | Feb 2007 | A1 |
| 20070050548 | Bali et al. | Mar 2007 | A1 |
| 20070050571 | Nakamura et al. | Mar 2007 | A1 |
| 20070061508 | Zweighaft | Mar 2007 | A1 |
| 20070069318 | Chow et al. | Mar 2007 | A1 |
| 20070086260 | Sinclair | Apr 2007 | A1 |
| 20070088666 | Saito | Apr 2007 | A1 |
| 20070118676 | Kano et al. | May 2007 | A1 |
| 20070118713 | Guterman et al. | May 2007 | A1 |
| 20070124474 | Margulis | May 2007 | A1 |
| 20070124540 | van Riel | May 2007 | A1 |
| 20070143532 | Gorobets et al. | Jun 2007 | A1 |
| 20070143560 | Gorobets | Jun 2007 | A1 |
| 20070143566 | Gorobets | Jun 2007 | A1 |
| 20070147356 | Malas et al. | Jun 2007 | A1 |
| 20070150689 | Pandit et al. | Jun 2007 | A1 |
| 20070156998 | Gorobets | Jul 2007 | A1 |
| 20070168698 | Coulson et al. | Jul 2007 | A1 |
| 20070198770 | Horii et al. | Aug 2007 | A1 |
| 20070204128 | Lee et al. | Aug 2007 | A1 |
| 20070208790 | Reuter et al. | Sep 2007 | A1 |
| 20070214320 | Ruia et al. | Sep 2007 | A1 |
| 20070233455 | Zimmer et al. | Oct 2007 | A1 |
| 20070233937 | Coulson et al. | Oct 2007 | A1 |
| 20070250660 | Gill et al. | Oct 2007 | A1 |
| 20070250838 | Belady et al. | Oct 2007 | A1 |
| 20070260608 | Hertzberg et al. | Nov 2007 | A1 |
| 20070261030 | Wadhwa | Nov 2007 | A1 |
| 20070263514 | Iwata et al. | Nov 2007 | A1 |
| 20070266037 | Terry et al. | Nov 2007 | A1 |
| 20070271468 | McKenney et al. | Nov 2007 | A1 |
| 20070274150 | Gorobets | Nov 2007 | A1 |
| 20070276897 | Tameshige et al. | Nov 2007 | A1 |
| 20070300008 | Rogers et al. | Dec 2007 | A1 |
| 20080005748 | Matthew et al. | Jan 2008 | A1 |
| 20080010395 | Mylly et al. | Jan 2008 | A1 |
| 20080043769 | Hirai | Feb 2008 | A1 |
| 20080052377 | Light | Feb 2008 | A1 |
| 20080052477 | Lee et al. | Feb 2008 | A1 |
| 20080059752 | Serizawa | Mar 2008 | A1 |
| 20080091876 | Fujibayashi et al. | Apr 2008 | A1 |
| 20080098159 | Song | Apr 2008 | A1 |
| 20080104321 | Kamisetty et al. | May 2008 | A1 |
| 20080109090 | Esmaili et al. | May 2008 | A1 |
| 20080120469 | Kornegay | May 2008 | A1 |
| 20080126507 | Wilkinson | May 2008 | A1 |
| 20080126700 | El-Batal et al. | May 2008 | A1 |
| 20080126852 | Brandyberry et al. | May 2008 | A1 |
| 20080133963 | Katano et al. | Jun 2008 | A1 |
| 20080137658 | Wang | Jun 2008 | A1 |
| 20080140737 | Garst et al. | Jun 2008 | A1 |
| 20080140819 | Bailey et al. | Jun 2008 | A1 |
| 20080205286 | Li et al. | Aug 2008 | A1 |
| 20080229045 | Qi | Sep 2008 | A1 |
| 20080235443 | Chow et al. | Sep 2008 | A1 |
| 20080243966 | Croisettier et al. | Oct 2008 | A1 |
| 20080263259 | Sadovsky et al. | Oct 2008 | A1 |
| 20080263305 | Shu et al. | Oct 2008 | A1 |
| 20080263569 | Shu et al. | Oct 2008 | A1 |
| 20080271039 | Rolia et al. | Oct 2008 | A1 |
| 20080276040 | Moritoki | Nov 2008 | A1 |
| 20090070526 | Tetrick | Mar 2009 | A1 |
| 20090083478 | Kunimatsu et al. | Mar 2009 | A1 |
| 20090083485 | Cheng | Mar 2009 | A1 |
| 20090089485 | Yeh | Apr 2009 | A1 |
| 20090125650 | Sebire | May 2009 | A1 |
| 20090125700 | Kisel | May 2009 | A1 |
| 20090132621 | Jensen et al. | May 2009 | A1 |
| 20090150599 | Bennett | Jun 2009 | A1 |
| 20090150605 | Flynn et al. | Jun 2009 | A1 |
| 20090150641 | Flynn et al. | Jun 2009 | A1 |
| 20090228637 | Moon | Sep 2009 | A1 |
| 20090248763 | Rajan et al. | Oct 2009 | A1 |
| 20090248922 | Hinohara et al. | Oct 2009 | A1 |
| 20090276588 | Murase | Nov 2009 | A1 |
| 20090276654 | Butterworth | Nov 2009 | A1 |
| 20090287887 | Matsuki et al. | Nov 2009 | A1 |
| 20090292861 | Kanevsky et al. | Nov 2009 | A1 |
| 20090294847 | Maruyama et al. | Nov 2009 | A1 |
| 20090300277 | Jeddoloh | Dec 2009 | A1 |
| 20090307424 | Galloway et al. | Dec 2009 | A1 |
| 20090313453 | Stefanus et al. | Dec 2009 | A1 |
| 20090327602 | Moore et al. | Dec 2009 | A1 |
| 20090327804 | Moshayedi | Dec 2009 | A1 |
| 20100005072 | Pitts | Jan 2010 | A1 |
| 20100005228 | Fukutomi et al. | Jan 2010 | A1 |
| 20100017556 | Chin | Jan 2010 | A1 |
| 20100017568 | Wadhawan et al. | Jan 2010 | A1 |
| 20100023674 | Aviles | Jan 2010 | A1 |
| 20100023682 | Lee et al. | Jan 2010 | A1 |
| 20100032676 | Moon | Jan 2010 | A1 |
| 20100030946 | Kano et al. | Feb 2010 | A1 |
| 20100036840 | Pitts | Feb 2010 | A1 |
| 20100042805 | Recio et al. | Feb 2010 | A1 |
| 20100070701 | Iyigun et al. | Mar 2010 | A1 |
| 20100070725 | Prahlad et al. | Mar 2010 | A1 |
| 20100070747 | Iyigun et al. | Mar 2010 | A1 |
| 20100070982 | Pitts | Mar 2010 | A1 |
| 20100076936 | Rajan | Mar 2010 | A1 |
| 20100077194 | Zhao et al. | Mar 2010 | A1 |
| 20100082774 | Pitts | Apr 2010 | A1 |
| 20100095059 | Kisley et al. | Apr 2010 | A1 |
| 20100169542 | Sinclair | Jul 2010 | A1 |
| 20100199036 | Siewert et al. | Aug 2010 | A1 |
| 20100205231 | Cousins | Aug 2010 | A1 |
| 20100205335 | Phan et al. | Aug 2010 | A1 |
| 20100211737 | Flynn et al. | Aug 2010 | A1 |
| 20100217916 | Guo et al. | Aug 2010 | A1 |
| 20100228903 | Chandrasekaran et al. | Sep 2010 | A1 |
| 20100235597 | Arakawa | Sep 2010 | A1 |
| 20100262738 | Swing et al. | Oct 2010 | A1 |
| 20100262740 | Borchers et al. | Oct 2010 | A1 |
| 20100262757 | Sprinkle et al. | Oct 2010 | A1 |
| 20100262758 | Swing et al. | Oct 2010 | A1 |
| 20100262759 | Borchers et al. | Oct 2010 | A1 |
| 20100262760 | Swing et al. | Oct 2010 | A1 |
| 20100262761 | Borchers et al. | Oct 2010 | A1 |
| 20100262762 | Borchers et al. | Oct 2010 | A1 |
| 20100262766 | Sprinkle et al. | Oct 2010 | A1 |
| 20100262767 | Borchers et al. | Oct 2010 | A1 |
| 20100262773 | Borchers et al. | Oct 2010 | A1 |
| 20100262894 | Swing et al. | Oct 2010 | A1 |
| 20100262979 | Borchers et al. | Oct 2010 | A1 |
| 20110022819 | Post et al. | Jan 2011 | A1 |
| 20110107033 | Grigoriev et al. | May 2011 | A1 |
| 20110179162 | Mayo et al. | Jul 2011 | A1 |
| 20110225342 | Sharma et al. | Sep 2011 | A1 |
| 20110231857 | Zaroo et al. | Sep 2011 | A1 |
| 20110238546 | Certain et al. | Sep 2011 | A1 |
| 20110265083 | Davis | Oct 2011 | A1 |
| 20110314202 | Iyigun et al. | Dec 2011 | A1 |
| 20110320733 | Sanford et al. | Dec 2011 | A1 |
| 20120159081 | Agrawal et al. | Jun 2012 | A1 |
| 20120173824 | Iyigun et al. | Jul 2012 | A1 |
| 20120226866 | Bozek et al. | Sep 2012 | A1 |
| 20120254824 | Bansod | Oct 2012 | A1 |
| 20120278588 | Adams et al. | Nov 2012 | A1 |
| 20130232303 | Quan | Sep 2013 | A1 |
| 20130339958 | Droste et al. | Dec 2013 | A1 |
| 20140136872 | Cooper et al. | May 2014 | A1 |
| 20140156938 | Galchev et al. | Jun 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 1771495 | May 2006 | CN |
| 1100001 | May 2001 | EP |
| 1418502 | May 2004 | EP |
| 1814039 | Mar 2009 | EP |
| 123416 | Sep 2001 | GB |
| 4242848 | Aug 1992 | JP |
| 8153014 | Jun 1996 | JP |
| 200259525 | Sep 2000 | JP |
| 2009122850 | Jun 2009 | JP |
| WO9419746 | Sep 1994 | WO |
| WO9518407 | Jul 1995 | WO |
| WO9612225 | Apr 1996 | WO |
| WO0131512 | May 2001 | WO |
| WO0201365 | Jan 2002 | WO |
| 2004061645 | Jul 2004 | WO |
| WO2004099989 | Nov 2004 | WO |
| WO2005103878 | Nov 2005 | WO |
| WO2006062511 | Jun 2006 | WO |
| WO2006065626 | Jun 2006 | WO |
| WO2008130799 | Mar 2008 | WO |
| WO2008073421 | Jun 2008 | WO |
| WO2011106394 | Sep 2011 | WO |
| Entry |
|---|
| WIPO, International Search Report and Written Opinion for PCT/US2012/039189, mailed Dec. 27, 2012. |
| Albert Noll et al., CellVM: A Homogeneous Virtual Machine System for a Heterogeneous Single-Chip Multiprocessor; Technical Report No. 06-17, Donald Bren School of Information and Computer Science, University of California, Irvine; Nov. 2006. |
| NEVEX Virtual Technologies, CacheWorks Data Sheet, http://www.nevex.com/wp-content/uploads/2010/12/Data-Sheet3.pdf, Published Dec. 1, 2010, Visited Aug. 1, 2012. |
| David C. Steere et al., “Efficient User-Level File Cache Management on the Sun Vnode Interface,” School of Computer Science, Carnegie Mellon University, CMU-CS-90-126, Usenix Conference Proceedings, Jun. 1990. |
| Pin Lu et al., Virtual machine memory access tracing with hypervisor exclusive cache, Proceedings of the USENIX Annual Technical Conference 2007 (ATC'07), Article No. 3, 15 pages. |
| Mark Friedman et al., File Cache Performace and Tuning, Windows 2000 Performance Guide, O'Reilly & Associates, Inc., http://msdn.microsoft.com/en-us/library/ms369863.aspx, Published Jan. 2002, Visited Aug. 3, 2012. |
| Microsoft, Filter Driver Development Guide, http://download.microsoft.com/download/e/b/a/eba1050f-a31d-436b-9281-92cdfeae4b45/FilterDriverDeveloperGuide.doc, 2004, Published 2004, Visited Aug. 3, 2012. |
| Microsoft, File Cache Management, Windows Embedded CE 6.0 R3, msdn.microsoft.com/en-us/subscriptions/aa911545.aspx. Published Aug. 28, 2008, Visited Aug. 3, 2012. |
| D. Muntz et al., Multi-level Caching in Distributed File Systems, CITI Technical Report 91-3, Aug. 16, 1991. |
| Omesh Tickoo et al, Modeling Virtual Machine Performance: Challenges and Approaches, SIGMETRICS Perform. Eval. Rev. 37, 3 (Jan. 2010), 55-60. DOI=10.1145/1710115.1710126 http://doi.acm.org/10.1145/1710115.1710126. |
| Craig Linn, Windows I/O Performance: Cache Manager and File System Considerations, CMGA Proceedings, Sep. 6, 2006. |
| Adabas, Adabas Caching Configuration and Tuning, http://documentation.softwareag.com/adabas/ada821mfr/addons/acf/config/cfgover.htm, Published Sep. 2009, Visited Aug. 3, 2012. |
| Adabas, Adabas Caching Facility, http://www.softwareage.com/es/Images/Adabas—Caching—Faility—tcm24-71167.pdf, Published 2008, Visited Aug. 3, 2012. |
| VMware, Introduction to VMware vSphere, http://www.vmware.com/pdf/vsphere4/r40/vsp—40—intro—vs.pdf, Published 2009, Visited Aug. 1, 2012. |
| VMware, Virtual Disk API Programming Guide, Virtual Disk Development Kit 1.2, Published Nov. 2010, Visited Aug. 3, 2012. |
| Atlantis Computing Technology, Caching, http://atlantiscomputing.com/technology/caching, published 2012, accessed Aug. 1, 2012. |
| Richard Rosen, IntelliCache, Scalability and consumer SSDs, blogs.citrix.com/2012/01/03/intellicache-scalability-and-consumer-ssds, Jan. 3, 2012, accessed Aug. 3, 2012. |
| Christian Ferber, XenDesktop and local storage+IntelliCache, Jun. 22, 2011, blogs.citrix.com/2011/06/22/xendesktop-and-local-storage-intellicache/, accessed Aug. 3, 2012. |
| Citrix, XenServer—6.0.0 Installation Guide, Mar. 2, 2012, http://support.citrix.com/servlet/KbServlet/download/28750-102-673824/XenServer-6.0.0-installation.pdf. accessed Aug. 3, 2012. |
| Adabas, Adabas Caching ASSO, Data, Work, Aug. 26, 2011, http://communities.softw areag.com/web/guest/pwiki/-/wiki/Main/.../pop—up?—36—viewMode=print, Oct. 2008, accessed Aug. 3, 2012. |
| Adabas, File Level Caching, http://documentation.softwareag.com/adabas/ada824mfr/addons/acf/services/file-level-caching.htm, accessed Aug. 3, 2012. |
| Actel, “Actel Fusion FPGAs Supporting Intelligent Peripheral Management Interface (IPMI) Applications,” http://www.actel.com/documents/Fusion—IPMI—AN.pdf, Oct. 1, 2006, visited Mar. 11, 2010. |
| Agigatech, Bulletproof Memory for RAID Servers, Part 1, http://agigatech.com/blog/bulletproof-memory-for-raid-servers-part-1/, last visited Feb. 16, 2010. |
| Anonymous, “Method for Fault Tolerance in Nonvolatile Storage”, http://ip.com, IP.com No. IPCOM000042269D, 2005. |
| Ari, “Performance Boosting and Workload Isolation in Storage Area Networks with SanCache,” Hewlett Packard Laboratories, Proceedings of the 23rd IEEE / 14th SA Goddard Conference on Mass Storage Systems and Technologies (MSST 2006), May 2006, pp. 263-27. |
| Arpaci-Dusseau, “Removing the Costs of Indirection in Flash-based SSDs with Nameless Writes,” Jun. 2010, HotStorage'10, Boston, MA. |
| Asine, “ASPMC-660 Rugged IDE Flash Drive PMC Module,” http://www.asinegroup.com/products/aspmc660.html, copyright 2002, visited Nov. 8, 2009. |
| Bandulet “Object-Based Storage Devices,” Jul. 2007 http://developers.sun.com/solaris/articles/osd.htme, visited Dec. 1, 2011. |
| Barrall et al., U.S. Appl. No. 60/625,495, “Dynamically Expandable and Contractible Fault-Tolerant Storage System Permitting Variously Sized Storage Devices and Method,” filed Nov. 5, 2004. |
| Barrall et al., U.S. Appl. No. 60/718,768, “Dynamically Adaptable Fault-Tolerant Storage System,” filed Sep. 20, 2005. |
| BiTMICRO, “BiTMICRO Introduces E-Disk PMC Flash Disk Module at Military & aerospace Electronics East 2004,” http://www. bitmicro.com/press.sub, published May 18, 2004, visited Mar. 8, 2011. |
| Bonnet, “Flash Device Support for Database Management,” published Jan. 9, 2011. |
| Brandon, Jr., “Sparse Matrices in CS Education,” Journal of Computing Sciences in Colleges, vol. 24 Issue 5, May 2009, pp. 93-98. |
| Casey, “San Cache: SSD in the San,” Storage Inc., http://www.solidata.com/resourses/pdf/storageing.pdf, 2000, visited May 20, 2011. |
| Casey, “Solid State File-Caching for Performance and Scalability,” SolidData Quarter 1 2000, http://www.storagesearch.—com/3dram.html, visited May 20, 2011. |
| Clustered Storage Solutions: “Products,” http://www.clusteredstorage.com/clustered—storage—solutions.HTML, last visited Feb. 16, 2010. |
| Coburn, “NV-Heaps: Making Persistent Objects Fast and Safe with Next-Generation, Non-Volatile Memories”, ACM 978-1-4503-0266-1/11/0, published Mar. 5, 2011. |
| Data Direct Networks, “White Paper: S2A9550 Overview,” www.//datadirectnet. com, 2007. |
| EEEL-6892, Lecture 18, “Virtual Computers,” Mar. 2010. |
| ELNEC, “NAND Flash Memories and Programming NAND Flash Memories Using ELNEC Device Programmers, Application Note,” published Mar. 1, 2007. |
| Friedman, Mark, et al., “File Cache Performance and Tuning, Windows 2000 Performance Guide, O'Reilly & Associates, Inc., http://msdn.microsoft.com/en-us/library/ms369863.aspx,” published Jan. 2002, visited Aug. 3, 2012. |
| Gal, “A Transactional Flash File System for Microcontrollers,” 2005 USENIX Annual Technical Conference, published Apr. 10, 2009. |
| Garfinkel, “One Big File Is Not Enough: A Critical Evaluation of the Dominant Free-Space Sanitization Technique,” 6th Workshop on Privacy Enhancing Technologies. Cambridge, United Kingdom, published Jun. 1, 2006. |
| Gill, “WOW: Wise Ordering for Writes—Combining Spatial and Temporal Locality in Non-Volatile Caches,” IBM, Fast ″05: 4th USENIX Conference on File and Storage Technologies, 2005. |
| Gutmann, “Secure Deletion of Data from Magnetic and Solid-State Memory”, Usenix, 14 pages, San Jose, CA, published Jul. 1, 1996. |
| Huffman, “Non-Volatile Memory Host Controller Interface,” Apr. 14, 2008, 65 pgs. |
| Hynix Semiconductor, Intel Corporation, Micron Technology, Inc. Phison Electronics Corp., Sony Corporation, Spansion, Stmicroelectronics, “Open NAND Flash Interface Specification,” Revision 2.0, Feb. 27, 2008. |
| HYTSOR: “Making SSDs the Survival of the Fittest in High-Performance Storage Systems,” ics10-Paper 102, Feb. 2010. |
| IBM, “Method to Improve Reliability of SSD Arrays,” Nov. 2009. |
| Information Technology, “SCSI Object-Based Storage Device Commands,” 2 (OSD-2), Project T10/1729-D, Revision 4, published Jul. 30, 2004, printed Jul. 24, 2008. |
| Intel, “Non-Volatile Memory Host Controller Interface (NVMHCI) 1.0,” Apr. 14, 2008. |
| Johnson, “An Introduction to Block Device Drivers,” Jan. 1, 1995. |
| Kawaguchi, “A Flash-Memory Based File System,” TCON'95 Proceedings of the USENIX 1995 Technical Conference Proceedings, p. 13. |
| Linn, Craig, “Windows I/O Performance: Cache Manager and File System Considerations,” CMGA Proceedings, Sep. 6, 2006. |
| Lu, Pin, “Virtual Machine Memory Access Tracing with Hypervisor Exclusive Cache,” Departmentn of Computer Science, University of Rochester, 2007. |
| Mesnier, “Object-Based Storage,” IEEE Communications Magazine, Aug. 2003, pp. 84-90. |
| Micron Technology, Inc., “NAND Flash 101: An Introduction to ND Flash and How to Design It in to Your Next Product (TN-29-19),” http://www.micron.com/˜/media/Documents/Products/Technical%20Note/ND%20Flash/145tn2919—nd—101.pdf, 2006, visited May 10, 2010. |
| Micron, “TN-29-08: Technical Note, Hamming Codes for NAND Flash Memory Devices,” Mar. 10, 2010. |
| Micron, “TN-29-17: NAND Flash Design and Use Considerations,” Mar. 10, 2010. |
| Micron, “TN-29-42: Wear-Leveling Techniques in NAND Flash Devices,” Mar. 10, 2010. |
| Microsoft, Data Set Management Commands Proposal for ATA8-ACS2, published Oct. 5, 2007, Rev. 3. |
| Microsoft, “How NTFS Works,” Apr. 9, 2010. |
| Morgenstern, David, “Is There a Flash Memory RAID in your Future?”, http://www.eweek.com— eWeek, Ziff Davis Enterprise Holdings Inc., Nov. 8, 2006, visited Mar. 18, 2010. |
| Novell, “File System Primer”, http://wiki.novell.com/index.php/File—System—Primer, 2006, visited Oct. 18, 2006. |
| PerfectCacheServer, “Automatic Disk Caching,” http://www. Raxco. Com/business/perfectcache—server.aspx, last visited Oct. 31, 2012. |
| PIVOT3, “RAIGE Cluster: Technology Overview,” White Paper, www.pivot3.com, Jun. 2007. |
| Plank, “A Tutorial on Reed-Solomon Coding for Fault Tolerance in RAID-like System,” Department of Computer Science, University of Tennessee, pp. 995-1012, Sep. 1997. |
| Porter, “Operating System Transactions,” ACM 978-1-60558-752-3/09/10, published Oct. 1, 2009. |
| Probert, “Windows Kernel Internals Cache Manager,” Microsoft Corporation, http://www.i.u.tokyo.ac.jp/edu/training/ss/ lecture/new-documents/Lectures/15-CacheManager/CacheManager.pdf, printed May 15, 2010. |
| Ranaweera, 05-270RO, SAT: Write Same (10) command (41h), T10/05, Jul. 7, 2005, www.t10.org/ftp/t10/document.05/05-270r0.pdf, last visited Apr. 11, 2013. |
| Rosen, Richard, “IntelliCache, Scalability and consumer SSDs,” blogs.citrix.com/2012/01/03/intellicache-scalability-and-consumer-ssds, Jan. 3, 2012, accessed Aug. 3, 2012. |
| Rosenblum, “The Design and Implementation of a Log-Structured File System,” ACM Transactions on Computer Systems, vol. 10 Issue 1, Feb. 1992. |
| Samsung Electronics, “Introduction to Samsung's Linux Flash File System—RFS Application Note”, Version 1.0, Nov. 2006. |
| Seagate Technology LLC, “The Advantages of Object-Based Storage-Secure, Scalable, Dynamic Storage Devices, Seagate Research Technology Paper, TP-536” Apr. 2005. |
| Sears, “Stasis: Flexible Transactional Storage,” OSDI '06: 7th USENIX Symposium on Operating Systems Design and Implementation, published Nov. 6, 2006. |
| Seltzer, “File System Performance and Transaction Support”, University of California at Berkeley, published Jan. 1, 1992. |
| Seltzer, “Transaction Support in a Log-Structured File System”, Harvard University Division of Applied Sciences, published Jan. 1, 1993 (Chapter 5, pp. 52-69). |
| Seltzer, “Transaction Support in Read Optimized and Write Optimized File Systems,” Proceedings of the 16th VLDB Conference, Brisbane, Australia, published Jan. 1, 1990. |
| Shimpi, Anand, The SSD Anthology: Understanding SSDs and New Drives from OCZ, Mar. 18, 2009, 69 pgs. |
| Shu, “Data Set Management Commands Proposals for ATA8-ACS2,” Dec. 12, 2007, http://www.t13.org.Documents/Uploaded Docu ments/docs2008/e07154r6-Data—Set—Management—Proposal—for—ATA-ACS2.pdf, printed Apr. 5, 2010. |
| Singer, Dan, “Implementing MLC ND Flash for Cost-Effective, High Capacity Memory,” M-Systems, White Paper, 91-SR014-02-8L, Rev. 1.1, Sep. 2003. |
| Solidata, “Best Practices Guide, Sybase: Maximizing Performance through Solid State File-Caching,” http:// solidata.com/resources/pdf/bp-sybase.pdf. May 2000, cited May 18, 2011. |
| Spansion, “Data Management Software (DMS) for AMD Simultaneous Read/Write Flash Memory Devices”, published Jul. 7, 2003. |
| Spillane, “Enabling Transactional File Access via Lightweight Kernel Extensions”, Stony Brook University, IBM T. J. Watson Research Center, published Feb. 25, 2009. |
| State Intellectual Property Office, Office Action, CN Application No. 200780050970.0, issued Jun. 29, 2011. |
| State Intellectual Property Office, Office Action, Chinese Nationalization of U.S. Appl. No. 11/952,123. |
| State Intellectual Property Office, Office Action, CN Application No. 200780050970.0, issued Oct. 28, 2010. |
| State Intellectual Property Office, Office Action, CN Application No. 200780051020.X, issued Nov. 11, 2010. |
| State Intellectual Property Office, Office Action, CN Application No. 200780050983.8, issued May 18, 2011. |
| State Intellectual Property Office, Office Action, CN Application No. 200780051020.X, issued Jul. 6, 2011. |
| State Intellectual Property Office, Office Action, CN Application No. 200780051020.X, issued Nov. 7, 2011. |
| State Intellectual Property Office, Office Action, CN Application No. 200780050970.0, issued Jan. 5, 2012. |
| Steere, David et al., Efficient User-Level File Cache Management on the Sun Vnode Interface, School of Computer Science, Carnegie Mellon University, Apr. 18, 1990. |
| Superspeed, “New Super Cache 5 on Servers,” http:// www.superspeed.com/servers/supercache.php, last visited Oct. 31, 2013. |
| Tal, “NAND vs. NOR Flash Technology,” M-Systems, www2.electronicproducts.com/PrintArticle.aspx?ArticleURL=FEBMSY1.feb2002.html, visited Nov. 22, 2010. |
| Terry et al., U.S. Appl. No. 60/797,127, “Filesystem-aware Block Storage System, Apparatus, and Method,” filed May 3, 2006. |
| USTPO, Interview Summary for U.S. Appl. No. 10/372,734, mailed Feb. 28, 2006. |
| USPTO, Notice of Allowance for U.S. Appl. No. 12/986,117, Mailed Apr. 4, 2013. |
| USPTO, Notice of Allowance for U.S. Appl. No. 12/986,117 Mailed Jun. 5, 2013. |
| USPTO, Office Action for U.S. Appl. No. 12/879,004 mailed Feb. 25, 2013. |
| USPTO, Office Action for U.S. Appl. No. 13/607,486 mailed Jan. 10, 2013. |
| USPTO, Office Action for U.S. Appl. No. 10/372,734, mailed Sep. 1, 2005. |
| USPTO, Office Action for U.S. Appl. No. 11/952,113, mailed Dec. 15, 2010. |
| USPTO, Office Action for U.S. Appl. No. 12/711,113, mailed Jun. 6, 2012. |
| USPTO, Office Action for U.S. Appl. No. 12/711,113, mailed Nov. 23, 2012. |
| USPTO, Office Action for U.S. Appl. No. 13/607,486 mailed May 2, 2013. |
| USPTO, Office Action for U.S. Appl. No. 13/118,237 mailed Apr. 22, 2013. |
| USPTO, Notice of Allowance, U.S. Appl. No. 11/952,109, issued May 1, 2013. |
| USPTO, Office Action, U.S. Appl. No. 11/952,109, mailed Nov. 29, 2011. |
| Van Hensbergen, IBM Research Report, “Dynamic Policy Disk Caching for Storage Networking,” IBM Research Division, Computer Science, RC24123 (WO611-189), Nov. 28, 2006. |
| VMware, Introduction to VMware vSphere, http://www.vmware.com/pdf/vsphere4/r40/vsp—40—intro—vs.pdf, 2009, accessed Aug. 1, 2012. |
| VMware, Virtual Disk API Programming Guide, Virtual Disk Development Kit 1.2, Nov. 2010, accessed Aug. 3, 2012. |
| Volos, “Mnemosyne: Lightweight Persistent Memory”, ACM 978-1-4503-0266-1/11/03, published Mar. 5, 2011. |
| Wacha, “Improving RAID-Based Storage Systems with Flash Memory,” First Annual ISSDM/SRL Research Symposium, Oct. 20-21, 2009. |
| Walp, “System Integrated Flash Storage,” Microsoft Corporation, 2008, http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T559—WHO8.pptx, Printed Apr. 6, 2010, 8 pgs. |
| Wang, “OBFS: A File System for Object-based Storage Devices”, 21st IEE/12th SA Goddard Conference on Mass Storage Systems and Technologies, Apr. 2004. |
| Wikipedia, “Object Storage Device,” http://en.wikipedia.org/wiki/Object-storage-device, last visited Apr. 29, 2010. |
| Winnett, Brad, “S2A9550 Overview,” White Paper, http://www.ddn.com/pdfs/ddn—s2a—9550—white—paper.pdf, Jul. 2006, 27 pgs. |
| WIPO, International Preliminary Report of Patentability for PCT/US2007/086691, mailed Feb. 16, 2009. |
| WIPO, International Preliminary Report on Patentability for PCT/US2007/086688, mailed Mar. 16, 2009. |
| WIPO, International Preliminary Report on Patentability for PCT/US2007/086701, mailed Mar. 16, 2009. |
| WIPO, International Preliminary Report on Patentability for PCT/US2007/086687, mailed Mar. 18, 2009. |
| WIPO, International Preliminary Report on Patentability for PCT/US2007/025048, mailed Jun. 10, 2009. |
| WIPO, International Preliminary Report on Patentability for PCT/US2010/048325, mailed Mar. 13, 2012. |
| WIPO, International Search Report and Written Opinion for PCT/US2007/086691, mailed May 8, 2008. |
| WIPO, International Search Report and Written Opinion for PCT/US2007/025049, mailed May 14, 2008. |
| WIPO, International Search Report and Written Opinion for PCT/US2007/025048, mailed May 27, 2008. |
| WIPO, International Search Report and Written Opinion for PCT/US2007/086701, mailed Jun. 5, 2008. |
| WIPO, International Search Report and Written Opinion for PCT/US2007/086687, mailed Sep. 5, 2008. |
| WIPO, International Search Report and Written Opinion for PCT/US2011/65927, mailed Aug. 28, 2012. |
| WIPO, International Search Report and Written Opinion for PCT/US2012/029722, mailed Oct. 30, 2012. |
| WIPO, International Search Report for PCT/US2012/039189, mailed Dec. 27, 2012. |
| WIPO, International Search Report and Written Opinion PCT/US2010/025885, mailed Sep. 28, 2011. |
| WIPO, International Search Report PCT/US2012/050194, mailed Feb. 26, 2013. |
| Woodhouse, “JFFS: The Journaling Flash File System,” Ottawa Linux Symposium, http://sources.redhat.com/jffs2/jffs2.pdf, Jul. 2001. |
| Wright, “Extending ACID Semantics to the File System”, ACM Transactions on Storage, vol. 3, No. 2, published May 1, 2011, pp. 1-40. |
| Wu, “eNVy: A Non-Volatile, Main Memory Storage System,” ACM 0-89791-660-3/94/0010, ASPLOS-VI Proceedings of the sixth international conference on Architectural support for programming languages and operating systems, pp. 86-97, 1994. |
| Yang, “A DCD Filter Driver for Windows NT 4,” Proceedings of the 12th International Conference on Computer Applications in Industry and Engineering (CAINE-99), Atlanta, George, USA, Nov. 4-6, 1999. |
| Yerrick, “Block Device,” http://www.pineight.com/ds/block, last visited Mar. 1, 2010. |
| USPTO, Office Action for U.S. Appl. No. 13/287,998 mailed Jun. 10, 2014. |
| USPTO, Office Action for U.S. Appl. No. 14/262,581 mailed Jun. 19, 2014. |
| USPTO, Office Action Interview Summary for U.S. Appl. No. 13/541,659 mailed Aug. 26, 2014. |
| USPTO, Office Action for U.S. Appl. No. 13/687,979 mailed Sep. 9, 2014. |
| USPTO, Office Action for U.S. Appl. No. 13/288,005 mailed Jul. 8, 2014. |
| Number | Date | Country | |
|---|---|---|---|
| 20120304171 A1 | Nov 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61489230 | May 2011 | US |
