This application is related by subject matter to the U.S. patent application Ser. Nos. 12/260,041 and 12,260,054, both filed Oct. 28, 2008 and entitled “Quality of Service Management” and both of which are incorporated herein by reference in their entirety.
Modern data centers often have a multi-tier configuration wherein a front end server accesses one or more layers of middle-tier and back-tier servers for various services. One example of a back-end server is a storage array. Storage arrays form the backbone of modern data centers by providing consolidated data access to multiple applications simultaneously. Increasingly, organizations are moving towards consolidated storage, either using block-based access over a Storage Area Network (SAN) or file-based access over Network-Attached Storage (NAS) systems. A Storage Area Network is a network whose primary purpose is the transfer of data between computer systems and storage elements. Easy access from anywhere at anytime, ease of backup, flexibility in allocation and centralized administration are some of the advantages of storage arrays.
Quality of Service (QoS) refers to resource management methodologies whereby resources are allocated among a plurality of users or clients according to a policy. The policy may guarantee a minimum and/or maximum level of service (e.g., as a percentage of resources), or it may set limits and reservations which are expressed in absolute units, such as MHz or GHz for CPU resource allocation, GB for memory or disk resource allocation, and Mbps or Gbps for network bandwidth resource allocation. A “limit” is a maximum level of service expressed in terms of absolute units and a “reservation” is a minimum level of service expressed in terms of absolute units. It is also common to distribute services according to assigned resource “shares” (also known as “weights”) so that each client is provided a level of service that compares to its peers at the same ratio as the assigned shares. In addition, combinations of these policies are possible. Thus, QoS suggests an ability to evenly distribute services or arbitrarily assign priority to selected applications, users, or data flows. QoS management, however, may be complicated when a client that is not bound by a QoS policy accesses a shared resource. Such client may consume services at a rate that impacts the level of service received by the clients that are bound by the QoS policy. In some situations referred to herein as an “anomaly,” the workload presented by the client not bound by the QoS policy is so large that it interferes with the proper distribution of resources, such as IO access bandwidth, to clients in accordance with the QoS policy.
One or more embodiments of the invention provide a method for detecting an anomaly in a shared input/ouput (IO) resource that is accessed by a plurality of hosts or clients. The anomaly detection triggers a response from the hosts or clients as a way to protect against the effect of the anomaly.
In one embodiment, the anomaly is caused and detected when a host that is not bound by any QoS policy presents large workloads to a shared IO resource that is also accessed by hosts that are governed by QoS policy. When an anomaly is detected, the hosts that are governed by QoS policy respond to the anomaly by increasing their window sizes. The window sizes of these hosts may be increased to the maximum window size or in proportion to their QoS shares.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. In other instances, well known process operations and implementation details have not been described in detail in order to avoid unnecessarily obscuring the invention.
Interconnect network 120 may be a wide area network, a local area network, or a network hosting a protocol especially suited for storage arrays, such as Fibre Channel, iSCSI, HyperSCSI, etc. For example network 120 may comprise one or more of Fibre Channel switches. Storage array 130 may be any type of storage array such as a network-attached storage (NAS) filer or a blocked-based device over a storage area network (SAN). Storage array 130 may include a manager 132 and an array of devices 136 (e.g., disks). Manager 132 is a computer program executing on one or more processors, which may be implemented as an appliance or a plurality of appliances working in tandem. Manager 132, may, for example, comprise a plurality of storage processors, as generally understood in the art of storage arrays. While storage arrays are typically made up of a plurality of disks, it should be recognized that as prices for solid-state non-volatile storage devices fall, they are increasingly taking the place of rotating disk storage media. The use of the term, “disk” herein, should therefore not be construed as limited only to rotating disk storage media, but also what is become known as solid state disks, or “SSDs.”
Manager 132 maintains a request queue 134, which is a list of pending IO requests that may be satisfied in any order. Each request comprises a request to read and/or write data to or from storage array 130. Each read request identifies an address, address range or other identifier of the data to be read whereas write requests include data to be written along with an identifier for a location in the array where the data is to be written. Each request that is pending in request queue 134 corresponds to a request from one of hosts 110.
Although QoS policy for hosts 110 governs their accesses to storage array 130, unmanaged client 116 is not bound by any such policy. As a result, large workloads presented by unmanaged client 116 may cause an undesirable reduction in the available IO access bandwidth and interfere with the IO access bandwidth management of hosts 110. In one embodiment, hosts 110 are configured to detect when the IO access bandwidth between hosts 110 and storage array 130 is reduced. If the reductions are severe enough, they may be identified as an anomaly, and hosts 110 adapt to protect against the effects of the anomaly. In another embodiment, shown in
When issue queue 117 is implemented in hardware as part of the HBA for each host, issue queue 117 may have a maximum size that can be exceeded by the total number of requests from clients 112. To accommodate these “overflow” IO requests, a buffer (not shown) in the disk IO handler 114 may receive overflow IO requests from all clients when issue queue 117 is full. In one embodiment, the buffer is a first-in, first-out (FIFO) buffer. When an IO request is satisfied, a slot in the issue queue is opened, and the next (longest-residing) IO request in the buffer is sent to the HBA 118. Although in this embodiment the buffer is a FIFO buffer in which the longest-residing IO request is removed, other algorithms may be implemented, such as preferentially selecting IOs in series that are close to one other.
In another embodiment a separate buffer is provided for each client. When an IO request is issued by a host 110, a new IO request from one of the separate buffers takes its place in the issue queue 117. User-set shares (also sometimes referred to as “weights”) for each client on the host can be implemented by changing the frequency of IO request draw from the corresponding client. For example, if clients 1, 2, and 3 are given shares of 100, 200, and 300, respectively, then for every one IO request pulled from the buffer associated with client 1, two IO requests are pulled from the buffer associated with client 2 and three IO requests are pulled from the buffer associated with client 3. It should be noted that some HBAs may be configured to directly manage a plurality of issue queues, so that there could be separately managed individual issue queues for each client. Also, scheduling policies other than proportional sharing, such as scheduling algorithms based on priorities, tokens, credits, reservations, or limits associated with each client, may be implemented in this embodiment.
Each VM may include a guest operating system (GOS) and one or more applications (APP). The guest operating systems may be a commodity operating system such as Microsoft Windows® or a specialized operating system designed specifically to work with virtualization software 111 (sometimes referred to as a “paravirtualized OS”). In one embodiment, virtualization software 111 resides on a physical data storage medium (not shown) forming part of host 110, whereas virtual disks (not shown) for each client virtual machine are mapped by virtualization software 111 to files that reside remotely or locally. The guest operating system and applications access data at storage array 130 by way of a virtual host bus adapter (not shown) that is mapped by virtualization software 111 to host bus adapter 118. Note that this need not be a one-to-one mapping; e.g., there could be several virtual disk controllers in the guest and multiple physical HBAs on the host. In this case, the virtualization software may choose to send individual requests via different physical HBAs.
If one or more of hosts 110 have one or more VMs running, it may be desirable to assign a QoS share for each VM. For example, one host 110 may have two VMs, wherein one of the VMs requires faster response time. In this case, it would be desirable to provide greater QoS shares to the VM requiring the faster response time. A similar situation can occur for non-VM clients as well, wherein an operating system can give greater shares to one running application in preference to other running applications. Using the QoS policy it is possible, in some embodiments described below, to separately assign shares to individual clients.
Because storage array 130 is a shared resource, contention at storage array 130, e.g., at the array controller, cache, buses, and disk arms, may lead to unpredictable IO completion times. These features of storage array 130 complicate efforts to fairly distribute services among hosts 110 according to a QoS policy. Various techniques have been developed to fairly distribute accesses to a common resource among a plurality of hosts 110 according to QoS policy, one of which is described in U.S. patent application Ser. No. 12/260,041, filed Oct. 28, 2008, the entire contents of which are incorporated by reference herein. In U.S. patent application Ser. No. 12/260,041, a control algorithm is described that manages issue queue sizes, also referred to as “window sizes,” based in part on a calculation of a moving average latency, referred to as “current average latency” or CAL. CAL for host 110 at time t may be as computed according to Equation 1:
CAL(t)=(1−α)×L+α×CAL(t−1) (Eq. 1)
In Equation 1, L is the current latency at host 110. Current latency L is the issue queue residency time for the IO request most recently removed from issue queue 117 (
The control algorithm that manages window sizes implements Equation 2.
Equation 2 solves for a new window size w(t+1), where w(t+1) is the adjusted window size for time t+1; w(t) is the current window size; γ is a constant value; LATthreshold is a system-wide latency threshold selected to balance throughput with latency; LSYS(t) is the system-wide average latency across hosts 110 at time t; and β is a per-host value based on an assigned share representing a relative level of priority of the host relative to other hosts. The constant γ is a value selected between zero and one and defines how much influence the current window size has over the new window size. The lower the value of gamma, the more weight is given to the current window size w(t).
Embodiments of the present invention may implement any technique for implementing QoS policy, including the window size adjustment technique described above. However, unmanaged client 116 is not bound by any QoS policy, and so large workloads presented by unmanaged client 116 would cause a proportionally large reduction in the IO access bandwidth available to hosts 110 and throttling of IO requests by hosts. Embodiments of the present invention provide methods for detecting such situations and implementing counter-measures to protect against the effect of such situations on the IO access bandwidth management of hosts 110.
In one embodiment, when an anomaly is detected, the window sizes of all of hosts 100 are no longer adjusted in accordance with QoS policy, but instead are increased to its maximum size. In another embodiment, host 110 that has the largest QoS share is increased to its maximum size, and the remaining hosts 110 are allocated window sizes in relative proportion according to their QoS shares.
According to one or more embodiments of the present invention, an anomaly is detected based on various IO statistics. The first is a value known as a latency threshold, LATThreshold. This value may be set empirically based on the relationship between latency and throughput. LATThreshold may be a fixed constant that is the same for all hosts 110 or may be a programmable value. A typical conservative value for LATThreshold would be between 30 and 50 milliseconds. In one embodiment, LATThreshold is a user-adjustable parameter with a broad range, e.g., 5-100 milliseconds. User input could therefore be used to set the threshold based on application-specific requirements. In addition to QoS fairness, efficient utilization of the storage array and a work-conserving algorithm are important goals. In another embodiment, LATThreshold can also be adjusted by observing the long-term behavior of the workload. This observation may be performed by a central entity (e.g., manager 146) that can obtain latency and bandwidth information from all hosts 110 and observe the latency values that correspond to various peaks in the observed bandwidth.
Other IO statistics include current average latency and the number of outstanding IO requests (OIOs). These statistics are maintained for each host 110 and compiled at periodic intervals. Any technique for computing current average latency may be employed, including those disclosed in U.S. patent application Ser. No. 12/260,041. In addition, current average latency values that are normalized with respect to IO request sizes may be used.
In one embodiment, as illustrated in
An example of shared file 138 is shown below in TABLE 1. Shared file 138 includes a header portion 140 in which an anomaly flag is stored, and a body portion with a plurality of blocks 139. In each block 139, a history of each corresponding host's OIO values, current average latency values, and window size values are stored. Storing a history allow host 110 or manager 146 to differentiate between a persistent workload presented by unmanaged client 116 and a workload of short duration. The persistent workload should be recognized in order to configure hosts 110 to compete with unmanaged client 116 while a workload of short duration may be ignored. As shown in TABLE 1, shared file 138 stores IO statistics for five different time samples, e.g., t, t-1, t-2, t-3, and t-4.
Each host 110 is able to update its own IO statistics stored in the shared file 138. However, the entries in the shared file 138 may be read by any of hosts 110. Since each host 110 may update the statistics independently, the time associated with each sample may be different, i.e., t-2 for each of hosts 110 may be a different absolute time. In addition, each host 110 has read/write access to header portion 140. Consequently, any of hosts 110 can set or clear the anomaly flag, as well as respond to an anomaly condition even though the anomaly may have been detected by another host 110.
In another embodiment of the present invention, shared file 138 only stores the current values of the IO statistics. Referring to TABLE 1, this means that only the “t” values of OIO, latency, and window size will be maintained by shared file 138, and the history data (t-1, t-2, t-3, and t-4 data) are stored in memory at the respective hosts 110. In addition, the number of time samples covered by the history data is a parameter that is configurable by the hosts 110.
As an alternative to shared file 138, IO statistics may be maintained in raw blocks on a shared storage partition, or maintained individually at each host/client and communicated to each other using network communication.
For example, if the current average latency is not near the latency threshold (e.g., LATThreshold−k) (step 510), an anomaly is not detected. The value of k is a constant that is configurable and represents how near the current average latency needs to be to the LATThreshold in order to possibly trigger the anomaly detection and response. That is, if hosts 110 are not experiencing undesirably long latencies such that the current average latency of hosts 110 is less than the value represented by LATThreshold−k, an anomaly is not detected (step 535). On the other hand, if the current average latency is near the latency threshold (i.e., greater than the value represented by LATThreshold−k), and the conditions set forth in steps 515, 520, and 525 are met, an anomaly is detected and the anomaly flag is set (step 530). In other words, under the condition where the current average latency is near the latency threshold, if the aggregate window size has decreased sufficiently (i.e., ΔW>WAnomaly) over a certain time period (step 515), and the number of OIOs of all hosts 110 have also decreased by at least a certain amount (denoted as OIOAnomaly), but the latency has not decreased by more than a small threshold (denoted as LAnomaly), over the same period (i.e., ΔL>LAnomaly and ΔOIO>OIOAnomaly; steps 520 and 525), an anomaly is detected and the anomaly flag is set (step 530). If any of the conditions set forth in steps 515, 520, and 525 is not met, an anomaly is not detected (step 535).
In one embodiment, ΔW is computed as (Wp−Wc)/Wc, where Wp is the window size during a prior time period and We is the window size during the current time period; ΔL is computed as (Lma−Lp)/Lp, where Lp is the average latency during a prior time period and Lma is the moving average of the average latency; and ΔOIO is computed as (OIOp−OIOma)/OIOma, where OIOp is the number of OIOs of all hosts 110 during a prior time period and OIOma is the moving average of the number of OIOs of all hosts 110. The values of WAnomaly, LAnomaly, and OIOAnomaly may each be fixed or programmed. In one exemplary embodiment, these values are selected to be 0.3, −0.05, and 0.1, respectively.
In another embodiment, the change in the ratio of LATAverage to OIO may be computed and evaluated for drastic changes, which can be detected in various ways, including:
In other embodiments, an anomaly may be detected using any known pattern matching, machine learning, or statistical analysis techniques to identify cases where array performance continues to worsen despite continued throttling of IOs from managed clients.
After an anomaly is detected and the anomaly flag is set to TRUE, the anomaly condition is forced to persist for a first predefined period of time so that each host 110 will have time to respond to the anomaly in accordance with method 550. At the expiration of this first predefined time period, the anomaly flag is set to FALSE. After the anomaly flag has been set to FALSE, this condition is forced to persist for a second predefined period of time to prevent the same anomaly condition from triggering another response too early. The first and second predefined periods of time are configurable by the user and may be programmatically adjusted based on real-time behavior of the shared resource.
While embodiments described above are directed to detecting and responding to an anomaly in a system where multiple clients share a common storage resource, the same principles may be applied to a system where multiple clients share a common network, such as a LAN in which clients communicate using the TCP protocol. In a shared network situation, QoS policy may be implemented to provide a guaranteed minimum bit rate, delay, jitter, packet dropping probability, or error rate. If a group of clients that are subject to the QoS policy share the same network as an unmanaged client, i.e., a client that is not bound by any QoS policy, heavy network IO from the unmanaged client will negatively impact the level of service received by the clients that are bound by the QoS policy. Thus, additional embodiments of the invention are directed to providing a method for detecting an anomaly in a shared network and triggering a response to protect against the negative effects of the anomaly detected in the shared network.
The various embodiments described herein may employ various computer-implemented operations involving data stored in computer systems. For example, these operations may require physical manipulation of physical quantities—usually, though not necessarily, these quantities may take the form of electrical or magnetic signals, where they or representations of them are capable of being stored, transferred, combined, compared, or otherwise manipulated. Further, such manipulations are often referred to in terms, such as producing, identifying, determining, or comparing. Any operations described herein that form part of one or more embodiments of the invention may be useful machine operations. In addition, one or more embodiments of the invention also relate to a device or an apparatus for performing these operations. The apparatus may be specially constructed for specific required purposes, or it may be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
The various embodiments described herein may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like.
One or more embodiments of the present invention may be implemented as one or more computer programs or as one or more computer program modules embodied in one or more computer readable media. The term computer readable medium refers to any data storage device that can store data which can thereafter be input to a computer system—computer readable media may be based on any existing or subsequently developed technology for embodying computer programs in a manner that enables them to be read by a computer. Examples of a computer readable medium include a hard drive, network attached storage (NAS), read-only memory, random-access memory (e.g., a flash memory device), a CD (Compact Discs)—CD-ROM, a CD-R, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
Although one or more embodiments of the present invention have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be made within the scope of the claims. Accordingly, the described embodiments are to be considered as illustrative and not restrictive, and the scope of the claims is not to be limited to details given herein, but may be modified within the scope and equivalents of the claims. In the claims, elements and/or steps do not imply any particular order of operation, unless explicitly stated in the claims.
Virtualization systems in accordance with the various embodiments, may be implemented as hosted embodiments, non-hosted embodiments or as embodiments that tend to blur distinctions between the two, are all envisioned. Furthermore, various virtualization operations may be wholly or partially implemented in hardware. For example, a hardware implementation may employ a look-up table for modification of storage access requests to secure non-disk data.
Many variations, modifications, additions, and improvements are possible, regardless the degree of virtualization. The virtualization software can therefore include components of a host, console, or guest operating system that performs virtualization functions. Plural instances may be provided for components, operations or structures described herein as a single instance. Finally, boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention(s). In general, structures and functionality presented as separate components in exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements may fall within the scope of the appended claims(s).
Number | Name | Date | Kind |
---|---|---|---|
6654344 | Toporek et al. | Nov 2003 | B1 |
6675211 | Mamaghani et al. | Jan 2004 | B1 |
6886035 | Wolff | Apr 2005 | B2 |
7158536 | Ching et al. | Jan 2007 | B2 |
7213077 | Border | May 2007 | B2 |
7356679 | Le et al. | Apr 2008 | B1 |
7373417 | Abbasi et al. | May 2008 | B1 |
7420990 | Ching et al. | Sep 2008 | B2 |
7551623 | Feroz et al. | Jun 2009 | B1 |
7606887 | Stewart et al. | Oct 2009 | B1 |
7643418 | Varier et al. | Jan 2010 | B1 |
7720954 | Raja et al. | May 2010 | B2 |
7865584 | Grossner et al. | Jan 2011 | B2 |
7873060 | Luo et al. | Jan 2011 | B2 |
7912951 | Gulati et al. | Mar 2011 | B2 |
8064910 | Lee | Nov 2011 | B2 |
8127014 | Gulati et al. | Feb 2012 | B2 |
8140827 | Liang et al. | Mar 2012 | B2 |
8250197 | Gulati et al. | Aug 2012 | B2 |
20040057383 | Tseng et al. | Mar 2004 | A1 |
20040068461 | Schluetter | Apr 2004 | A1 |
20040194095 | Lumb et al. | Sep 2004 | A1 |
20050021591 | Boyd et al. | Jan 2005 | A1 |
20050021879 | Douglas | Jan 2005 | A1 |
20050141419 | Bergamasco et al. | Jun 2005 | A1 |
20060090163 | Karisson et al. | Apr 2006 | A1 |
20060168594 | Deverill et al. | Jul 2006 | A1 |
20070104188 | Kuc | May 2007 | A1 |
20070162462 | Zhang et al. | Jul 2007 | A1 |
20080046558 | Raja et al. | Feb 2008 | A1 |
20080141333 | Chen et al. | Jun 2008 | A1 |
20080244209 | Seelam et al. | Oct 2008 | A1 |
20080320278 | Liang et al. | Dec 2008 | A1 |
20090106602 | Piszczek et al. | Apr 2009 | A1 |
20090327481 | Rickard et al. | Dec 2009 | A1 |
20100191712 | Wolman et al. | Jul 2010 | A1 |
20100191851 | Raja et al. | Jul 2010 | A1 |
20100250742 | Leighton et al. | Sep 2010 | A1 |
20110022705 | Yellamraju et al. | Jan 2011 | A1 |
20110055390 | Malloy et al. | Mar 2011 | A1 |
Entry |
---|
U.S. Appl. No. 12/834,473, filed Jul. 12, 2010 in the name of Ajay Gulati et al. |