Patent Application
20140143594
In some computer systems, multiple physical disk drives can be grouped and accessed as a single logical volume. The logical volume is often implemented using a RAID (Redundant Array of Independent Disks) technique. RAID is a storage technology that combines multiple disk drive components into a logical unit. Data is distributed across the physical disk drives using a particular configuration or “RAID level.” Different RAID levels can be employed, depending on what level of redundancy (e.g., fault tolerance) and performance is required. When data redundancy is lost in a RAID system due to disk failure(s), the disk array must recover redundancy before additional failures occur to avoid data loss.
Some embodiments of the invention are described with respect to the following figures:
In an example, one or more of the logical volume(s) 108 comprises at least one RAID (Redundant Array of Independent Disks) volume (RAID volume(s) 107). That is, multiple ones of the physical disk drives 102 are combined to form RAID set(s) 107 such that the data is distributed across the selected physical disk drives 102 using a particular scheme or “RAID level.” Examples of RAID levels include RAID-0 (striping data across multiple disk drives), RAID-1 (mirroring data across multiple disk drives), RAID-5 (block-level striping with parity). Other examples of RAID levels include RAID-2, RAID-3, RAID-4, RAID-6, RAID-1MM (double mirror), RAID-50, RAID-60, and the like RAID set(s) are combined to form a logical volume 108. Each of the logical volume(s) 108 can include a fault-tolerant configuration of RAID set(s) (e.g., RAID-1 or RAID-5 set(s)), or a non-fault-tolerant configuration of RAID set(s) (e.g., RAID-0 set(s)). The controller 104 can control rebuild and scrub rates of RAID set(s) in order to meet a specified Annualized Data Loss Event Rate (ADLER) goal with a minimum of disruption to performance. ADLER provides a measure of data loss rate on an annualized basis. A similar metric is the Mean Time To Data Loss (MTTDL). While ADLER is described herein in various examples, it is to be understood that the ADLER can be converted to other known metrics, such as MTTDL.
The memory 202 can store code executable by the processor 204 to provide an operating environment (OE) 208 (e.g., firmware). A user can interact with the OE 208 to configure the controller 104 and the attached physical disk drives 102. For example, a user can interact with the OE 208 to select particular disk drives and organize them into RAID sets of a particular type (e.g., particular RAID level), and RAID sets into logical volumes. Alternatively, an external system (e.g., one of the computing systems 106) can interact with the OE 208 to configure the controller 104 and the attached physical drives 102 automatically without user interaction. The OE 208 can also employ a model 210 to control rebuild and scrub rates of RAID set(s), as described below. The OE 208 can use the model 210 to determine optimal rebuild and scrub rates for any given RAID volume, and implement the rebuild and scrub processes according to the selected rates.
Rebuild and scrub rates for a RAID volume can be fixed at a constant value or subject to limits based on other activity in the array. Such techniques can limit ADLER to a predictable array-specific upper bound. Such techniques, however, do not optimize the tradeoff between minimum ADLER and maximum performance. In an example implementation, the controller 104 applies a mathematical model to the rebuild and scrub rate optimization problem. The model includes various parameters that reflect array algorithms, configuration settings, and disk failure characteristics. Rebuild and scrub rates can then be optimized accordingly.
For the design-controlled variables 302, the parameters can be defined as follows:
For the supplier-controlled variables 304, the parameters can be defined as follows:
For the usage-controlled variables 306, the parameters can be defined as follows:
The optimizer 308 relates the ADLER, rebuild rate, and scrub rate using constant values derived from the design and configuration of the disks and RAID volume based on the parameters described above. The optimizer 308 determines optimal values for rebuild and scrub rates for a given RAID volume.
The following derivation of the model 210 assumes a single type and capacity of disk drive. Only one RAID level is modeled at a time, and full capacity utilization is assumed. These assumptions can be relaxed by computing the model on subsets of the capacity in the array and deriving optimal scrub rate and rebuild rate for those subsets.
Derivation of the model 210 can begin with a generic description of ADLER for example RAID levels. The notation P(<x>) is read as “the probability of event <x>”. The following combinations of probabilities represent paths leading to data loss, where “URE” means unrecoverable read error(s):
The parameters described above can be incorporated into the ADLER statements for the example RAID levels. The parameter notation defined above is annotated with “1”, “5”, “MM”, and “6” for RAID 1, 5, double mirror, and 6, respectively. For example, disk capacity C for a disk in a RAID1 array is denoted C1. Also, an additional probability, Pu, representing the probability of encountering at least one unrecoverable segment error during rebuild, is defined. The value of Pu is discussed below.
RAID1 ADLER (Events/Year per array)˜8760*(N1/MTBF)*{[(G1−1)*C1/(ER1*MTBF)]+Pu1}
RAID5 ADLER (Events/Year per array)˜8760*(N5/MTBF)*{[(G5−1)*C5/(ER5*MTBF)]+Pu5}
RAID1MM ADLER (Events/Year per array)˜8760*(N1MM/MTBF)*[(G1MM−1)*C1MM/(ER1MM*MTBF)]*{[(G1MM−2)*C1MM/(ER1MM*MTBF)]+Pu1MM}
RAID6 ADLER (Events/Year per array)˜8760*(N6/MTBF)*[(G6−1)*C6/(ER6*MTBF)]*{[(G6−2)C6/(ER6*MTBF)]+Pu6}
The factor “8760” in the above equations converts the results from per hour to per year. The following notation is used in the derivation of Pu:
The probability of encountering at least one unrecoverable segment error during rebuild, Pu, can be defined as follows:
Pu=1−(1−Psc)S
Psc=Pw−(Pw−Ps)*In[(1+q)/q], where q>=0, q=rw*h*C/ER
Pw=Pseg=Pbit*8*Sseg
Ps=(1−(1−e−hTS)/hTS)**rw*Pseg
where “In” denotes a natural logarithm operation, “e” is the constant base of the natural logarithm, and S represents the number of segments that need to be read back for the rebuild operation.
The above formula for Pu introduces complexity into the model 210 that can be removed by estimating Pu. For example, a Taylor series expansion can be used and the most significant terms in the result can be interpreted. This yields the following estimation:
Pu˜(K−1)*C*Pseg*Ld*(rw/2)*[(1/ER)+(1/SR)]
The Pu estimation can be recombined with the RAID specific ADLER equations above to yield the following final RAID specific ADLER equations. The notation is further simplified with the following substitutions:
ρ=1/ER
λ=1/MTBF
σ=1/SR
The RAID specific ADLER equations can be defined as:
RAID1 ADLER˜8760*N1*λ*C1*[(G1−1)*ρ1*λ+(K1−1)*Pseg*Ld1*(rw1/2)*(ρ1+σ1)]
RAID5 ADLER˜8760*N5*λ*C5*[(G5−1)*ρ5*λ+(K5−1)*Pseg*Ld5*(rw5/2)*(ρ5+σ5)]
RAID1MM ADLER˜8760*N1MM*λ*C1MM*(G1MM−1)*ρ1MM*λ*C1MM*[(G1MM−2)*ρ1MM*λ+(K1MM−1)*Pseg*Ld1MM*(rw1MM/2)*(ρ1MM+σ1MM)]
Note the significance of the rebuild and scrub rates (ER and SR expressed through ρ and σ) in the calculations. An additional refinement of these equations recognizes correlated failure by replacing every occurrence of “ρ*λ” with “ρ*λ*α”. Correlated failure refers to common cause failure where the occurrence of the first failure may trigger subsequent failures to occur faster in the system. This means that the second, third, and so on failures may occur at a higher rate.
At this point, there is a means of calculating the ADLER that results from a choice of rebuild and scrub rates given the rest of the configuration of a given RAID array or portion thereof. The relationship between rebuild and scrub rates can be solved for a given ADLER. Since there is only one equation with two unknowns, an additional policy must be used to choose actual scrub and rebuild values. In an example, one such policy is to try to minimize rebuild rate so long as the resulting scrub rate is not too high. The threshold representing “too high” can be chosen as the scrub rate that corresponds to a point of diminishing return on reducing rebuild rate. This can be done by taking a first derivative of the rebuild rate with respect to scrub rate and imposing a threshold on the derivative to find the point of diminishing return.
For example, in a RAID1 array, the above ADLER equation solved for ER is:
ER1˜[(8760*N1*λ*C1)*SR1*(G1−1)*λ+(K1−1)*Pseg*Ld1*(rw1/2)]/[(RAID1 ADLER)*SR1−8760*N1*λ*C1*(K1−1)*Pseg*Ld1*(rw1/2)]
The relationship between ER and SR for a fixed ADLER has the form of y=ax/(x+b), a shape illustrated in
This process can be repeated for each RAID mode. One of the implications of this approach is that rebuild rates and/or scrub rates for parts of the system that are more vulnerable will be higher than parts of the system that are less vulnerable. It is also good practice to rebuild more vulnerable parts of the system first when multiple parts are degraded. Finally, the model 210 can be enhanced to allow higher scrub and/or rebuild rates when input/output activity from other sources is low (e.g., below a defined threshold). The model 210 can then be used to insure that average scrub and rebuild rates do not fall below acceptable values when other input/output activity is more intense.
In the foregoing description, numerous details are set forth to provide an understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these details. While the invention has been disclosed with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover such modifications and variations as fall within the true spirit and scope of the invention.
