In order to protect against potential loss of data in a storage system, it is often advantageous to implement a replication scheme. Current replication schemes are only able to sustain a limited amount of error before data within the storage system is unable to be read.
Specific embodiments of the technology will now be described in detail with reference to the accompanying figures. In the following detailed description of embodiments of the technology, numerous specific details are set forth in order to provide a more thorough understanding of the technology. However, it will be apparent to one of ordinary skill in the art that the technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description
In the following description of
In general, embodiments of the technology relate to storing data in a storage array using a RAID scheme. More specifically, the RAID scheme uses information about the failure rates of the storage devices in order to select physical locations on the individual storage devices in which to store data.
For purposes of this technology, the term “RAID” as used herein refers to “Redundant Array of Independent Disks.” While “RAID” refers to any array of independent disks, embodiments of the technology may be implemented using any type of persistent storage device.
In one embodiment of the technology, a client (100A, 100M) is any system or process executing on a system that includes functionality to issue a read request or a write request to the RAID controller (104). In one embodiment of the technology, the clients (100A, 100M) may each include a processor (not shown), memory (not shown), and persistent storage (not shown). In one embodiment of the technology, the RAID controller (104) is configured to implement a RAID scheme, which includes writing data to the storage array (108) in a manner consistent with embodiments described in
In one embodiment of the technology, the RAID controller (104) is operatively connected to memory (106). The memory (106) may be any volatile memory including, but not limited to, Dynamic Random-Access Memory (DRAM), Synchronous DRAM, SDR SDRAM, and DDR SDRAM. In one embodiment of the technology, the memory (106) is configured to temporarily store various data (including parity data) prior to such data being stored in the storage array.
In one embodiment of the technology, the FPGA (102) (if present) includes functionality to calculate parity values (e.g., P parity value, Q parity value) for purposes of storing data in the storage array (108) and/or functionality to perform various calculations necessary to recover corrupted data stored using the RAID scheme. The RAID controller (104) may use the FPGA (102) to offload the processing of various data in accordance with one or more embodiments of the technology.
In one embodiment of the technology, the storage array (108) includes one or more storage devices (SDDs). Each storage device includes persistent storage media. Examples of storage devices include, but are not limited to, magnetic memory devices, optical memory devices, phase change memory devices, solid state memory devices, tape drives, any other suitable type of persistent memory device, or any combination thereof.
Those skilled in the art will appreciate that while
Those skilled in the art will appreciate that the technology is not limited to the configuration shown in
In one or more embodiments of the technology, each physical location (204) in the storage array is associated with a logical address (200) and a physical address (202). Further, for each physical address (202), the RAID controller may track the following information: (i) status (206), which indicates whether the block is a free block or an occupied block; (ii) stripe (208) membership, i.e., whether the physical location is part of a stripe (see e.g.,
In one embodiment of the technology, the individual failure rate (210) may be calculated by the RAID controller. Alternatively, the RAID controller may obtain the individual failure rates (210) from an external source. In another embodiment of the technology, the individual failure rate (210) of a given physical location may be a function of the number of program/erase (P/E) cycles that have been performed on the particular physical location. In such cases, the RAID controller may track the P/E cycles (e.g., tracked as P/E cycle value) for each of the physical locations in the storage array and then calculate the individual failure rate (210) for each of the physical locations using this information.
The P/E cycle value may represent: (i) the number of P/E cycles that have been performed on the physical location defined by the physical address or (ii) a P/E cycle range (e.g., 5,000-9,999 P/E cycles), where the number of P/E cycles that have been performed on the physical location (204) defined by the physical address (202) is within the P/E cycle range. In one embodiment of the technology, a P/E cycle is the writing of data to one or more pages in an erase block (i.e., the smallest addressable unit for erase operations, typically, a set of multiple pages) and the erasure of that block, in either order. In one embodiment of the technology, the physical location (204) corresponds to page. The P/E cycle values may be stored per physical location per set of physical locations, or and/or at any other level of granularity. In one embodiment of the technology, the P/E cycle values may be stored on a per page basis, a per block basis, on a per set of blocks basis, and/or at any other level of granularity. The RAID controller includes functionality to update, as appropriate, the P/E cycle values when data is written to (and/or erased from) physical locations in the storage array.
In step 306, the RAID controller determines a failure rate threshold based upon the individual failure rate data obtained in step 304. In one embodiment of the technology, the failure rate threshold is determined such that a sufficient number of stripes with an aggregate failure rate that is less than or equal to the failure rate threshold may be stored in the storage array. For example, if the failure rate threshold is set lower than the average individual failure rate for physical locations in the storage array, then only a limited number of physical locations may be used to generate stripes that have an aggregate failure rate that is less than or equal to the failure rate threshold. This may result in inefficient usage of the storage array. Alternatively, if the failure rate threshold is set higher than the average individual failure rate for physical locations in the storage array, then a large number of stripes that have an aggregate failure rate that is less than or equal to the failure rate threshold may be stored in the storage array. While this may result in a more efficient usage of the storage array, this may also result in stripes in the storage array that have a wide range of aggregate failure rate thresholds.
In one embodiment of the technology, the failure rate threshold is determined experimentally using the information about in steps 302 and 304. Alternatively, a model for determining failure rate thresholds for storage arrays may be implemented by the RAID controller, where the model takes the information obtained in steps 302 and 304 as input and outputs a failure rate threshold.
In one embodiment of the technology, the RAID controller may take an average of the individual failure rates (obtained in step 304) and determine an acceptable deviation from the average. The acceptable deviation may be, for example, 5%40% from the average. Those skilled in the art will appreciate that the acceptable deviation may cover a varying range based upon the load of the system, criticality of the data, or other profile information conveyed to the RAID controller by an administrator or other third party. The RAID controller may use the aforementioned acceptable deviation to filter out outliers in set of individual failure rates obtained in step 304. The failure rate threshold may then be determined using the resulting set of individual failure rates.
The process depicted in
In step 406, the RAID controller selects a set of free physical locations to be part of a new RAID stripe. The set of free physical locations may be determined by obtaining the set of free physical locations in the storage array and then selecting a number of physical locations corresponding stripe width. For example, if the stripe width is five, then five physical locations are selected. Further, the selection of the physical locations for the stripe may also take into account in the independent fault domains (IFDs) with which the physical locations are associated. In one or more embodiments of the technology, an IFD corresponds to a failure mode which results in the data at a given location being inaccessible. Each IFD corresponds to an independent mode of failure in the storage array. For example, if the data is stored in a storage array that includes SSD that include NAND flash, then the IFDs may be (i) SSD and (ii) NAND die within the SSD. Accordingly, the physical locations for the stripe, in one embodiment of the technology, are distributed across at least one IFD. For example, if the IFD is disks, then the physical locations for the stripe may be selected such that each physical location is on a separate disk.
In step 408, the RAID controller obtains the individual failure rate of each physical location in the new RAID stripe determined in step 406 and calculates the aggregate failure rate of the new RAID stripe based upon the individual failure rates. The aggregate failure rate may be calculated according to one or more mathematical formulas. For example, the aggregate failure rate may be the average of the individual failure rates.
In step 410, the RAID controller determines whether or not the calculated aggregate failure rate exceeds the failure rate threshold that determined in step 306. If the calculated aggregate failure rate is below the failure rate threshold, then the method proceeds to step 412; otherwise the method proceeds to step 418.
In one embodiment of the technology, in step 410, RAID controller may determine whether the aggregate failure rate of the stripe is greater than or equal to a minimum failure rate threshold and less than or equal to a maximum failure rate threshold. In this scenario, the aggregate failure rate threshold is within a range. The size of the range may vary based on the implementation of the technology.
Continuing with the discussion of
In step 416, the RAID controller notifies the client of a successful write operation.
Returning to step 410, if the calculated aggregate failure rate exceeds the failure rate threshold in step 410, the RAID controller proceeds to step 418. In step 418, the RAID controller determines whether there are other combinations of free physical locations that may be selected for the new RAID stripe. If there are other combinations available, then the process proceeds to step 406. If there are no other combinations available, then the RAID controller notifies the client in step 416 that the write operation has failed.
In one or more embodiments of the technology the failure to select a RAID stripe of locations that has an aggregate failure rate within the failure rate threshold may trigger the RAID controller to calculate a new failure rate threshold, i.e. proceed to step 302 of
In one or more embodiments of the technology, the RAID controller may pre-determine one or more RAID stripes that have an aggregate failure rate within the failure rate threshold. In other words, in one or more embodiments, the RAID controller may proactively perform, e.g., steps 406-410. In one or more embodiments, the RAID controller may randomly select free physical locations to compose one or more of the pre-determined RAID stripes that have an aggregate failure rate within the failure rate threshold. In one or more alternative embodiments, the selection of free physical locations for inclusion in a particular RAID stripe may be random, according to a particular pattern (e.g. linear, checkerboard, or other patterns), or according to some other selection scheme.
In one or more embodiments, the RAID controller may store a list of pre-determined RAID stripe(s) that have an aggregate failure rate within the failure rate threshold. Further, in one or more embodiments, the RAID controller may generate the list of pre-determined RAID stripe(s) during, e.g., the failure rate threshold determination as described in
In one or more embodiments, the RAID controller may use a list of pre-determined RAID stripes that are within the failure rate threshold. In other words, in one or more embodiments, the RAID controller may, in response to receiving a write request in step 402, select one of the pre-determined RAID stripes and then proceed to step 412.
The following section describes various examples in accordance with one or more embodiments of the technology. The examples are not intended to limit the scope of the technology.
Referring to
Consider a scenario in which all the physical locations are initially free and a write request is subsequently received. In response to receiving the write request, the RAID controller selects the first linear set of free physical locations A-E. See
Once the RAID controller selects the free physical locations A-E to compose the RAID stripe, the RAID controller calculates the aggregate failure rate of the RAID stripe by obtaining the individual failure rates of each physical location as discussed above. Assuming that the aggregate failure rate is less than the failure rate threshold, the RAID controller then calculates the parity and writes the parity and data to the physical locations A-E. However, if the aggregate failure rate is not less than the failure rate threshold, the RAID controller cannot write to the selected RAID stripe of free physical locations and must select a new RAID stripe.
In the instant example assume that the aggregate failure rate of free physical locations A-E is greater than a failure threshold rate. Accordingly, referring to
In the instant example assume that the aggregate failure rate of free physical locations A-E is greater than a failure threshold rate. Accordingly, referring to
Those skilled in the art will appreciate that while the technology has been described with respect to a single dimension RAID scheme, embodiments of the technology may be extended to any multi-dimensional RAID scheme. For example, embodiments of the technology may be extended to the multi-dimensional RAID scheme described in U.S. Pat. No. 8,316,260 (which is hereby incorporated by reference).
Advantageously, systems and methods implementing one or more embodiments of the technology may exhibit failure rate improvements on a per-stripe basis by a factor substantially between 4 and 5.
One or more embodiments of the technology may be implemented using instructions executed by one or more processors in the system. Further, such instructions may correspond to computer readable instructions that are stored on one or more non-transitory computer readable mediums.
While the technology has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the technology as disclosed herein. Accordingly, the scope of the technology should be limited only by the attached claims.
Number | Name | Date | Kind |
---|---|---|---|
6351838 | Amelia | Feb 2002 | B1 |
7398418 | Soran et al. | Jul 2008 | B2 |
7406621 | Lubbers et al. | Jul 2008 | B2 |
7543100 | Singhal et al. | Jun 2009 | B2 |
7752389 | Fan | Jul 2010 | B1 |
7934120 | Zohar et al. | Apr 2011 | B2 |
8078906 | Yochai et al. | Dec 2011 | B2 |
8145840 | Koul et al. | Mar 2012 | B2 |
8200887 | Bennett | Jun 2012 | B2 |
8489979 | Rub | Jul 2013 | B2 |
8782505 | Rub | Jul 2014 | B2 |
9529674 | Pereira | Dec 2016 | B2 |
20020161972 | Talagala et al. | Oct 2002 | A1 |
20040153961 | Park et al. | Aug 2004 | A1 |
20040225926 | Scales et al. | Nov 2004 | A1 |
20050166083 | Frey et al. | Jul 2005 | A1 |
20050223156 | Lubbers et al. | Oct 2005 | A1 |
20060085594 | Roberson et al. | Apr 2006 | A1 |
20080109602 | Ananthamurthy et al. | May 2008 | A1 |
20080168225 | O'Connor | Jul 2008 | A1 |
20090187786 | Jones et al. | Jul 2009 | A1 |
20100199125 | Reche | Aug 2010 | A1 |
20120079318 | Colgrove et al. | Mar 2012 | A1 |
20120089778 | Au et al. | Apr 2012 | A1 |
20120166712 | Lary | Jun 2012 | A1 |
20160148708 | Tuers | May 2016 | A1 |
Number | Date | Country |
---|---|---|
1577774 | Sep 2005 | EP |
2004-326759 | Nov 2004 | JP |
2010-508604 | Mar 2010 | JP |
2008054760 | May 2008 | WO |
Entry |
---|
Anvin H., “The mathematics of RAID-6”, retrieved from the internet at http://www.cs.utk.edu/″plank/plank/papers/CS-96-332.html, Jan. 20, 2004 (9 pages). |
Kazmi, A., “PCI Express™ Basics & Applications in Communication Systems,” PCI-SIG Developers Conference, 2004 (50 pages). |
Percival, D., “Multicast Over PCT Express®,” PCI-SIG Developer's Conference Europe, 2009 (33 pages). |
Huffman, A., NVM Express Revision 1.0b, Jul. 12, 2011 (126 pages). |
Regula, J., Using Non-transparent Bridging in PCI Express Systems, Jun. 1, 2004 (31 pages). |
International Search Report issued in PCT/US2013/033224 dated Jul. 31, 2013 (3 pages). |
Written Opinion of the International Searching Authority issued in PCT/US2013/033224 dated Jul. 31, 2013 (10 pages). |
International Preliminary Report on Patentablility in corresponding International Application No. PCT/US2013/033224 dated Oct. 2, 2014 (2 pages). |
Written Opinion of the International Searching Authority issued in PCT/US2013/033224 dated Oct. 2, 2014 (10 pages). |
Minoru Uehara, Orthogonal Raid with Multiple Parties in Virtual Large-Scale Disks, IPSJ SIG Technical Report , vol. 2011-DPS-149 No. 4, Nov. 24, 2011 [CD-ROM], IPSJ, Nov. 24, 2011, pp. 1-8 (10 pages). |
Office Action in corresponding Japanese application 2015-501902 dated Oct. 6, 2015 (6 pages). |