Patent Application
20060294300
The claimed invention relates generally to the field of data storage and more particularly, but not by way of limitation, to an apparatus and method for atomic data transactions with a nonvolatile cache memory.
Computer networking began proliferating when the data transfer rates of industry standard architectures could not keep pace with the data access rate of the 80386 processor made by Intel Corporation. Local area networks (LANs) evolved to storage area networks (SANs) by consolidating the data storage capacity in the network. Users have realized significant benefits by the consolidation of equipment and the associated data handled by the equipment in SANs, such as the capability of handling an order of magnitude more storage than would otherwise be possible with direct attached storage, and doing so at manageable costs.
More recently the movement has been toward a network-centric approach to controlling the data storage subsystems. That is, in the same way that the storage was consolidated, so too are the systems that control the functionality of the storage being offloaded from the servers and into the network itself. Host-based software, for example, can delegate maintenance and management tasks to intelligent switches or to a specialized network storage services platform. Appliance-based solutions eliminate the need for the software running in the hosts, and operate within computers placed as a node in the enterprise. Intelligent data storage subsystems self-deterministically allocate, manage, and protect its respective data storage capacity and present that capacity as a virtual storage space to the network to accommodate global storage requirements. This virtual storage space is able to be provisioned into multiple storage volumes. A distributed computing environment uses these intelligent storage devices for global provisioning as well as for global sparing in the event of failures. In any event, the intelligent network solutions can centralize such things as storage allocation routines, backup routines, and fault tolerance schemes independently of the hosts.
As file transactions are executed in such a network-centric system, storage controllers must direct the data to/from the appropriate locations on the physical storage media. This leads to complex caching and mapping constructs to make certain the right data is communicated to the right place. The structure of metadata can significantly impact storage system performance and reliability, particularly across different components in the storage system. One existing implementation employs a scatter-gather list. However, what is needed is more sophisticated mapping schemes associating the logical storage volume to the physical striped blocks of data stored in the data storage medium, and atomic write metadata utilizing the benefits of such improved mapping schemes. It is to this solution that embodiments of the present invention are directed.
Embodiments of the present invention are generally directed to an atomic write descriptor associated with stripe buffer list metadata.
In some embodiments an atomic write data structure is provided comprising a stripe buffer list descriptor defining state information of a stripe buffer list entry associating a virtual stripe of data storage space with cache data.
In some embodiments a method is provided comprising constructing metadata associating a virtual stripe of data storage space with cache data; modifying the metadata to reflect state changes related to a pending data transaction between a host or media update request and the cache; writing the modified metadata to an atomic write descriptor; and recalling the atomic write descriptor for transacting a selected atomic block size in an event of an interruption while executing the data transaction.
In some embodiments a distributed storage system is provided comprising a bulk data storage space in communication with a cache, and means for atomically transacting data between the storage space and the cache.
These and various other features and advantages which characterize the claimed invention will become apparent upon reading the following detailed description and upon reviewing the associated drawings.
It is further contemplated that the A host computer 102 and the A intelligent data storage subsystem 108 can be physically located at a first site, the B host computer 102 and B intelligent data storage subsystem 108 can be physically located at a second site, and the C host computer 102 can be yet at a third site, although such is merely illustrative and not limiting. All entities on the distributed computer system are connected over some type of computer network.
The policy processor 144 can be programmed to execute desired operations via the ISP 130. For example, the policy processor 144 can communicate with the list managers 162, 164, that is send and receive messages, via the CPS 166. Responses to the policy processor 144 can serve as interrupts signaling the reading of memory 148 registers.
The policy processor 144 can be programmed by a storage controller vendor to provide value-added functionality, such as but not limited to the managed reliability and recovery schemes discussed above. That is, the policy processor 144 can execute custom logic to perform sophisticated operations via the ISP 130. The policy processor 144 can send messages to any list manager and receive responses back from any list manager, all via the CPS 166. A message response issued by the ISP 130 can result, for example, in a PCI write to the policy processor 144. In this manner, the policy processor 144 can also access memory 148 and trigger actions within the ISP 144 using CPS 166 messages and the list managers.
Accordingly, a storage controller vendor can create custom storage service applications for execution on the policy processor 144. The storage service applications can access the functionality of the ISP 130, such as through ISP 130 register access and DMA over a PCI bus, and such as by opening and accessing memory windows via individual PCI read or write operations with the attendant overhead. Some implementations may also support a faster access than individual PCI access. In some embodiments, for example, the policy processor 144 can be implemented with a PowerPC processor that accesses the ISP 130 functionality through cache writes across the PCI bus, although other processors and access methods may be employed.
In some embodiments the ISP 130 is coupled through a host fibre channel interface to a host (e.g., an application server), and through one or more storage device FC interfaces to the data storage capacity 109. The ISP 302 communicates storage transactions (e.g. SCSI exchanges) between the host and the storage capacity 109. It should be understood that other network connectivity may also exist between the host and the storage capacity 109.
For example, a data storage transaction (e.g. a SCSI write request) can be received over the host fibre channel interface. The host fibre channel interface processes the pending transaction from the request frame. The request data includes a fibre channel Origin eXchange IDentifier (OXID), which identifies a given fibre channel exchange. An exchange represents a sequence of one or more non-concurrent related requests and responses passing between a pair of fibre channel ports, encapsulating a conversation, such as a SCSI task. Exchanges may be bidirectional and may be short or long lived. The OXID is extracted from the request frame and used to process the request data.
A counterpart to the OXID is the Responder eXchange IDentifier (RXID), which identifies the context to be used when responding to the request. For example, a data frame from one of the storage devices through the storage device interface is processed by the ISP 130. The RXID is extracted from the data frame and used to locate the context needed to route the response data.
Collectively, OXIDs and RXIDs are referred to as exchange identifiers, although other exchange identifiers may also be employed. In some embodiments, the exchange identifier is used to locate a context for a given transaction. The context provides continuity among individual frames in a single transaction. For example, when a transaction is initiated for a SCSI write command, a context is established for the transaction. In addition, the receiving controller allocates or locates metadata, described below, for the associated stripe and populates the appropriate portion of the metadata using available data buffers from buffer memory. For subsequent stages in the transaction, the storage system can retrieve the context from memory using the exchange identifier and continue processing the exchange using the context data. When the exchange is complete, the metadata structures and data buffers for the transaction may be maintained for some time to act as a write-back cache-as long as the cached data remains valid, the ISP 130 can service a read request from its cache, rather than retrieving the data from physical storage. The validity of cached data is managed using known caching techniques.
Data striping allocates portions of multiple data storage discs within the storage capacity 109, so that multiple stripes can be accessed simultaneously. For example, rather than reading a large file from a single disc, it is much faster to chop the file up into pieces, store each of the pieces in different stripes on different discs, and then use all the discs to read back the file in parallel. For example,
The data stripe 180 is the basic building block used to virtualize a range of blocks representative of a given set of virtual block addresses (VBAs) or parity for some collection of data stripes. The VBAs are mapped to logical block addresses (LBAs), which specify stripe locations on the physical storage media. Storage requests and responses are communicated relative to VBAs, and a media-side component of the ISP 130 maps the VBAs to and from LBAs during physical media access. A VBA can also include an offset into a given stripe, so as to allow for partial population of the stripe.
The underlying metadata structure is a stripe buffer list (SBL). The ISPs 130 store the SBLs, such as in memory 168, and implement the SBL configuration for describing data in the storage system cache that is associated with a particular data stripe. In some embodiments the SBLs are allocated to distinct pools (e.g. a primary pool and a secondary pool) to support mirroring. The various stripe buffer lists are structured as an array, so a context structure can specify an index (SBL index) of the appropriate SBL. Accordingly, each SBL references a plurality of data buffers in which data read from or destined to a particular stripe is cached. Furthermore, each SBL is aligned with a virtual disk stripe, such that the starting VBA of the SBL is evenly divisible by the disk stripe size and the maximum amount of buffer space referenced by the SBL corresponds to the disk stripe size. In some embodiments, each data buffer is sized to 2112 bytes and is capable of holding 4 sectors of file data, although other sizes are also contemplated.
The context structure can also specify a mirror flag that indicates to which pool of data buffers and metadata the data transaction is designated. For example, if the mirror flag indicates ‘0,’ then the ISP 130 caches the data using an SBL from its primary pool having the SBL index given by the context. If the mirror flag indicates ‘1,’ then an SBL from its secondary pool is used. As such, the SBL index, and the mirror flag are used to access the appropriate SBL and other associated metadata.
The context structure can also specify the starting sector offset within the disk stripe associated with the data transaction. The offset, therefore, also specifies an offset into the SBL. In this manner, the ISP 130 is able to determine where in the SBL to begin processing (e.g., writing data to certain data buffers referenced by the SBL or reading data from certain data buffers referenced by the SBL).
Other metadata structures, discussed in more detail below, include a stripe buffer list descriptor (SBLD), a stripe buffer list present bits (SPB), and a stripe buffer list dirty bits (SDB), govern how the data buffers are associated with the storage media and how the data buffers are handled by the caching mechanisms. An SBLD table includes entries that associate individual SBLs with ranges of VBAs (i.e., with a virtual disk stripe) on a given logical unit. A given SBLD table entry is referenced by the same SBL index used to identify the appropriate SBL for a given data transaction. In some embodiments there are two SBLD tables stored in memory, one for each pool.
An exemplary format for an eight-byte SBLD table entry is shown below: Byte Location Description Address Bit Locations
The state field in the exemplary SBLD table entry is defined as:
For each SBL entry, there is also a corresponding entry in an associated SPB table, which is indexable by the SBL index. Each SPB table entry consists of a selected number of bits for indicating whether the sector data for a corresponding 10 VBA is valid or about to be valid. For the present example, the particular bit in a given field can be referenced using a right shift of the value 0x8000 based on the lower order 4 bits of the VBA for the given sector. A bit mask for a range with a given starting VBA and a length L within a given field is computed as:
MASK=(0xFFFF<<(16-L))>>(VBA & 15)
The particular field for a given VBA is computed as:
INDEX=(VBA>>4) & 15
For each SBL entry, there is also a corresponding entry in an associated SDB table, which is likewise indexable by the SBL index. Each SDB table entry consists of a selected number of bits for indicating whether the sector data for a corresponding VBA is dirty (i.e. write back cached) and, therefore, likely needs to be flushed out of the cache in the buffer memory. For the present example, the particular bit in a given field is referenced using a right shift of the value 0x8000 based on the lower order 4 bits of the VBA for the given sector. A bit mask for a range with a given starting VBA and a length L within a given field is computed as:
MASK=(0xFFFF<<(16-L))>>(VBA & 15)
The particular field for a given VBA is computed as:
INDEX=(VBA>>4) & 15
The context data is communicated from the ISP 130 through the CPS 166 to an address generator, which computes the appropriate address for the appropriate SBL, based on the SBL index and pool. In some embodiments, the context is communicated to the address generated in a message packet that includes, without limitation, the SBL index, an SBL offset (which identifies the offset into the SBL from which the first valid buffer index is read), a sector offset (which indicates the starting sector within the first valid data buffer), and a sector count (which represents the number of sectors to be transferred from the ISP 130 to the buffer memory). For example, with 32 sectors per data buffer, the SBL offset corresponds to VBA[10:5] and the starting sector index corresponds to VBA[4:0]. With 4 sectors per data buffer, the SBL offset corresponds to VBA[7:2] and the starting sector index corresponds to VBA[1:0].
Responsive to receipt of the context, the address generator decodes the message packet and accesses the identified SBL, according to the SBL offset, to obtain the specified data buffer index. Then, using the data buffer index and the sector offset, the address generator then computes a data buffer memory address. The sector offset and the number of sectors in a buffer are used to determine the remaining number of sectors in the data buffer. The transfer size associated with the data buffer memory address is the lesser of the remaining data buffer space and the total transfer size (stored in the message packet). The data buffer address and the transfer size are written to an address, such as FIFO, of a data buffer manager.
As described above, each SBL structure 192, 194, 196 is aligned with a virtual disk stripe and one or more physical disk stripes on target storage media. When the storage controller receives data (whether from a host or from a storage device), the data is stored by the receiving controller (i.e., FC interface) in frame buffers and organized for processing, such as a FIFO. The FCC then sends a message to the appropriate address generator (AG) logic through the CPS. The message provides the information necessary to identify which SBL 191 to use and where to start within the SBL 191. The data needed to construct the message is located using the exchange ID extracted from the frame header. The data in the frame is moved to memory. In a similar fashion, when the storage controller transmits data (whether to a host or to a storage device), the transmitting controller extracts the context from the request data provided by the policy processor 144.
An SPB table 200 and an SDB table 202 allow for cache management of the data buffers. The SPB table 200 includes a bit for each sector of each SBL 191 to indicate whether the cached data for a given sector is or is about to be valid (i.e., has current data for the corresponding sector). It also indicates that the buffer index is valid. In a subsequent transaction requesting a read operation to the same sector, if the SPB table 200 indicates that the data is valid, then the request can be serviced from the buffer memory instead of requiring a round trip to the storage medium. The SDB table 202 includes a bit for each sector of the SBL 191 to indicate whether the cached data for a given sector is dirty (i.e., needs to be written from the data buffer). For example, if the storage controller loses power in the middle of an exchange, the SDB table 202 can be used to indicate that the data in the cache needs to be flushed to the appropriate target. Therefore, when the storage controller reboots, data integrity is maintained in the presence of a catastrophic event.
The SPB table 200 and SDB table 202 combined represent the logic shown below. The “present and dirty, but CRC invalid” state represents a condition where the data in the data buffers contain modified data (e.g. the CRCs do not match what is expected). In such a case, the CRC needs to be recomputed and updated.
The state information 218 contains a data integrity field (DIF) 220 that can be appended to the end of the user data for each sector as the data is being read from a host (e.g. SCSI target write). An invalid DIF is the exception to the general rule that the associated metadata is in a normal write back state, and forms part of a valid crash recovery record. The state information 218 can further contain parity placement 222 and presence 224 data, a flush before copy flag 226, an in use flag 228, and a dirty block count 230.
The method continues in block 248 where the transaction is executed idempotently. In decision block 250 it is determined whether the transaction has been interrupted. If the determination of block 250 is no, then the method ends. If the determination of block 250 is yes, however, then the AWD is recalled in block 254 and used to atomically complete the transaction.
The Miss case is the simplest and basically amounts to calculating new dirty counts for the SBLDs and setting the dirty bits in the SDBs. There are no previous dirty buffers to deal with and no SBLfroms to clean up. The SBreal for this case are found in the SBLreal index 268.
The Dirty case contemplates the SBLfrom because there was some previous dirty data within the range of blocks for the transaction. The SBLDs and SDBs for these SBLs will not be updated at any time during the atomic writing script. This is because the SBLfroms are used exclusively to hold the buffers with the new dirty data to be written. They will be returned to the free pool by the partner after the script completes. It is, however, necessary to clear the present bits in the associated SPBs that were set as part of allocation. Note that the partner may use the same SBLfrom for multiple simultaneous atomic writing operations on the same data stripe because the set of buffers must be, as a consequence of the algorithm, exclusive to the particular operation.
When there is previous dirty data, the essential part of the atomic writing method is to replace the old set of buffers with an entire new set of buffers. Because of this, the operation is preferably done on complete buffers. In order to achieve this condition the partner will need to take out an extended lock to block access to any sectors in the first and last buffers that are outside the range of blocks for the write (for which new data is being supplied by the host). Furthermore, these outside sectors must be copied if dirty, or marked as not present if currently marked present, or simply copied if present, whether or not they are dirty. To simplify the logic, any outside sectors that are present can be copied. Of course, there may not be any outside sectors if the transfer is properly aligned to buffer boundaries.
The RMW case is similar to the Miss case except that one or two SBLs are involved for maintaining row and possibly diagonal parity data.
For the branch denoted “A1” (block 720=yes), new parity data 224 is set in block 292, new column parity 222 is set in block 294, and the AWD 260 is then created in block 296. For the branch denoted “A2” (block 280=no and block 282 =no), the AWD 260 can be created directly. Finally, for the branch denoted “A3” (branch 280 =no and branch 282=yes), the SBLreal buffers are freed in block 298, the SPB for SBLfrom is cleared in block 300, and then the AWD 260 is created in block 296. It will be noted that the operations performed in blocks 298 and 300 can only precede the operation in block 296 because the freed SBLreal buffers wind up as dirty if the process is interrupted before the AWD is written.
The condition (Miss, Dirty, or RMW) is set in block 302, and the AWD 260 is written to memory in block 304. The pending transaction is then executed idempotently in block 306. Decision block 308 determines whether the execution is interrupted. If the determination of block 308 is no, control then branches to block 310 (
If it is determined in block 326 that the condition is not Dirty, then control branches to block 310 where the condition is changed to idle and the atomic writing method is ended. Contrarily, if the condition is Dirty, then the buffer indices of SBLfrom are copied to SBLreal and the condition is then set to idle in block 310, the atomic writing method being ultimately ended.
Summarizing, an AWD (such as 260) structure is provided comprising an SBLD (such as 199) defining state information (such as 218) of an SBL (such as 191) entry associating a virtual stripe of data storage space with cache data.
In some embodiments the AWD comprises a block indicator (such as 264) defining a number of blocks in a pending data transaction between the storage space and the cache. In some embodiments the AWD comprises an offset indicator (such as 266) defining a starting address of the number of blocks in the data buffer. In some embodiments the AWD comprises a first index (such as 268) to a first SBL entry associated with a state of the respective stripe after the data transaction. AWD comprises a second index (such as 270) to a second SBL entry associated with a state of the respective stripe before the data transaction. In some embodiments the AWD comprises a condition indicator (such as 272) defining a characteristic type of pending data transaction, such as a miss sequence, a previous dirty sequence, and a random array of independent drives update parity sequence. The condition indicator must be written last. The single store operation makes the entire sequence “atomic.”
In some embodiments a method for atomic writing is provided, comprising constructing metadata associating a virtual stripe of data storage space with cache data (such as 242); modifying the metadata to reflect state changes related to a pending data transaction between the data storage space and the cache (such as 244); writing the modified metadata to an atomic write descriptor (such as 246); and recalling the atomic write descriptor for transacting a selected atomic block size in an event of an interruption while executing the data transaction (such as 254).
The method writing step can be characterized by writing the block indicator defining the number of data blocks in the transaction, writing the offset indicator defining a starting address of the number of data blocks in the data buffer, defining a condition indicator relating to a characteristic type of data transaction, updating state information of parity data in a redundant array of independent drives, updating state information of dirty data previously existing in the cache, updating a bit map relating locations of dirty data within the data buffer sectors, and updating a bit map relating locations of present data within the data buffer sectors. The constructing step can be characterized by defining a SBL aligning the virtual stripes of the data storage space to the cache data, and wherein the writing step comprises defining a first index to a first stripe buffer list entry associated with a state of the respective stripe after the data transaction, and defining a second index to a second stripe buffer list entry associated with a state of the respective stripe before the data transaction.
In some embodiments a distributed storage system is provided comprising a bulk data storage space in communication with a cache, and means for atomically transacting data between the storage space and the cache. The means for atomically transacting can be characterized by storing an atomic write descriptor relating virtual stripes of the storage space to cache data. The means for atomically transacting can be characterized by the atomic write descriptor indexing stored information defining states of the storage space both before and after a pending data transaction.
It is to be understood that even though numerous characteristics and advantages of various embodiments of the present invention have been set forth in the foregoing description, together with details of the structure and function of various embodiments of the invention, this detailed description is illustrative only, and changes may be made in detail, especially in matters of structure and arrangements of parts within the principles of the present invention to the full extent indicated by the broad general meaning of the terms in which the appended claims are expressed. For example, the particular elements may vary depending on the number and size of data storage devices making up an MDA, without departing from the spirit and scope of the present invention.
In addition, although the embodiments described herein are directed to a data storage system, it will be appreciated by those skilled in the art that the claimed subject matter is not so limited and various other processing systems can utilize the embodiments of the present invention without departing from the spirit and scope of the claimed invention.
