Patent Grant
7664889
The present invention relates to computer systems; more particularly, the present invention relates to computer system interaction with storage systems.
Serial attached storage protocols, such as serial ATA (SATA) and serial Small Computer System Interface (SCSI) (SAS) are becoming more prevalent for connecting storage devices to a computer system. In computer systems implementing such serial storage devices, one storage device in the system may communicate with others. For example, a device requesting data (referred to as the initiator device) may receive data from a target device.
A storage device typically includes a direct memory access (DMA) engine to perform data transfer operations to assist in transferring input/output (I/O) data from (to) a host memory and to (from) a local memory. The DMA engine usually includes a Data Integrity Engine that performs desired data integrity checks (e.g., Cyclic Redundancy Checking (CRC), Application Tag check and/or Reference Tag check etc.). The Data Integrity Engine operates on data blocks which may or may not have data integrity field (DIF). A data block with no DIF typically includes a 512 B (or 520 B, 2 KB, 4 KB etc.). User Data Field and a data block with DIF is a data block appended with an 8 B Data Integrity Field (DIF), also referred to as Block Guard (BG).
When BG is enabled and either an Insert or Strip function is selected, the number of bytes transferred on the local interface of the DMA will not equal the number of bytes transferred on the host interface. Consequently, a programmer of the DMA engine accounts for the expansion or compression of the data stream across descriptors when the Insert or Strip functions are selected. Typically, the programming of the DMA engine (e.g., generating the descriptors by keeping track of the exact amount of user data versus the data integrity data that is being transferred across from the source to the destination) is done in firmware. Programming the DMA engine via firmware is a burden on the local processor, which results in less optimal overall performance. Also, the processor cannot keep up with the ever-increasing link rates, thus affecting the overall system I/O performance.
The invention is illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements, and in which:
A hardware automated mechanism for generating DMA descriptors is described. In the following detailed description of the present invention numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.
Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
In a further embodiment, a chipset 107 is also coupled to interface 105. Chipset 107 includes a memory control hub (MCH) 110. MCH 110 may include a memory controller 112 that is coupled to a main system memory 115. Main system memory 115 stores data and sequences of instructions that are executed by CPU 102 or any other device included in system 100. In one embodiment, main system memory 115 includes dynamic random access memory (DRAM); however, main system memory 115 may be implemented using other memory types. Additional devices may also be coupled to interface 105, such as multiple CPUs and/or multiple system memories.
MCH 110 is coupled to an input/output control hub (ICH) 140 via a hub interface. ICH 140 provides an interface to input/output (I/O) devices within computer system 100. ICH 140 may support standard I/O operations on I/O busses such as peripheral component interconnect (PCI), accelerated graphics port (AGP), universal serial bus (USB), low pin count (LPC) bus, or any other kind of I/O bus (not shown).
According to one embodiment, ICH 140 includes a host bus adapter (HBA) 144. HBA 144 serves as a controller implemented to control access to one or more storage devices 150. In one embodiment, storage device 150 is a serial SCSI (SSP) drive. However in other embodiments, hard disk drive 150 may be implemented as other serial drives. Thus, HBA 144 is capable of controlling different device types.
According to one embodiment, HBA 144 includes a storage controller. A storage controller includes one or more storage links with corresponding transport layers (TL's) that process input/output (I/O) control and data frames both on the transmission (Tx) and receiver (Rx) sides of HBA 144.
Referring to
DM 260 generates descriptors and keeps track of their execution based on the requests made by either the TxTL or the RxTL. Storage controller 200 also includes an I/O context cache memory 240 and an I/O context cache controller 250. DMA engine 230 works on several DMA work queues that temporarily store DMA transactions that are to be performed, usually of varying priorities. The data being moved is initiated by setting up work entries in a DMA engine 230 work queue.
In operation the RxTL or TxTL requests for an I/O sequence to process from an I/O sequence scheduler (not shown) when the I/O sequence scheduler is idle. I/O sequences include one or more I/O tasks that are to be processed. Further, there is a request for task context for the particular I/O sequence from the cache controller 250. A task context is a data stream of one or more tasks that are to be processed. If the I/O sequence is a data frame, the RxTL/TxTL 210 passes on a corresponding context to a Descriptor Generator (not shown) within DM 260 that creates the descriptors for DMA engine 230 based on the requests from TL 210.
In one embodiment, DMA engine 230 features a Data Integrity Field (DIF) or Block Guard (BG) to enable DMA engine 230 to verify, verify and strip, verify and update or calculate and insert a DIF to a source data block depending on control bits set as part of the I/O context. In such an embodiment, the BG is 8 bytes. However in other embodiments, the DIF may be of any length based on the application.
A descriptor may include a Host Address (HA), Host Byte Count (HBC), Local Address (LA) and Local Byte Count (LBC).
According to one embodiment, the storage controller 200 implements a hardware automated mechanism for generating DMA descriptor(s) by keeping track of actual user data and the DIF, when BG is enabled, being transferred to and from the host memory from and to the local memory for each I/O depending on the BG function that is performed. In such an embodiment, the DMA descriptors are generated by considering the lengths of the data buffers and their address locations in both the host memory and local memory.
Further, in the cases where the expanded block data frame received by the storage protocol engine ends with only a current running DIF being received and when a BG function like STRIP or UPDATE/VERIFY or INSERTION is enabled, DM 260 stores the current running DIF in the I/O context. Thus, when the next frame for that particular I/O sequence arrives, DM 260 fetches the corresponding I/O context and includes the retrieved current running DIF data along with received data to synthesize new descriptors.
In yet another embodiment, DM 260 stores partial descriptor component information (e.g., the current scatter gather list pointer (SGL_PTR), host buffer offset, local buffer offset, total bytes transferred etc.) pertaining to an I/O sequence when the transaction layer can multiplex multiple I/O sequences or data streams to a single DM. In such an embodiment, relevant partial descriptor information is stored as part of each of the I/O contexts.
Further the partial descriptor information is stored in a context memory and corresponding I/O is fetched either from the context cache 240 or from the context memory when the DM 260 resumes the descriptor synthesis for the particular I/O sequence or data stream.
In one embodiment, DM 260 includes a data integrity engine that performs data integrity functions on the source data stream for the data blocks. An Insert function converts compressed data blocks into expanded data blocks. Based on user data fields from the compressed data blocks, the DIF is generated and inserted into the data stream subsequent to their respective compressed data blocks.
A Strip function converts expanded data blocks into compressed data blocks. Based on the user data fields from the expanded data blocks, the DIF is generated and verified against the DIF from the respective expanded data blocks. Following the verification, the DIF is removed from the expanded data blocks in the transferred data stream.
An Update function verifies and updates the DIF of expanded data blocks. Based on the user data fields from the expanded data blocks, the DIF is generated and verified against the DIF from the respective expanded data blocks. Subsequently, the generated DIF replaces the verified DIF in the transferred data stream. Use of the Update and Strip functions assumes that the source data includes expanded data blocks while the Insert function assumes that the source data includes compressed data blocks.
A Verify function verifies the DIF of expanded data blocks. Like with the Update function, based on the user data fields from the expanded data blocks, the DIF is generated and only verified against the DIF from the respective expanded data blocks. Unlike with the Update function, no data is modified during the data transfer with Verify function.
According to one embodiment, if STRIP or UPDATE/VERIFY functions are enabled, the total source data byte count should be a multiple of 520 (512 B data plus 8 B BG) and if INSERT is enabled, the total source data byte count should be a multiple of 512. Based on the direction of data movement and the BG function being enabled, there are mechanisms implemented in the Data Integrity Engine for the descriptor generation. These mechanisms include Local to Host STRIP, Local to Host INSERT, Local to Host UPDATE, Host to Local INSERT, Host to Local STRIP, and Host to Local UPDATE.
If however XC is not zero, it is determined whether the host address (HA)+XC+DIF, if enabled, is greater than a next host boundary buffer (HBB) (e.g., the HBB has been exceeded), decision block 430. If HA+XC+DIF is not greater than HBB, it is determined whether the local address (LA)+XC+DIF, if enabled, is greater than a next local boundary buffer (LBB) (e.g., the LBB has been exceeded), decision block 440.
If LA+XC+DIF is not greater than HBB, a descriptor is generated based on the XC when generating HA, a host byte count (HBC), LA and a local byte count (LBC), including DIF if enabled, the I/O Context Fields values for the particular context are updated and marked as done, processing block 450. Subsequently, control is returned to decision block 420.
If LA+XC+DIF is not greater than LBB, a descriptor is generated based on the next LBB when generating HA, HBC, LA and LBC, including DIF if enabled, and the I/O Context Fields are updated, processing block 480. Subsequently, control is returned to decision block 420.
Returning to decision block 430, if HA+XC+DIF>HBB, it is determined whether LA+XC+DIF, if enabled, is greater than a LBB, decision block 460. If LA+XC+DIF>LBB, it is determined whether the calculated LBC (LBB−LA)<the calculated HBC (HBB−HA), decision block 470.
If the calculated LBC is greater than the calculated HBC, control is again forwarded to process block 480 where a descriptor is generated based on the next LBB when generating HA, HBC, LA and LBC, including DIF if enabled, and the I/O Context Fields are updated. Subsequently control is returned to decision block 420.
If the calculated LBC is not greater than the calculated HBC, a descriptor is generated based on the next HBB when generating HA, HBC, LA and LBC, including DIF if enabled, and the I/O Context Fields are updated, processing block 490. Subsequently, control is returned to decision block 420. Returning to decision block 460, if LA+XC+DIF is not greater than LBB, control is again forwarded to decision block 490.
Table 1 below illustrates results for descriptor generation for a 5200 B transfer of data from local memory buffers to the host memory with STRIP enabled, assuming that the length of the local memory buffers is 1 Kb, the length of the host memory buffers is 2 KB, the data block size is 512 B and the BG is 8 B.
In this scenario, local buffers L0-L5 store (1024 B), (1024 B), (1024 B), (1024 B), (1024 B) and (80 B), respectively, for a total of 5200 B. The host memory buffer A/L pairs are A0/2 k, A1/2 k, A2/2 k for a total of 5120 B. Thus a first descriptor 1 is generated where the host address (HA) is A0, the local address (LA) is L0, the local byte count (LBC) is 1024 (1016 B data and 8 B BG) and the host byte count (HBC) is 1016, equivalent to the actual amount of data moved to the host buffer.
When the next descriptor to move the next frame stored in local buffer L1, the HA is A0+1016, while LA is L1, LBC is 1024 and HBC is 1008. As illustrated in this scenario, the DM keeps track of the exact byte count transferred from local memory to host memory.
For descriptor 3 24 Bytes of data is transferred from local memory to host memory due to host buffer boundary crossing. The DM should stop on the host buffer boundary and continue working on the next descriptor (descriptor 4) from the next host buffer boundary. Descriptors 6 and 7 have similar reasons for being separated.
Table 2 illustrates results for descriptor generation for a 5200 B transfer of data from host memory buffers to the local memory with STRIP enabled, assuming that the length of the local memory buffers is 1 KB, the length of the host Memory buffers is 2 KB, the data block size is 512 B and the BG is 8 B.
Descriptor 2 stops at host buffer boundary and descriptor 3 continues on the next data transfer but stops at local buffer boundary. Similarly, descriptor 5 stops at host buffer boundary and descriptor 6 continues on the next data transfer but stops at local buffer boundary.
Table 3 below illustrates results for descriptor generation for a 5120 Byte transfer of data from host memory buffers to the local memory with INSERTION enabled, assuming that the length of the local memory buffers is 1 KB, the length of the host memory buffers is 2 KB, the data block size is 512 B and the BG is 8 B. This implements a total of 5200 Byte local buffer space available for transfer.
In this scenario, local buffers L0-L5 store (1024 B), (1024 B), (1024 B), (1024 B), (1024 B) and (80 B), respectively, for a total of 5200 B. The host memory buffer A/L pairs are A0/2 k, A1/2 k, A2/2 k for a total of 5120 B. Thus, a first descriptor 1 is generated where the host address (HA) is A0, the host byte count (HBC) is 1016, the local address (LA) is L0 and the local byte count (LBC) is 1024 (1016 B data and 8 B BG).
When the next descriptor to move the next data chunk is stored into local buffer L1, the HA is A0+1016, while LA is L1, LBC is 1024 and HBC is 1008. As illustrated in this scenario, the DM keeps track of the exact byte count transferred from host memory to local memory.
Descriptor 3 transfers 24 Bytes of data from host to local due to host buffer boundary crossing. The DM should stop on the host buffer boundary and continue working on the next descriptor (descriptor 4) from the next host buffer boundary. Descriptors 6 and 7 have similar reasons to be separated.
Note that, the insertion from host to local data transfer is just the reverse process of the stripping from local to host data transfer. Also note that, the above mechanism uses local buffer size as the basis to perform the calculation (e.g., the HBC of descriptor is calculated based on the local buffer size including data integrity field). The other or similar method is to use the host buffer size as basis to perform LBC and HBC calculations.
The above-described hardware automated descriptor generation and management of data transfers significantly benefits I/O performance since the task is offloaded from firmware to improve frame processing latency, reduce the firmware's overhead, and improve overall system I/O performance.
Whereas many alterations and modifications of the present invention will no doubt become apparent to a person of ordinary skill in the art after having read the foregoing description, it is to be understood that any particular embodiment shown and described by way of illustration is in no way intended to be considered limiting. Therefore, references to details of various embodiments are not intended to limit the scope of the claims, which in themselves recite only those features regarded as essential to the invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4577314 | Chu et al. | Mar 1986 | A |
| 7185266 | Blightman et al. | Feb 2007 | B2 |
| 20030110325 | Roach et al. | Jun 2003 | A1 |
| 20060047904 | Rohde et al. | Mar 2006 | A1 |
| 20060136619 | Edirisooriya et al. | Jun 2006 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20070073923 A1 | Mar 2007 | US |
