Patent Grant
7536584
The present invention relates in general to the field of Serial-Attached-SCSI (SAS) systems, and particularly to SAS expanders.
Serial-Attached-SCSI (SAS) systems are becoming more and more common in modern computer systems. SAS systems include SAS initiator devices and SAS target devices as does its parent, the Small Computer Systems Interface (SCSI). SAS target devices are typically storage devices, such as disk drives, that receive commands from SAS initiator devices, such as SAS host bus adapters in host computers or SAS I/O controllers in Redundant Arrays of Inexpensive Disks (RAID) controllers.
Implementations and uses of SAS are described in detail in the following documents, each of which is incorporated by reference in its entirety for all intents and purposes:
SAS systems are built on point-to-point serial connections between SAS devices. Each point-to-point connection is referred to as a link, or lane, and the two endpoints are referred to as a PHY. A PHY contains a transceiver and electrically interfaces to a link to communicate with another PHY at the other end of the link. The link, or lane, includes two differential signal pairs, one in each direction. A SAS port includes one or more PHYs. A SAS port that has more than one PHY grouped together is referred to as a wide port, and the more than one links coupling the two wide ports are referred to as a wide link. Wide ports and links provide increased data transfer rates between SAS endpoints.
The simplest SAS topology is a single SAS initiator having a SAS port that is connected by a single SAS link to a SAS port of a single SAS target. However, it is desirable in many applications, such as a high data availability RAID system, to enable one or more SAS initiators to communicate with multiple SAS target devices. In addition to initiators and targets, SAS includes a third type of device, expanders, which are employed in SAS systems to achieve more complex topologies. SAS expanders perform switch-like functions, such as routing, to enable SAS initiators and targets to communicate via the SAS point-to-point connections.
The present inventors have observed various problems in complex topology SAS systems when a component is marginal or goes bad, such as a SAS device that generates logical errors, improper PHY analog settings, a bad or marginal PHY, or a bad or marginal link, which may include bad or marginal cables, connectors, or printed circuit board assembly traces. Some of the manifestations of the faulty components include intermittent communication errors between SAS devices, complete loss of a SAS link, or failure of an entire SAS domain. Another manifestation is the inability for an initiator to see a SAS target in the topology due to intermittent failures that cause a SAS device to work sufficiently well to be allowed into the topology, but to be sufficiently faulty to prevent effective communication between SAS devices.
One method of dealing with these problems is an initiator-based solution. The initiator may attempt to identify the faulty component and send a command through the SAS domain to disable, or bypass, various PHYs in the domain in a trial-and-error approach until the initiator has isolated the problem. However, the present inventors have observed some failure scenarios which cannot be satisfactorily remedied by the initiator-based approach. For example, assume a component fails in an intermittent fashion, such as a marginal PHY, that causes a SAS expander to first detect that a SAS link is operating properly, to subsequently detect that the link is not operating properly, and to continue this sequence for a relatively long time. According to the SAS standard, the SAS expander is required to transmit a BROADCAST primitive on each of its SAS ports to notify other SAS devices of the change of status within the SAS domain. Each time a SAS initiator receives the BROADCAST primitive it is required to perform a SAS discover process to discover the device type, SAS address, and supported protocols of each SAS device in the SAS domain and to configure routing tables within the SAS expanders as needed. The SAS discover process can take a relatively large amount of time. If the SAS expander transmits BROADCAST primitives due to the operational to non-operational link transitions according to a period that is comparable to the SAS discover process time, then consequently the SAS initiator may be unable to effectively send commands though the SAS domain to identify and remedy the problem. Or, even if the initiator is successful in identifying and fixing the problem, the SAS domain may have been effectively unavailable for providing user data transfers for an unacceptable length of time.
Therefore, what is needed is a solution to improve the data availability in SAS systems, which are subject to the foregoing problems.
The present invention provides an intelligent SAS expander that automatically detects faulty communications on one of its links and isolates the fault by disabling its PHY that is connected to the link on which it detects the faulty communications. In one embodiment, the intelligent SAS expander also reports the disabling of the PHY. In one embodiment, the intelligent SAS expander also recovers by re-enabling the previously disabled PHY after corrective action has been taken. The intelligent SAS expander monitors and analyzes various fault detection parameters to detect the faulty communications and isolate the faulty component by disabling the PHY.
In one aspect, the present invention provides a serial-attached-SCSI (SAS) expander for facilitating communication between SAS devices. The SAS expander includes a plurality of SAS PHYs, for transceiving signals with the SAS devices on a corresponding plurality of SAS links coupled to the plurality of SAS PHYs. The SAS expander also includes status registers, coupled to the plurality of SAS PHYs, configured to provide fault detection parameters concerning communications on the SAS links. The SAS expander also includes a microprocessor, coupled to the status registers. The microprocessor identifies faulty communications on one of the SAS links based on the fault detection parameters provided by the status registers. The microprocessor also disables a corresponding one of the plurality of SAS PHYs coupled to the one of the SAS links on which the microprocessor identified the faulty communications.
In another aspect, the present invention provides a method for increasing data availability in a serial-attached-SCSI (SAS) system including a SAS expander, having PHYs, for facilitating communications between a SAS initiator and a plurality of SAS target storage devices. The method includes the SAS expander identifying faulty communications on a SAS link connected to a PHY of the SAS expander. The method also includes the SAS expander disabling the SAS expander PHY, in response to identifying the faulty communications.
In another aspect, the present invention provides a serial-attached-SCSI (SAS) system. The system includes a SAS initiator, a plurality of SAS target storage devices, and a SAS expander, connected to the SAS initiator and to the plurality of SAS target devices by a plurality of SAS links. The SAS expander includes a plurality of PHYs, for coupling to the plurality of SAS links. The SAS expander identifies faulty communications on one of the plurality of SAS links and responsively disables one of the plurality of PHYs connected to the identified one of the plurality of SAS links.
An advantage of the present invention is that it provides a more direct, deterministic, and robust method for responding to faults in SAS systems than a SAS initiator-based solution. In particular, the intelligent SAS expanders described herein may directly disable a PHY associated with a detected fault by directly accessing registers within the SAS expander that control the PHY, without relying on the transmission of commands through the SAS domain to disable the PHY as required by an initiator-based solution, which, as described herein, might be ineffective or unacceptable with some failure modes.
Referring now to
Advantageously, the SAS expanders 102 of the SAS system 100 are intelligent SAS expanders 102 that include the ability to identify faulty communications on a SAS link 112 connected to one of the SAS expander 102 PHYs. Furthermore, the intelligent SAS expanders 102 include the ability to disable the identified PHY to isolate the faulty component, which may be the PHY itself, from the rest of the SAS system 100. Additionally, the intelligent SAS expanders 102 include the ability to report the disabled PHY. Finally, the intelligent SAS expanders 102 include the ability to recover from faulty condition. In one embodiment, a user notifies the SAS expander 102 that corrective action has been taken, such as replacing the faulty component (e.g., faulty cable, faulty SAS disk 106, or other faulty component), and the SAS expander 102 responsively repairs the communication between the SAS expander 102 and the other device by re-enabling the previously disabled PHY. In one embodiment, the SAS expander 102 is intelligent enough to automatically detect that a user has remedied the fault, and responsively re-enables the PHY. In one embodiment, the SAS expander 102 is intelligent enough to automatically take action to remedy the fault, such as by adjusting the PHY analog settings until reliable communications are restored. The SAS expander 102 includes a microprocessor that monitors status registers associated with the PHYs to identify faulty communications on a SAS link 112, writes to control registers to disable and re-enable the PHYs, and performs the reporting function, as discussed in detail below.
The SAS system 100 of
Advantageously, the SAS system 100 of
As mentioned above, the SAS links 112 may include various components, such as cables, connectors, and printed circuit board assemblies that include signal conductors. In one embodiment, the SAS expander 102 comprises a PM8388 SXP 24×3G 24-port SAS expander available from PMC-Sierra, Inc., of Santa Clara, Calif., in which the present inventors have modified the code 222 (discussed below) to perform the fault identification, isolation, reporting, and repairing steps of the present invention described herein. In other embodiments, the SAS expander 102 comprises a modified version of the following PMC-Sierra models: PM8387 SXP 36×3G 36-port SAS expander, PM8399 SXP 24×3GSEC 24-port SAS expander, or PM8398 SXP 36×3GSEC 24-port SAS expander.
Although
Referring now to
Each SAS port 216 includes one or more SAS PHYs 208 that is connected to one of the SAS links 112 of
The SAS expander 102 also includes a set of control and status registers (CSRs) 206 associated with each PHY 208, which the microprocessor 202 reads and writes to monitor fault detection parameters 300 (described below with respect to
The SAS expander 102 also includes multiplexed data paths (such as a crossbar) and switching circuitry (not shown) that interconnect the various PHYs 208 to enable them to transfer commands and data from one PHY 208 to another to perform the switching function of the SAS expander 102. The SAS expander 102 may also include buffering circuits associated with each of the PHYs 208 for buffering the commands and data when received in a port 216 and when waiting to be transmitted out a port 216. The commands and data are routed through the network between the ports 216 based on routing table information, which in one embodiment is stored in the memory 204.
Referring now to
The microprocessor 202 maintains a corresponding threshold 396 for each of the counters 392. Some of the thresholds 396 are stored in the CSRs 206, namely the disparity error interval threshold 362 and the code violation error interval threshold 364, and the SAS expander 102 hardware automatically compares them with the corresponding counter 392 value and generates an interrupt if the threshold is exceeded. The thresholds 224 corresponding to the other counter 392 values are stored in the memory 204, and the microprocessor 202 periodically compares the counter 392 values, or accumulated counts derived from the periodically sampled counter 392 values, with the thresholds 224 to identify faulty communications on the SAS links 112.
The counters 392 include an invalid DWORD count 302, which indicates the number of invalid DWORDs received outside PHY reset sequences; a disparity error count 304, which indicates the number of running disparity errors received outside PHY reset sequences; a code violation count 306, which indicates the number of times a decode error was detected on a bit stream; a loss of DWORD synchronization count 308, which indicates the number of times the PHY 208 has restarted the link reset sequence because it lost dword synchronization (i.e., the number of times the PHY 208 went from PHY ready state to COMINIT state); a PHY reset failed count 312, which indicates the number of times the PHY 208 has failed to obtain dword synchronization during final SAS speed negotiation; a CRC error count 314, which indicates the number of CRC DWORD errors detected for received IDENTIFY and OPEN address frames; an in connection CRC error count 316, which indicates the number of in connection CRC errors; and a PHY change count 318, which indicates the number of PHY change events that have been generated.
The interrupt indicators 218 include a PHY ready interrupt 322, which indicates the PHY 208 has finished initialization and is ready to transmit and receive data (A PHY 208 becomes ready only after COMINIT has been detected); a COMINIT interrupt 324, which indicates a valid COMINIT out of band (OOB) sequence has been successfully negotiated; a elastic store overflow interrupt 326, which indicates a valid DWORD was received and the internal elastic store, or buffer, is full; a disparity error interrupt 328, which indicates the disparity error interval threshold 362 has been exceeded during the number of clock cycles specified in the performance monitoring period 372; a code violation error interrupt 332, which indicates the code violation error interval threshold 364 has been exceeded during the number of clock cycles specified in the performance monitoring period 372; a DWORD synchronization loss interrupt 334, which indicates DWORD synchronization on the PHY 208 was lost and consequently the PHY 208 has restarted the link reset sequence.
The states 394 include a link connected state 342, which indicates whether the port 216 is in a connected state; a DWORD synchronization lost state 344, which indicates the PHY 208 has currently lost DWORD synchronization; an init passed state 346, which indicates whether the port 216 has successfully completed the link initialization sequence; a device present state 348, which indicates whether a device is connected to the PHY 208; an attached device type state 352, which indicates whether a SAS or SATA device was detected as being connected; a rate state 354, which indicates whether the final negotiated line rate is 1.5 or 3.0 Gbits/sec; a PHY reset limit saturation state 356, which indicates that the PHY 208 reset threshold has been reached.
In one embodiment, the SAS expander 102 is configured to receive from the RAID controllers 104 SCSI Enclosure Services (SES) pages that set and get the various fault detection parameters 300, that get the status of the PHYs 208, and that directly enable or disable individual PHYs 208. In one embodiment, control and status information, such as SES pages, may be sent via an out-of-band communication path between the SAS expanders 102 within an enclosure 114, such as an I2C connection or other communication path. The out-of-band communication path may be advantageously employed if the SAS expander 102 has disabled all PHYs 208 connecting the SAS expander 102 to an upstream SAS expander 102, such as might occur if the SAS cable connecting them is faulty. The disabling SAS expander 102 may communicate to the other SAS expander 102 in the enclosure 114 status information indicating that it has disabled the PHYs 208. In this situation, to avoid rebooting, the user may cause the other SAS expander 102 in the enclosure 114 to broadcast an SES page via the out-of-band communication path to the PHY-disabled SAS expander 102 instructing the SAS expander 102 to re-enable the disabled PHYs 208 after the cable has been replaced. The out-of-band communication path is particularly useful for the SAS expanders 102 within an enclosure 114 that do not have an inter-expander SAS link 112, which may not be present because the SAS specification does not allow loops within the SAS topology. Furthermore, the SAS expander 102 includes default values of the fault detection parameters 300 that are stored in a non-volatile memory of the SAS expander 102 and that are employed at boot time of the SAS expander 102. The default values may be modified by the RAID controllers 104 or by the microprocessor 202 during operation.
Referring now to
At block 402, the microprocessor 202 of
At block 404, the microprocessor 202 identifies faulty communications on a SAS link 112 connected to one of its PHYs 208 based on the monitoring at block 402. The microprocessor 202 analyzes the fault detection parameters 300 according to isolation rules embodied in the code 222 for fault indications to determine whether there is a need to disable a PHY 208. The identification of the faulty communications may include various criteria as discussed herein. An isolation rule may be triggered by one or more of the various counts exceeding a threshold, by detection that a PHY 208 has reached one or more particular states, that one or more particular events have occurred as indicated by one or more of the interrupt indicators 218, and various combinations thereof. In one embodiment, the microprocessor 202 only identifies faulty communications related to a PHY 208 if the PHY 208 is enabled. In one embodiment, the microprocessor 202 only identifies faulty communications related to a PHY 208 if isolation is allowed for the PHY 208. In one embodiment, the SAS expander 102 receives SES pages from the RAID controllers 104 to selectively enable and disable individual PHYs 208 and to selectively allow and disallow isolation of individual PHYs 208. Flow proceeds to block 406.
At block 406, the microprocessor 202 writes to a control register 206 to disable the PHY 208 identified at block 404. Flow proceeds to block 408.
At block 408, the SAS expander 102 reports the fact that the PHY 208 was disabled to one or both of the RAID controllers 104. In one embodiment, the SAS expander 102 also reports the reason the PHY 208 was disabled. In one embodiment, the SAS expander 102 also reports all threshold values used by the SAS expander 102 to make a determination to disable the PHY 208. In one embodiment, the SAS expander 102 reports by transmitting an SES diagnostic page to the RAID controller 104. In one embodiment, the SAS expander 102 reports by transmitting a Serial Management Protocol (SMP) message to the RAID controller 104. In one embodiment, the SAS expander 102 provides an interface to the RAID controllers 104 to enable the RAID controllers 104 to obtain the status of each PHY 208 and the current error counts, state, and events described herein. Flow proceeds to block 412.
At block 412, the RAID controller 104 reports that the PHY 208 was disabled to a user. In one embodiment, the RAID controller 104 reports to the user via a management interface. In one embodiment, the RAID controller 104 reports to the user by reporting to one or both of the hosts 108, which in turn notify the user. Flow proceeds to block 414.
At block 414, assuming the disabled PHY 208 is part of a wide port 216, communications between the SAS expander 102 port and the SAS device connected to the port 216 continue via the remaining PHYs 208 of the port 216 and associated SAS links 112 that are functioning properly. It is noted that the SAS system 100 may experience a proportionally lower data throughput due to the disabled PHY 208 and its respective SAS link 112. However, advantageously, by disabling the PHY 208 associated with the faulty SAS link 112 (or the PHY 208 itself may have been faulty), the likelihood that the SAS system 100 will continue functioning normally is increased, thereby improving the availability of the data on the SAS disks 106 to the hosts 108, rather than experiencing the various problems discussed herein. Flow ends at block 414.
Referring now to
Flow begins at block 402. Blocks 402 through 412 of
At block 514, the hosts 108 continue to access the SAS disks 106 implicated by the PHY 208 disabled at block 406 via an alternate pathway that does not include the disabled PHY 208. With respect to the SAS system 100 of
Referring now to
Flow begins at block 402. Blocks 402 through 412 of
At block 614, the user takes action to correct the faulty component in response to the reporting of the disabled PHY 208 at block 412. Examples of action that the user may take to correct the faulty component include, but are not limited to, replacing a cable, replacing a connector, replacing a SAS disk 106, replacing a SAS expander 102, replacing a RAID controller 104, and reconfiguring a PHY 208, such as to adjust its analog settings. Flow proceeds to block 616.
At block 616, the user notifies one of the RAID controllers 104 that he has taken the corrective action at block 614. In one embodiment, the user notifies the RAID controller 104 via a management interface. In one embodiment, the user notifies one of the hosts 108, which in turn notifies the RAID controller 104. Flow proceeds to block 618.
At block 618, the RAID controller 104 notifies the SAS expander 102 that the corrective action was taken. In one embodiment, the SAS expander 102 is notified by receiving a SCSI Enclosure Services (SES) diagnostic page from the RAID controller 104. In one embodiment, the SAS expander 102 is notified by receiving a Serial Management Protocol (SMP) message from the RAID controller 104. In one embodiment, the RAID controller 104 notifies the SAS expander 102 by explicitly instructing the SAS expander 102 to re-enable the PHY 208. Flow proceeds to block 622.
At block 622, the microprocessor 202 writes to a control register 206 to re-enable the PHY 208 that was previously disabled at block 406, in response to the notification that the corrective action was taken. Flow ends at block 622.
In one embodiment, the microprocessor 202 foregoes disabling the PHY 208 at block 406 if the PHY 208 is linked to another SAS expander 102 that is downstream from a RAID controller 104 linked to the SAS expander 102 that detected the fault. This advantageously simplifies recovery of certain failure modes on a SAS topology involving cascaded SAS expanders 102, such as the SAS system 100 of
Referring now to
Flow begins at block 402. Blocks 402 through 412 and 614 of
At block 716, the microprocessor 202 automatically detects that the user took the corrective action at block 614. In one embodiment, the user corrective action automatically detected by the microprocessor 202 is a user replacing a cable. The microprocessor 202 automatically detects the cable replacement by detecting a change of state from link not connected to link connected via the link connected state 342 fault detection parameter 300. In one embodiment, the user corrective action automatically detected by the microprocessor 202 is a user replacing a SAS disk 106 or a SATA disk. The microprocessor 202 automatically detects the disk replacement by detecting a change of state from device not present to device present via the device present state 348 fault detection parameter 300 and detects whether the replaced disk is a SAS disk or a SATA disk via the attached device type state 352. In one embodiment, the user corrective action automatically detected by the microprocessor 202 is a user replacing a SAS expander 102. The microprocessor 202 automatically detects the SAS expander 102 replacement by detecting a change of state from device not present to device present via the device present state 348 fault detection parameter 300 of a PHY 208 connected to the replaced SAS expander 102 via the inter-expander SAS link 112 of
At block 718, the microprocessor 202 writes to a control register 206 to re-enable the PHY 208 that was previously disabled at block 406, in response to the automatic detection at block 716 that the corrective action was taken by the user. Flow ends at block 718.
Referring now to
Flow begins at block 402. Blocks 402 through 412 of
At block 814, the microprocessor 202 automatically takes corrective action. In one embodiment, the automatic corrective action taken by the microprocessor 202 is to automatically adjust the PHY 208 analog settings, which may cause the SAS link 112 to start functioning properly if, for example, the cable length has been changed since the last time the PHY 208 analog settings were set. Flow proceeds to block 816.
At block 816, the microprocessor 202 writes to a control register 206 to re-enable the PHY 208 that was previously disabled at block 406. Flow proceeds to decision block 818.
At decision block 818, the microprocessor 202 determines whether normal communications have been restored on the SAS link 112 after re-enabling the PHY 208. If so, flow ends; otherwise, flow proceeds to block 822.
At block 822, the microprocessor 202 disables the PHY 208 again. Flow returns from block 822 to block 814.
In one embodiment, the microprocessor 202 maintains a retry count threshold, and once the microprocessor 202 has performed the steps in the loop at blocks 814 to 822 a number of times that exceeds the retry threshold, the microprocessor 202 leaves the PHY 208 disabled and stops trying to automatically repair the fault until it detects an event indicating that it should re-enable the PHY 208.
In one embodiment, the microprocessor 202 increases the period of the steps performed in the loop at blocks 814 to 822 each time it disables the PHY 208 at block 822 in order to reduce the number of SAS discover processes that must be performed in response to the PHY 208 disabling/re-enabling. A management application client performs a SAS discover process to discover all the SAS devices and expander devices in the SAS domain (i.e., determining their device types, SAS addresses, and supported protocols). A SAS initiator device uses this information to determine SAS addresses to which it is able to establish connections. A self-configuring expander device uses this information to fill in its expander route table. Additionally, if there are multiple disabled PHYs 208 that need re-enabling, then the microprocessor 202 re-enables all of the disabled PHYs 208 at the same time in order to further reduce the number of SAS domain discover processes that must be performed.
Although the present invention and its objects, features, and advantages have been described in detail, other embodiments are encompassed by the invention. For example, although embodiments have been described in which the SAS initiators are RAID controllers, other embodiments are contemplated in which other types of SAS initiators are employed. Furthermore, although embodiments have been described in which the SAS targets are disk drives, other embodiments are contemplated in which other types of SAS storage devices are employed.
Finally, those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the spirit and scope of the invention as defined by the appended claims.
This application claims priority to U.S. Provisional Application No. 60/804,267, filed Jun. 8, 2006, which is hereby incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6078979 | Li et al. | Jun 2000 | A |
| 7290074 | Asano et al. | Oct 2007 | B2 |
| 7423995 | Elliott et al. | Sep 2008 | B1 |
| 20050005062 | Liu et al. | Jan 2005 | A1 |
| 20050071532 | Bakke et al. | Mar 2005 | A1 |
| 20060034358 | Okamura | Feb 2006 | A1 |
| 20060061369 | Marks et al. | Mar 2006 | A1 |
| 20060101171 | Grieff et al. | May 2006 | A1 |
| 20070011373 | Marks et al. | Jan 2007 | A1 |
| 20070061632 | Uddenberg et al. | Mar 2007 | A1 |
| 20070061681 | Carmichael | Mar 2007 | A1 |
| 20070070885 | Uddenberg et al. | Mar 2007 | A1 |
| 20070087615 | Paulsen et al. | Apr 2007 | A1 |
| 20070088978 | Lucas et al. | Apr 2007 | A1 |
| 20070100847 | Slutz et al. | May 2007 | A1 |
| 20070220204 | Nakajima et al. | Sep 2007 | A1 |
| Number | Date | Country |
|---|---|---|
| 1610225 | Dec 2005 | EP |
| WO02073903 | Sep 2002 | WO |
| WO02098059 | Dec 2002 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20080010530 A1 | Jan 2008 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60804267 | Jun 2006 | US |
