Patent Application
20050138221
1. Field
The embodiments relate to handling redundant paths among devices.
2. Description of the Related Art
An adaptor or multi-channel protocol controller enables a device coupled to the adaptor to communicate with one or more connected end devices over a connection according to a storage interconnect architecture, also known as a hardware interface, where a storage interconnect architecture defines a standard way to communicate and recognize such communications, such as Serial Attached Small Computer System Interface (SCSI) (SAS), Serial Advanced Technology Attachment (SATA), Fibre Channel, etc. Further details on the SAS architecture for devices and expanders is described in the technology specification “Information Technology—Serial Attached SCSI (SAS)”, reference no. ISO/IEC 14776-150:200x and ANSI INCITS.***:200x PHY layer (Jul. 9, 2003), published by ANSI and further details on the SATA architecture are described in the technology specification “Serial ATA: High Speed Serialized AT Attachment” Rev. 1.0A (January 2003).
Devices may communicate through a cable or through etched paths on a printed circuit board when the devices are embedded on the printed circuit board. These storage interconnect architectures allow a device to maintain one or more connections with end devices through a direct connection to the end device or through one or more expanders. In the SAS/SATA architecture, a SAS port is comprised of one or more SAS PHYs, where each SAS PHY interfaces a physical layer, i.e., the physical interface or connection, and a SAS link layer having multiple protocol link layer. Communications from the SAS PHYs in a port are processed by the transport layers for that port. There is one transport layer for each SAS port to interface with each type of application layer supported by the port. A “PHY” as defined in the SAS protocol is a device object that is used to interface to other devices and a physical interface.
An expander is a device that facilitates communication and provides for routing among multiple SAS devices, where multiple SAS devices and additional expanders connect to the ports on the expander, where each port has one or more SAS PHYs and corresponding physical interfaces. The expander also extends the distance of the connection between SAS devices. With an expander, a device connecting to a SAS PHY on the expander may be routed to another expander PHY connected to a SAS device. Further details on the SAS architecture for devices and expanders is described in the technology specification “Information Technology—Serial Attached SCSI (SAS)”, reference no. ISO/IEC 14776-150:200x and ANSI INCITS.***:200x PHY layer (Jul. 9, 2003), published by ANSI, which is referred to herein as the “SAS Specification”.
A port in an adaptor or expander contains one or more PHYs. Ports in a device are associated with PHYs based on the configuration that occurs during an identification transmission. An identification transmission are those one or more transmissions a device initiates to inform a connected device of information on the interface. A port is assigned one or more PHYs within a device for those PHYs within that device that are configured to use the same SAS address during the identification sequence and that connect to attached PHYs that also transmit the same address during the identification sequence. A wide port has multiple PHYs and a narrow port has only one PHY. A wide link comprises the set of physical links that connect the PHYs of a wide port to the corresponding PHYs in the corresponding remote wide port and a narrow link is the physical link that attaches a narrow port to a corresponding remote narrow port.
On an initiator device including SAS ports, a domain comprises all devices that can be reached through an initiator port, where the port may connect to multiple target devices through one or more expanders or directly. A routing loop occurs if at least two paths are available to a single SAS address in a single domain, so that there are multiple paths to the same device in the same domain, i.e., accessible through the same initiator port. The SAS specification provides that if a routing loop is detected, the loop may be broken by disabling all the expander PHYs attached to that target device except the expander PHY with the lowest PHY identifier in the expander device with the lowest SAS address.
Referring now to the drawings in which like reference numbers represent corresponding parts throughout:
In the following description, reference is made to the accompanying drawings which form a part hereof and which illustrate several embodiments. It is understood that other embodiments may be utilized and structural and operational changes may be made.
The operating system 10 may load a device driver 20a and 20b for each storage interface supported in the adaptor 12 to enable communication with a device communicating using the same supported storage interface and also load a bus interface 24, such as a Peripheral Component Interconnect (PCI) interface, to enable communication with a bus 26. The operating system 10 may load device drivers 20a and 20b supported by the adaptors 12a, 12b upon detecting the presence of the adaptors 12a, 12b, which may occur during initialization or dynamically, such as the case with plug-and-play device initialization. In the embodiment of
Each PHY 34 for port 30 further includes a SAS PHY layer 42 and a physical layer 44. The physical layer 44 comprises the physical interface, including the transmitter and receiver circuitry, paths, and connectors. As shown, the physical layer 44 is coupled to the PHY layer 42, where the PHY layer 42 provides for an encoding scheme, such as 8b10b to translate bits, and a clocking mechanism, such as a phased lock loop (PLL), to convert data between a serial and parallel format. The PHY layer 32a, 32b . . . 32n may include a serial-to-parallel converter to perform the serial-to-parallel conversion and a phased lock loop (PLL) to track the incoming data and provide the data clock of the incoming data to the serial-to-parallel converter to use when performing the conversion. Data is received at the adaptor 12 in a serial format, and is converted by the SAS PHY layer 42 to the parallel format for transmission within the adaptor 12. The SAS PHY layer 42 further provides for error detection, bit shift and amplitude reduction, and the out-of-band (OOB) signaling to establish an operational link with another SAS PHY in another device, speed negotiation with the PHY in the external device transmitting data to adaptor 12, etc. The layers may be implemented as program components executed from memory and/or implemented in hardware.
In the embodiments of
An adaptor 12 may further have one or more unique domain addresses, where different ports in an adaptor 12 can be organized into different domains or devices. The SAS address of a PHY may comprise the SAS address of the port to which the PHY is assigned and that port SAS address is used to identify and address the PHY to external devices. A port may be uniquely identified by the SAS address assigned to that port and the SAS address of the PHYs to which the PHYs in the port connect. An interface address comprises an address assigned to an interface, where an interface may comprise a PHY, a logical interface and/or a physical interface, etc., such that an interface provides a connection to another interface which may be on a remote device.
If (at block 106) the port is attached to an expander, e.g., 72a, then the initiator, e.g., 70, sends (at block 110) a request to obtain information about the attached expander, such as an SMP report general, to the attached expander and adds (at block 112) information on the attached expander to the discovery map 76. The initiator 70 further sends (at block 114) a discovery request, such as an SMP discovery request, to the expander to determine the SAS address of expanders, e.g., 72b, 72c, or end devices attached to the expander added to the discovery map 76. If (at block 116) the SAS address of the expander added to the discovery map 76 is not a duplicate to any other expander in the domain already included in the discovery map 76, then the SAS address of the expander is added (at block 118) to the discovery map 76; otherwise, the initiator 70 disables (at block 120) the port and its PHYs subject to the discovery operation. In certain embodiments, the initiator 70 may disable the port/PHY on the intermediary expander, e.g., 72a, connecting to the expander port/PHY having the duplicate address e.g., 72b, 72c. Alternatively, the initiator 70 may disable the port/PHY having the duplicate address, e.g., 72d. The path that is disabled may include the path having the duplicate expander port/PHY having the lowest PHY identifier in the intermediary expander having the lowest expander SAS address. Alternative techniques may be used to select a route/duplicate expander to disable and perform the disabling. For instance, the route may be disabled by disabling the port on one expander providing the duplicate paths and not disabling other PHYs/ports.
If (at block 122) there are more PHYs in the discovered expander, then control proceeds back to block 114 to discover any further expanders, e.g., 72b, 72c, and/or end devices attached to the additional PHYs in the expander, e.g., 72a. Otherwise, if there are no more PHYs on the expander to discover, then the initiator 70 configures (at block 124) the known and allowable expander route entries that conform to configuration rules, such as the rule requiring no duplicate paths to a SAS device in the same domain. In certain embodiments, the initiator 70 may perform a level ordered traversal or breadth first discovery, where all devices at a level are discovered before proceeding to discover and configure paths for the devices at a next level. After configuring all the routes for an expander and expanders connected to that expander that satisfy the discovery rules so as not to have duplicate addresses for an expander, a determination is made (at block 126) as to whether there are further expanders in the domain of the port being discovered. The initiator 70 would learn of further expanders, e.g., 72b, 72c, 72d, in response to the discovery request sent to the expander just discovered. If (at block 126) there are further expanders to discover in the domain, then control proceeds back to block 110 to perform discovery with respect to that additional expander, e.g., 72b, 72c, 72d. Otherwise, if (at block 126) there are no more expanders in the domain to discover, then a determination is made (at block 128) whether all expanders have been fully configured. If not, then the configuration of the expanders is completed (at block 130). From the yes branch of block 128 or block 130, control proceeds back to block 104 to perform discovery with respect to any further ports in the initiator 70.
In the embodiment of
If (at block 206) the port is attached to an expander, e.g., 72a, then the initiator, e.g., 70, discovers and configures (at block 210) the attached expander and adds (at block 212) information on the attached expander to the discovery map 76. The initiator 70 further sends (at block 214) a discovery request, such as an SMP discovery request, to the expander to determine the SAS address of expanders, e.g., 72b, 72c, or end devices attached to the expander added to the discovery map 76. The expander or end device discovered in response to the SMP discover request sent at block 214 is added (at block 216) to the discovery map 76. If (at block 222) there are more PHYs in the discovered expander, then control proceeds back to block 214 to discover any further expanders, e.g., 72b, 72c, and/or end devices attached to the additional PHYs in the expander, e.g., 72a, added to the map that have not been subject to discovery. Otherwise, if there are no more PHYs on the expander to discover, then the initiator 70 configures (at block 224) the known expander route entries. After configuring all the routes for the expander and expanders connected to that expander that satisfy the discovery rules so as not to have duplicate addresses for an expander, a determination is made (at block 226) as to whether there are further expanders in the domain of the port being discovered. The initiator 70 would learn of further expanders, e.g., 72b, 72c, 72d, in response to the discovery request sent to the expander just discovered. As discussed, in certain embodiments, the initiator 70 may perform a level ordered traversal or breadth first discovery, where all devices at a level are discovered before proceeding to discover and configure paths for the end devices and expanders at a next level of devices. If (at block 226) there are further expanders to discover in the domain, then control proceeds back to block 210 to discover and configure any further expander, e.g., 72b, 72c, 72d. Otherwise, if (at block 226) there are no more expanders in the domain to discover, then a determination is made (at block 228) whether all expanders have been fully configured. If not, then the configuration of the expanders is completed (at block 230). From the yes branch of block 228 or block 230, control proceeds back to block 204 to perform discovery with respect to any further ports in the initiator 70.
After discovering and configuring expanders at all the ports, the initiator 70 searches (at block 230) each domain for expanders having duplicate SAS addresses, e.g., expanders 72b or 72c. If duplicates are found, selected expander PHYs/ports are disabled. As discussed, in certain embodiments, the initiator 70 may disable the ports connecting to the expander having the duplicate address, which may involve disabling all but the PHY/port having a lowest PHY identifier in an expander having the lowest expander SAS address. Alternative techniques may be used to select a route/duplicate expander to disable and perform the disabling.
With the operations of
In the embodiments of
At block 258, commands are sent to cause a configuration where the selected intermediary device is configured as the path to the connected device and the path to the discovered device other than through the selected intermediary device other than the selected intermediary device is disabled. In certain embodiments, the commands that cause the disabling of the at least one path through the at least one intermediary device other than the selected intermediary device further comprise determining at least one interface on the at least one intermediary device coupled to the path to the discovered device to disable and sending a command to disable the determined interface.
In still further embodiments, sending the commands to cause the configuration to disable the at least one path to the discovered device comprises disabling at least one interface on the intermediary device other than the selected device that connects to one corresponding interface on the discovered device. In alternative embodiments, sending the commands to cause the configuration to disable the at least one path to the discovered device comprises disabling at least one interface on the discovered device that connects to the intermediary device other than the selected intermediary device.
In certain embodiments, the devices may comprise SAS devices, the intermediary devices may comprise SAS expanders, and wherein redundant paths may occur when there are multiple paths to a same SAS address in a same domain.
In alternative embodiments, a discovery map indicating end and intermediary devices discovered as accessible is generated and the generated discovery map is processed to determine one or more devices to which there are redundant paths. For each determined device to which there are redundant paths, one intermediary device is selected to provide one path to the determined device. Commands are sent to cause a configuration where the selected intermediary device is configured as the path to the determined device and the at least one path to the determined device through at least one intermediary device other than the selected intermediary device is disabled.
Described embodiments provide techniques to detect redundant paths through intermediary devices, such as SAS expanders, during device discovery so that only one path to a device is configured and any possible redundant paths to the same target device are disabled.
The described embodiments may be implemented as a method, apparatus or article of manufacture using programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof. The term “article of manufacture” and “circuitry” as used herein refers to a state machine, code or logic implemented in hardware logic (e.g., an integrated circuit chip, Programmable Gate Array (PGA), Application Specific Integrated Circuit (ASIC), etc.) or a computer readable medium, such as magnetic storage medium (e.g., hard disk drives, floppy disks, tape, etc.), optical storage (CD-ROMs, optical disks, etc.), volatile and non-volatile memory devices (e.g., EEPROMs, ROMs, PROMs, RAMs, DRAMs, SRAMs, firmware, programmable logic, etc.). Code in the computer readable medium is accessed and executed by a processor. When the code or logic is executed by a processor, the circuitry may include the medium including the code or logic as well as the processor that executes the code loaded from the medium. The code in which preferred embodiments are implemented may further be accessible through a transmission media or from a file server over a network. In such cases, the article of manufacture in which the code is implemented may comprise a transmission media, such as a network transmission line, wireless transmission media, signals propagating through space, radio waves, infrared signals, etc. Thus, the “article of manufacture” may comprise the medium in which the code is embodied. Additionally, the “article of manufacture” may comprise a combination of hardware and software components in which the code is embodied, processed, and executed. Of course, those skilled in the art will recognize that many modifications may be made to this configuration, and that the article of manufacture may comprise any information bearing medium known in the art. Additionally, the devices, adaptors, etc., may be implemented in one or more integrated circuits on the adaptor or on the motherboard.
Additionally, the expander, PHYs, and protocol engines may be implemented in one or more integrated circuits on the adaptor or on the motherboard.
In the described embodiments, layers were shown as operating within specific components, such as the expander and devices, such as the initiator and target end devices. In alternative implementations, layers may be implemented in a manner different than shown.
In described embodiments, if there are redundant paths through multiple intermediary devices to one port/PHY, then the disabled one or more paths comprise all one or more redundant paths but the path through the port/PHY on the intermediary expander having the lowest PHY identifier in the intermediary expander having the lowest SAS address. In alternative embodiments, different techniques may be used to select expanders and PHYs providing redundant paths to a device to disable or not configure than selecting all but the PHY/Port with the lowest PHY identifier in the intermediary expander having the lowest SAS address.
In certain implementations, the device driver and network adaptor embodiments may be included in a computer system including a storage controller, such as a SCSI, Redundant Array of Independent Disk (RAID), etc., controller, that manages access to a non-volatile storage device, such as a magnetic disk drive, tape media, optical disk, etc. In alternative implementations, the network adaptor embodiments may be included in a system that does not include a storage controller, such as certain hubs and switches.
In described embodiments, the storage interfaces supported by the adaptors comprised SATA and SAS. In additional embodiments, other storage interfaces may be supported. Additionally, the adaptor was described as supporting certain transport protocols, e.g. SSP, STP, and SMP. In further implementations, the adaptor may support additional transport protocols used for transmissions with the supported storage interfaces.
The illustrated operations of
The adaptor 12 may be implemented on a network card, such as a Peripheral Component Interconnect (PCI) card or some other I/O card, or on integrated circuit components mounted on a system motherboard or backplane.
The foregoing description of various embodiments has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claims to the precise form disclosed. Many modifications and variations are possible in light of the above teaching.
