1. Field of the Invention
The invention relates generally to a zoning scheme implemented in a blade server chassis. More specifically, this invention relates to a method for partitioning Serial Attached Small Computer System Interface (SCSI) or SAS storage within a blade server chassis, where SAS storage blades are securely shared by server blades through the implementation of a “pair-based zoning” scheme.
2. Background Art
Due to ever-increasing demand for high density computing power, along with a need to secure data content and simultaneously deliver data efficiently, there arises a necessity of connecting groups of targets in blade server environments. SAS has proven to be very much of interest in addressing these storage connectivity issues because of low cost and interconnectivity beyond that of traditional SCSI. By way of employing expanders, support of up to 214 or 16384 devices is provided.
Thus, the capability of linking multiple hosts and targets can be achieved. In the case of blade server environments, allowing targets to access resources from servers requires controlling the sharing of the resources. A mechanism for either grouping devices together or isolating devices from each other needs to be implemented in order to achieve correctness of operation in data management. This is accomplished by “zoning.” Zoning can render Hard Disk Drives (HDDs) owned by one host (i.e. OS on a processor blade) unavailable for access by other hosts.
Zoning is a recent addition to the SAS architecture and is defined in the SAS specification. Before the advent of SAS-2, the second generation SAS, some developed pre-SAS-2 zoning approaches are implemented in the expanders used in Constellation systems. It is expected that later versions of these expanders that are compliant to the SAS-2 specification will be available. The interfaces employed shall be used on both the preceding and compliant versions.
In general, in one aspect, the invention relates to a method for partitioning SAS storage within a blade server chassis, and partitioning SAS expansion ports within the blade server chassis. The blade server chassis may be capable of housing N server blades, N storage blades, or any combination thereof up to a total of N blades. Connectivity between SAS storage blades and server blades may be provided via a pair of redundant, dual-domained SAS switches. The SAS switches may also include multiple expansion ports.
In one aspect of the invention, in order for SAS storage blades to be securely shared by server blades, a “pair-based” zoning may be implemented, whereby if a server blade and a storage blade occupy neighboring slots in the blade server chassis, the pair of server-storage blades may be set to belong in the same zone.
In another aspect of the invention, in order for SAS expansion ports to be securely shared by server blades, a “slot-ordered” zoning may be implemented, whereby if a server blade is located in an even slot, exclusive access to a single SAS expansion port may be provided.
Other aspects and advantages of the invention will be apparent from the following description and the appended claims.
a-2b show the steps involved in setting zoning permissions in the unmanaged mode in accordance with one embodiment of the invention such that a Zoning Permission Table is completed.
Specific embodiments of the invention will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.
In the following detailed description of embodiments of the invention, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.
In general, embodiments of the present invention describe a specific method for partitioning SAS storage within a blade server chassis, and partitioning SAS expansion ports within the blade server chassis. In one or more embodiments of the invention, the blade server chassis may be capable of housing N server blades, N SAS storage blades, or any combination thereof up to a total of N blades. Each storage blade constitutes a leaf node in a SAS tree and each server blade is a root node of the SAS tree. In one embodiment, connectivity between SAS storage blades and server blades may be provided via a pair of redundant, dual-domained SAS switches. In addition to providing connectivity between server blades and storage blades, the aforementioned SAS switches may also include multiple expansion ports, which may be used to connect to another SAS switch or to external SAS storage.
As an example in accordance with the above embodiment, a Sun Constellation blade chassis C10 may have ten blade slots. The C10 chassis may also additionally have twenty I/O card slots, two shared I/O module bays, and a Chassis Management Module (CMM) slot. Each blade slot may accept two types of blades, for example, a processor blade (server blade) or a storage blade. If a storage blade is present, at least one Network Express Module (NEM) may be installed to make the Hard Disk Drives (HDDs) on the storage blades available to the server blades.
In one or more embodiments of the invention, in order that storage blades may be securely shared by server blades, i.e., one server blade may have exclusive access to certain storage blades while other server blades may have exclusive access to certain other storage blades, the blade server chassis may implement a “pair-based” zoning scheme, whereby if a server blade and a storage blade occupy neighboring slots in the blade chassis, the pair of server and storage blades are said to belong in the same zone.
Without zoning or a storage sharing scheme, an operating system on a server blade may discover all storage blades in the blade server chassis and continue to overwrite stored data resulting in data corruption and incorrect system behavior.
In one or more embodiments of the invention, management of the storage components of the Constellation system 100 may be divided into two functions. In one embodiment, the first function may be the Zoning Manager (ZM), which may handle the zoning of the SAS domains. The ZM may run on a Constellation Management Module (CMM) and may communicate to expanders on storage blades or expanders on NEMs 130 via bi-directional two-wire I2C links. In one or more embodiments, the ZM may be used to divide the SAS fabric into separate zone groups, each zone group consisting of a processor complex, i.e., a processor blade and a set of HDDs, either on a storage blade or on a JBOD enclosure attached via an external port on the NEM. In one or more embodiments, management of storage resources within the zones may be done using a utility referred to as the Management Client running on the processor complex owning the zone. The management client may communicate over the SAS links with the storage blades and NEMs using the industry standard SCSI Enclosure Services (SES) and Serial Management Protocol (SMP) interfaces. The management client may provide for management of the storage Blades and NEMs (HDD, storage and NEM LEDs, reporting temperature and voltage on these boards, etc.).
In one or more embodiments of the invention, there may be two zoning modes defined: managed and unmanaged. In one embodiment, in the unmanaged mode, zoning may be enabled, and slots 0 and 1 may constitute the first pair-based zone, slots 2 and 3 may constitute the second pair-based zone, and so forth. This may be termed “pair-based” zoning, and in accordance with one embodiment, if a server (processor) blade and storage blade occupy neighboring slots, the pair of server-storage blades are said to belong in the same zone. No other server blade may access the aforementioned storage blade. It may be possible that HDDs on a storage blade are unavailable for access by any server blade. It is to be noted that it may also be possible for a server blade to neighbor another server blade. In accordance with the aforementioned embodiment, the two server blades are said to be in the same zone, despite there being no storage blades for use by either server blade. All the slots in the blade server chassis may form a series of non-overlapping zones, starting with slot 0.
In one or more embodiments of the invention, in order for SAS expansion ports to be securely shared by server blades, i.e., one server blade may have exclusive access to certain SAS expansion ports while other server blades may have exclusive access to other SAS expansion ports, the blade server chassis implements a “slot-ordered” zoning scheme, whereby if a server blade is located in an even slot (0, 2, 4, 6, and so on), the server blade may have exclusive access to a single SAS expansion port. The assignment of SAS expansion port to server blade may map the lowest-number SAS expansion port to the lowest numbered even slot. In other words, processor blade slot 0 may further have access to external port 0 (e.g. on both NEMs of
In one or more embodiments, in the unmanaged mode, configuring the NEMs or the storage blades may be based on the locally stored configuration and without intervention from the ZM. In one embodiment, the ZM's responsibility may be to change the configuration based on a direction from a System Administrator. The configuration state may be stored in the NEMs and storage blade boards and may be used for power-up and hot plug configuration.
In the abovementioned embodiment, the issue of plugging an NEM or storage blade with an incorrect stored configuration (i.e., blade coming from some other system or another slot in this system) may be handled using the SAS address of the attached devices to confirm whether a given device may be zoned into the system. For example, if a storage blade was taken from another system, the SAS addresses of the storage blade expander may not match the addresses stored by the NEM that the storage blade may be connected to. Thus, the PHYs, i.e., link layer connectors to physical devices, of the NEM connecting to the aforementioned storage blade may be placed by the NEM in zone group 0, whose purpose is to prevent a host from discovering the HDDs on the storage blade.
Similarly, if an NEM coming from a different system is added, the Host Bus Adapter (HBA) addresses on the processor blades may not match, and thereby all the PHYs connected to the processor blades may be in zone group 0, preventing the processor blades from discovering anything at all. Further, in accordance with the same embodiment, if an expander is added with zoning disabled, the inter-expander links to this expander may be programmed to no access (group 0).
Whenever an expander is powered on, and the expander is in unmanaged mode in accordance with one embodiment, a series of actions may be taken to set the zoning permissions by the expander. The PHY of an expander may be attached to an end device (HDD slot or processor blade) or may be a PHY of an external NEM port.
In Step 202, an expander may check for a presence thereof on a storage blade. If the expander is on a storage blade, the zoning state may be set to “Enabled” in Step 204. At Step 206, if the PHY of the storage blade expander is attached to an HDD slot, the zone group for the PHY may be set as per a table shown in
In Step 202, if the expander is not present on a storage blade, the expander may check for a presence thereof on an NEM in Step 210, and the zoning state may be set to “Enabled” in Step 212. At Step 214, if the PHY of the NEM expander is attached to a processor blade or external port, the zone group for the PHY may be set as per the table shown in
Thus, the Zoning Permission Table of
In one or more embodiments, in the managed mode, the zoning configuration may be managed by the ZM. The ZM itself may be stateless, and may manipulate the zoning state kept by each of the expanders. All changes to the zoning configuration may be done by the ZM. In one embodiment, when in managed mode, the state stored by the expanders may be restored by after a power cycle or link reset. This restoration may involve some historical checking in order to provide security against restoring incorrect state when a new blade, HDD or NEM may be installed.
In one or more embodiments of the invention, a general critical requirement is that the system may be able to boot without the presence of a ZM. Another requirement is that one client may not be able to access another client's storage resources. To satisfy both these requirements, it may be necessary for the expanders to verify that the device attached to each PHY has not changed during any link reset sequence, which may include a unit power cycle or hot-plug event. This behavior is supported by the SAS-2 specification.
It is to be noted that the above steps may also apply when the attached device is another expander to ensure that if a new module (storage or processor blade, NEM, or external JBOD) is added during a power cycle of the blade server chassis, the module may not have access to, and not provide access to other resources in the system. At the same time, the zoning configuration may be persistent for modules that are not changed during the power cycle.
In one or more embodiments, additional zoning steps may be summarized as follows.
When a processor blade is added, and the expander is in unmanaged mode, the PHYs on the NEM which connect to the added processor blade may be assigned as per the table in
When a processor Blade is added, and the expander is in Managed Mode, the PHYs on the NEM expander which connect to the added processor blade may be assigned to either zone group 0 (no access) or the zone group last assigned to the PHY, i.e., value before last link reset.
When an HDD is added, and the expander is in unmanaged mode, the storage PHY connected to the added HDD may be assigned to zone group 0.
When an HDD is added, and the expander is in managed mode, the storage PHY that connects to the added HDD may be assigned to either zone group 0 (no access) or the zone group that was last assigned to the PHY, i.e., value before last link reset.
Whenever a processor or storage blade, an NEM, an HDD or external JBOD is removed, the zone group of the PHYs attached to these modules may not be changed. The zone group may be adjusted when a module is added.
In one or more embodiments, when a CMM is removed the ZM may become unavailable, but the system continues to function normally. Other than the addition of components already known to the configuration, or the swapping of components between slots and bays, no storage configuration changes may be allowed when a ZM is not available.
When the CMM is added to the system, and the ZM process starts, there may be no changes to the system. Any changes may be made by the System Administrator.
Additional rules and guidelines may be provided in one or more embodiments for zone group assignment. The rules serve to simplify the zoning process. As an example, zone groups 100-127 may be reserved for unmanaged mode and may not be used in managed mode. The two ports of a controller on a processor blade may be assigned the same zone group. An HDD may be assigned the same zone as the processor that owns the HDD.
While the invention has been described with respect to an exemplary embodiment of a blade server environment, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.