Patent Application
20140241379
This invention relates to the use of the Ethernet physical and link layers to transport Fibre Channel (FC) frames, and more particularly to a novel FC Over Ethernet (FCOE) protocol for encapsulating FC frames within Ethernet frames, in one embodiment within a blade server environment.
FC is the primary interconnect used for networked storage. Ethernet is the primary interconnect used for Local Area Networks (LANs). Both FC and Ethernet have previously been viewed as separate parallel protocols and complete solutions for different purposes. Furthermore, because Ethernet is known to be more unreliable than FC (Ethernet devices discard packets under certain conditions such as congestion), and FC provides for the reliable transport of frames, the two protocols have historically been viewed as incompatible solutions.
The historically separate and parallel nature of FC and Ethernet is illustrated in
More recently, however, efforts have been made to combine the FC and Ethernet protocols. For example, FC over Transmission Control Protocol/Internet Protocol (TCP/IP) (FCIP), (described in Request For Comments (RFC) 3643 and RFC3821, available at www.ietf.org), and internet FC Protocol (iFCP) (described in RFC4172), provide a method for transporting FC frames over TCP/IP/Ethernet. However, FCIP is fundamentally different from FCOE in that FCIP is a point-to-point protocol that does not route packets—all packets sent out at the transmitting end are received at the receiving end. iFCP uses TCP/IP as a transport. FCIP and iFCP also allow scaling to Wide Area Networks (WANs) and to the global Internet. However as a result of these extended capabilities, these protocols (FCIP and iFCP) are complex, expensive to implement, and have relatively low performance.
Accordingly, there is a need to layer FC over Ethernet is a manner that is less complex, more inexpensive and yet higher performing than FCIP or iFCP. In addition, it is desirable for this solution to be able to take advantage of the prevalence of legacy Ethernet switching devices and legacy FC software drivers, and allow a single physical adapter and a single wire to handle both Ethernet and storage traffic while sharing part of the switching infrastructure. There is also a need to take advantage of the wide availability of low cost Ethernet switching Application Specific Integrated Circuits (ASICs) and boxes to allow development of low cost FC switches and fabrics when FC is layered over Ethernet. Furthermore, given that 10 Gigabit (10 G) FC is not yet deployed in any significant volume, there is a need to develop FC over 10 G Ethernet (10 GbE) as a standard way of implementing FC at a 10 Gigabit data rate.
The present invention is directed to using Ethernet as an underlying transport for FC frames, in a protocol referred to herein as Fibre Channel Over Ethernet (FCOE). In FCOE, the Ethernet physical and link layers are utilized for transport of FC frames. In particular, the FC physical layer and part of the FC-2 link layer are replaced with the Ethernet physical and link layers, so that FC frames can be encapsulated and transported within Ethernet frames in a way that preserves the FC higher layers and is transparent to those layers. In FCOE, at the higher levels (e.g. the driver level, the software level, and the functional level), communications appear to be standard FC. However, at the lowest levels (e.g. the physical layer), the communications appear to be standard Ethernet, and as such, standard Ethernet switching hardware can be used.
In the FCOE protocol, each FC frame is encapsulated within an Ethernet frame. In general, the payload of the FCOE frame contains type information from the FC Start Of Frame (SOF) indicator, the FC header, an optional FC payload, an optional FC Cyclic Redundancy Check (CRC) field, type information from the FC End Of Frame (EOF) indicator, and optional padding for short packets.
The FC SOF and EOF indicators do not merely delimit the start and end of a FC frame, they also carry information. The type information represented by the FC SOF indicators and FC EOF indicators must be carried in the FCOE frame, because Ethernet SOF and EOF indicators do not contain this additional information. Therefore, in the FCOE protocol, particular numerical values are assigned to the various FC SOF and EOF indicator types, and these values are stored as FC SOF type information and FC EOF type information in the FCOE payload.
The FC header of a FC frame indicates the 24-bit destination address of the FC frame. Similarly, the Media Access Control (MAC) header of an Ethernet frame indicates the destination address of the Ethernet frame. Therefore, the FCOE protocol requires a translation from the 24-bit FC address to an Ethernet MAC address. In FCOE, the Address Resolution Protocol (ARP), which specifies how to map an address from a higher level protocol to a lower level protocol and is used frequently to map Internet Protocol (IP) addresses to MAC addresses, may be used to map the FC address to an Ethernet MAC address located within the FCOE MAC header.
The Ethernet MAC header (and therefore also the FCOE MAC header) includes a 6-byte destination address field, a 6-byte source address field, a 2-byte Ethertype field, and optionally a 4-byte Virtual Local Area Network (VLAN) field. The Ethertype field indicates the payload type. In addition, for the FCOE protocol, a new Ethertype value indicates that the Ethernet frame is a FCOE frame as opposed to a standard Ethernet frame.
In one system-level embodiment, an Ethernet network carrying FCOE replaces a standard FC network so that all participating devices are communicating using the FCOE protocol. The switches are Ethernet switches, additionally configured to provide FC fabric services. These fabric services may be provided by a FC fabric server. The functionality implemented in the FC fabric server corresponds to the FC fabric services functionality traditionally implemented in FC switches.
In another system level embodiment, devices implementing the FCOE protocol may be implemented in a blade server. The blade server includes a backplane with Ethernet (e.g. 10 GbE) pathways, and FCOE is carried on the pathways. In this blade server embodiment, FCOE represents only a short segment in the overall system. The entire backplane may be Ethernet over which both storage and networking traffic can be run. The Ethernet links are connected to an Ethernet switch, which may have a standard Ethernet port that leaves the blade server chassis and is utilized for network communications. The Ethernet switch receives data from all blades and switches it to the correct destination, either external networking via the Ethernet port, or if it is storage traffic, to a FCOE/FC converter and then out to a FC network via a FC port. The FCOE/FC converter is coupled to a FC switch, which may be located external to the blade chassis through a standard FC connection. The FC switch then routes the FC frames to FC devices over a FC fabric. The FC switch may be additionally configured to provide FC fabric services.
In the two embodiments described above, any of the devices implementing the FCOE protocol may utilize FCOE-specific hardware, or a processor and firmware capable of implementing the FCOE protocol. One device capable of implementing the FCOE protocol is the Intelligent Network Processor (INP) described in U.S. application Ser. No. 11/433,728 filed on May 11, 2006 and entitled “Intelligent Network Processor and Method of Using Intelligent Network Processor,” the contents of which are incorporated by reference herein. One or more processors and firmware within the INP may be programmed to implement the FCOE protocol in the devices described above.
In particular, in a FCOE blade server utilizing INPs, the blade server chassis includes a plurality of blades, each blade containing an INP for implementing an FCOE I/O controller function. The blades are connected to two I/O modules across a redundant Ethernet backplane (e.g. 10 GbE), each I/O module being a card within the blade server chassis and containing an INP for implementing the Ethernet switch function, the FCOE/FC converter function, and the FC switch function. The Ethernet backplane provides a unified backplane transport for both LAN and SAN applications.
In the following description of preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which it is shown by way of illustration specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the preferred embodiments of the present invention.
Embodiments of the present invention are directed to using Ethernet as an underlying transport for FC frames, in a novel protocol referred to herein as Fibre Channel Over Ethernet (FCOE).
The FC protocol is a layered protocol, typically used as a Small Computer System Interface (SCSI) transport. As illustrated in
Embodiments of the present invention utilize the Ethernet physical and link layers for transport of FC frames. As shown in
One advantage of using FCOE according to embodiments of the present invention is the ability to leverage the Ethernet physical layer and make use of existing Ethernet infrastructure such as Ethernet switch ASICs and optics instead of FC, which due to the volumes associated with Ethernet, can be much less expensive. Another advantage of the present invention is that with FCOE, networking and storage traffic can be implemented over a single Ethernet link. There is no need to have separate FC and network connectivity because FCOE uses standard Ethernet switching.
In the FCOE protocol according to embodiments of the present invention, each FC frame is encapsulated within an Ethernet Frame. In general, the payload 326 of the FCOE frame 304 contains type information 340 from the FC SOF indicator 306, the FC header 308, an optional FC payload 310, an optional FC CRC field 312, type information 342 from the FC EOF indicator 314, and optional padding 328 for short packets.
The FC SOF and EOF indicators 306 and 314 do not merely delimit the start and end of a FC frame, they also carry information. There are a number of different types of FC SOF indicators 306 and FC EOF indicators 314, each indicator type representing certain information. For example, FC frames are grouped into sequences, so some of the FC SOF indicator types indicate whether the FC frame is the start of a new sequence or part of an existing sequence. The class of the transaction is also represented by a particular FC SOF indicator type. Similarly, an example of a particular FC EOF indicator type is an abort EOF, which indicates that an error occurred within the frame.
The various FC SOF indicator types and FC EOF indicator types and the information represented by each are described in RFC 3643, for example, the contents of which are incorporated by reference herein. SOF types include SOFf, SOFi2, SOFn2, SOFi3, SOFn3, SOFi4, SOFn4, and SOFc4 (active Class-4), where the “i” and “n” designation refer to the first frame of a sequence and subsequent frames in a sequence, respectively, and {2, 3, 4, f} refer to the FC Class of service. EOF types include EOFn (normal—not the last frame of sequence), EOFt (terminate—the last frame of a sequence), EOFni (normal invalid—FC routing error detected), EOFa (abort—sender aborted, receiver should discard), EOFdt (disconnect terminate—Used for Class-4 to disconnect), EOFdti (disconnect terminate invalid—Class-4), EOFrt (remove terminate—removes Class-4 circuit), and EORrti (remove terminate invalid—Class-4).
In embodiments of the present invention, the type information represented by the FC SOF indicators 306 and FC EOF indicators 314 must be carried in the FCOE frame, because Ethernet SOF and EOF indicators 316 and 324 do not contain this additional information. Therefore, in the FCOE protocol according to embodiments of the present invention, particular numerical values are assigned to the various FC SOF and EOF indicator types, and these values are stored as FC SOF type information 340 and FC EOF type information 342 in the FCOE payload 326.
The FC header 308 of a FC frame 300 indicates a 24-bit destination address and a 24-bit source address of the FC frame. Similarly, the MAC header 318 of an Ethernet frame 302 indicates the destination address and source address of the Ethernet frame. Note that each FC port will ship with a factory configured MAC address as well as its FC world wide name. Therefore, the FCOE protocol requires a translation from the 24-bit FC destination address to an Ethernet MAC destination address, and a translation from the 24-bit FC source address to an Ethernet MAC source address. According to embodiments of the present invention, the Address Resolution Protocol (ARP), which specifies how to map an address from a higher level protocol to a lower level protocol and is used frequently to map IP addresses to MAC addresses, may be used to map the FC destination address to an Ethernet MAC destination address 332 located within the FCOE MAC header 330, and map the FC source address to an Ethernet MAC source address 334 located within the FCOE MAC header 330. However, it should be understood that the ARP protocol is just one example protocol, and that other protocols may be used to map an FC address to a MAC address. The ARP protocol is defined in RFC826, available at www.ietf.org and incorporated herein by reference. It should be understood that FC discovery will remain unchanged as ARP is layered below FC and works transparently.
Note that using the ARP protocol to map the FC destination address to an Ethernet MAC destination address is advantageous over other methods such as a hardware-derived address, because such methods are not compatible with FC login processes as specified in the Ethernet Standard IEEE 802.3, the contents of which are incorporated by reference herein. For example, U.S. Patent Application Publication 2006/0098681A1 describes a hardware-derived address in which a 48-bit Ethernet destination MAC address is comprised of the 24-bit destination FC identification field and a 24-bit Organization Unique Identifier (OUI) code that has been registered to indicate the FCOE protocol, and a 48-bit Ethernet source MAC address is similarly comprised of the 24-bit source FC identification (ID) field and the 24-bit OUI code that has been registered to indicate the FCOE protocol. However, such a methodology is not compatible with the FC fabric login process, where a zero is initially stored as the source FC ID of a device and a subsequent response from a switch connected over the FC fabric is required to provide the actual source FC ID for the device. This FC fabric login process cannot be performed using FCOE. Embodiments of the present invention avoid this problem and are compatible with IEEE 802.3 because the Ethernet MAC address is physically assigned in the normal way, so upon power up an Ethernet MAC address is available to be mapped to a FC ID.
The Ethernet MAC header 318 (and therefore also the FCOE MAC header 330) includes the 6-byte destination address field 332, the 6-byte source address field 334, a 2-byte Ethertype field 336, and optionally a 4-byte VLAN field 338. The Ethertype field 336 indicates the payload type. In addition, for the FCOE protocol, a new Ethertype value is provided to indicate that the Ethernet frame is a FCOE frame 304 as opposed to a standard Ethernet frame. In addition to the new FCOE Ethertype according to embodiments of the present invention, other conventional Ethertypes include, but are not limited to, the standard Ethernet Ethertype, an ARP Ethertype, and a pause frame Ethertype. However, if all destinations in the system are FCOE in an end-to-end FCOE system embodiment (described in further detail below), a new FCOE Ethertype field may not be needed.
Although the FC CRC field 312 does not need to be captured in the FCOE payload 326 because Ethernet already has a CRC field 322, nevertheless in alternative embodiments of the present invention the FC CRC field 312 may be included in the FCOE payload to assist in identifying errors in the translation between FC and FCOE in certain system embodiments. For example, including the FC CRC field 312 may be helpful in a situation where a FC initiator generates a FC frame and an original FC CRC, the FC frame is translated from FC to FCOE in a converter and subsequently translated back from FCOE to a FC frame in another converter, and then the FC frame is received by a FC target. When the FC frame is received by the FC target, a new FC CRC is generated and checked against the original FC CRC. If, during the translations between FCOE and FC, an error was generated, the new and original CRCs will not match, indicating the presence of an error.
Conventional FC uses a credit-based flow control mechanism for link level flow control. With link level flow control, FC primitive sequences grant credits to remote FC devices to send FC frames. The remote FC devices will consume credits as FC frames are sent. The use of credits avoids the dropping of frames under congestion.
In contrast, Ethernet conventionally does not provide for flow control, so that if an Ethernet frame is sent and the remote end does not have room for it, the frame is dropped. It is the responsibility of a higher level protocol to retransmit the Ethernet frame at a later time. However, there is an option defined in the Ethernet IEEE 802.3x standard, incorporated herein by reference, that does provide for flow control based on the concept of pause frames. The use of pause frames allows for the implementation of an on/off type flow control, wherein the transmitting device will stop sending Ethernet frames when it receives an indication that the remote device's queues are full. Credit based flow control is theoretically superior to on/off flow control in that it requires buffering equal to link bandwidth times round trip time, allowing full utilization of the link. On/off flow control requires three to five times this much buffering to achieve full link performance with no packet drops and some safety margin. In practice, however, this is not a problem. With practical Ethernet implementations, the amount of buffering required is still small enough to be easily integrated into the Ethernet interface with low hardware cost.
Therefore, in embodiments of the present invention, Ethernet pause frames are used for flow control. The use of Ethernet pause frames is advantageous because standard Ethernet switches understand the concept of pause frames, allowing the present invention to be as compatible with existing standard Ethernet infrastructure as possible. However, it should be understood that other methods of flow control such as credit-based flow control, a method of flow control well-understood by those skilled in the art, could be utilized instead of Ethernet pause frames.
Standard Ethernet frames currently have a 1500 byte payload. However, full size FC payloads are currently 2112 bytes long. If a FC frame is embedded inside a standard Ethernet frame, only a maximum of 1476 bytes would be available for the FC payload. Therefore, in embodiments of the present invention, FC frames can be restricted to the 1476 byte payload, or alternatively a full size 2112 byte FC payload can be embedded into an oversize Ethernet frame (a “jumbo frame,” according to Ethernet terminology). Jumbo frames allow FC frames to remain the same size, so they are easier to use in FCOE implementations. However, reducing the size of FC frames to 1476 bytes is not visible to higher layers, because a FC payload is typically divided into sequences, and the sequences would simply have more frames in them.
In one system-level embodiment of the present invention, an Ethernet network carrying FCOE replaces a standard FC network so that all participating devices are communicating using the FCOE protocol. The switches are Ethernet switches, additionally configured to provide FC fabric services. These fabric services may be provided by a FC fabric server. It is well-understood to those skilled in the art that some FC frames will contain a reserved address for frames requiring special handling. These addresses include 0xfffffe, 0xfffffd, and 0xfffffc. In embodiments of the present invention, a FC fabric server is implemented that claims these addresses and processes the associated frames. Other fabric services include initialization sequences with login packets and State Change Notifications (SCNs). The functionality implemented in the FC fabric server corresponds to the FC fabric services functionality traditionally implemented in FC switches.
In another system level embodiment of the present invention, devices implementing the FCOE protocol may be implemented in a blade server. The blade server would include a backplane with Ethernet (e.g. 10 Gigabit Ethernet (10 GbE)) pathways, and FCOE would be carried on the pathways. In this blade server embodiment, FCOE would represent only a short segment in the overall system.
The Ethernet switch 612 may be another card located in the back panel of the blade server chassis 604. Alternatively, the Ethernet switch 612 may be just an ASIC on another card in the blade server chassis 604. The FCOE/FC converter 616 may also be implemented by an ASIC on the same card as the Ethernet switch 612.
Therefore, in the embodiment of
It should be understood that embodiments of the present invention are based on the Ethernet protocol, not the IP protocol. Because FCOE cannot be routed based on IP addresses, only Ethernet MAC addresses, FCOE may be most advantageous on LANs. This is in contrast to iSCSI or FCIP which can be routed over the Internet. The advantage of the FCOE protocol according to embodiments of the present invention is that it is faster, simpler and easier to implement with relatively low cost Ethernet hardware. FCOE is also simpler because it doesn't implement the TCP/IP protocol, which is more complex.
In the embodiments of
Although the present invention has been fully described in connection with embodiments thereof with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the present invention as defined by the appended claims.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11514665 | Sep 2006 | US |
| Child | 14272785 | US |
