Patent Grant
11469922
The invention relates to computer networks and, more particularly, data center networks.
In a typical cloud-based data center, a large collection of interconnected servers provides computing and/or storage capacity for execution of various applications. For example, a data center may comprise a facility that hosts applications and services for subscribers, i.e., customers of the data center. The data center may, for example, host all of the infrastructure equipment, such as compute nodes, networking and storage systems, power systems and environmental control systems.
In most data centers, clusters of storage systems and application servers are interconnected via a high-speed switch fabric provided by one or more tiers of physical network switches and routers. Data centers vary greatly in size, with some public data centers containing hundreds of thousands of servers, and are usually distributed across multiple geographies for redundancy. A typical data center switch fabric includes multiple tiers of interconnected switches and routers. In current implementations, packets for a given packet flow between a source server and a destination server or storage system are always forwarded from the source to the destination along a single path through the routers and switches comprising the switching fabric.
This disclosure describes novel data center network systems. Various implementations are described in network systems may use any of a number of multiple paths that exist in a switch fabric between two servers of a data center to carry the packets for a single flow between the two servers. Example implementations of network systems for data centers are described in which a switch fabric provides full mesh interconnectivity such that any of the servers may communicate packet data for a given packet flow to any other of the servers using any of a number of parallel data paths within the data center switch fabric. For example, network architectures and techniques are described that facilitate, in example implementations, spraying of individual packets for a given packet flow across some or all of the multiple parallel data paths in the data center switch fabric and reordering of the packets for delivery to the destination. In example implementations, the techniques may provide full mesh interconnectivity between the servers of the data center and may nevertheless be non-blocking and drop-free and provide significantly better (e.g., 3× or more) switch fabric utilization compared to existing approaches.
Various data center network architectures and configurations are described. In some examples, according to the techniques described herein, access nodes coupled to the servers, intermediate permutation (e.g., electrical and/or optical) devices, and core switches of the switch fabric may be configured and arranged in such a way that the parallel data paths in the switch fabric provide full mesh (any-to-any) interconnections between any pairwise combination of the access nodes, even in massive data centers having hundreds of thousands of servers. The permutation devices couple the access nodes to the core switches by optical links to communicate the data packets between the access nodes and the core switches as optical signals. Each of the permutation devices comprises a set of input ports and a set of output ports to direct signals between the access nodes and the core switches to communicate the data packets. Each of the permutation devices is configured such that communications received from input ports are permuted across the output ports based on wavelength so as to provide full-mesh connectivity between the edge-facing ports and the core-facing ports without optical interference. In other examples, the permutation device may be replaced by an electronic switch or router. In yet other implementations, the permutation device may be used in conjunction with electronic switches and routers to interconnect even a larger number of servers.
In some example implementations, this disclosure is directed to an optical permutor that operates as an optical interconnect for transporting optical communications between network devices, such as devices within a data center. As described herein, the optical permutor provides a plurality of input ports that receive respective optical input signals, each potentially carrying optical communications at a plurality of different wavelengths. Internal optical elements of the optical permutor are configured such that optical communications received from input ports are “permutated” across output optical ports based on wavelength so as to provide full-mesh connectivity between the ports and in a manner that guarantees no optical interference due to wavelength collision. That is, the optical permutor is configured to ensure that optical communications received from any one of the optical input ports can be directed to any one of the optical output ports without optical interference with any simultaneous communications on any of the other input ports. Moreover, the optical permutor may be bi-directional. In this way, the optical permutors may provide bi-directional, full-mesh point-to-point connectivity for transporting optical communications.
In other example implementations, a system comprises a plurality of servers; a plurality of access nodes, each of the access nodes coupled to a subset of the servers to communicate data packets between the servers; and an electrical permutation device coupled to a subset of the access nodes and configured to communicate the data packets to other access nodes within the plurality of access nodes, wherein the electrical permutation device comprises a set of input ports and a set of output ports to communicate the data packets between the subset of the access nodes, wherein each of the input ports receives data packets of a plurality of packet flows that each have a unique source address for the packet flows received on the same input port, and wherein the electrical permutation device is configured to permute, based on the input ports, the plurality of packet flows received on each of the input ports across the output ports of the electrical permutation device to provide connectivity between the input ports and each of the output ports such that each output port receives a different unique permutation of the input ports and the respective source addresses of the packet flows.
In another example, this disclosure describes a method comprising: interconnecting a plurality of servers by an intermediate network comprising: a plurality of access nodes, each of the access nodes coupled to a subset of the servers to communicate data packets between the servers, an electrical permutation device coupled to a subset of the access nodes and configured to communicate the data packets to other access nodes within the plurality of access nodes, wherein the electrical permutation device comprises a set of input ports and a set of output ports to communicate the data packets between the subset of the access nodes, wherein each of the input ports receives data packets of a plurality of packet flows that each have a unique source address for the packet flows received on the same input port, and wherein the electrical permutation device is configured to permute, based on the input ports, the plurality of packet flows received on each of the input ports across the output ports of the electrical permutation device to provide connectivity between the input ports and each of the output ports such that each output port receives a different unique permutation of the input ports and the respective source addresses of the packet flows; and communicating a packet flow between the servers across the intermediate network.
The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.
In some examples, data center 10 may represent one of many geographically distributed network data centers. In the example of
In this example, data center 10 includes a set of storage systems and application servers 12 interconnected via a high-speed switch fabric 14. In some examples, servers 12 are arranged into multiple different server groups, each including any number of servers up to, for example, n servers 121-12n. Servers 12 provide computation and storage facilities for applications and data associated with customers 11 and may be physical (bare-metal) servers, virtual machines running on physical servers, virtualized containers running on physical servers, or combinations thereof.
In the example of
Although not shown, data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection, and/or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices.
In the example of
In example implementations, access nodes 17 are configurable to operate in a standalone network appliance having one or more access nodes. For example, access nodes 17 may be arranged into multiple different access node groups 19, each including any number of access nodes up to, for example, x access nodes 171-17x. As such, multiple access nodes 17 may be grouped (e.g., within a single electronic device or network appliance), referred to herein as an access node group 19, for providing services to a group of servers supported by the set of access nodes internal to the device. In one example, an access node group 19 may comprise four access nodes 17, each supporting four servers so as to support a group of sixteen servers.
In the example of
As one example, each access node group 19 of multiple access nodes 17 may be configured as standalone network device, and may be implemented as a two rack unit (2 RU) device that occupies two rack units (e.g., slots) of an equipment rack. In another example, access node 17 may be integrated within a server, such as a single 1 RU server in which four CPUs are coupled to the forwarding ASICs described herein on a mother board deployed within a common computing device. In yet another example, one or more of access nodes 17 and servers 12 may be integrated in a suitable size (e.g., 10 RU) frame that may, in such an example, become a network storage compute unit (NSCU) for data center 10. For example, an access node 17 may be integrated within a mother board of a server 12 or otherwise co-located with a server in a single chassis.
According to the techniques herein, example implementations are described in which access nodes 17 interface and utilize switch fabric 14 so as to provide full mesh (any-to-any) interconnectivity such that any of servers 12 may communicate packet data for a given packet flow to any other of the servers using any of a number of parallel data paths within the data center 10. For example, example network architectures and techniques are described in which access nodes, in example implementations, spray individual packets for packet flows between the access nodes and across some or all of the multiple parallel data paths in the data center switch fabric 14 and reorder the packets for delivery to the destinations so as to provide full mesh connectivity.
In this way, according to the techniques herein, example implementations are described in which access nodes 17 interface and utilize switch fabric 14 so as to provide full mesh (any-to-any) interconnectivity such that any of servers 12 may communicate packet data for a given packet flow to any other of the servers using any of a number of parallel data paths within the data center 10. For example, example network architectures and techniques are described in which access nodes, in example implementations, spray individual packets for packet flows between the access nodes and across some or all of the multiple parallel data paths in the data center switch fabric 14 and reorder the packets for delivery to the destinations so as to provide full mesh connectivity.
As described herein, the techniques of this disclosure introduce a new data transmission protocol referred to as a Fabric Control Protocol (FCP) that may be used by the different operational networking components of any of access nodes 17 to facilitate communication of data across switch fabric 14. As further described, FCP is an end-to-end admission control protocol in which, in one example, a sender explicitly requests a receiver with the intention to transfer a certain number of bytes of payload data. In response, the receiver issues a grant based on its buffer resources, QoS, and/or a measure of fabric congestion. In general, FCP enables spray of packets of a flow to all paths between a source and a destination node, and may provide any of the advantages and techniques described herein, including resilience against request/grant packet loss, adaptive and low latency fabric implementations, fault recovery, reduced or minimal protocol overhead cost, support for unsolicited packet transfer, support for FCP capable/incapable nodes to coexist, flow-aware fair bandwidth distribution, transmit buffer management through adaptive request window scaling, receive buffer occupancy based grant management, improved end to end QoS, security through encryption and end to end authentication and/or improved ECN marking support. More details on the FCP are available in U.S. Provisional Patent Application No. 62/566,060, filed Sep. 29, 2017, entitled “Fabric Control Protocol for Data Center Networks with Packet Spraying Over Multiple Alternate Data Paths,” the entire content of which is incorporated herein by reference.
The techniques may provide certain advantages. For example, the techniques may increase significantly the bandwidth utilization of the underlying switch fabric 14. Moreover, in example implementations, the techniques may provide full mesh interconnectivity between the servers of the data center and may nevertheless be non-blocking and drop-free.
Although access nodes 17 are described in
In some example implementations, each access node 17 may, therefore, have multiple parallel data paths for reaching any given other access node 17 and the servers 12 reachable through those access nodes. In some examples, rather than being limited to sending all of the packets of a given flow along a single path in the switch fabric, switch fabric 14 may be configured such that access nodes 17 may, for any given packet flow between servers 12, spray the packets of the packet flow across all or a subset of the M parallel data paths of switch fabric 14 by which a given destination access node 17 for a destination server 12 can be reached.
According to the disclosed techniques, access nodes 17 may spray the packets of individual packet flows across the M paths end-to-end forming a virtual tunnel between a source access node and a destination access node. In this way, the number of layers included in switch fabric 14 or the number of hops along the Mparallel data paths, may not matter for implementation of the packet spraying techniques described in this disclosure.
The technique of spraying packets of individual packet flows across all or a subset of the Mparallel data paths of switch fabric 14, however, enables the number of layers of network devices within switch fabric 14 to be reduced, e.g., to a bare minimum of one. Further, it enables fabric architectures in which the switches are not connected to each other, reducing the likelihood of failure dependence between two switches and thereby increasing the reliability of the switch fabric. Flattening switch fabric 14 may reduce cost by eliminating layers of network devices that require power and reduce latency by eliminating layers of network devices that perform packet switching. In one example, the flattened topology of switch fabric 14 may result in a core layer that includes only one level of spine switches, e.g., core switches 22, that may not communicate directly with one another but form a single hop along the Mparallel data paths. In this example, any access node 17 sourcing traffic into switch fabric 14 may reach any other access node 17 by a single, one-hop L3 lookup by one of core switches 22.
An access node 17 sourcing a packet flow for a source server 12 may use any technique for spraying the packets across the available parallel data paths, such as available bandwidth, random, round-robin, hash-based or other mechanism that may be designed to maximize, for example, utilization of bandwidth or otherwise avoid congestion. In some example implementations, flow-based load balancing need not necessarily be utilized and more effective bandwidth utilization may be used by allowing packets of a given packet flow (five tuple) sourced by a server 12 to traverse different paths of switch fabric 14 between access nodes 17 coupled to the source and destinations servers. The respective destination access node 17 associated with the destination server 12 may be configured to reorder the variable length IP packets of the packet flows and deliver the packets to the destination server in the sequence in which they were sent.
In some example implementations, each access node 17 implements at least four different operational networking components or functions: (1) a source component operable to receive traffic from server 12, (2) a source switching component operable to switch source traffic to other source switching components of different access nodes 17 (possibly of different access node groups) or to core switches 22, (3) a destination switching component operable to switch inbound traffic received from other source switching components or from cores switches 22 and (4) a destination component operable to reorder packet flows and provide the packet flows to destination servers 12.
In this example, servers 12 are connected to source components of the access nodes 17 to inject traffic into the switch fabric 14, and servers 12 are similarly coupled to the destination components within the access nodes 17 to receive traffic therefrom. Because of the full-mesh, parallel data paths provided by switch fabric 14, each source switching component and destination switching component within a given access node 17 need not perform L2/L3 switching. Instead, access nodes 17 may apply spraying algorithms to spray packets of a packet flow, e.g., available bandwidth, randomly, round-robin, based on QoS/scheduling or otherwise to efficiently forward packets without, in some examples, requiring packet analysis and lookup operations.
Destination switching components of access nodes 17 may provide a limited lookup necessary only to select the proper output port for forwarding packets to local servers 12. As such, with respect to full routing tables for the data center, only core switches 22 may need to perform full lookup operations. Thus, switch fabric 14 provides a highly-scalable, flat, high-speed interconnect in which servers 12 are, in some embodiments, effectively one L2/L3 hop from any other server 12 within the data center.
Access nodes 17 may need to connect to a fair number of core switches 22 in order to communicate packet data to any other of access nodes 17 and the servers 12 accessible through those access nodes. In some cases, to provide a link multiplier effect, access nodes 17 may connect to core switches 22 via top of rack (TOR) Ethernet switches, electrical permutation devices, or optical permutation (OP) devices (not shown in
Flow-based routing and switching over Equal Cost Multi-Path (ECMP) paths through a network may be susceptible to highly variable load-dependent latency. For example, the network may include many small bandwidth flows and a few large bandwidth flows. In the case of routing and switching over ECMP paths, the source access node may select the same path for two of the large bandwidth flows leading to large latencies over that path. In order to avoid this issue and keep latency low across the network, an administrator may be forced to keep the utilization of the network below 25-30%, for example. The techniques described in this disclosure of configuring access nodes 17 to spray packets of individual packet flows across all available paths enables higher network utilization, e.g., 85-90%, while maintaining bounded or limited latencies. The packet spraying techniques enable a source access node 17 to fairly distribute packets of a given flow across all the available paths while taking link failures into account. In this way, regardless of the bandwidth size of the given flow, the load can be fairly spread across the available paths through the network to avoid over utilization of a particular path. The disclosed techniques enable the same amount of networking devices to pass three times the amount of data traffic through the network while maintaining low latency characteristics and reducing a number of layers of network devices that consume energy.
As shown in the example of
As described, each access node group 19 may be configured as standalone network device, and may be implemented as a device configured for installation within a compute rack, a storage rack or a converged rack. In general, each access node group 19 may be configured to operate as a high-performance I/O hub designed to aggregate and process network and/or storage I/O for multiple servers 12. As described above, the set of access nodes 17 within each of the access node groups 19 provide highly-programmable, specialized I/O processing circuits for handling networking and communications operations on behalf of servers 12. In addition, in some examples, each of access node groups 19 may include storage devices 27, such as high-speed solid-state hard drives, configured to provide network accessible storage for use by applications executing on the servers. Each access node group 19 including its set of access nodes 17, storage devices 27, and the set of servers 12 supported by the access nodes 17 of that access node group may be referred to herein as a network storage compute unit (NSCU) 40.
Although access node group 19 is illustrated in
In one example implementation, access nodes 17 within access node group 19 connect to servers 52 and solid state storage 41 using Peripheral Component Interconnect express (PCIe) links 48, 50, and connect to other access nodes and the datacenter switch fabric 14 using Ethernet links 42, 44, 46. For example, each of access nodes 17 may support six high-speed Ethernet connections, including two externally-available Ethernet connections 42 for communicating with the switch fabric, one externally-available Ethernet connection 44 for communicating with other access nodes in other access node groups, and three internal Ethernet connections 46 for communicating with other access nodes 17 in the same access node group 19. In one example, each of externally-available connections 42 may be a 100 Gigabit Ethernet (GE) connection. In this example, access node group 19 has 8×100 GE externally-available ports to connect to the switch fabric 14.
Within access node group 19, connections 42 may be copper, i.e., electrical, links arranged as 8×25 GE links between each of access nodes 17 and optical ports of access node group 19. Between access node group 19 and the switch fabric, connections 42 may be optical Ethernet connections coupled to the optical ports of access node group 19. The optical Ethernet connections may connect to one or more optical devices within the switch fabric, e.g., optical permutation devices described in more detail below. The optical Ethernet connections may support more bandwidth than electrical connections without increasing the number of cables in the switch fabric. For example, each optical cable coupled to access node group 19 may carry 4×100 GE optical fibers with each fiber carrying optical signals at four different wavelengths or lambdas. In other examples, the externally-available connections 42 may remain as electrical Ethernet connections to the switch fabric.
The four remaining Ethernet connections supported by each of access nodes 17 include one Ethernet connection 44 for communication with other access nodes within other access node groups, and three Ethernet connections 46 for communication with the other three access nodes within the same access node group 19. In some examples, connections 44 may be referred to as “inter-access node group links” and connections 46 may be referred to as “intra-access node group links.”
Ethernet connections 44, 46 provide full-mesh connectivity between access nodes within a given structural unit. In one example, such a structural unit may be referred to herein as a logical rack (e.g., a half-rack or a half physical rack) that includes two NSCUs 40 having two AGNs 19 and supports an 8-way mesh of eight access nodes 17 for those AGNs. In this particular example, connections 46 would provide full-mesh connectivity between the four access nodes 17 within the same access node group 19, and connections 44 would provide full-mesh connectivity between each of access nodes 17 and four other access nodes within one other access node group of the logical rack (i.e., structural unit). In addition, access node group 19 may have enough, e.g., sixteen, externally-available Ethernet ports to connect to the four access nodes in the other access node group.
In the case of an 8-way mesh of access nodes, i.e., a logical rack of two NSCUs 40, each of access nodes 17 may be connected to each of the other seven access nodes by a 50 GE connection. For example, each of connections 46 between the four access nodes 17 within the same access node group 19 may be a 50 GE connection arranged as 2×25 GE links. Each of connections 44 between the four access nodes 17 and the four access nodes in the other access node group may include four 50 GE links. In some examples, each of the four 50 GE links may be arranged as 2×25 GE links such that each of connections 44 includes 8×25 GE links to the other access nodes in the other access node group. This example is described in more detail below with respect to
In another example, Ethernet connections 44, 46 provide full-mesh connectivity between access nodes within a given structural unit that is a full-rack or a full physical rack that includes four NSCUs 40 having four AGNs 19 and supports a 16-way mesh of access nodes 17 for those AGNs. In this example, connections 46 provide full-mesh connectivity between the four access nodes 17 within the same access node group 19, and connections 44 provide full-mesh connectivity between each of access nodes 17 and twelve other access nodes within three other access node group. In addition, access node group 19 may have enough, e.g., forty-eight, externally-available Ethernet ports to connect to the four access nodes in the other access node group.
In the case of a 16-way mesh of access nodes, each of access nodes 17 may be connected to each of the other fifteen access nodes by a 25 GE connection, for example. In other words, in this example, each of connections 46 between the four access nodes 17 within the same access node group 19 may be a single 25 GE link. Each of connections 44 between the four access nodes 17 and the twelve other access nodes in the three other access node groups may include 12×25 GE links.
As shown in
In one example, solid state storage 41 may include twenty-four SSD devices with six SSD devices for each of access nodes 17. The twenty-four SSD devices may be arranged in four rows of six SSD devices with each row of SSD devices being connected to one of access nodes 17. Each of the SSD devices may provide up to 16 Terabytes (TB) of storage for a total of 384 TB per access node group 19. As described in more detail below, in some cases, a physical rack may include four access node groups 19 and their supported servers 52. In that case, a typical physical rack may support approximately 1.5 Petabytes (PB) of local solid state storage. In another example, solid state storage 41 may include up to 32 U.2×4 SSD devices. In other examples, NSCU 40 may support other SSD devices, e.g., 2.5″ Serial ATA (SATA) SSDs, mini-SATA (mSATA) SSDs, M.2 SSDs, and the like.
In the above described example in which each of the access nodes 17 is included on an individual access node sled with local storage for the access node, each of the access node sleds may include four SSD devices and some additional storage that may be hard drive or solid state drive devices. In this example, the four SSD devices and the additional storage may provide approximately the same amount of storage per access node as the six SSD devices described in the previous example.
In one example, each of access nodes 17 supports a total of 96 PCIe lanes. In this example, each of connections 48 may be an 8×4-lane PCI Gen 3.0 connection via which each of access nodes 17 may communicate with up to eight SSD devices within solid state storage 41. In addition, each of connections 50 between a given access node 17 and the four server nodes 12 within the server 52 supported by the access node 17 may be a 4×16-lane PCIe Gen 3.0 connection. In this example, access node group 19 has a total of 256 external facing PCIe links that interface with servers 52. In some scenarios, access nodes 17 may support redundant server connectivity such that each of access nodes 17 connects to eight server nodes 12 within two different servers 52 using an 8×8-lane PCIe Gen 3.0 connection.
In another example, each of access nodes 17 supports a total of 64 PCIe lanes. In this example, each of connections 48 may be an 8×4-lane PCI Gen 3.0 connection via which each of access nodes 17 may communicate with up to eight SSD devices within solid state storage 41. In addition, each of connections 50 between a given access node 17 and the four server nodes 12 within the server 52 supported by the access node 17 may be a 4×8-lane PCIe Gen 4.0 connection. In this example, access node group 19 has a total of 128 external facing PCIe links that interface with servers 52.
Each of access node groups 19 connects to servers 52 using PCIe links 50, and to switch fabric 14 using Ethernet links 42. Access node groups 191 and 192 may each include four access nodes connected to each other using Ethernet links and local solid state storage connected to the access nodes using PCIe links as described above with respect to
In addition, each of access node groups 19 supports PCIe connections 50 to servers 52. In one example, each of connections 50 may be a 4×16-lane PCIe Gen 3.0 connection such that access node group 19 has a total of 256 externally-available PCIe links that interface with servers 52. In another example, each of connections 50 may be a 4×8-lane PCIe Gen 4.0 connection for communication between access nodes within access node group 19 and server nodes within servers 52. In either example, connections 50 may provide a raw throughput of 512 Gigabits per access node 19 or approximately 128 Gigabits of bandwidth per server node without accounting for any overhead bandwidth costs.
As discussed above with respect to
In the illustrated configuration of an 8-way mesh interconnecting two access node groups 19, each access node 17 connects via full mesh connectivity to each of the other seven access nodes in the cluster. The mesh topology between access nodes 17 includes intra-access node group links 46 between the four access nodes included in the same access node group 19, and inter-access node group links 44 between access nodes 171-174 in access node group 191 and access nodes 175-178 in access node group 192. Although illustrated as a single connection between each of access nodes 17, each of connections 44, 46 are bidirectional such that each access node connects to each other access node in the cluster via a separate link.
Each of access nodes 171-174 within first access node group 191 has three intra-access node group connections 46 to the other access nodes in first access node group 191. As illustrated in first access node group 191, access node 171 supports connection 46A to access node 174, connection 46B to access node 173, and connection 46C to access node 172. Access node 172 supports connection 46A to access node 171, connection 46D to access node 174, and connection 46E to access node 173. Access node 173 supports connection 46B to access node 171, connection 46E to access node 172, and connection 46F to access node 174. Access node 174 supports connection 46A to access node 171, connection 46D to access node 172, and connection 46F to access node 173. The access nodes 175-178 are similarly connected within second access node group 192.
Each of access nodes 171-174 within first access node group 191 also has four inter-access node group connections 44 to the access nodes 175-178 in second access node group 192. As illustrated in
Each of access nodes 17 may be configured to support up to 400 Gigabits of bandwidth to connect to other access nodes in the cluster. In the illustrated example, each of access nodes 17 may support up to eight 50 GE links to the other access nodes. In this example, since each of access nodes 17 only connects to seven other access nodes, 50 Gigabits of bandwidth may be leftover and used for managing the access node. In some examples, each of connections 44, 46 may be single 50 GE connections. In other examples, each of connections 44, 46 may be 2×25 GE connections. In still other examples, each of intra-access node group connections 46 may be 2×25 GE connections, and each of inter-access node group connections 44 may be single 50 GE connections to reduce a number of inter-box cables. For example, from each access node 171-174 within first access node group 191, 4×50 GE links go off box to connect to access nodes 175-178 in second access node group 192. In some examples, the 4×50 GE links may be taken out from each of the access nodes 17 using DAC cables.
In the illustrated example, rack 70 includes four access node groups 191-194 that are each separate network appliances 2 RU in height. Each of the access node groups 19 includes four access nodes and may be configured as shown in the example of
In this example, each of the access node groups 19 supports sixteen server nodes. For example, access node group 191 supports server nodes A1-A16, access node group 192 supports server nodes B1-B16, access node group 193 supports server nodes C1-C16, and access node group 194 supports server nodes D1-D16. A server node may be a dual-socket or dual-processor server sled that is ½Rack in width and 1 RU in height. As described with respect to
Access node groups 19 and servers 52 are arranged into NSCUs 40 from
NSCUs 40 may be arranged into logical racks 60, i.e., half physical racks, from
Logical racks 60 within rack 70 may be connected to the switch fabric directly or through an intermediate top of rack device 72. As noted above, in one example, TOR device 72 comprises a top of rack Ethernet switch. In other examples, TOR device 72 comprises an optical permutor that transports optical signals between access nodes 17 and core switches 22 and that is configured such that optical communications are “permuted” based on wavelength so as to provide full-mesh connectivity between the upstream and downstream ports without any optical interference.
In the illustrated example, each of the access node groups 19 may connect to TOR device 72 via one or more of the 8×100 GE links supported by the access node group to reach the switch fabric. In one case, the two logical racks 60 within rack 70 may each connect to one or more ports of TOR device 72, and TOR device 72 may also receive signals from one or more logical racks within neighboring physical racks. In other examples, rack 70 may not itself include TOR device 72, but instead logical racks 60 may connect to one or more TOR devices included in one or more neighboring physical racks.
For a standard rack size of 40 RU it may be desirable to stay within a typical power limit, such as a 15 kilowatt (kW) power limit. In the example of rack 70, not taking the additional 2 RU TOR device 72 into consideration, it may be possible to readily stay within or near the 15 kW power limit even with the sixty-four server nodes and the four access node groups. For example, each of the access node groups 19 may use approximately 1 kW of power resulting in approximately 4 kW of power for access node groups. In addition, each of the server nodes may use approximately 200 W of power resulting in around 12.8 kW of power for servers 52. In this example, the 40 RU arrangement of access node groups 19 and servers 52, therefore, uses around 16.8 kW of power.
In some examples, the different operational networking components of access node 17 may perform flow-based switching and ECMP based load balancing for Transmission Control Protocol (TCP) packet flows. Typically, however, ECMP load balances poorly as it randomly hashes the flows to paths such that a few large flows may be assigned to the same path and severely imbalance the fabric. In addition, ECMP relies on local path decisions and does not use any feedback about possible congestion or link failure downstream for any of the chosen paths.
The techniques described in this disclosure introduce a new data transmission protocol referred to as a Fabric Control Protocol (FCP) that may be used by the different operational networking components of access node 17. FCP is an end-to-end admission control protocol in which a sender explicitly requests a receiver with the intention to transfer a certain number of bytes of payload data. In response, the receiver issues a grant based on its buffer resources, QoS, and/or a measure of fabric congestion.
For example, the FCP includes admission control mechanisms through which a source node requests permission before transmitting a packet on the fabric to a destination node. For example, the source node sends a request message to the destination node requesting a certain number of bytes to be transferred, and the destination node sends a grant message to the source node after reserving the egress bandwidth. In addition, instead of the flow-based switching and ECMP forwarding used to send all packets of a TCP flow on the same path to avoid packet reordering, the FCP enables packets of an individual packet flow to be sprayed to all available links between a source node and a destination node. The source node assigns a packet sequence number to each packet of the flow, and the destination node uses the packet sequence numbers to put the incoming packets of the same flow in order.
SF component 30 of access node 17 is considered a source node of the fabric. According to the disclosed techniques, for FCP traffic, SF component 30 is configured to spray its input bandwidth (e.g., 200 Gbps) over links to multiple SX components of access nodes within a logical rack. For example, as described in more detail with respect to
SX component 32 of access node 17 may receive incoming packets from multiple SF components of access nodes within the logical rack, e.g., SF component 30 and seven other SF components of other access nodes within the logical rack. For FCP traffic, SX component 32 is also configured to spray its incoming bandwidth over links to multiple core switches in the fabric. For example, as described in more detail with respect to
DX component 34 of access node 17 may receive incoming packets from multiple core switches either directly or via one or more intermediate devices, e.g., TOR Ethernet switches, electrical permutation devices, or optical permutation devices. For example, DX component 34 may receive incoming packets from eight core switches, or four or eight intermediate devices. DX component 34 is configured to select a DF component to which to send the received packets. For example, DX component 34 may be connected to DF component 36 and seven other DF components of other access nodes within the logical rack. In some case, DX component 34 may become a congestion point because DX component 34 may receive a large amount of bandwidth (e.g., 200 Gbps) that is all to be sent to the same DF component. In the case of FCP traffic, DX component 34 may avoid long term congestion using the admission control mechanisms of FCP.
DF component 36 of access node 17 may receive incoming packets from multiple DX components of access nodes within the logical rack, e.g., DX component 34 and seven other DX components of other access nodes within the logical rack. DF component 36 is considered a destination node of the fabric. For FCP traffic, DF component 36 is configured to recorder packets of the same flow prior to transmitting the flow to a destination server 12.
In some examples, SX component 32 and DX component 34 of access node 17 may use the same forwarding table to perform packet switching. In this example, the personality of access node 17 and the nexthop identified by the forwarding table for the same destination IP address may depend on a source port type of the received data packet. For example, if a source packet is received from a SF component, access node 17 operates as SX component 32 and determines a nexthop to forward the source packet over the fabric toward a destination node. If a packet is received from a fabric-facing port, access node 17 operates as DX component 34 and determines a final nexthop to forward the incoming packet directly to a destination node. In some examples, the received packet may include an input tag that specifies its source port type.
As shown in
Thus, according to the disclosed techniques, upon receiving source traffic from one of servers 12, SF component 30A implemented by access node 171, for example, performs an 8-way spray of packets of the same flow across all available links to SX components 32 implemented by access nodes 17 included in logical rack 60. More specifically, SF component 30A sprays across one internal SX component 32A of the same access node 171 and seven external SX components 32B-32H of the other access nodes 172-178 within logical rack 60. In some implementations, this 8-way spray between SFs 30 and SXs 32 within logical rack 60 may be referred to as a first-stage spray. As described in other portions of this disclosure, a second-stage spray may be performed over a second-level network fanout within the switch fabric between access nodes 17 and core switches 22. For example, the second-stage spray may be performed through an intermediate device, such as a TOR Ethernet switch, an electric permutation device, or an optical permutation device, described in more detail below with respect to
In some examples, as described in more detail above, the first four access nodes 171-174 may be included in a first access node group 191 and the second four access nodes 174-178 may be included in a second access node group 192. The access nodes 17 within the first and second access node groups 19 may be connected to each other via a full-mesh in order to allow the 8-way spray between SFs 30 and SXs 32 within logical rack 60. In some examples, logical rack 60 including the two access nodes groups together with their supported servers 12 may be referred to as a half-rack or a half physical rack. In other examples, more or fewer access nodes may be connected together using full-mesh connectivity. In one example, sixteen access nodes 17 may be connected together in a full-mesh to enable a first-stage 16-way spray within a full physical rack.
According to the disclosed techniques, the switch fabric comprises a FCP fabric. The FCP fabric may be visualized as including multiple channels, e.g., a request channel, a grant channel, a FCP data channel and a non-FCP data channel. As illustrated in
The request channel within the FCP fabric may be used to carry FCP request messages from the source node to the destination node. Similar to the FCP data packets, the FCP request messages may be sprayed over all available paths toward the destination node, but the request messages do not need to be reordered. In response, the grant channel within the FCP fabric may be used to carry FCP grant messages from the destination node to source node. The FCP grant messages may also be sprayed over all available paths toward the source node, and the grant messages do not need to be reordered. The non-FCP data channel within the FCP fabric carries data packets that do not use the FCP protocol. The non-FCP data packets may be forwarded or routed using ECMP based load balancing, and, for a given flow identified by a five tuple, the packets are expected to be delivered in order to the destination node.
The example of
Upon receiving source FCP traffic from one of the servers 12, an SF component 30A of access node 171 in the first logical rack 601 performs an 8-way spray of packets of the FCP traffic flow across all available paths to SX components 32 implemented by the access nodes 17 in the first logical rack 601. As further illustrated in
Although illustrated in
According to the disclosed techniques, in one example implementation, each of SF components 30 and SX components 32 uses an FCP spray engine configured to apply a suitable load balancing scheme to spray the packets of a given FCP packet flow across all available links to a destination node. For example, the FCP spray engine may track a number of bytes transmitted on each link in order to select a least loaded link on which to forward a packet. In addition, the FCP spray engine may track link failures downstream to provide flow fairness by spraying packets in proportion to bandwidth weight on each active link. In this way, the spray of packets may not be uniform across the available links toward the destination node, but bandwidth will be balanced across the active links even over relatively short periods.
In this example, the source node, e.g., SF component 30A of access node 171, within first logical rack 601 sends a request message to the destination node, e.g., DF component 36B of access node 172, within second logical rack 602 requesting a certain weight or bandwidth and the destination node sends a grant message to the source node after reserving the egress bandwidth. The source node also determines whether any link failures have occurred between core switches 22 and logical rack 602 that includes the destination node. The source node may then use all active links in proportion to the source and destination bandwidths. As an example, assume there are N links between the source node and the destination node each with source bandwidth Sbi and destination bandwidth UN, where i=1 . . . N. The actual bandwidth from the source nodes to the destination node is equal to min(Sb, Db) determined on a link-by-link basis in order to take failures into account. More specifically, the source bandwidth (Sb) is equal to Σi=1NSbi, and destination bandwidth (Db) is equal to Σi=1NDbi, and the bandwidth (bi) of each link is equal to min(Sbi, Dbi). The weight of the bandwidth used on each link is equal to bi/Σi=1Nbi.
In the case of FCP traffic, SF components 30 and SX components 32 use the FCP spray engine to distribute packets of the FCP traffic flow based on the load on each link toward the destination node, proportion to its weight. The spray engine maintains credit memory to keep track of credits (i.e., available bandwidth) per nexthop member link, uses packet length included in an FCP header to deduct credits (i.e., reduce available bandwidth), and associates a given packet to the one of the active links having the most credits (i.e., the least loaded link). In this way, for FCP packets, the SF components 30 and SX components 32 spray packets across member links of a nexthop for a destination node in proportion to the member links' bandwidth weights.
Core switches 22 operate as the single hop along logical tunnel 100 between the source node, e.g., SF component 30A of access node 171, in first logical rack 601 and the destination node, e.g., DF component 36B of access node 172, in the second logical rack 602. Core switches 22 perform a full lookup operation for L2/L3 switching of the received packets. In this way, core switches 22 may forward all the packets for the same traffic flow toward the destination node, e.g., DF component 36B of access node 172, in the second logical rack 602 that supports the destination server 12. Although illustrated in
DX components 34 and DF components 36 of access nodes 17 within second logical rack 602 also have full mesh connectivity in that each DX component 34 is connected to all of the DF components 36 within second logical rack 602. When any of DX components 34 receive the packets of the traffic flow from core switches 22, the DX components 34 forward the packets on a direct path to DF component 36B of access node 172. DF component 36B may perform a limited lookup necessary only to select the proper output port for forwarding the packets to the destination server 12. In response to receiving the packets of the traffic flow, DF component 36B of access node 172 within second logical rack 602 reorders the packets of the traffic flow based on sequence numbers of the packets. As such, with respect to full routing tables for the data center, only the core switches 22 may need to perform full lookup operations. Thus, the switch fabric provides a highly-scalable, flat, high-speed interconnect in which servers are effectively one L2/L3 hop from any other server 12 within the data center.
A brief description of FCP and one example of its operation with respect to
As described above, FCP data packets are sent from a source node, e.g., SF component 30A of access node 171 within first logical rack 601, to a destination node, e.g., DF component 36B of access node 172 within second logical rack 602, via logical tunnel 100. Before any traffic is sent over tunnel 100 using FCP, the connection must be established between the end points. A control plane protocol executed by access nodes 17 may be used to set up a pair of tunnels, one in each direction, between the two FCP end points. The FCP tunnels are optionally secured (e.g., encrypted and authenticated). Tunnel 100 is considered to be unidirectional from the source node to the destination node, and a FCP partner tunnel may be established in the other direction from the destination node to the source node. The control plane protocol negotiates the capabilities (e.g., block size, MTU size, etc.) of both end points, and establishes the FCP connection between the end points by setting up tunnel 100 and its partner tunnel and an initializing queue state context for each tunnel.
Each of the end points is assigned a source tunnel ID and a corresponding destination tunnel ID. At each end point, a queue ID for a given tunnel queue is derived based on the assigned tunnel ID and priority. For example, each FCP end point may allocate a local tunnel handle from a pool of handles and communicate the handle to its FCP connection partner end point. The FCP partner tunnel handle is stored in a lookup table and referenced from the local tunnel handle. For the source end point, e.g., access node 171 within first logical rack 601, a source queue is identified by the local tunnel ID and priority, and a destination tunnel ID is identified from the lookup table based on the local tunnel ID. Similarly, for the destination end point, e.g., access node 172 within second logical rack 602, a destination queue is identified by the local tunnel ID and priority, and a source tunnel ID is identified from the lookup table based on the local tunnel ID.
FCP tunnel queues are defined as buckets of independent traffic streams that use FCP to transport payload across the FCP fabric. An FCP queue for a given tunnel is identified by the tunnel ID and priority, and the tunnel ID is identified by the source/destination end point pair for the given tunnel. Alternatively, the end points may use a mapping table to derive the tunnel ID and priority based on an internal FCP queue ID for the given tunnel. In some examples, an FCP fabric tunnel, e.g., logical tunnel 100, may support 1, 2, 4, or 8 queues per tunnel. The number of queues per tunnel is a FCP fabric property and may be configured at the time of deployment. All tunnels within the FCP fabric may support the same number of queues per tunnel. Each end point may support a maximum of 16,000 queues.
When the source node is communicating with the destination node, the source node encapsulates the packets using an FCP over UDP encapsulation. The FCP header carries fields identifying tunnel IDs, queue IDs, packet sequence numbers (PSNs) for packets, and request, grant, and data block sequence numbers between the two end points. At the destination node, the incoming tunnel ID is unique for all packets from the specific source node. The tunnel encapsulation carries the packet forwarding as well as the reordering information used by the destination node. A single tunnel carries packets for one or multiple queues between the source and destination nodes. Only the packets within the single tunnel are reordered based on sequence number tags that span across the queues of the same tunnel. The source node tags the packets with tunnel PSNs when they are sent over the tunnel toward the destination node. The destination node reorders the packets based on the tunnel ID and the PSNs. At the end of the reorder, the destination node strips the tunnel encapsulation and forwards the packets to the respective destination queues.
An example of how an IP packet entering FCP tunnel 100 at a source end point is transmitted to a destination end point is described here. A source server 12 having an IP address of A0 sends an IP packet for a destination server 12 having an IP address of B0. The source FCP endpoint, e.g., access node 171 within first logical rack 601, transmits an FCP request packet with source IP address A and destination IP address B. The FCP request packet has an FCP header to carry the Request Block Number (RBN) and other fields. The FCP request packet is transmitted over UDP over IP. The destination FCP end point, e.g., access node 172 within first logical rack 602, sends a FCP grant packet back to the source FCP end point. The FCP grant packet has an FCP header to carry the Grant Block Number (GBN) and other fields. The FCP grant packet is transmitted over UDP over IP. The source end point transmits the FCP data packet after receiving the FCP grant packet. The source end point appends a new (IP+UDP+FCP) data header on the input data packet. The destination end point removes the append (IP+UDP+FCP) data header before delivering the packet to the destination host server.
Furthermore, as described herein, each optical permutor 132 is configured such that optical communications received from downstream ports on each of several wavelengths 136 are “permuted” across upstream ports 138 based on wavelength so as to provide full-mesh connectivity between the upstream and downstream ports without any optical interference. That is, each optical permutor 132 is configured to ensure that optical communications received from any one of downstream servers 12 can be directed to any upstream-facing optical ports 138 without optical interference with any simultaneous communications from any other server 12. Moreover, optical permutors 132 may be bi-directional, i.e., similarly configured to permute communications from upstream ports 138 across downstream ports 136 such that no optical interference occurs on any of the downstream ports. In this way, optical permutors 132 provide bi-directional, full-mesh point-to-point connectivity for transporting communications for servers 12 to/from core switches 22 at the granularity of individual wavelengths.
For example, optical permutor 1321 is configured to optically direct optical communications from downstream-facing ports 1361-136x out upstream-facing ports 1381-138x such that each upstream port 138 carries a different one of the possible unique permutations of the combinations of downstream-facing ports 136 and the optical frequencies carried by those ports, where no single upstream-facing port 138 carries communications from servers 12 associated with the same wavelength. As such, in this example, each upstream-facing port 138 carries a non-interfering wavelength from each of the downstream facing ports 136, thus allowing a full mesh of communication. In
In this way, switch fabric 14 may provide full mesh interconnectivity such that any of servers 12 may communicate packet data to any other of the servers 12 using any of a number of parallel data paths. Moreover, according to the techniques described herein, switch fabric 14 may be configured and arranged in a way such that the parallel data paths in switch fabric 14 provides single L2/L3 hop, full mesh interconnections (bipartite graph) between servers 12, even in massive data centers having hundreds of thousands of servers. In some example implementations, each access node 17 may logically be connected to each core switch 22 and, therefore, have multiple parallel data paths for reaching any given other access node and the servers 12 reachable through those access nodes. As such, in this example, for M core switches 22, M possible data paths exist between each access node 17. Each access node 17 may be viewed as effectively directly connected to each core switch 22 (even though it is connected through an optical permutor) and thus any access node sourcing traffic into switch fabric 14 may reach any other access node 17 by a single, one-hop L3 lookup by an intermediate device (core switch).
Further example details of optical permutors are described in U.S. Provisional Appl. No. 62/478,414, filed Mar. 29, 2017, entitled “NON-BLOCKING, FULL-MESH DATA CENTER NETWORK HAVING OPTICAL PERMUTORS,” the entire contents of which are incorporated herein by reference.
In this example, network 200 represents a multi-tier network having M groups of Z physical network core switches 202A-1-202M-Z (collectively, “switches 202”) that are optically interconnected to O optical permutors 204-1-204-O (collectively, “OPs 204”), which in turn interconnect endpoints (e.g., servers 215) via Y groups of X access nodes 206A-1-206Y-X (collectively, “ANs 206”). Endpoints (e.g., servers 215) may include storage systems, application servers, compute servers, and network appliances such as firewalls and or gateways.
In the example of
Each optical permutor from OPs 204 receives light at a set of wavelengths from each of a set of multiple optical fibers coupled to the optical permutor and redistributes and outputs the wavelengths among each of another set of multiple optical fibers optically coupled to the optical permutor. Each optical permutor 204 may simultaneously input wavelengths from access nodes 206 for output to switches 202 and input wavelengths from switches 202 for output to access nodes 206.
In the example of
Network 200 may interconnect endpoints using one or more switching architectures, such as multi-tier multi-chassis link aggregation group (MC-LAG), virtual overlays, and IP fabric architectures. Each of switches 202 may represent a layer 2/layer 3 (e.g., Ethernet/IP) switch that participates in the one or more switching architectures configured for network 200 to provide point-to-point connectivity between pairs of access nodes 206. In the case of an IP fabric, each of switches 202 and access nodes 206 may execute a layer 3 routing protocol (e.g., BGP and/or OSPF) to exchange routes for subnets behind each of the access nodes 206.
In the example of
Each of access nodes 206 includes at least one optical interface to couple to a port of one of optical permutors 204. For example, access node 206A-1 is optically coupled to a port of optical permutor 204-1. As another example, access node 206A-2 is optically coupled to a port of optical permutor 204-2. In the example of
In the example of
Full mesh 220A of group 211A enables each pair of access nodes 206A-1-206A-X (“access nodes 206A”) to communicate directly with one another. Each of access nodes 206A may therefore reach each of optical permutors 204 either directly (via a direct optical coupling, e.g., access node 206A-1 with optical permutor 204-1) or indirectly via another of access nodes 206A. For instance, access node 206A-1 may reach optical permutor 204-O (and, by extension due to operation of optical permutor 204-O, switches 202M-1-202M-Z) via access node 206A-X. Access node 206A-1 may reach other optical permutors 204 via other access nodes 206A. Each of access nodes 206A therefore has point-to-point connectivity with each of switch groups 209. Access nodes 206 of groups 211B-211Y have similar topologies to access nodes 206A of group 211A. As a result of the techniques of this disclosure, therefore, each of access nodes 206 has point-to-point connectivity with each of switch groups 209.
The wavelength permutation performed by each of optical permutors 204 of permutation layer 212 may reduce a number of electrical switching operations required to perform layer 2 forwarding or layer 2/layer 3 forwarding of packets among pairs of access nodes 206. For example, access node 206A-1 may receive outbound packet data from a locally-coupled server 215 and that is destined for an endpoint associated with access node 206Y-1. Access node 206A-1 may select a particular transport wavelength on which to transmit the data on the optical link coupled to optical permutor 204-1, where the selected transport wavelength is permuted by optical permutor 204-1 as described herein for output on a particular optical link coupled to a switch of switching tier 210, where the switch is further coupled by another optical link to optical permutor 204-O. As a result, the switch may convert the optical signal of the selected transport wavelength carrying the data to an electrical signal and layer 2 or layer 2/layer 3 forward the data to the optical interface for optical permutor 204-O, which converts the electrical signal for the data to an optical signal for a transport wavelength that is permuted by optical permutor 204-O to access node 206Y-1. In this way, access node 206A-1 may transmit data to any other access node, such as access node 206Y-1, via network 200 with as few as a single intermediate electrical switching operation by switching tier 210.
In the example of
In the example of
As illustrated, the optical cables connect to the respective logical racks 60 via electro-optical circuits 226. Each logical rack 60 thus has four optical ports, one for each of electro-optical circuits 226. Electro-optical circuits 226 convert electrical signals into optical signals and convert optical signals into electrical signals. For example, in logical rack 601 of rack 701, EO1 may convert electrical signals from AN1 and AN2 of access node group 191 into optical signals for transmission over optical cable 230A to optical permutor 1321. Although not fully illustrated in
As described in detail above, each access node supports 2×100 GE connections toward the switch fabric such that a given logical rack 60 supports 16×100 GE connections from the eight access nodes AN1-AN8. The electro-optical circuits 226 within the given logical rack 60 convert the electrical signals carried on the 16×100 GE connections into optical signals for transmission over 4×400 GE optical cables to the four optical permutors 132. As an example, in logical rack 601 of physical rack 701, AN1 and AN2 together may have 4×100 GE connections for communicating to the switch fabric. Within access node group 191, the 4×100 GE connections are copper links to EO1. In some examples, these copper links may have finer granularity, e.g., 16×25 GE links. Upon converting the electrical signals received on the copper links to optical signals, EO1 sends the converted optical signals over a single 400 GE optical cable 230A to a downstream-facing port 224 of optical permutor 1321.
Each of optical permutors 132 also has sixteen upstream-facing ports 222 to connect to sixteen core switches CX1-CX1622 within a given one of switch groups 2201-2204. As described in more detail below with respect to
Each of switch groups 2201-2204 has a set of switches 22 that are each optically coupled to a same one of optical permutors 1321-1324 via a respective optical cable 223. For example, each of the upstream-facing ports 222 may support a 400 GE optical cable 223 between optical permutor 1321 and one of cores switches 22 within switch group 2201. Core switches 22 may convert optical signals received on optical cables 223 into electrical signals prior performing full lookup and switching functions on the received traffic. Prior to forwarding the traffic back to the one of optical permutors 132 via optical cables 223, core switches 22 may convert the traffic back into optical signals.
In the example of
Logical racks 60 of each of the other racks 702-708 are similarly connected to each of the four optical permutors 132 via optical cables. For example, one of optical cables 240 from logical rack 601 of rack 708 connects to the second-to-last downstream-facing port 224 of each of optical permutors 1321-1324, and one of optical cables 242 from logical rack 602 of rack 708 connects to the last downstream-facing port 224 of each of optical permutors 1321-1324. Each of the optical cables 230, 232, 240, 242 may be 400 GE optical cables. In some examples, each of the 400 GE optical cables may include four 100 GE optical fibers that each carry multiplexed optical signals having four different wavelengths or lambdas.
As described above, upon receipt of traffic from logical racks 60 on downstream-facing ports 224, optical permutors 132 spray the traffic across all upstream-facing ports 222. Optical permutors 132 then forward the traffic on upstream-facing ports 222 to each of sixteen core switches within a same switch group 220. For example, optical permutor 1321 transmits the traffic from each upstream-facing port 222 to one of core switches CX1-CX16 in switch group 2201 along optical cables 223. Each of optical cables 223 may be a 400 GE optical cable.
Optical permutors 302 operate substantially similar to optical permutors 132 described above, but have eight upstream-facing ports 322 and eight downstream-facing ports 324. In the example network cluster architecture 318 of
In the example of
Each of the optical cables 330, 332, 334, 340 may be 400 GE optical cables. In some examples, each of the 400 GE optical cables may include four 100 GE optical fibers that each carry multiplexed optical signals having four different wavelengths or lambdas. As described in detail above, each physical rack 70 includes four access node groups that each include four access nodes. Each access node supports 2×100 GE connections toward the switch fabric such that a given physical rack 70 supports 32×100 GE connections from the sixteen access nodes. Electro-optical circuits within the given physical rack 70 convert the electrical signals carried on the 32×100 GE connections into optical signals for transmission over 8×400 GE optical cables to the eight optical permutors 302.
As described above, upon receipt of traffic from racks 70 on downstream-facing ports 324, optical permutors 302 spray the traffic across all upstream-facing ports 322. Optical permutors 302 then forward the traffic on upstream-facing ports 322 to each of eight core switches within a same switch group 320. For example, optical permutor 3021 transmits the traffic from each upstream-facing port 322 to one of core switches CX1-CX8 in switch group 3201 along optical cables 323. Each of optical cables 323 may be a 400 GE optical cable.
The example network cluster architectures 218, 318 illustrated in
Each of clusters 350 includes a plurality of core switches 22 and a plurality of access nodes 17 that are each coupled to a plurality of servers 121-12n. Although not shown in
As illustrated in
Border gateway devices 354 enable clusters 350 to connect to each other and to the outside world via a level of routers 356 and service provider network 360, e.g., the Internet or another public WAN network. For example, border gateway devices 354 may be substantially similar to gateway device 20 from
In the illustrated example of
In one example of a packet being forwarded between cluster 350A and 350B, at time T0, an access node 17A within cluster 350A sends the packet to access node 352A over one of multiple links across fabric 14A. At time T1, access node 352A receives the packet from fabric 14A. At time T2, access node 352A sends the packet to border gateway device 354A over the PCIe connection. Border gateway device 354A performs a forwarding lookup on the destination IP address of the packet, and determine that the packet is destined for cluster 350B behind border gateway device 354A. At time T3, border gateway 354A within cluster 350A sends the packet to border gateway device 354B within cluster 350B over one of multiple links across service provider network 360. For example, border gateway device 354A may send the packet over one of multiple paths to routers 356A, 356B using either the packet spraying techniques described in this disclosure or ECMP. At time T4, border gateway device 354B within cluster 350B receives the packet from routers 356A, 356B. Border gateway device 354B performs a forwarding lookup on the destination IP address of the packet, and sends the packet to access node 352B over the PCIe connection. At time T5, access node 352B sends the packet to access node 17B within cluster 350B over one of multiple links across fabric 14B. At time T6, access node 17B receives the packet from fabric 14B. Access node 17B performs a forwarding lookup on the destination IP address of the packet, and sends the packet to one of servers 12 at time T7.
In the example of
For example, in
The following provides a complete example for one implementation of optical permutor 400 of
Further, each of the four optical fiber pairs for each of input ports P1-P16 is coupled to a different access node 17, thereby providing bidirectional optical connectivity from 64 different access nodes.
Table 2 lists one example configuration for optical permutor 400 for optical communications in the core-facing direction. That is, Table 2 illustrates an example configuration of optical permutor 400 for producing, on the optical fibers of core-facing output ports P17-P32, a set of 64 unique permutations for combinations of optical input ports P1-P16 and optical wavelengths L1-L4 carried by those input ports, where no single optical output port carries multiple optical communications having the same wavelength. For example, the first column of Table 2 lists the wavelengths L1-L4 carried by the four fibers F1-F4 of each input optical interfaces for ports P0-P16 while the right column lists the unique and non-interfering permutation of input port fiber/wavelength combination output on each optical output interface of ports P17-P32.
Continuing the example, Table 3 lists an example configuration for optical permutor 400 with respect to optical communications in the reverse, downstream direction, i.e., from core switches 22 to access nodes 17. That is, Table 3 illustrates an example configuration of optical permutor 400 for producing, on the optical fibers of rack-facing output ports P1-P16, a set of 64 unique permutations for combinations of core-facing input ports P16-P32 and optical wavelengths L1-L4 carried by those input ports, where no single optical output port carries multiple optical communications having the same wavelength.
Table 4 lists a second example configuration for optical permutor 400 for optical communications in the core-facing direction. As with Table 2 above, Table 4 illustrates an example configuration of optical permutor 400 for producing, on the optical fibers of core-facing output ports P17-P32, a set of 64 unique permutations for combinations of optical input ports P1-P16 and optical wavelengths L1-L4 carried by those input ports, where no single optical output port carries multiple optical communications having the same wavelength. Similar to Table 2 above, the first column of Table 4 lists the wavelengths L1-L4 carried by the four fibers F1-F4 of each input optical interfaces for ports P0-P16 while the right column lists another example of unique and non-interfering permutation of input port fiber/wavelength combination output on each optical output interface of ports P17-P32.
Continuing the example, Table 5 lists another example configuration for optical permutor 400 with respect to optical communications in the reverse, downstream direction, i.e., from core switches 22 to access nodes 17. Like Table 3 above, Table 5 illustrates another example configuration of optical permutor 400 for producing, on the optical fibers of rack-facing output ports P1-P16, a set of 64 unique permutations for combinations of core-facing input ports P16-P32 and optical wavelengths L1-L4 carried by those input ports, where no single optical output port carries multiple optical communications having the same wavelength.
Optical permutor 500 includes a respective one of optical demultiplexers 600A-600N (herein, “optical demuxes 600”) for each optical input interface 520, where the optical demultiplexer is configured to demultiplex the optical communications for a given optical input onto internal optical pathways 640 based on the bandwidth of the optical communications. For example, optical demux 600A separates the optical communications received on optical input interface 520A onto a set of internal optical pathways 640A based on wavelengths λ1,1, λ1,2, . . . λ1,n. Optical demux 600B separates the optical communications received on optical input interface 520B onto a set of internal optical pathways 640B based on wavelengths λ2,1, λ2,2, . . . λ2,n. Each optical demux 600 operates in a similar fashion to separate the optical communications received from the receptive input optical interface 520 so as to direct the optical communications through internal optical pathways 640 toward optical output ports 540A-540N (herein, “optical output ports 540”).
Optical permutor 500 includes a respective one of optical multiplexers 620A-620N (herein, “optical muxes 620”) for each optical output port 540, where the optical multiplexer receives as input optical signals from optical pathways 640 that lead to each optical demux 600. In other words, optical pathways 640 internal to optical permutor 500 provide a full-mesh of N2 optical interconnects between optical demuxes 600 and optical muxes 620. Each optical multiplexer 620 receives N optical pathways as input and combines the optical signals carried by the N optical pathways into a single optical signal for output onto a respective optical fiber.
Moreover, optical demuxes 600 are each configured such that optical communications received from input interface ports 520 are “permuted” across optical output ports 540 based on wavelength so as to provide full-mesh connectivity between the ports and in a way that ensures optical interference is avoided. That is, each optical demux 600 is configured to ensure that each optical output port 54 receives a different one of the possible unique permutations of the combinations of optical input ports 520 and the optical frequencies carried by those ports and where no single optical output port 540 carries communications having the same wavelength.
For example, optical demux 600A may be configured to direct the optical signal having wavelength λ1,1 to optical mux 620A, wavelength λ1,2 to optical mux 620B, wavelength λ1,3 to optical mux 620C, . . . and wavelength λ1,n to optical mux 620N. Optical demux 600B is configured to deliver a different (second) permutation of optical signals by outputting wavelength λ2,n to optical mux 620A, wavelength λ2,1 to optical mux 620B, wavelength λ2,2 to optical mux 620C, . . . and wavelength λ2,n−1 to optical mux 620N. Optical demux 600C is configured to deliver a different (third) permutation of optical signals by outputting wavelength λ3,n−1 to optical mux 620A, wavelength λ3,n−2 to optical mux 620B, wavelength λ3,n−3 to optical mux 620C, . . . and wavelength λ3,n−2 to optical mux 620N. This example configuration pattern continues through optical demux 600N, which is configured to deliver a different (Nth) permutation of optical signals by outputting wavelength λN,2 to optical mux 620A, wavelength λN,3 to optical mux 620B, wavelength λN,4 to optical mux 620C, . . . and wavelength λN,1 to optical mux 620N.
In the example implementation, optical pathways 640 are arranged such that the different permutations of input interface/wavelengths are delivered to optical muxes 620. In other words, each optical demux 600 may be configured to operate in a similar manner, such as λ1 being provided to a first port of the demux, λ2 being provided to a second port of the demux . . . , and λn, being provided to an Nth port of the demux. Optical pathways 640 are arranged to optically deliver a specific permutation of wavelengths to each optical mux 620 such that any communications from any one of optical demuxes 600 can reach any optical mux 620 and, moreover, each permutation of wavelengths is selected to avoid any interference between the signals, i.e., be non-overlapping.
For example, as shown in
In this way, a different permutation of input optical interface/wavelength combination is provided to each optical mux 620 and, moreover, each one of the permutations provided to the respective optical mux is guaranteed to include optical communications having non-overlapping wavelengths.
Optical permutor 500 illustrates one example implementation of the techniques described herein. In other example implementations, each optical interface 520 need not receive all N wavelengths from a single optical fiber. For example, different subsets of the N wavelengths can be provided by multiple fibers, which would then be combined (e.g., by a multiplexer) and subsequently permuted as described herein. As one example, optical permutor 500 may have 2N optical inputs 520 so as to receive 2N optical fibers, where a first subset of N optical fibers carries wavelengths λ1 . . . λn/2 and a second subset of N optical fibers carries wavelengths λn/2+1 . . . λn Light from pairs of the optical inputs from the first and second set may be combined to form optical inputs carrying N wavelengths, which may then be permuted as shown in the example of
In the example implementation, optical permutor 500, including optical input ports 520, optical demuxes 600, optical pathways 640, optical muxes 620 and optical output ports 540 may be implemented as one or more application specific integrated circuit (ASIC), such as a photonic integrated circuit or an integrated optical circuit. In other words, the optical functions described herein may be integrated on a single chip, thereby providing an integrated optical permutor that may be incorporated into electronic cards, devices and systems.
In the example implementation of
In other examples, optical permutors 132, 400, 500 may make use of star couplers and waveguide grating routers described in Kaminow, “Optical Integrated Circuits: A Personal Perspective,” Journal of Lightwave Technology, vol. 26, no. 9, May 1, 2008, the entire contents of which are incorporated herein by reference.
In some example implementations, intra-access node group permutators may be used to provide multiplexed packet spraying between access nodes, which may further increase the scalability of the network systems described herein. That is, in some examples, one or more first stage permutation devices may be coupled between access nodes of the same access node group to increase the fan out as individual access nodes spray packets for packet flows across other access nodes within the same access node group.
In general, electrical permutor 2000 permutes packet-based communications (e.g., packet flows) from a set of input ports 2020 across a set of output ports 2040. As further explained, electrical permutor 2000 is an interconnect device that multiplexes packet-based communications between access nodes, such as access nodes of a common access node group that service one or more racks of servers. Source components (SF) 2005 and source switching components (SX) 2007 of the access nodes of access node group 2001 are shown in the example of
In this way, the SF component for each access node may be configured to spray packets it receives from servers toward all other access nodes in access node group 2001 such that the packets transmitted to electrical permutor 2000 from any given access node over any link 2002 is a mixture of packets from any of the servers supported by the respective access node. As described in further detail below, electrical permutor 2000 operates to provide seamless connectivity from any of the source components (SFs) to any of the source switching components (SXs) within access node group 2001. Moreover, use of electrical permutor 2000 may achieve significantly higher fanout from an individual SF to an increased number of SXs, thus effectively increasing the size of access node group 2001, without increasing the requirement for additional cabling or network ports.
For example, in the example of
More specifically, example electrical permutor 2000 includes a plurality of input ports 2020A-2020P (herein, “input ports 2020”) to receive electrical signals from a respective access node, where each of the electrical signals for the different links carries packets having up to n unique source network addresses from the transmitting SF component of the respective access node. In this way, communications received by electrical permutor 2000 on each of links 2002 may be viewed as n unique electrical streams of packets designated as ESp,n, where the subscript p represents one of the P inputs ports and the subscript n represents the source network address used by the transmitting access node that initially received the packet from one of the servers and is spraying the packet toward the SXs of the access node group. Thus, using this nomenclature, input 2020A receives n electrical streams (ES) carrying variable-sized packets sprayed across the access node group(s), where the unique fixed-bandwidth electrical streams flowing through input port 2020A utilize n different source network addresses and are designated as electrical streams ES1,1, ES1,2, . . . ES1,n. Similarly, input 2020B receives n electrical streams carrying variable-sized packets sprayed across the access node group, where the different electrical streams are designated ES2,1, ES2,2, . . . ES2,n. This pattern continues for all P input ports connected access nodes, where each access node is assigned n different network addresses for use as source network addresses.
As shown in
For example, in
In this way, electrical permutor 2000 is configured to direct communications from input ports 2020 out output ports 2040 such that each output port 2040 carries a different one of the possible unique permutations of the combinations of input ports 2020 and the electrical streams carried by those ports for the different network addresses used by the access nodes of the access node group, where each output port 2020 carries one of the electrical streams from every one of the access nodes and for each of the different input ports. As such, in this example, each output port 2040 carries an electrical stream of packet flows from each of the access nodes of the access node group, thus allowing a full mesh of communication. Electrical permutor 2000 may be used to achieve enhanced fanout in a first level of packet spraying within an access node group prior to a first layer three (L3) lookup and forwarding of the packets by the SX component of the access nodes within the access node group, and in some example implementations may be deployed in conjunction with any of the optical permutors described herein (e.g., OPs 132, OPs 204, OPs 302, OP 400 and OP 500) to provide bi-directional, full-mesh point-to-point connectivity for transporting communications for servers 12 to/from core switches 22. In some examples, electrical permutor 2000 has only a minimal packet buffer to absorb any collision of packets from multiple input ports simultaneously, and, in some examples, no output port 2040 is oversubscribed for bandwidth.
As such, an access network or switch fabric 14 (see
As described above, network implementations using the electrical permutor may be useful so as to increase the radix (fanout) of any access node group. For example, large network implementations may easily be achieved, such as easily achieving larger access nodes groups (e.g., 32, 64 or 128) to support large number of racks (e.g., 1024 racks) as one example.
In the example of
In general, each EP operates similar to the functionality described in reference to EP 2000 of
In general, an example packet originating from a server is received by a respective access node (step 1). The SF component of the access node receiving the outbound packet sprays the packet to any other of the access nodes in the access node group (step 2). At this time, the access node selects the appropriate source MAC address to be used when encapsulating the packet within a tunnel packet and forwards the packet toward one of the electrical permutors (step 3).
Upon receiving the packet from the source access node, the EP multiplexes the packet to its appropriate output port based on the low order bits of the source network address (e.g., MAC address) (step 4). The receiving EP delivers packet directly to its corresponding port without permutation (step 5). For example, for the EP receiving packets from the transmitting EP, all electrical streams of packets received on input port 0 would be delivered to output port 0, all electrical streams for packets received on input port 1 would be delivered to output port 1, and the like. In this way, the pair of EP's mimic direct connection between access nodes.
Upon receiving the sprayed packet, the SX component of the receiving access node within the access node group performs a layer 3 lookup on the destination IP address for the tunneled packet to determine a destination access node for terminating the transport of the packet (step 6). If the access node receiving the sprayed packet is the destination access node such that the destination server is a local server to that access node, the receiving access nodes performs packet extraction from the tunnel packet, packet reorder and delivery to the destination server (step 7). If the destination access node is a different access node located within Rack Group 0 (i.e., within the same access node group), the access node forwards the packet to the local access node (e.g., via the EP's as discussed herein) for packet extraction, packet reorder and delivery to the destination server (step 8). Otherwise, the access node forwards the packet toward the Rack Group having the destination access node by way of one of the core switches (step 9), for packet reordering and delivery to the destination server (step 10).
The example described with respect to
In general, the Ingress Port Block receives input communications, such as Ethernet packets, and implements an Ethernet MAC with Physical Coding Sublayer and Forward Error Correction code. Upon receiving an Ethernet frame, the Ingress Port Block sends the received ethernet frame to the parser block for parsing purpose.
The parser block parses the incoming Ethernet frame and is configured to determine the egress port to forward the frame in accordance with the permutation requirements described herein.
The Packet Writer block of the electrical permutor writes incoming Ethernet frames in the Packet Buffer. Hardware logic manages the buffer in fixed size cell units of 256 bytes. The Packet Writer Block interfaces with a Free Cell Manager to obtain addresses of free cells within the Packet Buffer. The Packet Writer Block uses the free cell addresses received from the Free Cell Manager to write incoming frames for temporary storage.
The Packet Write Arbiter provides a controlled interface to the Packet Buffer and multiplexes accesses from the different packet writers. For example, the Packet Write Arbiter arbitrates the packet buffer address port and multiplexes incoming cells to be written to packet buffer memory. The Packet Write Arbiter creates a packet in form of list of cells within the Packet Buffer and enqueues the cell addresses in the Egress Packet Queue block for subsequent transmission.
In one example, the Packet Buffer has a limited size to buffer (n−1)*n*packet MTU size to guarantee zero packet loss, where n represents the number of outputs ports. In the case where a shared buffer is used, as shown in
The Egress Packet Queue block implements a set of (e.g., 8 or 16) packet queues for each egress port of electrical permutor 2150. Packet lists are implemented in form of link-lists. The egress packet queue structure shares the storage.
An Egress Scheduler accesses the cell addresses to be popped from the Egress Packet Queues and schedules the packet from non-empty queues for transmission out the corresponding egress port. The Egress Scheduler sends the packet with its description information to the corresponding packet reader block for reading the corresponding packet from the Packet Buffer.
For example, the Packet Reader block of electrical permutor 2150 has responsibility to traverse over the link-list of the packet to be transmitted and create addresses to the packet buffer to read the cell information. After reading the data from the packet buffer, the Packet Reader block sends the cell address to free cell manager to free the packet buffer space previously occupied by the packet.
The Packet Read Address Mux multiplexes different read addresses and issue a read command to the Packet Buffer block for reading the packet data. The packet data read from the packet buffer is sent to the egress port group, which transmits the packet data with appropriate Ethernet CRC.
In the example of
As shown in this example, a set of access nodes 17 exchange control plane messages to establish a logical tunnel over a plurality of parallel network paths that provide packet-based connectivity between the access nodes (3101). For example, with respect to
Once the logical tunnel is established, one of the access nodes (referred to as the ‘transmitting access node’ in
Upon receiving a grant from the access node associated with a destination of the packet data to be transmitted (3108), the transmitting access nodes encapsulates the outbound packets within tunnel packets, thereby forming each tunnel packet to have a tunnel header for traversing the logical tunnel and a payload containing one or more of the outbound packets (3104).
Upon forming the tunnel packets, the transmitting access node forwards the tunnel packets by spraying the tunnel packets over the multiple, parallel paths through switch fabric 14 by which the receiving access node is reachable (3106). In some example implementations, the transmitting access node may, prior to forwarding the tunnel packets to switch fabric 14, spray the tunnel packets across multiple access nodes that, for example, form one or more access node groups (e.g., within one or more rack groups proximate to the transmitting access node, as one example), thereby providing a first level fanout for distributing the tunnel packets across the parallel paths. See, for example,
Upon receipt, the receiving access node extracts the original packets that are encapsulated within the tunnel packets (3109), reorders the original packets in the order sent by the application or storage server, and delivers the packets to the intended destination server (3110).
This application is a continuation application of and claims priority to U.S. patent application Ser. No. 15/939,041 filed on Mar. 28, 2018, now U.S. Pat. No. 10,637,685, which claims the benefit of U.S. Provisional Appl. No. 62/557,988, filed Sep. 13, 2017, U.S. Provisional Appl. No. 62/514,583, filed Jun. 2, 2017, and U.S. Provisional Appl. No. 62/478,414, filed Mar. 29, 2017, the entire contents of all of these applications is hereby incorporated herein by reference
| Number | Name | Date | Kind |
|---|---|---|---|
| 4872157 | Hemmady et al. | Oct 1989 | A |
| 4872159 | Hemmady et al. | Oct 1989 | A |
| 5301324 | Dewey et al. | Apr 1994 | A |
| 5812549 | Sethu | Sep 1998 | A |
| 5828860 | Miyaoku et al. | Oct 1998 | A |
| 6021473 | Davis et al. | Feb 2000 | A |
| 6055579 | Goyal et al. | Apr 2000 | A |
| 6314491 | Freerksen et al. | Nov 2001 | B1 |
| 6842906 | Bowman-Amuah | Jan 2005 | B1 |
| 6901451 | Miyoshi et al. | May 2005 | B1 |
| 6901500 | Hussain | May 2005 | B1 |
| 6993630 | Williams et al. | Jan 2006 | B1 |
| 7035914 | Payne et al. | Apr 2006 | B1 |
| 7275103 | Thrasher et al. | Sep 2007 | B1 |
| 7289964 | Bowman-Amuah et al. | Oct 2007 | B1 |
| 7342887 | Sindhu | Mar 2008 | B1 |
| 7480304 | Yeh et al. | Jan 2009 | B2 |
| 7486678 | Devanagondi et al. | Feb 2009 | B1 |
| 7664110 | Lovett et al. | Feb 2010 | B1 |
| 7733781 | Petersen | Jun 2010 | B2 |
| 7822731 | Yu et al. | Oct 2010 | B1 |
| 7843907 | Abou-Emara et al. | Nov 2010 | B1 |
| 7965624 | Ripa et al. | Jun 2011 | B2 |
| 8560757 | Pangbom et al. | Oct 2013 | B2 |
| 8582440 | Ofelt et al. | Nov 2013 | B2 |
| 8599863 | Davis | Dec 2013 | B2 |
| 8625427 | Terry et al. | Jan 2014 | B1 |
| 8737410 | Davis et al. | May 2014 | B2 |
| 8798077 | Mehra et al. | Aug 2014 | B2 |
| 8848728 | Revah et al. | Sep 2014 | B1 |
| 8850101 | Pangborn et al. | Sep 2014 | B2 |
| 8850125 | Pangborn et al. | Sep 2014 | B2 |
| 8918631 | Kumar et al. | Dec 2014 | B1 |
| 8966152 | Bouchard et al. | Feb 2015 | B2 |
| 9065860 | Pangborn et al. | Jun 2015 | B2 |
| 9118984 | DeCusatis et al. | Aug 2015 | B2 |
| 9154376 | Aziz | Oct 2015 | B2 |
| 9225628 | Zahavi | Dec 2015 | B2 |
| 9262225 | Davis et al. | Feb 2016 | B2 |
| 9282384 | Graves | Mar 2016 | B1 |
| 9294304 | Sindhu | Mar 2016 | B2 |
| 9294398 | DeCusatis et al. | May 2016 | B2 |
| 9369408 | Raghavan et al. | Jun 2016 | B1 |
| 9405550 | Biran et al. | Aug 2016 | B2 |
| 9565114 | Kabbani et al. | Feb 2017 | B1 |
| 9569366 | Pangborn et al. | Feb 2017 | B2 |
| 9632936 | Zuckerman et al. | Apr 2017 | B1 |
| 9800495 | Lu | Oct 2017 | B2 |
| 9853901 | Kampmann et al. | Dec 2017 | B2 |
| 9866427 | Yadav et al. | Jan 2018 | B2 |
| 9876735 | Davis et al. | Jan 2018 | B2 |
| 9946671 | Tawri et al. | Apr 2018 | B1 |
| 10003552 | Kumar et al. | Jun 2018 | B2 |
| 10135731 | Davis et al. | Nov 2018 | B2 |
| 10140245 | Davis et al. | Nov 2018 | B2 |
| 10304154 | Appu et al. | May 2019 | B2 |
| 10387179 | Hildebrant et al. | Aug 2019 | B1 |
| 10425707 | Sindhu et al. | Sep 2019 | B2 |
| 10540288 | Noureddine et al. | Jan 2020 | B2 |
| 10565112 | Noureddine et al. | Feb 2020 | B2 |
| 10637685 | Goel et al. | Apr 2020 | B2 |
| 10645187 | Goyal et al. | May 2020 | B2 |
| 10659254 | Sindhu et al. | May 2020 | B2 |
| 10686729 | Sindhu et al. | Jun 2020 | B2 |
| 10725825 | Sindhu et al. | Jul 2020 | B2 |
| 10904367 | Goel et al. | Jan 2021 | B2 |
| 20020015387 | Houh | Feb 2002 | A1 |
| 20020049859 | Bruckert et al. | Apr 2002 | A1 |
| 20020075862 | Mayes | Jun 2002 | A1 |
| 20020094151 | Li | Jul 2002 | A1 |
| 20020118415 | Dasylva et al. | Aug 2002 | A1 |
| 20020126634 | Mansharamani et al. | Sep 2002 | A1 |
| 20020126671 | Ellis et al. | Sep 2002 | A1 |
| 20030043798 | Pugel | Mar 2003 | A1 |
| 20030091271 | Dragone | May 2003 | A1 |
| 20030229839 | Wang et al. | Dec 2003 | A1 |
| 20040236912 | Glasco | Nov 2004 | A1 |
| 20050166086 | Watanabe | Jul 2005 | A1 |
| 20050259632 | Malpani et al. | Nov 2005 | A1 |
| 20060029323 | Nikonov et al. | Feb 2006 | A1 |
| 20060056406 | Bouchard et al. | Mar 2006 | A1 |
| 20060112226 | Hady et al. | May 2006 | A1 |
| 20060277421 | Balestriere | Dec 2006 | A1 |
| 20070073966 | Corbin | Mar 2007 | A1 |
| 20070172235 | Snider et al. | Jul 2007 | A1 |
| 20070192545 | Gara et al. | Aug 2007 | A1 |
| 20070198656 | Mazzaferri et al. | Aug 2007 | A1 |
| 20070255849 | Zheng | Nov 2007 | A1 |
| 20070255906 | Handgen et al. | Nov 2007 | A1 |
| 20080002702 | Bajic | Jan 2008 | A1 |
| 20080138067 | Beshai | Jun 2008 | A1 |
| 20080244231 | Kunze et al. | Oct 2008 | A1 |
| 20090024836 | Shen et al. | Jan 2009 | A1 |
| 20090083263 | Felch et al. | Mar 2009 | A1 |
| 20090135832 | Fan et al. | May 2009 | A1 |
| 20090228890 | Vaitovirta et al. | Sep 2009 | A1 |
| 20090234987 | Lee et al. | Sep 2009 | A1 |
| 20090285228 | Bagepalli et al. | Nov 2009 | A1 |
| 20090303880 | Maltz et al. | Dec 2009 | A1 |
| 20100061391 | Sindhu et al. | Mar 2010 | A1 |
| 20100125903 | Devarajan et al. | May 2010 | A1 |
| 20100290458 | Assarpour | Nov 2010 | A1 |
| 20100318725 | Kwon | Dec 2010 | A1 |
| 20110289179 | Pekcan et al. | Jan 2011 | A1 |
| 20110055827 | Lin et al. | Mar 2011 | A1 |
| 20110113184 | Chu | Mar 2011 | A1 |
| 20110170553 | Beecroft et al. | Jul 2011 | A1 |
| 20110173392 | Gara et al. | Jul 2011 | A1 |
| 20110202658 | Okuno et al. | Aug 2011 | A1 |
| 20110225594 | Iyengar et al. | Sep 2011 | A1 |
| 20110228783 | Flynn et al. | Sep 2011 | A1 |
| 20110238923 | Hooker et al. | Sep 2011 | A1 |
| 20110289180 | Sonnier et al. | Nov 2011 | A1 |
| 20110289279 | Sonnier et al. | Nov 2011 | A1 |
| 20120030431 | Anderson et al. | Feb 2012 | A1 |
| 20120036178 | Gavini et al. | Feb 2012 | A1 |
| 20120096211 | Davis et al. | Apr 2012 | A1 |
| 20120176890 | Balus et al. | Jul 2012 | A1 |
| 20120207165 | Davis | Aug 2012 | A1 |
| 20120254587 | Biran et al. | Oct 2012 | A1 |
| 20120314710 | Shikano | Dec 2012 | A1 |
| 20130003725 | Hendel et al. | Jan 2013 | A1 |
| 20130024875 | Wang et al. | Jan 2013 | A1 |
| 20130028083 | Yoshida et al. | Jan 2013 | A1 |
| 20130088971 | Anantharam et al. | Apr 2013 | A1 |
| 20130145375 | Kang | Jun 2013 | A1 |
| 20130191443 | Gan et al. | Jul 2013 | A1 |
| 20130258912 | Zimmerman et al. | Oct 2013 | A1 |
| 20130330076 | Liboiron-Ladouceur et al. | Dec 2013 | A1 |
| 20130346789 | Brunel et al. | Dec 2013 | A1 |
| 20140023080 | Zhang et al. | Jan 2014 | A1 |
| 20140040909 | Winser et al. | Feb 2014 | A1 |
| 20140044128 | Suresh et al. | Feb 2014 | A1 |
| 20140059537 | Kamble et al. | Feb 2014 | A1 |
| 20140075085 | Schroder et al. | Mar 2014 | A1 |
| 20140161450 | Graves et al. | Jun 2014 | A1 |
| 20140187317 | Kohler et al. | Jul 2014 | A1 |
| 20140258479 | Tenginakai et al. | Sep 2014 | A1 |
| 20140269351 | Graves et al. | Sep 2014 | A1 |
| 20140310467 | Shalf et al. | Oct 2014 | A1 |
| 20140359044 | Davis et al. | Dec 2014 | A1 |
| 20150009994 | Keesara | Jan 2015 | A1 |
| 20150019702 | Kancherla | Jan 2015 | A1 |
| 20150037032 | Xu et al. | Feb 2015 | A1 |
| 20150043330 | Hu et al. | Feb 2015 | A1 |
| 20150117860 | Braun | Apr 2015 | A1 |
| 20150143045 | Han et al. | May 2015 | A1 |
| 20150143073 | Winser et al. | May 2015 | A1 |
| 20150163171 | Sindhu et al. | Jun 2015 | A1 |
| 20150180603 | Darling et al. | Jun 2015 | A1 |
| 20150186313 | Sodhi et al. | Jul 2015 | A1 |
| 20150222533 | Birrittella et al. | Aug 2015 | A1 |
| 20150244617 | Nakil et al. | Aug 2015 | A1 |
| 20150256405 | Janardhanan et al. | Sep 2015 | A1 |
| 20150278148 | Sindhu | Oct 2015 | A1 |
| 20150278984 | Koker et al. | Oct 2015 | A1 |
| 20150280939 | Sindhu | Oct 2015 | A1 |
| 20150281128 | Sindhu | Oct 2015 | A1 |
| 20150324205 | Eisen et al. | Nov 2015 | A1 |
| 20150325272 | Murphy | Nov 2015 | A1 |
| 20150334034 | Smedley et al. | Nov 2015 | A1 |
| 20150334202 | Frydman et al. | Nov 2015 | A1 |
| 20150378776 | Lippett | Dec 2015 | A1 |
| 20150381528 | Davis et al. | Dec 2015 | A9 |
| 20160056911 | Ye et al. | Feb 2016 | A1 |
| 20160062800 | Stanfill et al. | Mar 2016 | A1 |
| 20160092362 | Barron et al. | Mar 2016 | A1 |
| 20160164625 | Gronvall et al. | Jun 2016 | A1 |
| 20160210159 | Wilson et al. | Jul 2016 | A1 |
| 20160239415 | Davis et al. | Aug 2016 | A1 |
| 20160241430 | Yadav et al. | Aug 2016 | A1 |
| 20160337723 | Graves | Nov 2016 | A1 |
| 20160364333 | Brown et al. | Dec 2016 | A1 |
| 20160364334 | Asaro et al. | Dec 2016 | A1 |
| 20160380885 | Jani et al. | Dec 2016 | A1 |
| 20170005921 | Liu et al. | Jan 2017 | A1 |
| 20170031719 | Clark et al. | Feb 2017 | A1 |
| 20170032011 | Song et al. | Feb 2017 | A1 |
| 20170060615 | Thakkar et al. | Mar 2017 | A1 |
| 20170061566 | Min et al. | Mar 2017 | A1 |
| 20170068639 | Davis et al. | Mar 2017 | A1 |
| 20170235581 | Nickolls et al. | Aug 2017 | A1 |
| 20170265220 | Andreoli-Fang et al. | Sep 2017 | A1 |
| 20170286143 | Wagner et al. | Oct 2017 | A1 |
| 20170286157 | Hasting et al. | Oct 2017 | A1 |
| 20170346766 | Dutta | Nov 2017 | A1 |
| 20180011739 | Pothula et al. | Jan 2018 | A1 |
| 20180024771 | Miller et al. | Jan 2018 | A1 |
| 20180026901 | Sugunadass | Jan 2018 | A1 |
| 20180115494 | Bhatia et al. | Apr 2018 | A1 |
| 20180152317 | Chang et al. | May 2018 | A1 |
| 20180234300 | Mayya et al. | Aug 2018 | A1 |
| 20180239702 | Farmahini Farahani et al. | Aug 2018 | A1 |
| 20180287818 | Goel et al. | Oct 2018 | A1 |
| 20180287965 | Sindhu | Oct 2018 | A1 |
| 20180288505 | Sindhu et al. | Oct 2018 | A1 |
| 20180293168 | Noureddine et al. | Oct 2018 | A1 |
| 20180300928 | Koker et al. | Oct 2018 | A1 |
| 20180307494 | Ould-Ahmed-Vall et al. | Oct 2018 | A1 |
| 20180307535 | Suzuki et al. | Oct 2018 | A1 |
| 20180322386 | Sridharan et al. | Nov 2018 | A1 |
| 20180357169 | Lai | Dec 2018 | A1 |
| 20190005176 | Illikkal et al. | Jan 2019 | A1 |
| 20190012278 | Sindhu et al. | Jan 2019 | A1 |
| 20190012350 | Sindhu et al. | Jan 2019 | A1 |
| 20190013965 | Sindhu et al. | Jan 2019 | A1 |
| 20190018806 | Koufaty et al. | Jan 2019 | A1 |
| 20190042292 | Palermo et al. | Feb 2019 | A1 |
| 20190042518 | Marolia et al. | Feb 2019 | A1 |
| 20190095333 | Heirman et al. | Mar 2019 | A1 |
| 20190102311 | Gupta et al. | Apr 2019 | A1 |
| 20190104057 | Goel et al. | Apr 2019 | A1 |
| 20190104206 | Goel et al. | Apr 2019 | A1 |
| 20190104207 | Goel et al. | Apr 2019 | A1 |
| 20190158428 | Gray et al. | May 2019 | A1 |
| 20190188079 | Kohli | Jun 2019 | A1 |
| 20190243765 | Sindhu et al. | Aug 2019 | A1 |
| 20190363989 | Shalev et al. | Nov 2019 | A1 |
| 20200021664 | Goyal et al. | Jan 2020 | A1 |
| 20200021898 | Sindhu et al. | Jan 2020 | A1 |
| 20200119903 | Thomas et al. | Apr 2020 | A1 |
| 20200133771 | Goyal et al. | Apr 2020 | A1 |
| 20200145680 | Dikshit et al. | May 2020 | A1 |
| 20200151101 | Noureddine et al. | May 2020 | A1 |
| 20200159568 | Goyal et al. | May 2020 | A1 |
| 20200159859 | Beckman et al. | May 2020 | A1 |
| 20200169513 | Goel et al. | May 2020 | A1 |
| 20200183841 | Noureddine et al. | Jun 2020 | A1 |
| 20200259682 | Goel et al. | Aug 2020 | A1 |
| 20200280462 | Sindhu et al. | Sep 2020 | A1 |
| 20200314026 | Sindhu et al. | Oct 2020 | A1 |
| 20200356414 | Sindhu et al. | Nov 2020 | A1 |
| 20210176347 | Goel et al. | Jun 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 104052618 | Sep 2014 | CN |
| 104954251 | Sep 2015 | CN |
| 105024844 | Nov 2015 | CN |
| 105847148 | Aug 2016 | CN |
| 105900406 | Aug 2016 | CN |
| 1079571 | Feb 2001 | EP |
| 1489796 | Dec 2004 | EP |
| 1501246 | Jan 2005 | EP |
| 2289206 | Mar 2011 | EP |
| 2928134 | Jul 2015 | EP |
| 2009114554 | Sep 2009 | WO |
| 2013184846 | Dec 2013 | WO |
| 2014178854 | Nov 2014 | WO |
| 2016037262 | Mar 2016 | WO |
| 2019014268 | Jan 2019 | WO |
| Entry |
|---|
| Final Office Action from U.S. Appl. No. 16/901,991, dated Mar. 28, 2022, 11 pp. |
| Office Action from U.S. Appl. No. 16/901,991, dated Nov. 3, 2021, 13 pp. |
| Office Action from U.S. Appl. No. 17/301,842, dated Feb. 3, 2022, 27 pp. |
| Response to Office Action dated Feb. 3, 2022, from U.S. Appl. No. 17/301,842, filed May 3, 2022, 10 pp. |
| Response to Office Action dated Nov. 3, 2021, from U.S. Appl. No. 16/901,991, filed Feb. 2, 2022, 8 pp. |
| Response to Second Written Opinion dated Feb. 12, 2019, from counterpart European Application No. PCT/US2018/024918, filed Apr. 12, 2019, 4 pp. |
| Notice of Allowance from U.S. Appl. No. 16/579,520, dated Dec. 24, 2020, 12 pp. |
| U.S. Appl. No. 17/301,842, filed Apr. 15, 2021, naming inventors Sindhu et al. |
| “QFX10000 Switches System Architecture,” White Paper, Juniper Networks, Apr. 2015, 15 pp. |
| Adya et al., “Cooperative Task Management without Manual Stack Management,” Proceedings of the 2002 Usenix Annual Technical Conference, Jun. 2002, 14 pp. |
| Al-Fares et al., “Hedera: Dynamic Flow Scheduling for Data Center Networks,” NSDI'10 Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation, Apr. 28-30, 2010, 15 pp. |
| Alizadeh et al., “CONGA: Distributed Congestion-Aware Load Balancing for Datacenters,” SIGCOMM '14 Proceedings of the 2014 ACM Conference on SIGCOMM, Aug. 17-22, 2014, pp. 503-514. |
| Bakkum et al., “Accelerating SQL Database Operations on a GPU with CUDA,” Proceedings of the 3rd Workshop on Genral-Purpose Computation on Graphics Processing Units, Mar. 14, 2010, 10 pp. |
| Banga et al., “Better operating system features for faster network servers,” ACM Sigmetrics Performance Evaluation Review, vol. 26, Issue 3, Dec. 1998, 11 pp. |
| Barroso et al., “Attack of the killer Microseconds,” Communications of the ACM, vol. 60, No. 4, Apr. 2017, 7 pp. |
| Benson et al., “MicroTE: Fine Grained Traffic Engineering for Data Centers,” CoNEXT '11 Proceedings of the Seventh Conference on emerging Networking EXperiments and Technologies Article No. 8, Dec. 6-9, 2011, 12 pp. |
| Benson et al., “Network Traffic Characteristics of Data Centers in the Wild,” IMC '10 Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, Nov. 1-30, 2010, pp. 267-280. |
| Deutsch, “Deflate Compressed Data Format Specification version 1.3,” IETF Network Working Group, RFC 1951, May 1996, 15 pp. |
| Ford et al., “TCP Extensions for Multipath Operation with Multiple Addresses,” Internet Engineering Task Force (IETF), RFC 6824, Jan. 2013, 64 pp. |
| Friedman et al., “Programming with Continuations,” Technical Report 151, Nov. 1983, 14 pp. |
| Gay et al., “The nesC Language: A Holistic Approach to Networked Embedded Systems,” accessed from http://nescc.sourceforge.net, last updated Dec. 14, 2004, 11 pp. |
| Halbwachs et al., “The Synchronous Data Flow Programming Language LUSTRE,” Proceedings of the IEEE, vol. 79, No. 9, Sep. 1991, 16 pp. |
| Haynes et al., “Continuations and Coroutines,” Technical Report No. 158, Jun. 1984, 19 pp. |
| Hewitt, “Viewing Control Structures as Patterns of Passing Messages,” Massachusetts Institute of Technology, Artificial Intelligence Laboratory, Dec. 1976, 61 pp. |
| Hseush et al., Data Path Debugging: Data-Oriented Debugging for a Concurrent Programming Language, PADD 88 Proceedings of the 1988 ACM SIGPLAN and SIGOPS workshop on Parallel and distributed debugging, May 5-6, 1988, 12 pp. |
| Huang et al., “Erasure Coding in Windows Azure Storage,” 2012 USENIX Annual Technical Conference, Jun. 13-15, 2012, 12 pp. |
| Hurson, “Advances in Computers, Volume 92,” Jan. 13, 2014, Academic Press, XP055510879, 94-95 pp. |
| Isen et al., “ESKIMO—Energy Savings using Semantic Knowledge of Inconsequential Memory Occupancy for DRAM subsystem,” 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Dec. 12-16, 2009, 10 pp. |
| Kahn et al., “Actors as a Special Case of Concurrent Constraint Programming,” ECOOP/OOPSLA '90 Proceedings, Oct. 21-25, 1990, 10 pp. |
| Kaminow, “Optical Integrated Circuits: A Personal Perspective,” Journal of Lightwave Technology, vol. 26, No. 9, May 1, 2008, pp. 994-1004. |
| Kandula et al., “Dynamic Load Balancing Without Packet Reordering,” SIGCOMM Computer Communication Review, vol. 37, No. 2, Apr. 2007, pp. 53-62. |
| Kandula et al., “The Nature of Datacenter Traffic: Measurements & Analysis,” IMC '09 Proceedings of the 9th ACM SIGCOMM conference on Internet measurement, Nov. 4-6, 2009, pp. 202-208. |
| Kelly et al., A Block Diagram Compiler, The Bell System Technical Journal, Dec. 7, 1960, 10 pp. |
| Kounavis et al., “Programming the data path in network processor-based routers,” Software—Practice and Experience, Oct. 21, 2003, 38 pp. |
| Larus et al., “Using Cohort Scheduling to Enhance Server Performance,” Usenix Annual Technical Conference, Jun. 2002, 12 pp. |
| Levis et al., “Tiny OS: An Operating System for Sensor Networks,” Ambient Intelligence, Jan. 2005, 34 pp. |
| Lin et al., A Parameterized Dataflow Language Extension for Embedded Streaming Systems, 2008 International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation, Jul. 21-24, 2008, 8 pp. |
| Mishra et al., “Thread-based vs Event-based Implementation of a Group Communication Service,” Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing, Mar. 30-Apr. 3, 1998, 5 pp. |
| Raiciu et al., “Improving Datacenter Performance and Robustness with Multipath TCP,” ACM SIGCOMM Computer Communication Review—SIGCOMM '11, vol. 41, No. 4, Aug. 2011, pp. 266-277. |
| Schroeder et al., “Flash Reliability in Production: The Expected and the Unexpected,” 14th USENIX Conference on File and Storage Technologies (FAST '16), Feb. 22-25, 2016, 15 pp. |
| Varela et al., “The Salsa Programming Language 2.0.0alpha Release Tutorial,” Tensselaer Polytechnic Institute, Nov. 2009, 52 pp. |
| von Behren et al., “Why Events Are A Bad Idea (for high-concurrency servers),” Proceedings of HotOS IX, May 2003, 6 pp. |
| Wang et al., “A Spatial and Temporal Locality-Aware Adaptive Cache Design with Network Optimization for Tiled Many-Core Architectures,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 25. No. 9, Sep. 2017, pp. 2419-2433. |
| Welsh et al., “SEDA: An Architecture for Well-Conditioned, Scalable Internet Services,” Eighteenth Symposium on Operating Systems Principles, Oct. 21-24, 2001, 14 pp. |
| Zhu et al., “Congestion Control for Large-Scale RDMA Deployments,” SIGCOMM '15 Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, Aug. 17-21, 2015, pp. 523-536. |
| International Search Report and Written Opinion of International Application No. PCT/US2018/024918, dated Jun. 21, 2018, 19 pp. |
| Response to Written Opinion dated Jun. 21, 2018, from International Application No. PCT/US2018/024918, dated Jan. 29, 2019, 24 pp. |
| Second Written Opinion of International Application No. PCT/US2018/024918, dated Feb. 12, 2019, 5 pp. |
| International Preliminary Report on Patentability from International Application No. PCT/US2018/024918, dated Jun. 12, 2019, 17 pp. |
| Prosecution History from U.S. Appl. No. 15/939,041, dated Dec. 11, 2019 through Mar. 25, 2020, 14 pp. |
| Prosecution History from U.S. Appl. No. 15/938,767, dated Sep. 10, 2018 through Jul. 31, 2019, 46 pp. |
| Prosecution History from U.S. Appl. No. 16/579,520, dated Jun. 26, 2020 through Sep. 25, 2020, 30 pp. |
| Prosecution History from U.S. Appl. No. 15/939,227, dated Jun. 26, 2019 through Apr. 23, 2020, 51 pp. |
| Prosecution History from U.S. Appl. No. 16/147,070, dated Dec. 31, 2019 through Sep. 18, 2020, 63 pp. |
| Prosecution History from U.S. Appl. No. 16/031,676, dated Aug. 22, 2019 through Jan. 13, 2020, 31 pp. |
| U.S. Appl. No. 16/901,991, filed Jun. 15, 2020, naming inventors Sindhu et al. |
| U.S. Appl. No. 16/939,617, filed Jul. 27, 2020, naming inventors Sindhu et al. |
| Advisory Action from U.S. Appl. No. 16/901,991 dated Jun. 27, 2022, 5 pp. |
| Final Office Action from U.S. Appl. No. 17/301,842 dated Jun. 24, 2022, 30 pp. |
| Response to Final Office Action dated Mar. 28, 2022 from U.S. Appl. No. 16/901,991, filed Jun. 28, 2022, p. 10. |
| Response to Final Office Action dated Mar. 28, 2022, from U.S. Appl. No. 16/901,991, filed May 31, 2022, 9 pp. |
| Number | Date | Country | |
|---|---|---|---|
| 20200259682 A1 | Aug 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62557988 | Sep 2017 | US | |
| 62514583 | Jun 2017 | US | |
| 62478414 | Mar 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15939041 | Mar 2018 | US |
| Child | 16859765 | US |
