An interconnection network is a set of communication lines and intermediate nodes between a given set of source nodes and a given set of terminal or terminating nodes, such as multi-processor computer elements known as cells. The intermediate nodes in an interconnection network typically comprise switches and hubs, referred to as nodes, between the sources and terminals. Static routing data tables have been used to identify routes used in the interconnection network. When a link between nodes is discovered to have failed, a static route around data table is used to determine a new link to route data across. The route around data table has routing information for a link if the link is the only link that fails. If multiple links fail, the table is not guaranteed to produce routes that ensure a fully connected interconnection network. Multiple links failing result in a fatal error, and shut down of the network.
Previously, systems route an interconnection network using fixed tables encoded in binary code. The code examines links in an order that the hardware supports. The hard coded tables are used to determine what values to write into the routing registers and which outbound port should be used when a bad link is discovered. This prior method is limited to working around a single bad link. A single bad link is guaranteed to have an alternate path, and a simple table lookup suffices. While having greater than one bad link might still have viable alternate paths, it is not guaranteed and the system is restricted to working around a single bad link.
In the following description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments that may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that structural, logical and electrical changes may be made without departing from the scope of the present invention. The following description is, therefore, not to be taken in a limited sense, and the scope of the present invention is defined by the appended claims.
The functions or algorithms described herein are implemented in software or a combination of software and human-implemented procedures in one embodiment. The software comprises computer executable instructions stored on computer readable media such as memory or other type of storage devices. The term “computer readable media” is also used to represent carrier waves on which the software is transmitted. Further, such functions correspond to modules, which are software, hardware, firmware or any combination thereof. Multiple functions are performed in one or more modules as desired, and the embodiments described are merely examples. The software is executed on a digital signal processor, ASIC, microprocessor, or other type of processor operating on a computer system, such as a personal computer, server or other computer system.
Configuration of an interconnection network can take place initially, or upon detection of failed links. A routing algorithm is provided a list of graph edges as an input that represents an interconnection fabric that is available. The routing algorithm explores the fabric in whatever order those links were provided. Each processor in the network writes the portion of fabric necessary for itself to function. As each edge is explored, a graph is updated to indicate the configuration of the hardware. The graph data is then used to write routing registers and handle bad links. Consistent rules are used to process the graph. The types of edges in the graph are those that lead to a terminating vs. non-terminating vertex.
Lines 125, 130, 135, 145 and 160 are represented by broken lines, signifying broken links or connection that have failed for one reason or another. Solid lines are used to represent good, operative links 140, 150 and 155.
Each link is represented by an edge with two vertices in one embodiment. The vertices correspond to the Quads. The edges form a graph utilizing graph theory. The graph can be queried to find routes around multiple broken links. In
Edges in the graph are internally represented in a relational database in the following format in one embodiment:
This format includes two different types of vertices, route through, and non-route through vertices, corresponding respectively to non-terminating and terminating vertices in
Ports have also been added to the edge structure. Vertices use the ports to track where the edges are. This means that they can distinguish between multiple edges that are connected to the same two vertices. These two notions, different types of vertices and the addition of ports, allow graph theory to work with the interconnection networks having these types of links. A color attribute is also provided, and is indicative of whether the link is operative or inoperative.
In one embodiment, the Cell_ID is an identifier of a vertex, such as a local vertex number. A cell is typically a computing element. It may contain several processors and some memory. In one embodiment, the Cell_ID is a number, such as between zero and fifteen that uniquely identifies a cell.
A graph type is an identifier that specifies which graph is being provided. The graph may be provided by inquiring to a cabinet that contains the cells, and may be predetermined based on the type of cell configuration provided in the cabinet. In one embodiment, a few predetermined topologies are available, and correspond expected predefined graphs.
The inputs are used at the beginning of phase one at 310 by a cell in the fabric, where a terminating vertex to the fabric is not initialized. A link is opened to a neighbor. If there is an unexpected neighbor, a fatal error is encountered at 315, and the cell halts. Likely causes of this include a bad backplane type serial configuration of a terminating vertex. At 310, if a link failure, or a terminating vertex is encountered, exploration of the vertex edge is done, because one cannot go further from such a terminating vertex.
If a non-terminating vertex is discovered, a link to the neighbor is opened at 335, and the input graph is matched to an actual graph being trimmed at 330. Also, the next edge is obtained, with the state returning to 310, and the exploration of the next edge. While opening a link to a neighbor at 335, if an illegal topology is found, an unexpected neighbor off a vertex-vertex link has been encountered at 340. The unexpected neighbor could be the result of a mis-cabling or a bad serial configuration of a vertex. The former may be more likely. The actual graph edge is terminated, and state 330 is entered for continued exploration of the input graph.
If, while opening a link to a neighbor at 335, a link failure is encountered, the actual graph is trimmed at 345, and shared routing may be performed using a canned route around table. Shared routing may be avoided if an unexpected neighbor is found, because this is usually a fatal condition. State 330 is then entered for further exploration of the input graph.
If the end of the graph is encountered at 330, state 320 is entered. If no more fabrics are available from 320, a phase one output is provided at 350 comprising an actual graph for each plane of the interconnection network, and routing for the terminating vertices across each of the planes is done. Shared routing may also be performed even if there are broken links found.
Phase one may be performed independent of both processor architecture and platform type. Cells may route things that are interesting to their outbound traffic. In one embodiment, different planes may be treated as independent entities. Terminating vertices have three ports each connected to a different non-terminating vertex port in one embodiment. An edge is the connection between two ports inclusive.
If the Topology is ok, the fabric is checked to determine if it is fully connected at 420. If at least one fully connected graph is present, control is returned back to the cell initialization process at 425.
When a link fails, the existing trimmed fabric graph may be used to traverse the edges, and find a new way to connect the network in an operational manner. The original expected topological graph may also be used, as the process will sort out which connections or links are good, and create a graph of an operational network fabric.
Computer-readable instructions for implementing the above processes may be stored on a computer-readable medium. The instructions are executable by a processing unit of a computer. A vertex may comprise such a computer. A hard drive, CD-ROM, and RAM are some examples of articles including a computer-readable medium.
The Abstract is provided to comply with 37 C.F.R. §1.72(b) to allow the reader to quickly ascertain the nature and gist of the technical disclosure. The Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.
Number | Name | Date | Kind |
---|---|---|---|
5714766 | Chen et al. | Feb 1998 | A |
5841673 | Kobayashi et al. | Nov 1998 | A |
5848242 | Behaghel et al. | Dec 1998 | A |
5937295 | Chen et al. | Aug 1999 | A |
5971596 | Nishikawa | Oct 1999 | A |
6047331 | Medard et al. | Apr 2000 | A |
6159620 | Heath et al. | Dec 2000 | A |
6282170 | Bentall et al. | Aug 2001 | B1 |
6477515 | Boroujerdi et al. | Nov 2002 | B1 |
6608815 | Huang et al. | Aug 2003 | B1 |
6701344 | Holt et al. | Mar 2004 | B1 |
6728205 | Finn et al. | Apr 2004 | B1 |
6741552 | McCrosky et al. | May 2004 | B1 |
6760302 | Ellinas et al. | Jul 2004 | B1 |
6791939 | Steele et al. | Sep 2004 | B1 |
6816585 | Blatt et al. | Nov 2004 | B1 |
6829216 | Nakata | Dec 2004 | B1 |
6976087 | Westfall et al. | Dec 2005 | B1 |
7286489 | Ades | Oct 2007 | B2 |
7308198 | Chudak et al. | Dec 2007 | B1 |
7379061 | Castonguay et al. | May 2008 | B2 |
7385917 | Mo et al. | Jun 2008 | B1 |
7391732 | Cortez et al. | Jun 2008 | B1 |
20020167898 | Thang et al. | Nov 2002 | A1 |
20030009582 | Qiao et al. | Jan 2003 | A1 |
20030058798 | Fleischer et al. | Mar 2003 | A1 |
20030112831 | Williams | Jun 2003 | A1 |
20030212941 | Gillies et al. | Nov 2003 | A1 |
20030227877 | Kar et al. | Dec 2003 | A1 |
20040000861 | Dorfman | Jan 2004 | A1 |
20040004938 | Buddhikot et al. | Jan 2004 | A1 |
20040059812 | Assa | Mar 2004 | A1 |
20040151124 | Hauser et al. | Aug 2004 | A1 |
20050049988 | Dahlquist et al. | Mar 2005 | A1 |
20050099419 | Castonguay et al. | May 2005 | A1 |
20050132318 | Kidd et al. | Jun 2005 | A1 |
20050243711 | Alicherry et al. | Nov 2005 | A1 |
20050286411 | Alicherry et al. | Dec 2005 | A1 |
20060136721 | Bruestle et al. | Jun 2006 | A1 |
20070027666 | Frankel | Feb 2007 | A1 |
20070177512 | Alexander | Aug 2007 | A1 |
20070177609 | Alexander, Jr. | Aug 2007 | A1 |
20070189191 | Ades | Aug 2007 | A1 |
Number | Date | Country | |
---|---|---|---|
20060056288 A1 | Mar 2006 | US |