This section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present invention that are described and claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
Symmetric multiprocessing (“SMP”) is the processing of computer instructions and/or programs by multiple processors under the control of a single operating system (“OS”) using a common memory and/or input/output (“I/O”) devices. By leveraging the processing power of multiple independent processors, such as sixty four processors for example, SMP systems may be able to generate significant computing power. As such, SMP systems can provide a more economical alternative to super computers or mainframes that typically rely on a small number of more expensive, custom-designed processors.
SMP systems employ multiple interconnected processors that cooperate and communicate with each other. There are a variety of factors, however, that can affect how efficiently the processors within an SMP system can communicate with each other, and, thus, how efficiently the SMP system can operate. One factor that affects the communication between the processors in an SMP system is the available data rate of the connections between the processors, which is referred to as the bandwidth. Higher bandwidth connections between processors enable more data to be communicated between two processors in a given period of time as compared to lower bandwidth connections. As such, higher bandwidth connections facilitate more efficient (i.e., faster) SMP systems. Similarly, SMP systems may also benefit from shorter transmission times, referred to as latencies, between the processors. For example, two processors may be able to cooperate more efficiently if they are directly coupled to one another versus if they are coupled to one another through a switch or other signal routing system. This is the case because transmitting data through the switch or other signal routing system can introduce transmission delays that are not present when signals are transmitted directly from one processor to another. Lastly, the efficiency of an SMP system may be also be affected by the redundancy of the connections between the processors. Increased redundancy can mitigate the effects of outages, malfunctions, and/or maintenance, and, consequently, can increase the robustness and computing power of an SMP system.
The embodiments described herein may be directed towards increasing bandwidth, decreasing latencies, and/or increasing redundancy in an SMP system.
Advantages of the invention may become apparent upon reading the following detailed description and upon reference to the drawings in which:
One or more specific embodiments of the present invention will be described below. In an effort to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers'specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.
The embodiments described herein may be directed towards computer topologies and architectures that may be employed with a wide range of currently-available processors to create symmetric multiprocessing (“SMP”) systems that exhibit higher bandwidths, lower latencies, and/or greater redundancies than conventional systems. For example, as will be described in greater detail below, in one embodiment, there is provided an SMP system composed of two groups of thirty-two central processing units (“CPUs”), such that all of the CPUs within a group of CPUs can communicate with each other over no more than a single crossbar switch (referred to as a “crossbar hop”) and all of the CPUs within the SMP system can communicate over no more two crossbar hops.
Turning now to the drawings and referring initially to
As illustrated in
In one embodiment, the CPUs 14a-d may be a processor from the Itanium Processor Family produced by Intel. Other examples of suitable CPUs 14a-d may include the Alpha EV7, produced by Alpha Processors, the Opteron produced by Advanced Micro Devices, and the Power 4/5 produced by International Business Machines. As described above, the CPUs 14a-d may be configured to communicate with one another, with input/output (“I/O”) devices, or with other components via the point-to-point data links 18a-k. In one embodiment, each of the CPUs 14a-d may include anywhere from two to twenty point-to-point data links 18a-k. For example, in the embodiment illustrated in
As described above, the CPUs 14a-d may be interconnected with each other via the data links 18a. The data links 18a may be wires, cables, fiber optic lines, or traces that connect to point-to-point data ports on the CPUs 14a-d. In one embodiment, the data links 18a may include pairs of wires configured to transmit SERDES data between SERDES ports on the CPUs 14a-d. In the embodiment illustrated in
The cell boards 12a and 12b may also include data agents 16a, 16b, 16c, and 16d (hereafter “16a-d”). The data agents 16a-d may include one or more integrated circuits (and their related memory and/or storage) that are configured to relay information between the CPUs 14a-d and other CPUs 14a-d, I/O devices, and/or other components of an SMP system. As illustrated, the data agents 16a-d may be coupled to the CPUs 14a-d by data links 18b-k. As with the data links 18a, the data links 18b-k may be wires, cables, fiber optic lines, or traces that couple to the point-to-point data ports on the CPUs 14a-d. In one embodiment, the data links 18a may include pairs of wires configured to transmit SERDES data between SERDES ports on the CPUs 14a-d and SERDES ports on the data agents 16a-d.
As will be described further below, the data agents 16a-d may expand the communication capabilities of the CPUs 14a-d beyond the number of data links 18a-k located on each of the CPUs 14a-d by enabling the CPUs 14a-d to communicate with other components in an SMP system via a switch or other signal routing system. It will be appreciated that conventional CPUs that employ point-to-point data links are typically only configured to be able to communicate with other CPUs that are directly coupled to the conventional CPU itself. Advantageously, the data agent may remove this conventional restriction and enable the CPUs 14a-d to communicate with more CPUs that the CPUs 14a-d have point-to-point data ports. For example, if the CPUs 14a-d each have eight point-to-point data links 18a-k, each of the CPUs 14a-d could conventionally only be connected to eight other CPUs 14a-d. The data agents 16a-d, however, are configured to increase the number of CPUs 14a-d that one of the CPUs, such as the CPU 14a for example, can communicate with by coupling the CPU 14a to a router or switch, such as a crossbar assembly 34, that is described further below in regard to
In alternate embodiments, a different number of data agents 16 may be employed on the cell boards 12a and 12b. For example, a single data agent 16 may serve both of the CPUs 14 on each of the cell boards 12a and 12b, or each of the CPUs 14 may have two or more data agents 16. In still other embodiments, the functionality of the data agents 16 may be integrated into the CPUs 14a-d. In addition, it will be appreciated that while the CPUs 14a and 14b and the data agents 16a and 16b are illustrated as disposed on a single PCB (the cell board 12a, for example), these elements can be disposed on different PCBs. The same holds true for the elements disposed on the cell board 12b.
Turning next to
As illustrated in
The data agents 16 within the first cabinet 32a and the second cabinet 32b may be coupled to the crossbars 34a-d via data links 36a, 36b, 36c, 36d (hereafter “36a-d ”) that are identical or similar to the data links 18a-k, described above in regard to
As described above, the cell boards 10a-p may be coupled to the crossbar assemblies 34a-d, which are hereafter referred to more simply as the crossbars 34a-d. In various embodiments, the crossbars may comprise 8-port crossbars, 10-port crossbars, 12-port crossbars, 16-port crossbars, 20-port crossbars, and so forth. One exemplary crossbar is the crossbars that are employed with sx1000 chipset produced by Hewlett Packard. The crossbars 34a-d are switches configured to receive data from one of the data agents 16 within the cabinets 32a and 32b or from another crossbar 34a-d, and to transmit the received data to either another one of the crossbars 34a-d or to another data agent 16. For example, if a CPU 14a within the cell board pair 10a wants to communicate with a CPU 14b within the cell board pair 10h, the CPU 14a may transmit a signal to the data agent 16a, (or 16b) within the cell board pair 10a. The data agent 16a, within the cell board 10a would then communicate the signal to the crossbar 34a, which would transmit the signal to the data agent 16b (or 16a) within the cell board 10h. This transmission of the signal through the crossbar 34a may be referred as a “crossbar hop.” The data agent 16b within the cell board 10h would then transmit the signal to the CPU 14b on the cell board 10h. In other words, advantageously a signal can be transmitted from one CPU 14a-d to another CPU 14a-d within one of the cabinets 32a or 32b over no more than one crossbar hop, which greatly reduces the latency of the SMP system 30 over conventional SMP systems.
A similar process occurs if one the CPUs 14 within first cabinet 32a wants to communicate with one of the CPUs 14 within the second cabinet or vice-versa. The main difference is that whereas it is possible for one of the CPUs 14 to communicate with any other CPU 14 within the same cabinet with only a single crossbar hop or less (see above), transmitting signals between the cabinets 32a and 32 takes two crossbar hops. For example, again looking at the CPU 14a within the cell board pair 10a, if the CPU 14a wants to communicate a signal to the CPU 14c within the cell board pair 10n (which is in the other cabinet), the CPU 14a may begin by transmitting the signal to the data agent 16a, (or 16b) within the cell board pair 10a. The data agent 16a, may then transmit the signal to the crossbar 34a, which will determine that the signal is intended for a CPU 14c within the second cabinet 32b. The crossbar 34a will then transmit the signal to the crossbar 34d (i.e., the closest crossbar to the CPU 14c) via data links 38 (see below). The crossbar 34d may then transmit the signal to the data agent 16c (or 16d) within the cell board pair 10n, which will transmit the signal to the CPU 14c.
Another advantage of the exemplary SMP system 30 is the number of redundant data paths within the system 30. For example, as described above, a signal from the CPU 14a within the cell board pair 10a to the CPU 14c within the cell board pair 10n may travel via the crossbars 34a and 34d. Alternatively, however, the signal may also be transmitted from the CPU 14a across the data link 18a to the CPUs 14c or 14d and then to the cell board pair 10n via the crossbars 34b and 34d. In still another possibility, the signal could be transmitted from the crossbar 34a to the crossbar 34c and then be transmitted across the cell board 12a within the cell board pair 10n to the CPU 14c. It will be appreciated that the above-described signal routing possibilities merely are three of many possibilities.
As described above, the crossbars 34a-d may be utilized to transmit data between cell board pairs 10 within a single cabinet 32a, b or between two or more cabinets 32a, b. In order to be able to simultaneously transmit signals amongst various pairs of CPUs 14, the crossbars 34a-d may employ multiple connections (referred to as “crossbar switch planes”), each of which is able to relay a transmission between a pair of data agents 16. In one embodiment, each of the data agents 16a-d may have at least one switch plane to communicate with other like-positioned data agents on other cell boards. For example, a CPU 14a on the cell board pair 10a may be communicating with a CPU 14b on the cell board pair 10b on one crossbar switch plane, while the CPU 14a on the cell board pair 10c is communicating with the CPU 14a on the cell board pair 10j, and so forth. The crossbar 34a may have at least one switch plane for each of the data agents 16a, in the first cabinet 32a to use to communicate. In one embodiment, the crossbar 34a has eight switch planes per data agent 16.
In addition, in some embodiments, the data agents 16 may be able to employ multiple crossbar switch planes for a single transmission. For example, one of the data agents 16 may divide a transmission between any two CPUs 14 across multiple crossbar switch planes to boost the bandwidth available between the two CPUs 14. As such, multiple crossbar switch planes provide redundancy and bandwidth to the SMP system 30.
As described above, the crossbars 34a-d may be interconnected by the data links 38. As with the data links 18a-k and 36, the data links 38 may be wires, cables, or traces that are suitable for coupling the crossbars 34a-d together. In one embodiment, the data links 38 may include pairs of wires configured to transmit SERDES data. In another embodiment, the data links 38 may include fiber optic cable or another suitable high speed transmission medium.
In addition to interconnecting the cell board pairs 10 within the first cabinet 32a and the second cabinet 32b, the crossbars 34a-d may also provide connectivity between the cell board pairs 10a-10p and one or more input/output (“I/O”) devices 40. As illustrated in
Turning next to
In the physical implementation illustrated in
One advantage of the physical implementation of the SMP system 30 illustrated in
As described above, the cell board pair 10 illustrated in
Unlike the embodiment illustrated in
While the invention may be susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, it should be understood that the invention is not intended to be limited to the particular forms disclosed. Rather, the invention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the invention as defined by the following appended claims.