This application is related to and claims priority from International Application No. PCT/CN2007/003367 filed on Nov. 29, 2007 and entitled, “MODIFYING SYSTEM ROUTING INFORMATION IN LINK BASED SYSTEMS”; which is entirely incorporated herein by reference for all purposes.
The present disclosure generally relates to the field of electronics. More particularly, an embodiment of the invention relates to techniques for modifying system routing information in link based systems.
RAS (Reliability, Availability, and Serviceability) has become a critical feature for modern computer system, especially in the server platforms. In a link based system, such as CSI (Common System Interface), the successful implementation of RAS features such as socket (or link) hot-plug depends on reconfiguration of routing data during runtime. Generally, routing data regarding immediate neighbors of a member of a link based system may be stored in storage devices local to each member of the linked based system. Routing data reconfiguration operations may be handled transparently to the OS (Operation System) by utilizing processing time that would otherwise be used by the OS. Since the OS has its own latency requirement, minimizing the routing table reconfiguration time becomes a key criterion in RAS implementations.
The detailed description is provided with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of various embodiments. However, some embodiments may be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the particular embodiments. Various aspects of embodiments of the invention may be performed using various means, such as integrated semiconductor circuits (“hardware”), computer-readable instructions organized into one or more programs (“software”) or some combination of hardware and software. For the purposes of this disclosure reference to “logic” shall mean either hardware, software, or some combination thereof.
Some of the embodiments discussed herein may allow for more efficient and/or faster change of system routing configuration in a link based system (such as a CSI system), e.g., to improve RAS. More particularly, some techniques enable relatively large computing systems (such as blade servers having many routes between various nodes which may also be referred to herein as agents such as discussed with reference to
More particularly,
As illustrated in
In one embodiment, the system 100 may support a layered protocol scheme, which may include a physical layer, a link layer, a routing layer, a transport layer, and/or a protocol layer. The fabric 104 may further facilitate transmission of data (e.g., in form of packets) from one protocol (e.g., caching processor or caching aware memory controller) to another protocol for a point-to-point or shared network. Also, in some embodiments, the network fabric 104 may provide communication that adheres to one or more cache coherent protocols.
Furthermore, as shown by the direction of arrows in
In
For the purpose of explaining one embodiment, assume that CPU3 in
1. Select one CPU as the Monarch CPU, which is responsible for executing most of SMI events handling code. (Below, CPU0 is taken as Monarch CPU for example);
2. Monarch CPU quiesces the whole system to pause all the traffic going through the CSI links;
3. Monarch CPU computes the new values for the system configuration RTA registers and updates them;
4. Monarch CPU performs other system configuration operations, e.g., computing the new values for SAD registers and updating them, disabling links to the hot-removed socket, etc.; and
5. Monarch CPU de-quiesces the system and releases the non-monarch processors and returns from the SMI. The system continues to run.
In step 3 above, to change system routings, the re-computing routing is performed by first obtaining the new topology of the routing fabric of the socket (or link) being removed. Second, the new values for all RTA registers may be computed. Finally, all RTA registers are updated. This approach is rather inefficient because it will have to go through all the RTA registers, computing their values and updating them, even though they preserve the old value after the hot-plug event, e.g., those RTA registers for routings between CPU0 and CPU1 in the example of
Referring to
For example, Table 1 below illustrates an example table corresponding to
Moreover, such routing information may be the intermediate result to compute the final RTA register value. For a link based system, the routing fabric on each component may be implemented with ports, entries, and virtual channels, etc. So, a number of successive computations may be performed. Also, depending on the implementation, some or all of the routing table information may be computed at system startup or otherwise before you before-hand. Other implementations may compute at least some of the routing table information during a hot-plug event. Furthermore, in some embodiments, the new topology may be obtained by various means at operation 302, such as based on information provided by a third-part system management/assistant agent, being discovered dynamically, etc.
At an operation 304, any necessary modifications may be determined. For example, since each cell in the Routing-Data-Table contains the routing information from the source to the destination, if a cell value does not change before and after the hot-plug event, the corresponding RTA register values will not need to change either; otherwise, RTA register values are changed to route the transitions to new ports or paths.
For example, Table 2 below illustrates an example of (New) Routing-Data-Table for CPU3 to be removed from
0.1
For example, by comparing Table 2 with Table 1, it becomes apparent that removal of CPU3 results in changes to 6 cells, while 15 cells preserve their previous values. In some embodiments, the current table may be stored in a different memory device than the new table (e.g., the tables may be stored in different memory subsystems or caches discussed with reference to
For example, assuming that for each cell time T is needed for the successive computation and hardware updating, then the time spent is reduced from (15+6)T=21T to 6T and it is 3.5 times faster. If one considers the hot-add case (e.g., adding CPU3), then the Table 1 corresponds to the Routing-Data-Table after the topology changes, while the Table 2 is the original configurations. This is because the cells from and to CPU3 should be counted, so the time spent is reduced from (21+10)T=31T to (6+10)T=16T, which is about 2 times faster. Accordingly, based on the determination made at operation 304, an operation 306 computes the new values for the filtered RTA registers. At an operation 308, the routing information may be updated based on the computed values of operation 306 (e.g., only filtered RTA registers of operation 304 are updated).
Because some embodiments may filter only changed RTA registers, as systems increase the number of their components, it becomes apparent that such embodiments will also result in better returns. For example, in
Further, in system 600 of
Accordingly, in some embodiments, an intermediate routing data table which contains routing information from a source to a destination may be used. By comparing data tables of before and after a hot-plug event, a minimal set of RTA registers need be computed and update to increase performance. Furthermore, even though socket hot-plug events are discussed herein as examples, embodiments discussed herein also work well with link hot-plug. The above example uses the SMI to describe the invention; similar techniques may be used in systems with the PMI.
The processor 702 may include one or more caches (not shown), which may be private and/or shared in various embodiments. Generally, a cache stores data corresponding to original data stored elsewhere or computed earlier. To reduce memory access latency, once data is stored in a cache, future use may be made by accessing a cached copy rather than refetching or recomputing the original data. The cache(s) may be any type of cache, such a level 1 (L1) cache, a level 2 (L2) cache, a level 3 (L-3), a mid-level cache, a last level cache (LLC), etc. to store electronic data (e.g., including instructions) that is utilized by one or more components of the system 700.
A chipset 706 may additionally be coupled to the interconnection network 704. Further, the chipset 706 may include a memory control hub (MCH) 708. The MCH 708 may include a memory controller 710 that is coupled to a memory 712. In an embodiment, the MCH may also include graphics logic and as a result may be referred to as a graphics MCH (GMCH). The memory 712 may store data, e.g., including sequences of instructions that are executed by the processor 702, or any other device in communication with components of the computing system 700. In an embodiment, the memory 712 may be the same or similar to the memory subsystems shown in
The MCH 708 may further include a graphics interface 714 coupled to a display device 716 (e.g., via a graphics accelerator in an embodiment). In one embodiment, the graphics interface 714 may be coupled to the display device 716 via an accelerated graphics port (AGP). In an embodiment of the invention, the display device 716 (such as a flat panel display) may be coupled to the graphics interface 714 through, for example, a signal converter that translates a digital representation of an image stored in a storage device such as video memory or system memory (e.g., memory 712) into display signals that are interpreted and displayed by the display 716.
As shown in
The bus 722 may be coupled to an audio device 726, one or more disk drive(s) 728, and a network adapter 730 (which may be a NIC in an embodiment). In one embodiment, the network adapter 730 or other devices coupled to the bus 722 may communicate with the chipset 706. Other devices may be coupled to the bus 722. Also, various components (such as the network adapter 730) may be coupled to the MCH 708 in some embodiments of the invention. In addition, the processor 702 and the MCH 708 may be combined to form a single chip.
Additionally, the computing system 700 may include volatile and/or nonvolatile memory (or storage). For example, nonvolatile memory may include one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), a disk drive (e.g., 728), a floppy disk, a compact disk ROM (CD-ROM), a digital versatile disk (DVD), flash memory, a magneto-optical disk, or other types of nonvolatile machine-readable media capable of storing electronic data (e.g., including instructions).
As illustrated in
In an embodiment, the processors 802 and 804 may be one of the processors 702 discussed with reference to
In at least one embodiment, one or more operations discussed with reference to
Chipset 820 may communicate with the bus 840 using a PtP interface circuit 841. The bus 840 may have one or more devices that communicate with it, such as a bus bridge 842 and I/O devices 843. Via a bus 844, the bus bridge 842 may communicate with other devices such as a keyboard/mouse 845, communication devices 846 (such as modems, network interface devices, or other communication devices that may communicate with the computer network 104), audio I/O device, and/or a data storage device 848. The data storage device 848 may store code 849 that may be executed by the processors 802 and/or 804.
In various embodiments of the invention, the operations discussed herein, e.g., with reference to
Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least an implementation. The appearances of the phrase “in one embodiment” in various places in the specification may or may not be all referring to the same embodiment.
Also, in the description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. In some embodiments of the invention, “connected” may be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements may not be in direct contact with each other, but may still cooperate or interact with each other.
Thus, although embodiments of the invention have been described in language specific to structural features and/or methodological acts, it is to be understood that claimed subject matter may not be limited to the specific features or acts described. Rather, the specific features and acts are disclosed as sample forms of implementing the claimed subject matter.
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/CN2007/003367 | 11/29/2007 | WO | 00 | 1/19/2012 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2009/070912 | 6/11/2009 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
5649108 | Spiegel et al. | Jul 1997 | A |
5784576 | Guthrie et al. | Jul 1998 | A |
6009488 | Kavipurapu | Dec 1999 | A |
6625659 | Aramizu et al. | Sep 2003 | B1 |
6647446 | James et al. | Nov 2003 | B1 |
6665297 | Hariguchi et al. | Dec 2003 | B1 |
6839350 | Inouchi et al. | Jan 2005 | B1 |
6956858 | Hariguchi et al. | Oct 2005 | B2 |
7200694 | Yakovlev et al. | Apr 2007 | B2 |
7415627 | Radhakrishnan et al. | Aug 2008 | B1 |
7493438 | Kaushik et al. | Feb 2009 | B2 |
7756017 | Goyal et al. | Jul 2010 | B2 |
7836238 | Freimuth et al. | Nov 2010 | B2 |
20020080798 | Hariguchi et al. | Jun 2002 | A1 |
20020129186 | Emerson et al. | Sep 2002 | A1 |
20030065752 | Kaushik et al. | Apr 2003 | A1 |
20030167367 | Kaushik et al. | Sep 2003 | A1 |
20030231633 | Aramizu et al. | Dec 2003 | A1 |
20030233474 | Yamamoto | Dec 2003 | A1 |
20050058119 | Inouchi et al. | Mar 2005 | A1 |
20050083936 | Ma | Apr 2005 | A1 |
20050157479 | Hayden | Jul 2005 | A1 |
20050289101 | Jayasimha | Dec 2005 | A1 |
20060206648 | Gupta et al. | Sep 2006 | A1 |
20070025355 | Cohen et al. | Feb 2007 | A1 |
20070118628 | Kumar et al. | May 2007 | A1 |
20070136504 | Wu | Jun 2007 | A1 |
20070153795 | Devereux et al. | Jul 2007 | A1 |
20070189284 | Kecskemeti | Aug 2007 | A1 |
20080005439 | Matsukuma | Jan 2008 | A1 |
20080259951 | Cardona et al. | Oct 2008 | A1 |
20090144476 | Cai et al. | Jun 2009 | A1 |
20100284330 | Huang | Nov 2010 | A1 |
20100287290 | Bramley et al. | Nov 2010 | A1 |
Number | Date | Country |
---|---|---|
1564987 | Jan 2005 | CN |
1889520 | Jan 2007 | CN |
1905512 | Jan 2007 | CN |
1989745 | Jun 2007 | CN |
2005-045591 | Feb 2005 | JP |
2006-012112 | Jan 2006 | JP |
2011-501921 | Jan 2011 | JP |
10-0778340 | Nov 2007 | KR |
2009049668 | Apr 2009 | WO |
2009070912 | Jun 2009 | WO |
Entry |
---|
Office Action received for Chinese Patent Application No. 200780101768.6, mailed on May 20, 2013, 8 pages including 5 pages of English Translation. |
Notice of Allowance received for Japanese Patent Application No. 2010-533409, mailed on May 28, 2013, 4 pages including 1 page of Foreign Associates reporting correspondence. |
Office Action received for Japanese Patent Application No. 2010-533409, mailed on Feb. 5, 2013, 2 pages of Office Action and 1 page of English Translation. |
Office Action Received for United Kingdom Patent Application No. 1009707.9, mailed on Nov. 7, 2011, 1 page of Office Action. |
International Search Report and Written Opinion Received for PCT Application No. PCT/CN2007/003367, mailed on Sep. 11, 2008, 9 pages. |
Office Action Received for Chinese Patent Application No. 200780101768.6, mailed on Feb. 3, 2012, 10 pages of Office Action and 16 pages of English Translation. |
International Preliminary Report on Patentability Received for the PCT Application No. PCT/CN2007/003367, issued on Jun. 1, 2010, 4 pages. |
Office Action Received for Chinese Patent Application No. 200780101768.6, mailed on Nov. 5, 2012, 3 pages of Office Action and 5 pages of English Translation. |
Office Action Received for Japanese Patent Application No. 2010-533409, mailed on Sep. 25, 2012, 2 pages of Office Action and 2 pages of English Translation. |
Office Action received for Chinese Patent Application No. 200780101768.6, mailed on Dec. 5, 2013, 11 Pages of Office Action and 17 Pages of English Translation. |
Office Action received for German Patent Application No. 11 2007 003 722.9, mailed on May 21, 2014, 6 pages of English Translation and 6 pages of Office Action. |
Office Action received for Chinese Patent Application No. 200780101768.6, mailed on Jul. 1, 2014, 5 pages of English Translation and 13 pages of CN Office Action. |
Office Action received for Chinese Patent Application No. 200780101768.6, mailed on Jan. 21, 2015. |
Certificate of Grant received for United Kingdom patent application No. 1009707.9, mailed on Feb. 15, 2012. |
Number | Date | Country | |
---|---|---|---|
20120151107 A1 | Jun 2012 | US |