Node card utilizing a same connector to communicate pluralities of signals

Information

  • Patent Grant
  • 9792249
  • Patent Number
    9,792,249
  • Date Filed
    Monday, June 29, 2015
    9 years ago
  • Date Issued
    Tuesday, October 17, 2017
    7 years ago
Abstract
A system and method for provisioning of modular compute resources within a system design are provided. In one embodiment, a node card or a system board may be used.
Description
FIELD

The disclosure relates generally to provisioning of modular compute resources within a system design.


BACKGROUND

Server systems generally provide a fixed number of options. For example, there are usually a fixed number of CPU sockets, memory DIMM slots, PCI Express 10 slots and a fixed number of hard drive bays, which often are delivered empty as they provide future upgradability. The customer is expected to gauge future needs and select a server chassis category that will serve present and future needs. Historically, and particularly with x86-class servers, predicting the future needs has been achievable because product improvements from one generation to another have been incremental.


With the advent of power optimized, scalable servers, the ability to predict future needs has become less obvious. For example, in this class of high-density, low-power servers within a 2 U chassis, it is possible to install on the order of 120 compute nodes in an incremental fashion. Using this server as a data storage device, the user may require only 4 compute nodes, but may desire 80 storage drives. Using the same server as a pure compute function focused on analytics, the user may require 120 compute nodes and no storage drives. The nature of scalable servers lends itself to much more diverse applications which require diverse system configurations. As the diversity increases over time, the ability to predict the system features that must scale becomes increasingly difficult.


It is desirable to provide smaller sub-units of a computer system that are modular and can be connected to each other to form larger, highly configurable scalable servers. Thus, it is desirable to create a system and method to modularly scale compute resources in these power-optimized, high density, scalable servers.





BRIEF DESCRIPTION OF THE DRAWINGS


FIG. 1 illustrates an example of a system board on which one or more node cards may be installed;



FIG. 2 illustrates an embodiment of the details of each node card;



FIG. 3 illustrates an example of a quad node card;



FIGS. 4 and 5 illustrate two examples of node cards with one or more connectors; and



FIG. 6 illustrates an example of a single server node card.





DETAILED DESCRIPTION OF ONE OR MORE EMBODIMENTS

The disclosure is particularly applicable to examples of the node cards illustrated and described below and it is in this context that the disclosure will be described. It will be appreciated, however, that the disclosure has broader applicability since the disclosed system and node cards can be implemented in different manners that are within the scope of the disclosure and may be used for any application since all of the various applications in which the system and node cards may be used are within the scope of the disclosure.



FIG. 1 illustrates an example of a system 40 that may include a system board 42 on which one or more node cards 46 may be installed. The system board 42 may be fit into a typical server chassis 44 and the system board may have the one or more node cards 46, such as one or more server node units (described below with reference to FIG. 2) plugged into the system board. There are a number of functions that are needed to complete a full classic server which includes Ethernet PHYs to interface the one or more ServerNodes 46 or a cluster of ServerNodes and server control functions (fan control, buttons etc. . . . ). The system board 42 is the component that ties the ServerNodes 46 to these components. The system board 42 is desirable if a hierarchical hardware partition is desired where the “building block” is smaller than the desired system, or when the “building block” is not standalone. The system board roles can include: Ethernet network connectivity, internal fabric connections between ServerNodes or groups a ServerNodes in a sub-system (the fabric design in FIG. 1) and chassis control and management. The system board is the component that connects the fabric links between ServerNodes and allows them to communicate with the external world. Once the fabric design, hardware partitioning and storage decisions have been made, the system board 42 can glue the system components together and the input/output (I/O) of the system may include: management data input/output (MDIO) for SFP communication, comboPHYs for internal fabric links, storage and Ethernet access, UART and JTAG ports for debug and SMBus and GPIOs for chassis component control and communication.


Now, several different examples of node cards that may be plugged into the system board are described in more detail. These node cards leverage highly integrated SoCs designed for Server applications, that enable density and system design options that has not been available to date. Cards can be defined that have the functionality of one or more servers and these Cards can be linked together to form clusters of servers in very dense implementations. A high level description of the Card would include a highly integrated SoC implementing the server functionality, DRAM memory, support circuitry such as voltage regulation, and clocks. The input/output of the card would be power and server to server interconnect and/or server to Ethernet PHY connectivity. SATA connections can also be added to interface to drives. An example of a node card is shown in FIG. 2 with one or more system on a chip (SoC) systems.


The fabric connections on each node card 46 can be designed to balance: usage of SoC PHYs, link redundancy, link bandwidth and flexibility in usage of the 8 links at the edge connectors. A node card 46 like that shown in FIG. 3 can be used in conjunction with a system board where the system board provides power to the node cards and connections to interconnect off the system board such as an Ethernet transceiver. The system board could house one or more node cards. In the case of housing more than one node card, the system board creates a cluster of Servers that utilize a server to server interconnect or fabric that is integrated in the SoC or a separate function on the card. This system board can be made in many forms, including industry standard form factors such as ATX or in customer form factors. The system board could be a blade or could fit into a standard chassis such as a 2 U or any other size.



FIG. 2 illustrates an example a node card 60. The node card may be a printed circuit board with a male physical connector, on which there is one or more servers that get power from some of the signals on the physical connector and use some of the signals on the connector for server to server communication or server to Ethernet PHY connections. In one embodiment, the physical connector may be PCIe connector. The node card may have an enable of the physical connector (see CARD_EN in FIG. 2) that enables the server. The node card may have regulators included on the PCB to provide regulated power supplies to various parts of the server off the power supply that is provided through one or more edge connectors, that may be implemented as PCIe connectors, and the enables (CARD_EN) may be connected to the regulators. The voltages on the node card may be 12V. The regulators may generate a common voltage that may be 3.3V (as shown in the example in FIG. 2), 1.8V, 0.9V and/or 1.35 or 1.5V. Each node card may have one or more SoCs 62, memory and appropriate regulators, but may also have multiple servers on the PCB including multiple SoC and multiple sets of DRAM and the DRAM is soldered on the PCB and signals are routed to the SoC. Alternatively, the DRAM is on a DIMM and the DIMM is connected to the PCB using a connector whose signals are routed to the SoC.


In the example in FIG. 2, the node card 60 may include one or more system on a chip (SOC) 62 (such as SOC0-SOC3 as shown in FIG. 2) and each SOC is part of a node 64, such as Node N0-N3 as shown, wherein the node may be a compute node, a storage node and the like. The SoCs on the node card may have heat sinks Each node 64 may further include one or more LEDs, memory (DDR, for example), a clock, a temperature sensor (TEMP) connected to the SoC, an SD slot and an SPI_FLASH slot as shown in FIG. 2. Thus, the node card may also have a storage card such as SD, uSD, MMC, eMMC that is connected to the SoC (as shown in the example below in FIG. 6). In one embodiment, a NAND or NOR can be used and connected to the SoC (such as in the examples in FIGS. 4-5 below) and/or a serial flash may be used and connected to the SoC.


The node card may also have one or more communication and/or storage connects 66, such as connects to various SATA devices, connects to XAUI interconnects and a UART that may be through an edge connector. In the node card, the server to server communication may be XAUI and one or more XAUI is routed to the edge connector and the XAUI signals are routed from the edge connector to the SoC and/or the XAUI signals are routed between SoCs on the PCB. In the node card, the server to server communication may be SGMII and one or more SGMII is routed to the edge connector and the SGMII signals are routed from the PCIe connector to the SoC or the SGMII signals are routed between SoCs on the PCB.


The node card may also have a SATA connector. The SATA signals may be routed from the SoC to the SATA connector or multiple SATA connectors are added to the PCB and multiple SATA connectors are routed from the SoC to the SATA connectors. The node card may also have a mini SATA on the Card or mSATA on the Card. The SATA may be routed to the edge connector from the SoC. In some embodiments, multiple SATA connections are made between the SoC and edge connector and PCIe x1 or x2, or x4, or x8 or x16 or x32 is used. The node card may use multiple edge connectors or any combination of multiple edge connectors such as x1 or x2, or x4, or x8 or x16 or x32. There may be a set of DC values, such as pull up and pull downs to set the slot identifier and the like and slow speed signals, and these DC values may be applied to the edge connector and routed onto the PCB for set up, control, ID or information and the DC values are routed to GPIOs on one or more SoCs.


The edge connector may also have signaling for JTAG and ALTBOOT (described below in more detail). The edge connector may also provide SLOT signaling, GPIO signaling and power (with an enable). The JTAG signals are routed from one or more SoCs to edge connector and the serial port and/or UART signals are routed from the edge connector to one or more SoCs. The SoC may have an addition signal or set of signals is routed to the edge connector that is used to arbitrate usage of the serial port or UART. In the system, a digital signal can be applied to the edge connector to cause an alternative boot procedure by connecting this signal from the edge connector to a signal on one or more SoCs that causes or enable an alternative boot. The digital signal or signals can be applied to the edge connector to cause an interrupt to the SoC or SoCs by connecting the SoC or SoCs to this digital signal on the connector. The system may have a level shifter(s) that is used on the PCB to translate a signal applied on the edge connector edge to a signal that is applied to the SoC(s). Furthermore, the digital signal that is routed from an SoC to the edge connector that resets and/or controls and/or provides info to an Ethernet phy or SFP that is not on the PCB and may be for reset, enable, disable, mdio, fault, los of signal and rate.



FIG. 3 illustrates an example of a quad node card 100. The quad node card 100 may have one or more systems on a chip 102 (SoC0-SoC3 in this example), one or more volatile memory devices 104, such as four 4 GB DDR3 Mini-DIMMs (1 per node) or DDR3L memory chips, one or more storage interfaces 106, such as sixteen SATA connectors (4 per node), one or more SD slots (one per node, MMC not supported) and one or more SPI flash chips (1 per node). The quad node card may be powered by 12V dc, supplied via edge connectors 108—all other voltages are internally generated by regulators. The quad node card may have server interconnect Fabric connections 110 routed via the edge connector 108, through a system board to which the node card is connected, to other node cards or external Ethernet transceivers and I2C and GPIO rout via the edge connector, per system board requirements. The quad node card 100 does not have ethernet PHY transceivers in some implementations, other implementations may choose to use Ethernet transceivers on the node card and route this as the interconnect and the node card is not a stand alone design, but may be used with a system board.


The quad Card example consists of 4 server nodes, each formed by a Calxeda® EnergyNode SoC, with its DIMM and local peripherals, which runs Linux independently from any other node. By design, these nodes can be directly interconnected to form a high bandwidth fabric, which provides network access through the system Ethernet ports. From the network view, the server nodes appear as independent servers; each available to take work on.



FIGS. 4 and 5 illustrate two examples of node cards 120, 130 with one or more connectors 108. The connectors may be a PCIe connector that makes a convenient physical interconnect between the node card and the system board, but any type of connector can be used. The connector type is selected based on its performance at the switching frequency of the fabric interconnect. For example, industry-standard Micro TCA connectors available from Tyco Electronics and Samtec operate up to 12 GHz. In the examples in FIGS. 4 and 5, the node card has the SOCs 102, the memory 104, the storage interfaces 106 and the fabric connector 110, but may also include one or more persistent memory devices 112, such as NAND flash. The node card definition can vary as seen below with variation in a number of SATA connectors and/or in a number of fabric interconnect for server to server communication. The type of PCIe connector in the node card could vary significantly based on quantity of interconnect and other signals desired in the design. FIGS. 4 and 5 shows two PCIe x16 connectors, but the node cards could vary using any quantity of PCIe connector and any type of PCIe (x1, x2, x4 etc. . . . ). Though not shown in FIG. 4 or 5 for brevity, since fabric connectivity exists with the node cards, the physical Ethernet interfaces depicted on the System Board 42 can also reside on the node cards.



FIG. 6 illustrates an example of a single server node card 140. The single server node card 140 may have one processor SoC 102, a 4 GB DDR3 DRAM 104 down (no DIMM), a microSD slot 114, a SATA data connector 106, a mSATA connector 116, one or more XAUI channels (four in this example) to the edge connector 108 for fabric connectivity and may be smaller than 2″.times.4″. This combination provides the compute, networking IO, system memory, and storage interfaces needed for a robust ARM server, in a form factor that is easily integrated into many chassis designs. This node card implements a x16 PCI connector with a custom electrical signaling interface that follows the Ethernet XAUI interface definition. The node card 140 may be a two-sided printed circuit board with components on each side as shown in FIG. 6.


While the foregoing has been with reference to a particular embodiment of the invention, it will be appreciated by those skilled in the art that changes in this embodiment may be made without departing from the principles and spirit of the disclosure, the scope of which is defined by the appended claims.

Claims
  • 1. A method comprising: communicating a plurality of signals between a substrate having a connector and an outside entity;receiving power, at one or more nodes connected to the substrate, from one or more first signals communicated over the connector;communicating with the outside entity using one or more second signals communicated over the connector;translating, by one or more level shifters, a signal on the connector to a signal applied to the one or more nodes, wherein at least one of the plurality of signals is configured to arbitrate usage of a serial port on a node card by the one or more nodes; andproviding a digital signal to the connector to cause an alternative boot procedure.
  • 2. The method of claim 1, wherein the alternative boot procedure is provided by connecting the digital signal from the connector to a signal on the one or more nodes.
  • 3. The method of claim 1, wherein the plurality of signals includes an enable signal that, upon assertion, enables the one or more nodes.
  • 4. The method of claim 1, further comprising: receiving, at one or more regulators connected to the substrate, power from one or more third signals communicated over the connector, wherein the plurality of signals includes an enable signal that enables the one or more regulators; andproviding, by the one or more regulators, a regulated voltage to the one or more nodes.
  • 5. The method of claim 1, further comprising routing one or more communication paths to the connector, wherein: the one or more communication paths are between the one or more nodes and the outside entity, andthe one or more communication paths are XAUI.
  • 6. The method of claim 1, further comprising routing one or more communication paths to the connector, wherein: the one or more communication paths are between the one or more nodes and the outside entity, andthe one or more communication paths are serial gigabit media independent interface (SGMII).
  • 7. The method of claim 1, wherein each of the one or more nodes is a server comprising a system on a chip (SOC), a memory, and a regulator; and wherein the method further comprises: routing a set of signals from the SOC to one or more serial advanced technology attachment (SATA) connectors through the connector.
  • 8. The method of claim 1, wherein each of the one or more nodes is a server comprising a system on a chip (SOC), a memory, and a regulator; and wherein the method further comprises: communicating, through the connector, a set of direct current (DC) values for one of set-up, control, identification, and information.
  • 9. The method of claim 8, further comprising routing the set of DC values to a set of general purpose input/output (GPIO) pins on each SOC.
  • 10. The method of claim 1, further comprising communicating a set of joint test action group (JTAG) signals to the one or more nodes.
  • 11. The method of claim 1, further comprising communicating, through the connector, at least one serial port signal and one or more universal asynchronous receiver/transmitter (UART) signals to the one or more nodes.
  • 12. A non-transitory computer readable medium having instructions stored thereon that, upon execution by a computing device, cause the computing device to perform operations, wherein the instructions comprise: instructions to communicate a plurality of signals between a substrate having a connector and an outside entity;instructions to receive power, at one or more nodes connected to the substrate, from one or more first signals communicated over the connector;instructions to communicate with the outside entity using one or more second signals communicated over the connector; andinstructions to translate, using one or more level shifters, a signal on the connector to a signal applied to the one or more nodes, wherein at least one of the plurality of signals is configured to arbitrate usage of a serial port on a node card by the one or more nodes; andwherein each of the one or more nodes is a server comprising a system on a chip (SOC), a memory, and a regulator; and wherein the instructions further comprise:instructions to communicate, through the connector, an alternative boot signal to the SOC to enable an alternative boot by the SOC.
  • 13. The non-transitory computer readable medium of claim 12, wherein each of the one or more nodes is a server comprising a system on a chip (SOC), a memory, and a regulator; and wherein the instructions further comprise: instructions to communicate, through the connector, at least one signal to interrupt the SOC.
  • 14. The non-transitory computer readable medium of claim 12, wherein each of the one or more nodes is a server comprising a system on a chip (SOC), a memory, and a regulator; and wherein the instructions further comprise: instructions to communicate, through the connector, a signal from the SOC to communicate with the outside entity.
  • 15. A system comprising: an outside entity; anda node card, wherein the node card comprises: a substrate having a connector configured to communicate a plurality of signals between the substrate and the outside entity;one or more nodes connected to the substrate configured to receive power from one or more first signals communicated over the connector, wherein the one or more nodes are further configured to communicate with the outside entity with one or more second signals over the connector;one or more level shifters configured to translate a signal on the connector to a signal applied to the one or more nodes; anda serial port on the node card, wherein at least one of the plurality of signals is configured to arbitrate usage of a serial port by the one or more nodes;wherein the connector receives a digital signal to cause an alternative boot procedure.
  • 16. The system of claim 15, wherein each of the one or more nodes is a server comprising a system on a chip (SOC), a memory, and a regulator.
  • 17. The system of claim 16, wherein the memory is one of a dynamic random access memory (DRAM) and a dual in-line memory module (DIMM), and wherein the memory is connected to the substrate and electrically connected to the SOC.
  • 18. The system of claim 16, wherein each server has a clock generator chip.
  • 19. The system of claim 16, wherein each server has a temperature sensor that is electrically connected to the SOC.
  • 20. The system of claim 16, wherein each server has a nonvolatile memory electrically connected to the SOC, and wherein the nonvolatile memory is one of a storage card, NAND flash, NOR flash, and serial flash.
  • 21. The system of claim 16, wherein each SOC has a heat sink.
  • 22. The system of claim 15, wherein the connector is one of a peripheral component interconnect express (PCIe) x1 connector, a PCIe x2 connector, a PCIe x4 connector, a PCIe x8 connector a PCIe x16 connector, and a PCIe x32 connector.
  • 23. The system of claim 15, wherein the node card further comprises an ethernet transceiver configured to act as an interconnect between the one or more nodes.
RELATED APPLICATION/PRIORITY CLAIMS

This application is a Continuation of U.S. application Ser. No. 13/527,498, filed Jun. 19, 2012, which claims the benefit under 35 USC 119(e) of U.S. Provisional Patent Application Ser. No. 61/553,555, filed on Oct. 31, 2011 and entitled, “System And Method For Modular Compute Provisioning In Large Scalable Processor Installations,” the entirety of which is incorporated herein by reference. This application is also related to U.S. patent application Ser. No. 13/527,505, filed on the same date and entitled, “System Board For System And Method For Modular Compute Provisioning In Large Scalable Processor Installations,” the entirety of which is also incorporated herein by reference.

US Referenced Citations (360)
Number Name Date Kind
5451936 Yang et al. Sep 1995 A
5594908 Hyatt Jan 1997 A
5623641 Kadoyashiki Apr 1997 A
5781187 Gephardt et al. Jul 1998 A
5901048 Hu May 1999 A
5908468 Hartmann Jun 1999 A
5968176 Nessett et al. Oct 1999 A
5971804 Gallagher et al. Oct 1999 A
6055618 Thorson Apr 2000 A
6141214 Ahn Oct 2000 A
6181699 Crinion et al. Jan 2001 B1
6192414 Horn Feb 2001 B1
6198741 Yoshizawa et al. Mar 2001 B1
6252878 Locklear Jun 2001 B1
6314487 Hahn et al. Nov 2001 B1
6314501 Gulick et al. Nov 2001 B1
6373841 Goh et al. Apr 2002 B1
6442137 Yu et al. Aug 2002 B1
6446192 Narasimhan et al. Sep 2002 B1
6452809 Jackson et al. Sep 2002 B1
6507586 Satran et al. Jan 2003 B1
6556952 Magro Apr 2003 B1
6574238 Thrysoe Jun 2003 B1
6711691 Howard et al. Mar 2004 B1
6766389 Hayter et al. Jul 2004 B2
6813676 Henry et al. Nov 2004 B1
6816750 Klaas Nov 2004 B1
6842430 Melnik Jan 2005 B1
6857026 Cain Feb 2005 B1
6963926 Robinson Nov 2005 B1
6963948 Gulick Nov 2005 B1
6977939 Joy et al. Dec 2005 B2
6988170 Barroso et al. Jan 2006 B2
6990063 Lenoski et al. Jan 2006 B1
7020695 Kundu et al. Mar 2006 B1
7032119 Fung Apr 2006 B2
7080078 Slaughter et al. Jul 2006 B1
7080283 Songer et al. Jul 2006 B1
7095738 Desanti Aug 2006 B1
7119591 Lin Oct 2006 B1
7143153 Black et al. Nov 2006 B1
7165120 Giles et al. Jan 2007 B1
7170315 Bakker et al. Jan 2007 B2
7180866 Chartre et al. Feb 2007 B1
7203063 Bash et al. Apr 2007 B2
7257655 Burney et al. Aug 2007 B1
7263288 Islam Aug 2007 B1
7274705 Chang et al. Sep 2007 B2
7278582 Siegel et al. Oct 2007 B1
7310319 Awsienko et al. Dec 2007 B2
7325050 O'Connor et al. Jan 2008 B2
7337333 O'Conner et al. Feb 2008 B2
7340777 Szor Mar 2008 B1
7353362 Georgiou et al. Apr 2008 B2
7382154 Ramos et al. Jun 2008 B2
7386888 Liang et al. Jun 2008 B2
7418534 Hayter et al. Aug 2008 B2
7437540 Paolucci et al. Oct 2008 B2
7447147 Nguyen et al. Nov 2008 B2
7447197 Terrell et al. Nov 2008 B2
7466712 Makishima et al. Dec 2008 B2
7467306 Cartes et al. Dec 2008 B2
7467358 Kang et al. Dec 2008 B2
7502884 Shah et al. Mar 2009 B1
7519843 Buterbaugh et al. Apr 2009 B1
7555666 Brundridge et al. Jun 2009 B2
7583661 Chaudhuri Sep 2009 B2
7586841 Vasseur Sep 2009 B2
7596144 Pong Sep 2009 B2
7599360 Edsall et al. Oct 2009 B2
7606225 Xie et al. Oct 2009 B2
7606245 Ma et al. Oct 2009 B2
7616646 Ma et al. Nov 2009 B1
7620057 Aloni et al. Nov 2009 B1
7644215 Wallace Jan 2010 B2
7657677 Huang et al. Feb 2010 B2
7657756 Hall Feb 2010 B2
7660922 Harriman Feb 2010 B2
7664110 Lovett et al. Feb 2010 B1
7673164 Agarwal Mar 2010 B1
7710936 Morales Barroso May 2010 B2
7719834 Miyamoto et al. May 2010 B2
7721125 Fung May 2010 B2
7751433 Dollo et al. Jul 2010 B2
7760720 Pullela et al. Jul 2010 B2
7761687 Blumrich et al. Jul 2010 B2
7783910 Felter et al. Aug 2010 B2
7791894 Bechtolsheim Sep 2010 B2
7792113 Foschiano et al. Sep 2010 B1
7796399 Clayton et al. Sep 2010 B2
7801132 Ofek et al. Sep 2010 B2
7802017 Uemura et al. Sep 2010 B2
7805575 Agarwal et al. Sep 2010 B1
7831839 Hatakeyama Nov 2010 B2
7840703 Arimilli et al. Nov 2010 B2
7865614 Lu et al. Jan 2011 B2
7925795 Tamir et al. Apr 2011 B2
7934005 Fascenda Apr 2011 B2
7970929 Mahalingaiah Jun 2011 B1
7975110 Spaur et al. Jul 2011 B1
7991817 Dehon et al. Aug 2011 B2
7991922 Hayter et al. Aug 2011 B2
7992151 Warrier et al. Aug 2011 B2
8019832 De Sousa et al. Sep 2011 B2
8060760 Shetty et al. Nov 2011 B2
8060775 Sharma et al. Nov 2011 B1
8082400 Chang et al. Dec 2011 B1
8108508 Goh et al. Jan 2012 B1
8122269 Houlihan et al. Feb 2012 B2
8132034 Lambert et al. Mar 2012 B2
8155113 Agarwal Apr 2012 B1
8156362 Branover et al. Apr 2012 B2
8165120 Maruccia et al. Apr 2012 B2
8170040 Konda May 2012 B2
8180996 Fullerton et al. May 2012 B2
8189612 Lemaire et al. May 2012 B2
8194659 Ban Jun 2012 B2
8199636 Rouyer et al. Jun 2012 B1
8205103 Kazama et al. Jun 2012 B2
8379425 Fukuoka et al. Feb 2013 B2
8397092 Karnowski Mar 2013 B2
8407428 Cheriton et al. Mar 2013 B2
8504791 Cheriton et al. Aug 2013 B2
RE44610 Krakirian et al. Nov 2013 E
8599863 Davis Dec 2013 B2
8684802 Gross et al. Apr 2014 B1
8738860 Griffin et al. May 2014 B1
8745275 Ikeya et al. Jun 2014 B2
8745302 Davis et al. Jun 2014 B2
8782321 Harriman et al. Jul 2014 B2
8812400 Faraboschi et al. Aug 2014 B2
8824485 Biswas et al. Sep 2014 B2
8854831 Arnouse Oct 2014 B2
8903964 Breslin Dec 2014 B2
9008079 Davis et al. Apr 2015 B2
9311269 Davis et al. Apr 2016 B2
9465771 Davis et al. Oct 2016 B2
20010046227 Matsuhira et al. Nov 2001 A1
20020004912 Fung Jan 2002 A1
20020040391 Chaiken et al. Apr 2002 A1
20020083352 Fujimoto et al. Jun 2002 A1
20020097732 Worster et al. Jul 2002 A1
20020107903 Richter et al. Aug 2002 A1
20020124128 Qiu Sep 2002 A1
20020159452 Foster et al. Oct 2002 A1
20020161917 Shapiro et al. Oct 2002 A1
20020172205 Tagore-Brage et al. Nov 2002 A1
20020186656 Vu Dec 2002 A1
20020194412 Bottom Dec 2002 A1
20020196611 Ho et al. Dec 2002 A1
20030007493 Oi et al. Jan 2003 A1
20030033547 Larson et al. Feb 2003 A1
20030041266 Ke et al. Feb 2003 A1
20030076832 Ni Apr 2003 A1
20030093255 Freyensee et al. May 2003 A1
20030093624 Arimilli et al. May 2003 A1
20030110262 Hasan et al. Jun 2003 A1
20030140190 Mahony et al. Jul 2003 A1
20030158940 Leigh Aug 2003 A1
20030159083 Fukuhara et al. Aug 2003 A1
20030172191 Williams Sep 2003 A1
20030188083 Kumar et al. Oct 2003 A1
20030193402 Post et al. Oct 2003 A1
20030202520 Witkowski et al. Oct 2003 A1
20030231624 Alappat et al. Dec 2003 A1
20040013113 Singh et al. Jan 2004 A1
20040017806 Yazdy et al. Jan 2004 A1
20040017808 Forbes et al. Jan 2004 A1
20040030938 Barr et al. Feb 2004 A1
20040068676 Larson et al. Apr 2004 A1
20040111612 Choi et al. Jun 2004 A1
20040141521 George Jul 2004 A1
20040165588 Pandya Aug 2004 A1
20040210693 Zeitler et al. Oct 2004 A1
20040215864 Arimilli et al. Oct 2004 A1
20040215991 McAfee et al. Oct 2004 A1
20040267486 Percer et al. Dec 2004 A1
20050015378 Gammel et al. Jan 2005 A1
20050018604 Dropps et al. Jan 2005 A1
20050018606 Dropps et al. Jan 2005 A1
20050018663 Dropps et al. Jan 2005 A1
20050021606 Davies et al. Jan 2005 A1
20050021728 Sugimoto Jan 2005 A1
20050030954 Dropps et al. Feb 2005 A1
20050033742 Kamvar et al. Feb 2005 A1
20050033890 Lee Feb 2005 A1
20050044195 Westfall Feb 2005 A1
20050077921 Percer et al. Apr 2005 A1
20050105538 Perera et al. May 2005 A1
20050141424 Lim et al. Jun 2005 A1
20050228852 Santos et al. Oct 2005 A1
20050240688 Moerman et al. Oct 2005 A1
20060002311 Iwanaga et al. Jan 2006 A1
20060013218 Shore et al. Jan 2006 A1
20060029053 Roberts et al. Feb 2006 A1
20060090025 Tufford et al. Apr 2006 A1
20060136570 Pandya Jun 2006 A1
20060140211 Huang et al. Jun 2006 A1
20060174342 Zaheer et al. Aug 2006 A1
20060179241 Clark et al. Aug 2006 A1
20060236371 Fish Oct 2006 A1
20060248359 Fung Nov 2006 A1
20060259734 Sheu et al. Nov 2006 A1
20060265609 Fung Nov 2006 A1
20070006001 Isobe et al. Jan 2007 A1
20070076653 Park et al. Apr 2007 A1
20070081315 Mondor Apr 2007 A1
20070094486 Moore et al. Apr 2007 A1
20070109968 Hussain et al. May 2007 A1
20070130397 Tsu Jun 2007 A1
20070174390 Silvain et al. Jul 2007 A1
20070180310 Johnson et al. Aug 2007 A1
20070209072 Chen Sep 2007 A1
20070226795 Conti et al. Sep 2007 A1
20070280230 Park Dec 2007 A1
20070286009 Norman Dec 2007 A1
20070288585 Sekiguchi et al. Dec 2007 A1
20080013453 Chiang et al. Jan 2008 A1
20080040463 Brown et al. Feb 2008 A1
20080052437 Loffink et al. Feb 2008 A1
20080059782 Kruse et al. Mar 2008 A1
20080075089 Evans et al. Mar 2008 A1
20080089358 Basso Apr 2008 A1
20080104264 Duerk et al. May 2008 A1
20080140771 Vass et al. Jun 2008 A1
20080140930 Hotchkiss Jun 2008 A1
20080159745 Segal Jul 2008 A1
20080162691 Zhang et al. Jul 2008 A1
20080183882 Flynn et al. Jul 2008 A1
20080186965 Zheng et al. Aug 2008 A1
20080199133 Takizawa et al. Aug 2008 A1
20080212273 Bechtolsheim Sep 2008 A1
20080212276 Bottom et al. Sep 2008 A1
20080217021 Lembcke et al. Sep 2008 A1
20080222434 Shimizu et al. Sep 2008 A1
20080235443 Chow et al. Sep 2008 A1
20080239649 Bradicich et al. Oct 2008 A1
20080243634 Dworkin et al. Oct 2008 A1
20080250181 Li et al. Oct 2008 A1
20080259555 Bechtolsheim Oct 2008 A1
20080259788 Wang et al. Oct 2008 A1
20080266793 Lee Oct 2008 A1
20080270599 Tamir et al. Oct 2008 A1
20080288660 Balasubramanian et al. Nov 2008 A1
20080288664 Pettey et al. Nov 2008 A1
20080288683 Ramey Nov 2008 A1
20080301794 Lee Dec 2008 A1
20080310848 Yasuda Dec 2008 A1
20080313369 Verdoorn et al. Dec 2008 A1
20080320161 Maruccia et al. Dec 2008 A1
20090021907 Mann et al. Jan 2009 A1
20090044036 Merkin Feb 2009 A1
20090063443 Arimilli et al. Mar 2009 A1
20090064287 Bagepalli et al. Mar 2009 A1
20090080428 Witkowski et al. Mar 2009 A1
20090097200 Sharma et al. Apr 2009 A1
20090113130 He et al. Apr 2009 A1
20090133129 Jeong et al. May 2009 A1
20090135751 Hodges et al. May 2009 A1
20090135835 Gallatin et al. May 2009 A1
20090158070 Gruendler Jun 2009 A1
20090172423 Song et al. Jul 2009 A1
20090198958 Arimilli et al. Aug 2009 A1
20090204834 Hendin et al. Aug 2009 A1
20090204837 Raval et al. Aug 2009 A1
20090216920 Lauterbach et al. Aug 2009 A1
20090219827 Chen et al. Sep 2009 A1
20090222884 Shaji et al. Sep 2009 A1
20090225751 Koenck et al. Sep 2009 A1
20090235104 Fung Sep 2009 A1
20090248943 Jiang et al. Oct 2009 A1
20090251867 Sharma et al. Oct 2009 A1
20090259863 Williams et al. Oct 2009 A1
20090259864 Li et al. Oct 2009 A1
20090265045 Coxe, III Oct 2009 A1
20090271656 Yokota et al. Oct 2009 A1
20090276666 Haley et al. Nov 2009 A1
20090279518 Falk et al. Nov 2009 A1
20090282274 Langgood et al. Nov 2009 A1
20090282419 Mejdrich et al. Nov 2009 A1
20090313390 Ahuja et al. Dec 2009 A1
20100005331 Somasundaram et al. Jan 2010 A1
20100008038 Coglitore Jan 2010 A1
20100008365 Porat Jan 2010 A1
20100026408 Shau Feb 2010 A1
20100040053 Gottumukkula et al. Feb 2010 A1
20100049822 Davies et al. Feb 2010 A1
20100051391 Jahkonen Mar 2010 A1
20100106987 Lambert et al. Apr 2010 A1
20100118880 Kunz et al. May 2010 A1
20100125742 Ohtani May 2010 A1
20100125915 Hall et al. May 2010 A1
20100138481 Behrens Jun 2010 A1
20100158005 Mukhopadhyay et al. Jun 2010 A1
20100161909 Nation et al. Jun 2010 A1
20100165983 Aybay et al. Jul 2010 A1
20100169479 Jeong et al. Jul 2010 A1
20100198972 Umbehocker Aug 2010 A1
20100218194 Dallman et al. Aug 2010 A1
20100220732 Hussain et al. Sep 2010 A1
20100250914 Abdul et al. Sep 2010 A1
20100265650 Chen et al. Oct 2010 A1
20100281246 Bristow et al. Nov 2010 A1
20100299548 Chadirchi et al. Nov 2010 A1
20100308897 Evoy et al. Dec 2010 A1
20100312910 Lin et al. Dec 2010 A1
20100312969 Yamazaki et al. Dec 2010 A1
20100318812 Auradkar et al. Dec 2010 A1
20110023104 Franklin Jan 2011 A1
20110026397 Saltsidis et al. Feb 2011 A1
20110029652 Chhuor et al. Feb 2011 A1
20110058573 Balakavi et al. Mar 2011 A1
20110075369 Sun et al. Mar 2011 A1
20110090633 Rabinovitz Apr 2011 A1
20110103391 Davis et al. May 2011 A1
20110113115 Chang et al. May 2011 A1
20110119344 Eustis May 2011 A1
20110123014 Smith May 2011 A1
20110138046 Bonnier et al. Jun 2011 A1
20110185370 Tamir et al. Jul 2011 A1
20110191514 Wu et al. Aug 2011 A1
20110191610 Agarwal et al. Aug 2011 A1
20110197012 Liao et al. Aug 2011 A1
20110210975 Wong et al. Sep 2011 A1
20110239014 Karnowski Sep 2011 A1
20110271159 Ahn et al. Nov 2011 A1
20110273840 Chen Nov 2011 A1
20110295991 Aida Dec 2011 A1
20110296141 Daffron Dec 2011 A1
20110320690 Petersen et al. Dec 2011 A1
20120011500 Faraboschi et al. Jan 2012 A1
20120020207 Corti et al. Jan 2012 A1
20120050981 Xu et al. Mar 2012 A1
20120054469 Ikeya et al. Mar 2012 A1
20120054511 Brinks et al. Mar 2012 A1
20120081850 Regimbal et al. Apr 2012 A1
20120096211 Davis et al. Apr 2012 A1
20120099265 Reber Apr 2012 A1
20120131201 Matthews et al. May 2012 A1
20120155168 Kim et al. Jun 2012 A1
20120198252 Kirschtein et al. Aug 2012 A1
20120207165 Davis Aug 2012 A1
20120297042 Davis et al. Nov 2012 A1
20130010639 Armstrong et al. Jan 2013 A1
20130024645 Cheriton et al. Jan 2013 A1
20130031331 Cheriton et al. Jan 2013 A1
20130058250 Casado et al. Mar 2013 A1
20130094499 Davis et al. Apr 2013 A1
20130097448 Davis et al. Apr 2013 A1
20130111107 Chang et al. May 2013 A1
20130148667 Hama et al. Jun 2013 A1
20130163605 Chandra et al. Jun 2013 A1
20130290643 Lim et al. Oct 2013 A1
20130290650 Chang et al. Oct 2013 A1
20130318269 Dalal et al. Nov 2013 A1
20140122833 Davis et al. May 2014 A1
20140359044 Davis et al. Dec 2014 A1
20140365596 Kanevsky et al. Dec 2014 A1
20150039840 Chandra et al. Feb 2015 A1
20150103826 Davis Apr 2015 A1
Foreign Referenced Citations (9)
Number Date Country
2005-223753 Aug 2005 JP
2005-536960 Dec 2005 JP
M377621 Apr 2010 TW
201017430 May 2010 TW
WO-2004021641 Mar 2004 WO
WO-2005013143 Feb 2005 WO
WO-2008000193 Jan 2008 WO
WO-2011044271 Apr 2011 WO
WO-2012037494 Mar 2012 WO
Non-Patent Literature Citations (119)
Entry
Notice of Allowance on U.S. Appl. No. 14/334,178 mailed Jun. 8, 2016.
Non-Final Office Action on U.S. Appl. No. 14/725,543 mailed Apr. 7, 2016.
Notice of Allowance on U.S. Appl. No. 13/624,725, mailed Mar. 30, 2016.
Das et al., “Unifying Packet and Circuit Switched Networks,” IEEE Globecom Workshops 2009, Nov. 30, 2009, pp. 1-6.
Final Office Action on U.S. Appl. No. 13/624,725 mailed Mar. 10, 2016.
Final Office Action on U.S. Appl. No. 13/662,759, mailed Feb. 22, 2016.
Non-Final Office Action on U.S. Appl. No. 12/889,721, mailed Feb. 24, 2016.
Final Office Action on U.S. Appl. No. 12/889,721 mailed Aug. 2, 2016.
Notice of Allowance on U.S. Appl. No. 14/725,543 mailed Jul. 21, 2016.
Non-Final Office Action on U.S. Appl. No. 13/234,054 mailed Oct. 20, 2016.
Notice of Allowance on U.S. Appl. No. 14/106,697 mailed Oct. 24, 2016.
Final Office Action on U.S. Appl. No. 14/052,723, mailed Dec. 3, 2015.
Non-Final Office Action on U.S. Appl. No. 14/334,178 mailed Dec. 18, 2015.
Non-Final Office Action on U.S. Appl. No. 14/334,931 Mailed Dec. 11, 2015.
Notice of Allowance on U.S. Appl. No. 13/692,741 mailed Dec. 4, 2015.
Advanced Switching Technology Tech Brief, published 2005, 2 pages.
Chapter 1 Overview of the Origin Family Architecture from Origin and Onyx2 Theory of Operations Manual, published 1997, 18 pages.
Cisco MDS 9000 Family Multiprotocol Services Module, published 2006, 13 pages.
Comparing the I2C BUS to the SMBUS, Maxim Integrated, Dec. 1, 2000, p. 1.
Deering, “IP Multicast Extensions for 4.3BSD UNIX and related Systems,” Jun. 1999, 5 pages.
Elghany et al., “High Throughput High Performance NoC Switch,” NORCHIP 2008, Nov. 2008, pp. 237-240.
Extended European Search Report for EP 10827330.1, mailed Jun. 5, 2013.
Final Office Action on U.S. Appl. No. 12/889,721, mailed Apr. 17, 2014.
Final Office Action on U.S. Appl. No. 13/692,741, mailed Mar. 11, 2015.
Final Office Action on U.S. Appl. No. 12/794,996, mailed Jun. 19, 2013.
Final Office Action on U.S. Appl. No. 12/889,721, mailed May 22, 2015.
Final Office Action on U.S. Appl. No. 13/234,054, mailed Apr. 16, 2015.
Final Office Action on U.S. Appl. No. 13/475,713, mailed Oct. 17, 2014.
Final Office Action on U.S. Appl. No. 13/475,722, mailed Oct. 20, 2014.
Final Office Action on U.S. Appl. No. 13/527,498, mailed Nov. 17, 2014.
Final Office Action on U.S. Appl. No. 13/527,505, mailed Dec. 5, 2014.
Final Office Action on U.S. Appl. No. 13/624,725, mailed Nov. 13, 2013.
Final Office Action on U.S. Appl. No. 13/624,731, mailed Jul. 25, 2014.
Final Office Action on U.S. Appl. No. 13/705,340, mailed Aug. 2, 2013.
Final Office Action on U.S. Appl. No. 13/705,414, mailed Aug. 9, 2013.
Final Office Action on U.S. Appl. No. 14/106,698, mailed Aug. 19, 2015.
Final Office Action on U.S. Appl. No. 14/334,931, mailed Jul. 9, 2015.
Final Office Action on U.S. Appl. No. 13/624,731, mailed Nov. 12, 2013.
fpga4fun.com,“What is JTAG?”, 2 pages, Jan. 31, 2010.
From AT to BTX: Motherboard Form Factor, Webopedia, Apr. 29, 2005, p. 1.
Grecu et al., “A Scalable Communication-Centric SoC Interconnect Architecture” Proceedings 5th International Symposium on Quality Electronic Design, 2005, pp. 343, 348 (full article included).
Hossain et al., “Extended Butterfly Fat Tree Interconnection (EFTI) Architecture for Network on CHIP,” 2005 IEEE Pacific Rim Conference on Communicatinos, Computers and Signal Processing, Aug. 2005, pp. 613-616.
HP Virtual Connect Traffic Flow—Technology brief, Jan. 2012, 22 pages.
International Preliminary Report on Patentability for PCT/US2009/044200, mailed Nov. 17, 2010.
International Preliminary Report on Patentability for PCT/US2012/038986 issued on Nov. 26, 2013.
International Preliminary Report on Patentability for PCT/US2012/061747, mailed Apr. 29, 2014.
International Preliminary Report on Patentability issued on PCT/US12/62608, issued May 6, 2014.
International Search Report and Written Opinion for PCT/US12/38987, mailed Aug. 16, 2012.
International Search Report and Written Opinion for PCT/US12/61747, mailed Mar. 1, 2013.
International Search Report and Written Opinion for PCT/US12/62608, mailed Jan. 18, 2013.
International Search Report and Written Opinion for PCT/US2010/053227, mailed May 10, 2012.
International Search Report and Written Opinion for PCT/US2011/051996, mailed Jan. 19, 2012.
International Search Report and Written Opinion on PCT/US09/44200, mailed Jul. 1, 2009.
International Search Report and Written Opinion on PCT/US2012/038986, mailed Mar. 14, 2013.
Jansen et al., “SATA-IO to Develop Specification for Mini Interface Connector” Press Release Sep. 21, 2009, Serial ATA3 pages.
Nawathe et al., “Implementation of an 8-Core, 64-Thread, Power Efficient SPARC Server on a Chip”, IEEE Journal of Solid-State Circuits, vol. 43, No. 1, Jan. 2008, pp. 6-20.
Non-Final Action on U.S. Appl. No. 13/728,362, mailed Feb. 21, 2014.
Non-Final Office Action on U.S. Appl. No. 12/889,721, mailed Jul. 2, 2013.
Non-Final Office Action on U.S. Appl. No. 13/475,722, mailed Jan. 17, 2014.
Non-Final Office Action on U.S. Appl. No. 12/794,996, mailed Sep. 17, 2012.
Non-Final Office Action on U.S. Appl. No. 12/889,721, mailed Oct. 11, 2012.
Non-Final Office Action on U.S. Appl. No. 12/889,721, mailed Sep. 29, 2014.
Non-Final Office Action on U.S. Appl. No. 13/234,054, mailed Oct. 23, 2014.
Non-Final Office Action on U.S. Appl. No. 13/234,054, mailed Aug. 6, 2015.
Non-Final Office Action on U.S. Appl. No. 13/284,855, mailed Dec. 19, 2013.
Non-Final Office Action on U.S. Appl. No. 13/453,086, mailed Mar. 12, 2013.
Non-Final Office Action on U.S. Appl. No. 13/475,713, mailed Apr. 1, 2014.
Non-Final Office Action on U.S. Appl. No. 13/527,505, mailed May 8, 2014.
Non-Final Office Action on U.S. Appl. No. 13/527,498, Mailed May 8, 2014.
Non-Final Office Action on U.S. Appl. No. 13/624,725, mailed Jan. 10, 2013.
Non-Final Office Action on U.S. Appl. No. 13/624,725, mailed Apr. 23, 2015.
Non-final office action on U.S. Appl. No. 13/624,731 mailed Jan. 29, 2013.
Non-Final Office Action on U.S. Appl. No. 13/662,759, mailed Nov. 6, 2014.
Non-Final Office Action on U.S. Appl. No. 13/692,741, mailed Sep. 4, 2014.
Non-Final Office Action on U.S. Appl. No. 13/692,741, mailed Jul. 1, 2015.
Non-Final Office Action on U.S. Appl. No. 13/705,286, mailed May 13, 2013.
Non-Final Office Action on U.S. Appl. No. 13/705,340, mailed Mar. 12, 2014.
Non-Final Office Action on U.S. Appl. No. 13/705,340, mailed Mar. 29, 2013.
Non-Final Office Action on U.S. Appl. No. 13/705,414, mailed Apr. 9, 2013.
Non-Final Office Action on U.S. Appl. No. 13/728,308, mailed May 14, 2015.
Non-Final Office Action on U.S. Appl. No. 13/728,428, mailed Jun. 12, 2015.
Non-Final Office Action on U.S. Appl. No. 14/052,723, mailed May 1, 2015.
Non-Final Office Action on U.S. Appl. No. 14/106,697, mailed Aug. 17, 2015.
Non-Final Office Action on U.S. Appl. No. 14/106,698, mailed Feb. 12, 2015.
Non-Final Office Action on U.S. Appl. No. 14/334,931, mailed Jan. 5, 2015.
Non-Final Office Action on U.S. Appl. No. 13/705,428, mailed Jul. 10, 2013.
Notice of Allowance on U.S. Appl. No. 13/453,086, mailed Jul. 18, 2013.
Notice of Allowance on U.S. Appl. No. 13/475,713, mailed Feb. 5, 2015.
Notice of Allowance on U.S. Appl. No. 13/475,722, mailed Feb. 27, 2015.
Notice of Allowance on U.S. Appl. No. 13/527,498, mailed Feb. 23, 2015.
Notice of Allowance on U.S. Appl. No. 13/527,505, mailed Mar. 6, 2015.
Notice of Allowance on U.S. Appl. No. 13/624,731, mailed Mar. 5, 2015.
Notice of Allowance on U.S. Appl. No. 13/705,340, mailed Dec. 3, 2014.
Notice of Allowance on U.S. Appl. No. 13/705,386, mailed Jan. 24, 2014.
Notice of Allowance on U.S. Appl. No. 13/705,414, mailed Nov. 4, 2013.
Notice of Allowance on U.S. Appl. No. 13/284,855, mailed Jul. 14, 2014.
Office Action on Taiwan Application 101139729, mailed May 25, 2015 (English translation not available).
Pande et al., “Design of a Switch for Network on Chip Applications,” May 25-28, 2003 Proceedings of the 2003 International Symposium on Circuits and Systems, vol. 5, pp. V217-V220.
Reexamination Report on Japanese Application 2012-536877, mailed Jan. 22, 2015 (English Translation not available).
Search Report on EP Application 10827330.1, mailed Feb. 12, 2015.
Venaas, “IPv4 Multicast Address Space Registry,” 2013, http://www.iana.org/assignments/multicast-addresses/multicast-addresses.xhtml.
Final Office Action on U.S. Appl. No. 13/234,054, mailed Jan. 26, 2016.
Final Office Action on U.S. Appl. No. 14/106,697 mailed Feb. 2, 2016.
Notice of Allowance on U.S. Appl. No. 13/728,428 mailed Jul. 18, 2016.
Final Office Action on U.S. Appl. No. 14/334,178, mailed Nov. 4, 2015.
Notice of Allowance U.S. Appl. No. 13/728,308, mailed Oct. 7, 2015.
Office Action on Taiwan Application 100133390, mailed Aug. 25, 2015 (English translation not available).
Final Office Action on U.S. Appl. No. 13/728,428 mailed May 6, 2016.
Notice of Allowance on U.S. Appl. No. 14/334,931 mailed May 20, 2016.
Notice of Allowance on U.S. Appl. No. 13/662,759 mailed May 10, 2016.
Non-Final Office Action on U.S. Appl. No. 15/270,418 mailed Apr. 21, 2017.
Notice of Allowance on U.S. Appl. No. 15/360,668, mailed May 5, 2017.
HP ProLiant SL6500 Scalable System, Family data sheet, HP Technical sheet, Sep. 2010 4 pages.
Non-Final Office Action on U.S. Appl. No. 15/281,462 mailed Feb. 10, 2017.
Non-Final Office Action on U.S. Appl. No. 14/809,723 mailed Dec. 30, 2016.
Notice of Allowance issued on U.S. Appl. No. 14/052,723, mailed Feb. 8, 2017.
Final Office Action on U.S. Appl. No. 13/234,054 dated May 31, 2017.
Final Office Action on U.S. Appl. No. 15/281,462 dated Jun. 13, 2017.
Non-Final Office Action on U.S. Appl. No. 15/254,111 dated Jun. 20, 2017.
Related Publications (1)
Number Date Country
20150378958 A1 Dec 2015 US
Provisional Applications (1)
Number Date Country
61553555 Oct 2011 US
Continuations (1)
Number Date Country
Parent 13527498 Jun 2012 US
Child 14753948 US