Patent Application
20090031067
1. Field of the Invention
The present invention relates to computer systems and, more specifically, to a computer system in which a root device communicates with a plurality of other devices.
2. Description of the Prior Art
In computer systems, increasing processor performance is driving the need for higher bandwidth on input-output (I/O) interfaces, memory interfaces and inter-processor interconnects. Generally, these interfaces are changing from multi-drop bidirectional parallel busses to point-to-point high speed unidirectional serial busses. The higher frequencies required to support the faster data rates can not be supported on multi-drop busses, driving topologies to point-to-point busses.
The directional turn-around penalties on high speed busses with multiple transfers in flight tend to be too extreme, resulting in a transition from shared bidirectional busses to separate unidirectional busses. The power and area expense of the electrical I/O used to drive each unidirectional high speed bus interconnect wire has driven the transition from wide parallel busses to narrow serial busses. On a wide parallel bus, a transaction can be sent in one bus clock cycle across the interface. On a serial bus, the transaction is broken into packets and driven on the bus over multiple clock cycles. The packets are reassembled at the receiving end to form the original transaction.
As serial high speed unidirectional busses become more common in computer systems, techniques must be developed to make them more efficient and reliable. Also, when serial high speed unidirectional busses develop faults or are temporarily congested, data transfer rates can be impacted.
Therefore, there is a need for a high speed unidirectional interconnect that provides both higher reliability and higher efficiency than existing topologies
The disadvantages of the prior art are overcome by the present invention which, in one aspect, is a data communications apparatus that includes a central device and a plurality of communication devices. The central device includes a plurality of central port pairs, in which each central port pair includes an input port and an output port. The plurality of communication devices is arranged in a spoke and ring configuration, in which each communication device is part of a communication spoke. Each communication spoke is in communication with a different central port pair. Each communication device is also a part of a communication ring, so that each communication device in a selected communication ring belongs to a different communication spoke.
In another aspect, the invention is a data memory apparatus that includes a memory controller and a plurality of buffered random access memories. Each memory is part of a selected one of a plurality of communication spokes that is coupled to the memory controller. Each memory is also part of a selected one of a plurality of communication rings. The memories are coupled so that no two memories are part of both a common communication spoke and a common communication ring.
In yet another aspect, the invention is a method of coupling communication devices to a central device. Each communication device includes at least three device port pairs and in which each device port pair includes an input port and an output port. The central device includes a plurality of central port pairs, each central port pair including an input port and an output port. A plurality of communication spokes is formed by coupling an input port of a first adjacent communication device to an output port of a second adjacent communication device and by coupling an output port of the first adjacent communication device to an input port of the second adjacent communication device. Each spoke is coupled to a different input port-output port pair of a central device. A plurality of communication rings is formed by coupling each first communication device to a different second communication device that is not included in a spoke in which the first communication device is included. A system in which each of the communication devices decides which other communication device to send data to, based on predefined criteria, is applied for routing data from the central device to the communication devices.
These and other aspects of the invention will become apparent from the following description of the preferred embodiments taken in conjunction with the following drawings. As would be obvious to one skilled in the art, many variations and modifications of the invention may be effected without departing from the spirit and scope of the novel concepts of the disclosure.
A preferred embodiment of the invention is now described in detail. Referring to the drawings, like numbers indicate like parts throughout the views. As used in the description herein and throughout the claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise: the meaning of “a,” “an,” and “the” includes plural reference, the meaning of “in” includes “in” and “on.”
As shown in
As shown in
Returning to
Returning to
In one embodiment, as shown in
One simple scheme for determining primary and alternate paths is destination-to-port mapping. At initialization, firmware explores the system topology and configures port mapping tables based on the topology it finds. Each DRAM in the topology is assigned an ID. The DRAM ID of the memory operation is included in a transaction packet. If the DRAM ID does not match the packet ID, the transaction packet is forwarded. If the DRAM ID matches the packet ID, the DRAM knows the transaction packet is for its local array. If the primary path fails, a secondary path is taken. If the secondary path fails, the next path in a priority table is chosen, if more than two paths exist. And so on, until all paths have been tried. The following table shows a forwarding hierarchy in accordance with this method:
If a transaction queue for one of the out ports becomes full, the priority routing map can be dynamically modified for periods of high congestion. For example, using the simple prioritized port routing map above, if a transaction comes to In Port 1, it would be normally routed Out Port A if port A was functional. With congestion management, a transaction that comes In Port 1 would be routed Out Port C even if Out Port A was functional, but only if Out Port A's transaction queue was full.
Transaction routing could also be done based on address ranges. In this embodiment, each DRAM 220 would be assigned an address range by firmware at initialization. The address range for each DRAM 220 would be loaded into registers in each DRAM 220. The address range would cover the storage locations in that DRAM 220. Transactions are sent with the memory address of the transaction. The DRAM 220 would execute the transaction if the address contained in the transaction packet mapped to its address range. If the memory address contained in the transaction packet did not match the DRAM's address range, the transaction would be forwarded. The priority of the port out of which it is forwarded would be looked up in a table. The table would be initialized with address ranges in the top row instead of destination ID tags. The following is an example of such a table for the DRAM 220 with ID 000:
The memory controller 210 could control all rerouting with no routing decisions done in the DRAMs 220, and with end-to-end retry at the path level. If the memory controller 210 did not get a response from the destination within a set period of time or a response was received indicating a bad interconnect, the memory controller would reissue the transaction using an alternate route.
The memory controller 210 would send routing information with the transaction. For example, the transaction could include information to go out port A on the first DRAM it travels through and port C on the second DRAM and then it is at the destination. On the return path, take port C on the first DRAM it travels through, port B at the second DRAM, and then it reaches the memory controller. There can be any number of levels of DRAMs that are traveled through and the number of levels to and from does not have to be equal.
This method of routing transactions by the memory controller 210 can be used to balance traffic and reduce congestion. If the memory controller 210 just sent several commands down the same path, it could send the next command down an alternate path, even if the path was longer, to reduce congestion.
While this embodiment uses memory interconnect as an example, the invention is not limited to memory interconnect. It may also apply, for example, to processor interconnect, I/O interconnect, storage interconnect, and any other application that supports redrive (daisy chaining) to add additional components. For memory interconnect, it may apply to the DRAM level or the traditional DIMM level (e.g., a Fully Buffered DIMM).
One embodiment, as shown in
While the above-described embodiments show spokes with only two communicating devices, it should be understood that the spokes can include more than just two devices. For example, as shown in
The above described embodiments, while including the preferred embodiment and the best mode of the invention known to the inventor at the time of filing, are given as illustrative examples only. It will be readily appreciated that many deviations may be made from the specific embodiments disclosed in this specification without departing from the spirit and scope of the invention. Accordingly, the scope of the invention is to be determined by the claims below rather than being limited to the specifically described embodiments above.
