Embodiments of the present invention relate to data communication, and more particularly to packet-based communication.
Most computer systems are formed of components coupled together using one or more buses, which are used to transmit information between the various system components. Present bus standards such as the Peripheral Component Interconnect (PCI) Specification, Rev. 2.1 (published Jun. 1, 1995) provide a multi-drop bus in which multiple devices are coupled to the same bus. Accordingly, it is easy to read or write to devices on the same bus.
However, as bus interface speeds increase, bus architectures are moving away from multi-drop architectures towards point-to-point architectures. In point-to-point architectures, peer-to-peer communication becomes more difficult as synchronization, ordering, and coherency of such communications becomes more difficult. One example of a point-to-point architecture is a PCI Express™ architecture in accordance with the PCI Express Base Specification, Rev. 1.0 (published Jul. 22, 2002).
Communication between serially connected devices typically involves buffering data to be sent in a transmitting device and then sending the data in a packetized form to a receiving device. Typically, an entire first packet is sent before a next packet can begin transmission. However, some systems provide for interleaving of multiple packets. Typically to handle such interleaving, a sideband channel is used to communicate information regarding the interleaving, including the channel used for interleaving, the length of the interleaving and so forth. Use of sideband resources negatively affects system performance and complicates packet interleaving. Accordingly, a need exists for improved packet interleaving.
Referring to
Referring to
In the embodiment of
As shown in
Also shown in
While the I/O hubs shown in
In various embodiments, each port of I/O hubs 20 and 30 may include a plurality of channels, often referred to herein as “virtual channels” that together may form one or more virtual networks and associated buffers to communicate data, control and status information between various devices. In one particular embodiment, each port may include up to at least six such channels, including a home (HOM) channel, a snoop (SNP) channel, a response (NDR) channel, a short message (NCS) channel, a data (NCB) channel and a data response (DR) channel, although the scope of the present invention is not so limited.
In other embodiments, additional or different virtual channels may be present in a desired protocol. Further, while discussed herein as being used within a coherent system, it is to be understood that other embodiments may be implemented in a non-coherent system to provide for deadlock-free routing of transactions. In some embodiments, the channels may keep traffic separated through various layers of the system, including, for example, physical, link, and routing layers, such that there are no dependencies.
Referring now to
First agent 110 may include a plurality of layers, including a physical layer 114, a link layer 118, a routing layer 124 and a protocol layer 125. Packets may be received in first agent 110 at physical layer 114, which may include a plurality of electrical transfer ports 112a and 112b. In various embodiments, physical layer 114 may be used to provide electrical transfer of information between two directly connected coherent agents via a fabric. While shown as including two such ports in the embodiment of
Packets communicated between agents may include a header and associated information (e.g., a “payload”). The packets are then forwarded to link layer 118, which may include a plurality of buffer flow control mechanisms. Shown in the embodiment of
Link layer 118, in various embodiments, may be used to provide reliable data transfer and flow control between two directly connected coherent agents, and also to virtualize a physical channel between the devices. Link layer 118 may be flow controlled to prevent overflow or underflow, and may also provide error detection and recovery. In certain embodiments, flow control may be implemented using packet-based and/or flit-based level credit/debit flow control. A flit may refer to a smallest unit of data transfer which, in one embodiment, may be 80 bits long. A packet corresponds to a logical grouping of one or more flits. In certain embodiments, a packet may be the smallest unit of information that includes routing and other information via a header. Each packet may contain a header and an optional data field. In certain embodiments, the header may be one to two flits long, and a data portion may extend to a cacheline size of the coherent system. In various embodiments, link layer 118 may virtualize a physical channel into multiple message classes and virtual networks. In such manner, a physical channel may be multiplexed among multiple virtual channels.
When flits are properly received, link layer 118 may provide the flits to routing layer 124. In various embodiments, routing layer 124 may provide a flexible and distributed manner of routing coherent packets from source to destination. The destination-based routing may be determined using routing tables within routing layer 124, which allows for adaptive routing and on-line reconfiguration of the routing tables. In one embodiment, a header generated by a transaction layer may include information that identifies a destination, an input port and a virtual network, and these values may be input into a routing table to determine the desired output port and the virtual network on which the transaction may flow. In various embodiments, routing tables may be stored in protected configuration space and programmed by firmware. Such routing tables may be programmed to enable deadlock-free routing for various topologies. Such destination-based routing may be based on a node identifier (node ID), which uniquely identifies an agent within the coherent system. Thus flits received from link layer 118 may be multiplexed in routing layer 124 through a switch 120 and provided to routing tables 122, which use, for example, a destination node ID field value to route the associated header and payload (if present) to the proper agent. If a packet is destined for agent 110, routing tables 122 will provide the packet to protocol layer 125. Alternately, if the packet is destined for another destination, the header and packet will be sent back through link layer 118 and physical layer 114 of agent 110 to a different agent.
In one embodiment, a protocol layer 125 (also referred to as a “transaction layer”) may include various protocol engines to form, order, and process packets for transmission through a coherent system. For example, in one embodiment, transaction layer 125 may include a coherence engine 127a, an interrupt engine 127C, and an I/O engine 127d, among others. Using such engines, transaction layer 125 may form a packet having a common system header (also referred to as a “system interconnect header”). The packets thus formed in transaction layer 125 may be provided to routing layer 124 on a given virtual channel.
In various embodiments of the present invention, protocol layer 125 may use a transaction ID associated with the header to order a transaction and perform desired processing thereon using various engines within protocol layer 125. Shown as representative engines in the embodiment of
Similar functionality may be present in second agent 150, which includes a corresponding physical layer 154, link layer 158, routing layer 164, and protocol layer 175. As further shown in
Further shown in
In various embodiments of the present invention to improve packet transmission, flits of one packet may be interleaved amongst flits of a different packet. Typically, interleaving in accordance with an embodiment of the present invention may be performed if a current packet being transmitted is a large packet, e.g., a data packet including many flits. Such flits are typically contiguously sent from a transmitter and received by a receiver. However, it may be desirable to transmit a relatively small packet (e.g., a packet with only one or a few flits) amongst the flits of this larger packet. To do so, an interleave indicator within the packet communication may be sent from sender to receiver to indicate presence of the interleaved packet. In some embodiments, interleaving may occur only during transmission of large packets. That is, interleaving of one small packet into transmission of another small packet may not be permitted, in some embodiments.
As several examples, insertion of a command or a special packet may occur during transmission of a current data packet or a data packet may be inserted into another data packet. The command packet may be a protocol message or a link layer message, in some embodiments. Typically, a command packet is sent to completion, and then the original packet may continue transmission. However, in some embodiments another interleaved packet may be transmitted contiguously after the first interleaved packet by setting of the interleave indicator for the new interleaved packet.
In yet other embodiments, two data packets may be interleaved so that two data streams are interleaved in a scheduled manner. In this manner, a minimum latency can be achieved when a sender can ready two streams at the same time.
Referring now to
During packet transmission, if interleaving is desired, the sending endpoint may receive a request to interleave at least one flit of an interleaved packet (block 230). For example, such a request may come from the sending endpoint itself. Interleaving may be used to rapidly send a short message, such as a command or status information or other special packet, during transmission of a longer message, such as data transmission. When an interleave request is received, the sending endpoint may transmit a flit of the interleaved packet. More specifically, the interleaved packet's flit may be sent with its interleave bit set (block 240). Beyond this interleave bit, no other information is sent to signify either a beginning or end of an interleave, nor on which channel the interleave is to occur. However, the packet header of the new packet may indicate a length of the new packet, as well as the new virtual channel on which the packet is sent. That is, in various embodiments an interleaved packet is sent on a different virtual channel than the packet currently being sent. As will be described further below, via the interleave indicator (i.e., the interleave bit), the receiving endpoint may determine that it has received an interleaved flit and process the packet including the interleaved flit (i.e., the second packet) accordingly.
Next it may be determined whether an additional interleaved flit of this interleaved packet is to be sent from the sending endpoint (diamond 250). If not, control passes to diamond 258. There it may be determined whether an additional interleaved packet is desired to be transmitted. If not, control returns to block 220 where a next flit of the first packet may be sent (block 220). As described above, this flit may be sent with its interleave bit cleared. If instead at diamond 258 it is determined that an additional interleaved packet is desired to be sent, control passes back to block 240, discussed above.
If instead at diamond 250 it is determined that an additional interleaved flit of the first interleaved packet is to be sent, control passes to block 255. There, the additional interleaved flit may be transmitted (block 255). In various embodiments, this additional interleaved flit may be sent with its interleave indicator cleared, indicating that the flit is part of the interleaved packet. From block 255, control passes back to diamond 250, discussed above.
As described above in various implementations, interleaving may occur only if a currently transmitting packet is of a particular size. Accordingly, prior to permitting an interleave operation, it may first be determined whether the currently transmitting packet is of a size is at least equal to a predetermined threshold. While such a threshold may vary in different embodiments, in some embodiments an interleaved packet may be at least longer than a remaining portion of the non-interleaved packet for interleaving to occur. While shown with these particular operations in the embodiment of
Accordingly flexible message transmission may be effected, as flits of different packets can be interleaved within one or more virtual channels for transmission between endpoints. Furthermore, the interleaved flits of different packets may be sent on multiple channels within one or more virtual networks.
Referring now to
If it has been determined that the flit is not interleaved, control may pass to block 275. There, the flit may be processed. As an example, the receiving endpoint may store a payload portion of the flit in a buffer corresponding to the packet of which the flit is a part. For example, the flit may be stored in a buffer associated with the virtual channel on which the flit was received. For example, a data buffer may be associated with data packets, which are generally not sent via interleaving. Furthermore, status information corresponding to the packet, which may be obtained from a header of the packet, may be stored in a current packet status storage of the receiving endpoint. As an example, this storage may include a current virtual channel indicator storage and a packet length storage. The current virtual channel indicator storage may store an indication of the virtual channel on which the flit was received, while the packet length storage may store an indication of the current packet's length (e.g., in flits). Upon processing the current flit, control may pass from block 275 to block 265 for reception of a next flit, which may or may not be part of the same packet.
If instead at diamond 270 it is determined that the currently received flit is part of an interleaved packet, control passes to block 280. There, status information of the prior packet may be stored (block 280). More specifically, if the prior packet had a flit length greater than the amount of received flits for the packet, status information that was previously stored in the current status storage of the receiving endpoint may be transferred to a prior status storage of the receiving endpoint.
In such manner, the prior packet, including payload flits and information corresponding to the packet (e.g., virtual channel and packet length), may be placed aside to allow the receiver to process the interleaved flit (block 285). In various embodiments, this flit processing may be performed as discussed above regarding a non-interleaved flit. However, the packet data, which may be a command, may be stored in a different buffer. That is, interleaved packets are typically sent on a different virtual channel than the packet in which it is interleaved. Accordingly, payload data of the interleaved flits may be stored in a buffer associated with the new virtual channel. For example, such a buffer may correspond to a command buffer with which interleaved packets are associated. Upon completion of processing, control may return to block 265 for receipt of a next flit, which again may be a interleaved or non-interleaved flit. In some implementations, one or more packets may thus be nested within a first packet, and further, double nesting of packets may occur.
Referring now to
While not shown in the embodiment of
As shown in
At a next time, time one, first agent 300 receives a one-flit packet on virtual channel B having its interleave bit set. Accordingly, the status information previously stored in current status storages 302 and 304 is moved to prior status storages 306 and 308, and the current packet information (i.e., current virtual channel B having a current length of one) is stored in current status storages 302 and 304. Similarly, second agent 310 receives a two-flit length packet on virtual channel A having its interleave bit set. Accordingly, status information previously stored in current status storages 312 and 314 is moved to prior status storages 316 and 318, and the current status information is placed in current status storages 312 and 314. Note that at time one, prior length status storage 318 has a length of two corresponding to the prior three-flit packet sent, decremented by one flit for the previously received flit.
At a next time two, first agent 300 again receives a one-flit length packet on virtual channel B having its interleave bit set. This interleaved flit is thus of a new interleaved packet, and the status information stored in first agent 300 thus remains the same. Also at time two, second agent 310 receives a second flit of the interleaved packet on virtual channel A having its interleave bit set to zero. Note that the interleave bit of this packet is zero, indicating it is part of the interleaved packet initiated at time one. Accordingly, the only status information that is changed in second agent 310 is that current length status storage 314 is decremented to a value of one.
At a next time three, first agent 300 receives a second flit of the original packet on virtual channel A having its interleave bit cleared. Accordingly, the prior status information previously stored in prior status storages 306 and 308 is moved into current status storages 302 and 304 (and the current length information is updated accordingly). Similarly, second agent 310 receives a second flit of its original packet on virtual channel B having its interleave bit cleared. Accordingly, its status storage information is manipulated in the same manner.
At time four, first agent 300 receives a one-flit packet on virtual channel B having its interleave bit set. Accordingly, the status information that was stored in current status storages 302 and 304 is moved to prior status storages 306 and 308 (note that the length information of status storage 308 has been updated accordingly), and the current status information is placed into current status storages 302 and 304. Also at time four, second agent 310 receives a third flit of its original packet on virtual channel B having its interleave bit cleared. Accordingly, current length status storage 314 is updated accordingly in second agent 310.
Still referring to
While shown with this example sequence in
Embodiments may be implemented in code and may be stored on a storage medium having stored thereon instructions which can be used to program a system to perform the instructions. The storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), flash memories, electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, or any other type of media suitable for storing electronic instructions.
While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.
This application is a continuation of U.S. patent application Ser. No. 11/211,063, filed Aug. 24, 2005, the content of which is hereby incorporated by reference.
Number | Name | Date | Kind |
---|---|---|---|
4542380 | Beckner et al. | Sep 1985 | A |
5303302 | Burrows | Apr 1994 | A |
5343473 | Cidon et al. | Aug 1994 | A |
5406504 | Denisco et al. | Apr 1995 | A |
5440752 | Lentz et al. | Aug 1995 | A |
5557608 | Calvignac et al. | Sep 1996 | A |
5781774 | Krick | Jul 1998 | A |
5838985 | Ohki | Nov 1998 | A |
5895487 | Boyd et al. | Apr 1999 | A |
5956729 | Goetz et al. | Sep 1999 | A |
5960455 | Bauman | Sep 1999 | A |
6009488 | Kavipurapu | Dec 1999 | A |
6209079 | Otani et al. | Mar 2001 | B1 |
6311258 | Gibson et al. | Oct 2001 | B1 |
6460080 | Shah et al. | Oct 2002 | B1 |
6493402 | Fimoff | Dec 2002 | B1 |
6516375 | Ajanovic | Feb 2003 | B1 |
6615319 | Khare | Sep 2003 | B2 |
6628615 | Joseph et al. | Sep 2003 | B1 |
6629219 | Manseau | Sep 2003 | B1 |
6633564 | Steer et al. | Oct 2003 | B1 |
6683884 | Howard | Jan 2004 | B1 |
6724721 | Cheriton | Apr 2004 | B1 |
6754209 | Stachura | Jun 2004 | B1 |
6772251 | Hastings | Aug 2004 | B1 |
6842827 | Blankenship | Jan 2005 | B2 |
6888843 | Keller | May 2005 | B2 |
6889284 | Nizar | May 2005 | B1 |
6891855 | Bruckman | May 2005 | B2 |
6901049 | Chapman | May 2005 | B1 |
6912251 | Ward et al. | Jun 2005 | B1 |
6912612 | Kapur | Jun 2005 | B2 |
6996100 | Haartsen | Feb 2006 | B1 |
7047374 | Sah | May 2006 | B2 |
7050410 | Kim | May 2006 | B1 |
7123621 | Niida et al. | Oct 2006 | B1 |
7142540 | Hendel et al. | Nov 2006 | B2 |
7203775 | Van Doren | Apr 2007 | B2 |
7218638 | Moll | May 2007 | B2 |
7225300 | Choquette et al. | May 2007 | B1 |
7245613 | Winkles et al. | Jul 2007 | B1 |
7248596 | Shimizu | Jul 2007 | B2 |
7318114 | Cypher | Jan 2008 | B1 |
7321596 | Ohnishi | Jan 2008 | B2 |
7395483 | Tourancheau | Jul 2008 | B1 |
7461218 | Muthrasanallur | Dec 2008 | B2 |
7558954 | Apostolopoulos et al. | Jul 2009 | B2 |
7698478 | Wang et al. | Apr 2010 | B2 |
7949794 | Ajanovic et al. | May 2011 | B2 |
20010013080 | Kameyama | Aug 2001 | A1 |
20020044560 | Dally et al. | Apr 2002 | A1 |
20020049901 | Carvey | Apr 2002 | A1 |
20030093631 | Osborne | May 2003 | A1 |
20030110422 | Naffziger | Jun 2003 | A1 |
20030128705 | Yi et al. | Jul 2003 | A1 |
20030163649 | Kapur | Aug 2003 | A1 |
20030177313 | Iyer et al. | Sep 2003 | A1 |
20030177320 | Sah | Sep 2003 | A1 |
20040022094 | Radhakrishnan | Feb 2004 | A1 |
20040114609 | Swarbrick | Jun 2004 | A1 |
20040160970 | Dally | Aug 2004 | A1 |
20050018609 | Dally | Jan 2005 | A1 |
20050088967 | Cen | Apr 2005 | A1 |
20050172091 | Rotithor et al. | Aug 2005 | A1 |
20050216638 | Smith et al. | Sep 2005 | A1 |
20050251599 | Hum et al. | Nov 2005 | A1 |
20050251611 | Creta et al. | Nov 2005 | A1 |
20050251612 | Creta et al. | Nov 2005 | A1 |
20050259696 | Steinman | Nov 2005 | A1 |
20080140909 | Flynn | Jun 2008 | A1 |
20090323547 | Caesar et al. | Dec 2009 | A1 |
Number | Date | Country |
---|---|---|
61241843 | Oct 1986 | JP |
Entry |
---|
PCI Express Base Specification Revision 1, Jul. 22, 2002, 423 pages. |
“PCI Express and Advanced Switching: Evolutionary Path to Building Next Generation Interconnects”. David Maythew and Krishnan. Proceedings of the 11th Symposium on High Performance Interconnects, Aug. 20-22, 2003. |
U.S. Appl. No. 11/211,046, filed Aug. 24, 2005, entitled “Buffering Data Packets According to Multiple Flow Control Schemes” by Aaron T. Sink and Herbert H.J. Hum. |
Number | Date | Country | |
---|---|---|---|
20130070779 A1 | Mar 2013 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 11211063 | Aug 2005 | US |
Child | 13676891 | US |