Embodiments of the present disclosure relate generally to multi-protocol tunneling across a multi-protocol I/O interconnect of a computer apparatus.
Conventional computer platform architectures include a variety of host controllers to implement a number of different types of I/O between computer platforms and peripheral devices that are connected to the platforms, and these computer platforms generally include protocol-specific connection interfaces that connect to the peripheral devices via protocol-specific plugs and cables. For example, a computer may include one or more of a USB-specific controller that connects to a peripheral device via a USB-specific connection interface, a display-specific controller (e.g., DisplayPort) that connects to a peripheral device via a display-specific connection interface, a PCI Express®-controller that connects to a peripheral device via a PCI Express®-specific connection interface, and so on.
Embodiments of the present disclosure will be described by way of example embodiments, but not limitations, illustrated in the accompanying drawings in which like references denote similar elements, and in which:
Various aspects of the illustrative embodiments will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. However, it will be apparent to those skilled in the art that alternate embodiments may be practiced with only some of the described aspects. For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to one skilled in the art that alternate embodiments may be practiced without the specific details. In other instances, well-known features are omitted or simplified in order not to obscure the illustrative embodiments.
Further, various operations will be described as multiple discrete operations, in turn, in a manner that is most helpful in understanding the illustrative embodiments; however, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations need not be performed in the order of presentation. Moreover, methods within the scope of this disclosure may include more or fewer steps than those described.
The phrase “in some embodiments” is used repeatedly. The phrase generally does not refer to the same embodiments; however, it may. The terms “comprising,” “having,” and “including” are synonymous, unless the context dictates otherwise. The phrase “A and/or B” means (A), (B), or (A and B). The phrase “A/B” means (A), (B), or (A and B), similar to the phrase “A and/or B”. The phrase “at least one of A, B and C” means (A), (B), (C), (A and B), (A and C), (B and C) or (A, B and C). The phrase “(A) B” means (B) or (A and B), that is, A is optional.
As shown in
In previously implemented computer apparatuses, an I/O link connecting a peripheral device to a computer system is protocol-specific with a protocol-specific connector port that allows a compatible peripheral device to be attached to the protocol-specific connector port (i.e., a USB keyboard device would be plugged into a USB port, a router device would be plugged into a LAN/Ethernet port, etc.) with a protocol-specific cable. Any single connector port would be limited to peripheral devices with a compatible plug and compatible protocol. Once a compatible peripheral device is plugged into the connector port, a communication link would be established between the peripheral device and a protocol-specific controller.
In the computer apparatus as described in the embodiment shown in
A non-protocol-specific connector port 112 may be configured to couple the I/O interconnect 108 with a connector port (not shown) of the device 110, allowing multiple device types to attach to the computer system 100 through a single physical connector port 112. Moreover, the I/O link between the device 110 and the I/O complex 106 may be configured to carry multiple I/O protocols (e.g., PCI Express®, USB, DisplayPort, HDMI®, etc.) simultaneously. In various embodiments, the connector port 112 may be capable of providing the full bandwidth of the link in both directions with no sharing of bandwidth between ports or between upstream and downstream directions. In various embodiments, the connection between the I/O interconnect 108 and the device 110 may support electrical connections, optical connections, or both.
The apparatus 100 may be a stand-alone device or may be incorporated into various systems including, but not limited to, various computing and/or consumer electronic devices/appliances, such as desktop computing device, a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.), mobile phones, smart phones, personal digital assistants, servers, workstations, set-top boxes, digital reorders, game consoles, digital media players, and digital cameras. A block diagram of an example system 200 is illustrated in
The system 200 may include communications interface(s) 217 operatively coupled to the bus 215 to provide an interface for system 200 to communicate over one or more networks and/or with any other suitable device. The communications interface(s) 217 may include any suitable hardware and/or firmware. The communications interface(s) 217 for one embodiment may include, for example, a network adapter, a wireless network adapter, a telephone modem, and/or a wireless modem. For wireless communications, the communications interface(s) 217 for one embodiment may include a wireless network interface controller 219 having one or more antennae 221 to establish and maintain a wireless communication link with one or more components of a wireless network. The system 200 may wirelessly communicate with the one or more components of the wireless network in accordance with any of one or more wireless network standards and/or protocols.
The system 100 may include a display device 223, such as, for example, a cathode ray tube (CRT), liquid crystal display (LCD), light emitting diode (LED), or other suitable display device, operatively coupled to the bus 215 for displaying information. In various embodiments, the display device 223 may be a peripheral device interconnected with the system 200. In various ones of these embodiments, such a peripheral display device may be interconnected with the I/O complex 206 by way of the multi-protocol port 212.
As described herein, for providing an I/O interconnect capable of carrying multiple I/O protocols, one or more of the various I/O interconnects described herein may include, among other things, a multi-protocol switching fabric 314 comprising a plurality of cross-bar switches, as shown in
Switch 316a may represent a first type of switch including null ports 320a configured to connect to a single optical or electrical link, while adapter ports 322a may be configured to connect to one or more mapped I/O protocol links. The adapter ports 322a may be used to connect mapped I/O protocol entities to the multi-protocol switching fabric 314. As used herein, the term “adapter” may be used to refer to the protocol adaptation function that may be built into the switch port to encapsulate the mapped I/O protocol packets into I/O packets that flow over the multi-protocol switching fabric 314.
Switch 316b may represent a second type of switch including only null ports 320b (like null ports 320a) configured to connect to a single optical or electrical link.
Although the switches 316a, 316b depicted in
In various embodiments, the multi-protocol switching fabric 314 may comprise one or more of the first type of switches 316a and one or more of the second type of switches 316b.
For implementing various multi-protocol tunneling between adapter ports of a switching fabric within the scope of the present disclosure, a connection manager (not illustrated) may be provided. The connection manager may be implemented in software, firmware, as logic within an I/O complex, as part of a system BIOS, or within an operating system running on a computer apparatus or system in which the I/O complex is included.
An example protocol stack for the multi-protocol interconnect architecture of an I/O complex is shown in
In various embodiments, and with reference to
An example implementation of the protocol layering is shown in
As shown, the adapter ports 522a1, 522c implement a first protocol layer (or frame layer) “protocol 1,” and adapter ports 522a2, 522d implement a second protocol layer (or frame layer) “protocol 2.” All ports implement the transport layer, while the physical layer is implemented by all null ports 520a, 520b, 520c, 520d.
As such, a link (e.g., link 532) between ports of switches may effectively be shared by multiple paths traversing the fabric between adapter ports of the multi-protocol switching fabric. In various embodiments, the multi-protocol interconnect architecture may be connection-oriented such that a path is configured end-to-end before data transfer takes place. The path may traverse one or more links through the multi-protocol switching fabric, and each hop, the path may be assigned a locally unique identifier that may be carried in the header of all the packets that are associated with the path. In various embodiments, packets belonging to the path may not be reordered within the multi-protocol switching fabric. Buffer allocation (flow control) and Quality of Service may be implemented on a per-path basis. As such, a path may provide virtual-wire semantics for a mapped I/O protocol across the multi-protocol switching fabric.
In various embodiments, the physical topology of a collection of switches (a domain) may be an arbitrarily interconnected graph.
As shown in
In various embodiments, the routing of configuration packets flowing downstream (in relation to the spanning tree) may be based on the topology ID of the target switch. The configuration packets may be routed in the transport layer packet header. In various embodiments, configuration packets flowing upstream may not use the topology ID and may simply be forwarded over the upstream port of each switch. Typically, every configuration packet carries a route string included in its payload. An example format of the route string is shown in
In various embodiments, each switch may be configured with its topology ID and its level in the spanning tree by the connection manager. Each switch may also be configured with the port number that points upstream to the connection manager of the domain either through hardware strapping or other similar mechanisms. In various embodiments, the topology ID, depth (in the tree), and upstream facing port may be configuration registers in the switch configuration space of every switch that are initialized by the connection manager during enumeration. An example format of the topology ID configuration register is shown in
Configuration packets flowing down the tree may be routed by the control port of a switch in accordance with one or more rules. For example, in various embodiments, the control port of the switch may be required to extract the port from the route string that corresponds to its configured level in the tree. In various embodiments, if the port is 0, the control port may be required to consume the packet. In various embodiments, if the port is non-zero, the control port may be required to forward the packet over the switch port that matches the port extracted from the route string. In various embodiments, configuration packets flowing up the spanning tree may simply be forwarded over the configured upstream facing port.
Multiple domains may interconnected in various embodiments.
In various embodiments, inter-domain links may be discovered either when the connection manager performs the initial discovery of the topology following power-on or by processing a hot-plug event. A link may be designated to be an inter-domain link when a read of the switch configuration space of the switch across the link results in an ERROR packet being sent that shows that the topology ID field has been previously assigned. When an inter-domain link is discovered, the connection manager may notify system software. The mechanism used to deliver the notification may be implementation-defined.
In various embodiments, the transport layer may only define the routing of inter-domain configuration packets between the two connection managers of the domains that are connected by an inter-domain link. Routing of configuration packets across multiple domains may be controlled by system software. When domains are daisy-chained, configuration packets passing from the originating domain may be delivered to the connection managers of every domain along the path to the target domain. The connection managers of the intermediate domains may pass the configuration packets to the system software which may be responsible for relaying the packet across the inter-domain link towards the target domain.
The routing of inter-domain REQUEST packets may be in accordance with one or more rules. For example, in various embodiments, system software on the originating domain may form REQUEST packet with a route string that points to the egress port of the domain that connects to the inter-domain link over which the packet must be forwarded and the CM bit may be set to 0. The packet may be required to be routed based on the route string at each hop within the domain and forwarded over the egress port across the inter-domain link. At the ingress port of the receiving domain, the control port may remap the route string to point to the ingress port over which the packet was received and the CM bit may be set to 1. In various embodiments, the packet may then be required to be routed to the connection manager of the receiving domain like other intra-domain configuration packets. The packet may be required to be delivered by the connection manager of the receiving domain to system software.
The routing of inter-domain RESPONSE packets may follow one or more of the same steps above. In various embodiments, system software that constructs the RESPONSE packet may use the route string in the corresponding REQUEST packet with the CM bit set to 0.
In various embodiments, the transport layer may employ a hierarchical, credit-based flow control scheme with respect to flow through the multi-protocol switching fabric to prevent or minimize overflow of receive buffers due to congestion. In various embodiments, the flow control scheme may allow a receiver to implement various buffer allocation strategies ranging from dedicated buffers per-path to shared buffer pools that are dynamically shared by multiple paths. In various embodiments, flow control may be turned off on a per-path basis. When flow control is turned off for a path, the path may be required to be provisioned with a receive buffer that can hold at least one maximum sized transport layer packet at each link.
In various embodiments, the I/O complex 1106 may be configured to connect the device 1110 with one or more protocol-specific controllers 1109a, 1109b, . . . 1109n via the I/O interconnect 1108 in order to tunnel multiple I/O protocols over a common link in a manner that is transparent to the OS software stacks of tunneled I/O protocols. The protocol-specific controllers 1109a, 1109b, . . . 1109n may be configured to then communicate with respective protocol-specific drivers in the OS for configuring the device 1110 as if the device 1110 was directly connected with the protocol-specific controller 1109a, 1109b, . . . 1109n.
For the implementation shown in
In various embodiments, the apparatus 1200 may be configured such that when the device 1210 is disconnected from the port 1212, a reverse sequence of events may occur. Specifically, the protocol-specific drivers 1211a, 1211b, . . . 1211n may process the protocol-specific unplug event, and then after the protocol-specific processing, the I/O driver 1213 may process the I/O unplug event.
Peripheral devices described herein (device 110, 210, 1110, or 1210, for example) may be any one of various types of devices, as noted earlier. In various embodiments, the peripheral device may be an expansion port (or other multi-protocol peripheral device) with which one or more other devices, with one or more I/O protocols, may be coupled. For example, for embodiments in which the peripheral device is an expansion port, the device may be simultaneously coupled with a PCI Express® device and a DisplayPort device, which may be coupled with an I/O complex through the expansion port device. In another example, the peripheral device may be a mobile or desktop computer system and one or more other devices may be coupled with the mobile or desktop computer system and with the I/O complex through the device. In various embodiments, multiple peripheral devices may be coupled together by daisy chaining the devices together.
In various embodiments, the peripheral device and/or the other devices coupled with the peripheral device may also include an I/O interconnect similar to one or more of the I/O interconnects 108, 208, 1108, 1208 described herein. As shown in
Processing for the method 1400 may start with block 1402 by identifying a plurality of switches of a switching fabric of a multi-protocol interconnect.
The method 1400 may proceed to block 1404 by creating a spanning tree representation of the plurality of switches.
The method 1400 may proceed to block 1406 by assigning unique identifications (IDs) to the switches of plurality of switches of the spanning tree. In various embodiments, the IDs may represent the relative positions of the switches within the spanning tree.
The method 1400 may proceed to block 1408 by storing the IDs and depth of the switches (in the spanning tree) in one or more registers of each of the switches.
The method 1400 may proceed to block 1410 by routing configuration packets through the spanning tree to the switches based at least in part on their respective IDs.
Processing for the method 1500 may start with block 1502 by determining whether a peripheral device has been plugged into a non-protocol-specific port of a computer apparatus including a multi-protocol tunneling I/O interconnect. Plugging may refer to a peripheral device being directly coupled with the non-protocol-specific port and/or a target peripheral device being directly coupled to some other peripheral device directly coupled with the non-protocol-specific port. In the latter embodiments, one or more other peripheral devices may be operatively disposed between the target peripheral device and the non-protocol-specific port. If no peripheral device has been plugged, then processing in block 1502 may repeat. In various embodiments, the computer apparatus may be configured to issue an interrupt signal indicating when a peripheral device has been plugged (e.g., hot-plugged).
Processing for the method 1500 may proceed to block 1504 by determining whether a data packet has been received. If no data packet has been received, then processing in block 1504 may repeat. In various embodiments, a data packet may be received from the peripheral device or from within the computer apparatus. In various embodiments, data packets within the computer apparatus may be received by the multi-protocol tunneling I/O interconnect from a protocol-specific controller (“host protocol-specific controller”) of the computer apparatus.
Processing for the method 1500 may proceed to block 1506 by determining whether the data packet was received from the peripheral device or from a host protocol-specific controller. If no data packet has been received, then processing in block 1506 may repeat.
If the data packet was received from the peripheral device, processing for the method 1500 may proceed to block 1508 by encapsulating packets of a first protocol into first transport layer packets configured to be routed through the switching fabric of the I/O interconnect. In various embodiments, packets of a second protocol, different from the first protocol, may also be encapsulated into second transport layer packets for routing through the switching fabric.
Processing for the method 1500 may proceed to block 1510 by simultaneously routing the first and second transport layer packets through the switching fabric of the I/O interconnect.
Processing for the method 1500 may proceed to block 1512 by decapsulating the transport layer packets. In various embodiments, decapsulation may be performed an adapter port of a switch of the switching fabric.
Processing for the method 1500 may proceed to block 1514 by routing the decapsulated packets to different host protocol-specific controllers of the computer apparatus.
If the data packet was received from the peripheral device, processing for the method 1500 may proceed from block 1506 to block 1516 by encapsulating packets of a first protocol into first transport layer packets configured to be routed through the switching fabric of the I/O interconnect. In various embodiments, packets of a second protocol, different from the first protocol, may also be encapsulated into second transport layer packets for routing through the switching fabric.
Processing for the method 1500 may proceed to block 1518 by simultaneously routing the first and second transport layer packets through the switching fabric of the I/O interconnect.
Processing for the method 1500 may proceed to block 1520 by decapsulating the transport layer packets. In various embodiments, decapsulation may be performed an adapter port of a switch of the switching fabric.
Processing for the method 1500 may proceed to block 1522 by routing the decapsulated packets to a peripheral device via a non-protocol-specific port of the computer apparatus.
In various embodiments, an article of manufacture may be employed to implement one or more methods as disclosed herein.
The storage medium 1602 may represent a broad range of persistent storage medium known in the art, including but not limited to flash memory, optical disks or magnetic disks. The programming instructions 1604, in particular, may enable an apparatus, in response to their execution by the apparatus, to perform various operations described herein. For example, the storage medium 1602 may include programming instructions 1604 configured to cause an apparatus to practice some or all aspects of multi-protocol tunneling of the methods of
Although various example methods, apparatus, systems, and articles of manufacture have been described herein, the scope of coverage of the present disclosure is not limited thereto. On the contrary, the present disclosure covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the appended claims either literally or under the doctrine of equivalents. For example, although the above discloses example systems including, among other components, software or firmware executed on hardware, it should be noted that such systems are merely illustrative and should not be considered as limiting. In particular, it is contemplated that any or all of the disclosed hardware, software, and/or firmware components could be embodied exclusively in hardware, exclusively in software, exclusively in firmware or in some combination of hardware, software, and/or firmware.
This Application is a Continuation of and claims the benefit of priority of U.S. patent application Ser. No. 15/407,946, filed Jan. 17, 2017, which in turn is a Continuation of and claims the benefit of priority of U.S. patent application Ser. No. 13/338,220, filed Dec. 27, 2011, and is hereby incorporated by reference.
Number | Name | Date | Kind |
---|---|---|---|
5242315 | O'Dea | Sep 1993 | A |
5251205 | Callon et al. | Oct 1993 | A |
5267337 | Kirma | Nov 1993 | A |
5419717 | Abendschein et al. | May 1995 | A |
5535036 | Grant | Jul 1996 | A |
5938736 | Muller et al. | Aug 1999 | A |
6014380 | Hendel et al. | Jan 2000 | A |
6108782 | Fletcher et al. | Aug 2000 | A |
6478625 | Tolmie et al. | Nov 2002 | B2 |
6536670 | Postman et al. | Mar 2003 | B1 |
6549966 | Dickens et al. | Apr 2003 | B1 |
6588938 | Lampert et al. | Jul 2003 | B1 |
6697379 | Jacquet et al. | Feb 2004 | B1 |
6783283 | Nishita | Aug 2004 | B2 |
6839771 | Bouchier et al. | Jan 2005 | B1 |
7081023 | Zhang et al. | Jul 2006 | B2 |
7095927 | Yamada et al. | Aug 2006 | B2 |
7096310 | Norden | Aug 2006 | B2 |
7171505 | Kuhlmann et al. | Jan 2007 | B2 |
7283481 | Huff | Oct 2007 | B2 |
7330468 | Tse-Au | Feb 2008 | B1 |
7412544 | Gibson et al. | Aug 2008 | B2 |
7437738 | Shah et al. | Oct 2008 | B2 |
7453892 | Buskirk | Nov 2008 | B2 |
7587536 | McLeod | Sep 2009 | B2 |
7646981 | Coffey | Jan 2010 | B2 |
7673080 | Yu et al. | Mar 2010 | B1 |
7677813 | Anrig et al. | Mar 2010 | B2 |
7734172 | Tse-Au | Jun 2010 | B2 |
7860119 | Enciso | Dec 2010 | B1 |
8051217 | Goodart et al. | Nov 2011 | B2 |
8121139 | Sunaga et al. | Feb 2012 | B2 |
8949471 | Hall | Feb 2015 | B2 |
20020049862 | Gladney et al. | Apr 2002 | A1 |
20020116564 | Paul | Aug 2002 | A1 |
20040081158 | Moll et al. | Apr 2004 | A1 |
20040230735 | Moll | Nov 2004 | A1 |
20050210177 | Norden | Sep 2005 | A1 |
20050238052 | Yamazaki | Oct 2005 | A1 |
20060050652 | Chappell | Mar 2006 | A1 |
20060126612 | Sandy et al. | Jun 2006 | A1 |
20060184711 | Pettey et al. | Aug 2006 | A1 |
20060203851 | Eidson | Sep 2006 | A1 |
20070005867 | Diamant | Jan 2007 | A1 |
20070171917 | Dai | Jul 2007 | A1 |
20070233916 | Seto | Oct 2007 | A1 |
20070249193 | Penumatcha et al. | Oct 2007 | A1 |
20080013569 | Boren | Jan 2008 | A1 |
20090010152 | Ofek et al. | Jan 2009 | A1 |
20090066704 | Daniel | Mar 2009 | A1 |
20090080428 | Witkowski et al. | Mar 2009 | A1 |
20090172185 | Chandra et al. | Jul 2009 | A1 |
20090198958 | Arimilli | Aug 2009 | A1 |
20090274162 | Gowda et al. | Nov 2009 | A1 |
20100049885 | Chandra et al. | Feb 2010 | A1 |
20100091775 | Yamamoto | Apr 2010 | A1 |
20100274907 | Hirasawa et al. | Oct 2010 | A1 |
20110035529 | Wang et al. | Feb 2011 | A1 |
20110055433 | Kishore et al. | Mar 2011 | A1 |
20110090797 | Beecroft | Apr 2011 | A1 |
20110090916 | Bitar | Apr 2011 | A1 |
20110202701 | Maitra | Aug 2011 | A1 |
20110219164 | Suzuki et al. | Sep 2011 | A1 |
20110228767 | Singla et al. | Sep 2011 | A1 |
20110228789 | Jia | Sep 2011 | A1 |
20110255533 | Gnanasekaran | Oct 2011 | A1 |
20110283017 | Alkhatib et al. | Nov 2011 | A1 |
20120284434 | Warren et al. | Nov 2012 | A1 |
20130042240 | Cardona et al. | Feb 2013 | A1 |
20130089097 | Filsfils et al. | Apr 2013 | A1 |
20130151840 | Kanigicherla | Jun 2013 | A1 |
Number | Date | Country |
---|---|---|
10233820 | Sep 1998 | JP |
2001358733 | Dec 2001 | JP |
20010076079 | Aug 2001 | KR |
2009085494 | Jul 2009 | WO |
2010021844 | Feb 2010 | WO |
Entry |
---|
Final Office Action for U.S. Appl. No. 13/338,220, dated Feb. 12, 2016, 24pages. |
Final Offce Action for U.S. Appl. No. 13/338,220, dated Jun. 17, 2014, 21 pages. |
International Preliminary Report on Patentability and Written Opinion received for International Application No. PCT/US2009/052831, dated Mar. 3, 2011. |
International Preliminary Report on Patentability for International Application No. PCT/US2008/084621, dated Jul. 8, 2010. |
International Search Report and Written Opinion received for International Application No. PCT/US2008/084621, dated May 18, 2009. |
International Search Report/Written Opinion for International Application No. PCT/US2009/052831, dated Mar. 15, 2010. |
Office Action for Chinese Application No. 200880122959.5, dated Mar. 19, 2012. |
Office Action for European Application No. 08868735.5, dated Sep. 29, 2011. |
Office Action for European Application No. 09808593.9, dated Sep. 13, 2011. |
Office Action for Japanese Application No. 2010-540703, dated Nov. 22. 2011. |
Office Action for Korean Application No. 10-2010-7016587, dated Apr. 26, 2012. |
Office Action for Korean Application No. 10-2010-7016587, dated Jul. 11, 2011. |
Office Action for Taiwan Application No. 97147418, dated Jun. 14, 2012. |
Office Action for U.S. Appl. No. 13/338,220, dated Jun. 5, 2015, 24pages. |
Office Action for U.S. Appl. No. 13/338,220, dated Oct. 9, 2013, 24 pages. |
Search Report for European Application No. 08868735.5, dated Sep. 2, 2011. |
Search Report for European Application No. 09808593.9, dated Aug. 19, 2011. |
Number | Date | Country | |
---|---|---|---|
20190166046 A1 | May 2019 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 15407946 | Jan 2017 | US |
Child | 16138612 | US | |
Parent | 13338220 | Dec 2011 | US |
Child | 15407946 | US |