Networked storage and computing systems have been introduced which store and process large amounts of data in enterprise-class storage environments. These networked storage systems typically provide access to bulk data storage over one or more network interfaces to end users or other external systems. In addition to storage of data, remote computing systems include various processing systems that can provide remote computing resources to end users. These networked storage systems and remote computing systems can be included in high-density installations, such as rack-mounted environments. Some computing devices employ Peripheral Component Interconnect Express (PCIe) interfaces to connect to peripherals and storage devices. However, typical PCIe implementations employ point-to-point host-device architectures.
In an embodiment, a connectivity card insertable into a connector of a host system is provided. The connectivity card includes a plurality of Peripheral Component Interconnect Express (PCIe) connectors configured to provide external PCIe ports on the connectivity card, each of the plurality of PCIe connectors capable of carrying PCIe traffic.
The connectivity card also includes a PCIe switch circuit configured to communicatively couple the plurality of connectors to a shared connectivity interface carried over a host connector of the connectivity card.
The connectivity card further includes control circuitry configured to monitor for connectivity issues that arise with regard to the plurality of PCIe connectors, and responsively mitigate the connectivity issues by at least reconfiguring a communication pathway in the PCIe switch circuit for at least a portion of the PCIe traffic affected by the connectivity issues.
In another embodiment, a method of operating a connectivity card insertable into a connector of a host system is provided. The method includes monitoring a plurality of external Peripheral Component Interconnect Express (PCIe) ports for connectivity issues, each of the plurality of PCIe ports capable of carrying PCIe traffic, and each PCIe port having a corresponding PCIe connector on the connectivity card, and detecting connectivity issues in one of the plurality of PCIe ports.
The method also includes mitigating the connectivity issues by reconfiguring a communication pathway in a PCIe switch circuit on the connectivity card for at least a portion of the PCIe traffic affected by the connectivity issues, the PCIe switch circuit configured to communicatively couple the plurality of PCIe ports to a shared connectivity interface carried over a host connector of the connectivity card.
In a further embodiment, one or more non-transitory computer-readable media having stored thereon program instructions to facilitate operating a connectivity card insertable into a connector of a host system is provided. The program instructions, when executed by a computing system, direct the computing system to at least monitor a plurality of external Peripheral Component Interconnect Express (PCIe) ports for connectivity issues, each of the plurality of PCIe ports capable of carrying PCIe traffic, and each PCIe port having a corresponding PCIe connector on the connectivity card, and to detect connectivity issues in one of the plurality of PCIe ports.
The program instructions further direct the computing system to mitigate the connectivity issues by reconfiguring a communication pathway in a PCIe switch circuit on the connectivity card for at least a portion of the PCIe traffic affected by the connectivity issues, the PCIe switch circuit configured to communicatively couple the plurality of PCIe ports to a shared connectivity interface carried over a host connector of the connectivity card.
Many aspects of the disclosure can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the several views. While several embodiments are described in connection with these drawings, the disclosure is not limited to the embodiments disclosed herein. On the contrary, the intent is to cover all alternatives, modifications, and equivalents.
PCIe link 122 is connected to cluster A 120 through PCIe connector 124, and to server 110 through PCIe connector 117. PCIe link 126 is connected to cluster A 120 through PCIe connector 128, and to server 110 through PCIe connector 117. PCIe link 132 is connected to cluster B 130 through PCIe connector 134, and to server 110 through PCIe connector 117. PCIe link 136 is connected to cluster B 130 through PCIe connector 138, and to server 110 through PCIe connector 117.
Server 110 includes motherboard 112, PCIe slot 114, and connectivity card 116. Connectivity card 116 includes four PCIe connectors 117. In a prior art example, if there are connectivity issues with PCIe link 122 (perhaps it has been disconnected from PCIe connector 124), the connectivity issues may cause a cascading failure problem for server 110, referred to as a host panic condition.
However, in an example embodiment of the present invention, connectivity card 116, which has been monitoring its four PCIe connectors 117 for connectivity issues, automatically switches traffic from PCIe link 122 to PCIe link 126 upon detecting the connectivity issues. Connectivity card 116 performs this redirection of traffic automatically and in such a way that server 110 is not disrupted by the connectivity issues.
Connectivity expansion card 210 can comprise a HHHL (half-height half-length) PCIe peripheral or expansion card. Other card sizes can be employed, such as a FHHL (full-height half-length) or FHFL (full-height full-length), or HHFL (half-height full length), among others. PCIe edge connector 214 is insertable into a mating PCIe socket of a host system, such as a motherboard or daughterboard of a computer or server system. Connectivity expansion card 210 can receive power over one or more power links provided by the host system over connector 214.
PCIe switch circuit 212 comprises one or more PCIe switch elements and distributes PCIe communications and traffic to one or more of ports 213. PCIe switch circuit 212 is also communicatively coupled to an on-card processor 211 or control system that handles traffic monitoring, traffic statistics communication, power monitoring, and status monitoring, among other operations.
PCIe-compatible connectors 213 can be used for cluster interconnect and can comprise mini-Serial Attached SCSI (SAS) HD connectors which are employed to carry PCIe signaling over mini-SAS cabling. Other example connectors include zQSFP+, microQSFP, and OCuLink-2 interconnect. In further examples, MiniSAS HD cables are employed that drive 12 Gb/s versus 6 Gb/s of standard SAS cables, since 12 Gb/s can support PCIe Gen 3. PCIe-compatible connectors 213 can comprise mini-SAS connectors that comprise mini-SAS jacks. Associated cabling for PCIe links 220 can comprise SAS cabling which can include associated shielding, wiring, sheathing, and termination connectors.
PCIe-compatible connectors 213 are grouped into a set of four (4) ports in the examples herein to allow connections to other switches, devices, or servers with up to a x16 bus, as well as supporting x1, x4, and x8 bus widths, among others. Each PCIe-compatible connector 213 can also have included indicators (such as light-emitting diodes (LEDs)) that shine through associated PCIe port connectors to help guide customers in connecting cables, troubleshooting, and indicating connectivity and errors.
PCIe switch circuit 212 communicates with a host system or host module (not pictured) over PCIe link 240. PCIe link 240 can comprise a PCIe link with multiple lanes, such as a “x4” PCIe link, although a different number of PCIe lanes can be employed. Additionally, more than one PCIe link 240 can be employed for load balancing, redundancy, and failover protection for connectivity expansion card 210. PCIe switch circuit 212 also communicates with four external PCIe-compatible connectors 213 over associated x4 PCIe links, although a different number of links and connectors can be employed. PCIe can support multiple bus widths, such as x1, x4, x8, and x16, among others, with each multiple of bus width comprising an additional “lane” for data transfer. An example embodiment of the present invention also supports transfer of sideband signaling, such as System Management Bus (SMBus) interfaces, as well as associated clocks, power, and bootstrapping, among other signaling.
PCIe switch circuit 212 comprises one or more PCIe crosspoint switches, which logically interconnect various ones of the associated PCIe links based at least on the traffic carried by each PCIe link. PCIe switch circuit 212 establishes switched connections between any PCIe interfaces handled by PCIe switch circuit 212. Each PCIe switch port can comprise a non-transparent (NT) or transparent port or Downstream Port Containment (DCP/eDPC). An NT port can allow some logical isolation between endpoints, while a transparent port does not allow logical isolation, and has the effect of connecting endpoints in a purely switched configuration. Access over an NT port or ports can include additional handshaking between the PCIe switch and the initiating endpoint to select a particular NT port or to allow visibility through the NT port. In other examples, a domain-based PCIe signaling distribution can be included which allows segregation of PCIe ports of a PCIe switch according to user-defined groups. In some examples, PCIe switch circuit 212 comprises a PLX Technology PEX8725 10-port, 24 lane PCIe switch chip. In some examples, PCIe switch circuit 212 comprises a PLX Technology PEX8796 24-port, 96 lane PCIe switch chip.
Although host PCIe link 240 is used in
Processor 211 can optionally communicate over at least PCIe link 215 and optional sideband links. Sideband links can include Universal Serial Bus (USB), SMBus, Inter-Integrated Circuit (I2C), controller area network bus (CAN), or any other communication interface, and in some examples are provided over portions of PCIe link 215. In
Processor 211 comprises one or more microprocessors, processing devices, multi-core processors, processing circuitry, or other processing system. Processor 211 can include one or more non-transitory memory devices, such as RAM, solid state storage, or other memory to store instructions that are executable by processor 211 to operate as discussed herein. In some examples, processor 211 comprises an ARM microcontroller, ARM microprocessor, field-programmable gate array (FPGA), application specific integrated circuit (ASIC), application specific processor, or other microprocessor or processing elements. In some examples, processor 211 comprises an ARM-compatible microprocessor or microcontroller, although other circuitry can be employed. Processor 211 can comprise any processing elements discussed below for control processor 400 of
Although processor 211 and PCIe switch circuit 212 are shown as separate elements in
Holdup circuitry can optionally be included in some examples on connectivity expansion card 210 to provide power to the connectivity card when input power has been lost or removed for the connectivity card. In some examples, the connectivity card is removed from an associated mating connector and input power is lost due to the removal. In other examples, power is lost to a host system into which connectivity expansion card 210 is connected, such as during a facility power outage or when an associated power supply fails. The various holdup circuitry can also be accompanied by a power controller circuit to selectively provide power to the elements of connectivity expansion card 210.
Control processor 400 includes communication interface 401 and processing system 410. Processing system 410 includes processing circuitry 411, random access memory (RAM) 412, and storage 413, although further elements can be included. Example contents of RAM 412 are further detailed in RAM space 420, and example contents of storage 413 are further detailed in storage system 460.
Processing circuitry 411 can be implemented within a single processing device but can also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing circuitry 411 include general purpose central processing units, microprocessors, application specific processors, and logic devices, as well as any other type of processing device. In some examples, processing circuitry 411 includes physically distributed processing devices, such as cloud computing systems.
Communication interface 401 includes one or more communication and network interfaces for communicating over communication links, networks, such as packet networks, the Internet, and the like. The communication interfaces can include PCIe interfaces, serial links, such as SPI links, I2C links, USB links, UART links, or one or more local or wide area network communication interfaces which can communicate over Ethernet or Internet protocol (IP) links. Communication interface 401 can include network interfaces configured to communicate using one or more network addresses, which can be associated with different network links. Examples of communication interface 401 include network interface card equipment, transceivers, modems, and other communication circuitry.
RAM 412 and storage 413 together can comprise a non-transitory data storage system, although variations are possible. RAM 412 and storage 413 can each comprise any storage media readable by processing circuitry 411 and capable of storing software. RAM 412 can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Storage 413 can include nonvolatile storage media, such as solid-state storage media, flash memory, phase change memory, or magnetic memory, including combinations thereof. RAM 412 and storage 413 can each be implemented as a single storage device but can also be implemented across multiple storage devices or sub-systems. RAM 412 and storage 413 can each comprise additional elements, such as controllers, capable of communicating with processing circuitry 311.
Software stored on or in RAM 412 or storage 413 can comprise computer program instructions, firmware, or some other form of machine-readable processing instructions having processes that when executed a processing system direct control processor 400 to operate as described herein. For example, software can drive processor 400 to monitor operating statistics and status for various PCIe traffic and other modules, monitor for connectivity issues that arise with regard to the plurality of PCIe connectors, and responsively mitigate the connectivity issues by at least reconfiguring a communication pathway in the PCIe switch circuit for at least a portion of the PCIe traffic affected by the connectivity issues, among other operations. The software can also include user software applications, application programming interfaces (APIs), or user interfaces. The software can be implemented as a single application or as multiple applications. In general, the software can, when loaded into a processing system and executed, transform the processing system from a general-purpose device into a special-purpose device customized as described herein.
RAM space 420 illustrates a detailed view of an example configuration of RAM 412. It should be understood that different configurations are possible. In an example embodiment of the present invention, applications 421 and custom kernel 422 can reside in flash memory and run in an execute in place (XIP) configuration. This enables quick boots while providing sufficient memory to execute telemetry and PCIe monitoring functions. Software applications 423-424 each comprise executable instructions which can be executed by processor 400 for operating a connectivity card or other circuitry according to the operations discussed herein. Specifically, PCIe monitor 423 monitors for connectivity issues that arise with regard to the plurality of PCIe connectors. PCIe redirect control 424 mitigates connectivity issues by at least reconfiguring a communication pathway in the PCIe switch circuit for at least a portion of the PCIe traffic affected by the connectivity issues.
Applications 421 and custom kernel 422 can reside in RAM space 420 during execution and operation of control processor 400, and can reside in storage system 460 during a powered-off state, among other locations and states. Applications 421 and custom kernel 422 can be loaded into RAM space 420 during a startup or boot procedure as described for computer operating systems and applications.
Storage system 460 illustrates a detailed view of an example configuration of storage 413. Storage system 460 can comprise flash memory such as NAND flash or NOR flash memory, phase change memory, magnetic memory, among other solid-state storage technologies. As shown in
Control processor 400 is generally intended to represent a computing system with which at least software 461 and 421-424 are deployed and executed in order to render or otherwise implement the operations described herein. However, control processor 400 can also represent any computing system on which at least software 461 and 421-424 can be staged and from where software 461 and 421-424 can be distributed, transported, downloaded, or otherwise provided to yet another computing system for deployment and execution, or yet additional distribution.
In
For example, in
The included descriptions and figures depict specific embodiments to teach those skilled in the art how to make and use the best mode. For the purpose of teaching inventive principles, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate variations from these embodiments that fall within the scope of the invention. Those skilled in the art will also appreciate that the features described above can be combined in various ways to form multiple embodiments. As a result, the invention is not limited to the specific embodiments described above, but only by the claims and their equivalents.
This application hereby claims the benefit of and priority to U.S. Provisional Patent Application No. 62/491,006, entitled “PCIe FABRIC CONNECTIVITY EXPANSION CARD,” filed on Apr. 27, 2017 and which is hereby incorporated by reference in its entirety.
Number | Name | Date | Kind |
---|---|---|---|
5828207 | Saadeh | Oct 1998 | A |
6061750 | Beardsley et al. | May 2000 | A |
6325636 | Hipp et al. | Dec 2001 | B1 |
7243145 | Poortman | Jul 2007 | B1 |
7260487 | Brey et al. | Aug 2007 | B2 |
7505889 | Salmonsen et al. | Mar 2009 | B2 |
7606960 | Munguia | Oct 2009 | B2 |
7725757 | Padweka et al. | May 2010 | B2 |
7877542 | Chow et al. | Jan 2011 | B2 |
8125919 | Khanka et al. | Feb 2012 | B1 |
8150800 | Webman et al. | Apr 2012 | B2 |
8656117 | Wong et al. | Feb 2014 | B1 |
8688926 | Breakstone et al. | Apr 2014 | B2 |
8756360 | Richard | Jun 2014 | B1 |
8880771 | Subramaniyan et al. | Nov 2014 | B2 |
9552316 | Desimone et al. | Jan 2017 | B2 |
9602437 | Bernath | Mar 2017 | B1 |
20020059428 | Susai et al. | May 2002 | A1 |
20030110423 | Helms et al. | Jun 2003 | A1 |
20030126478 | Burns et al. | Jul 2003 | A1 |
20050223136 | Tanaka et al. | Oct 2005 | A1 |
20060277206 | Bailey et al. | Dec 2006 | A1 |
20070067432 | Tarui et al. | Mar 2007 | A1 |
20080034153 | Lee et al. | Feb 2008 | A1 |
20080198744 | Menth | Aug 2008 | A1 |
20080281938 | Rai et al. | Nov 2008 | A1 |
20090006837 | Rothman et al. | Jan 2009 | A1 |
20090100280 | Lindsay | Apr 2009 | A1 |
20090190427 | Brittain et al. | Jul 2009 | A1 |
20090193201 | Brittain et al. | Jul 2009 | A1 |
20090193203 | Brittain et al. | Jul 2009 | A1 |
20090276551 | Brown et al. | Nov 2009 | A1 |
20100088467 | Lee et al. | Apr 2010 | A1 |
20110289510 | Lin et al. | Nov 2011 | A1 |
20110299317 | Shaeffer et al. | Dec 2011 | A1 |
20110320861 | Bayer et al. | Dec 2011 | A1 |
20120030544 | Fisher-Jeffes | Feb 2012 | A1 |
20120089854 | Breakstone et al. | Apr 2012 | A1 |
20120151118 | Flynn et al. | Jun 2012 | A1 |
20120166699 | Kumar et al. | Jun 2012 | A1 |
20120210163 | Cho | Aug 2012 | A1 |
20120317433 | Ellis et al. | Dec 2012 | A1 |
20130132643 | Huang | May 2013 | A1 |
20130185416 | Larkin et al. | Jul 2013 | A1 |
20140047166 | Asnaashari et al. | Feb 2014 | A1 |
20140056319 | Hellwig | Feb 2014 | A1 |
20140059265 | Iyer et al. | Feb 2014 | A1 |
20140075235 | Chandhoke et al. | Mar 2014 | A1 |
20140103955 | Avritch et al. | Apr 2014 | A1 |
20140108846 | Berke et al. | Apr 2014 | A1 |
20140353264 | Venugopal et al. | Dec 2014 | A1 |
20140365714 | Sweere et al. | Dec 2014 | A1 |
20150074322 | Galles | Mar 2015 | A1 |
20150121115 | Chandra et al. | Apr 2015 | A1 |
20150186437 | Molaro | Jul 2015 | A1 |
20150212755 | Asnaashari | Jul 2015 | A1 |
20150261710 | Billi | Sep 2015 | A1 |
20150304423 | Satoyama et al. | Oct 2015 | A1 |
20150309951 | Breakstone et al. | Oct 2015 | A1 |
20150347345 | Hellriegel | Dec 2015 | A1 |
20150373115 | Breakstone | Dec 2015 | A1 |
20160070661 | Huang et al. | Mar 2016 | A1 |
20160072640 | Yang et al. | Mar 2016 | A1 |
20160147628 | Arroyo et al. | May 2016 | A1 |
20160197996 | Barton et al. | Jul 2016 | A1 |
20160248631 | Duchesneau | Aug 2016 | A1 |
20180101500 | Heyd | Apr 2018 | A1 |
Number | Date | Country |
---|---|---|
201445317 | Dec 2014 | TW |
Entry |
---|
Aragon, Juan L. et al., “Control Speculation for Energy-Efficient Next-Generation Superscalar Processors,” IEEE Transactions on Computers, vol. 55, No. 3, pp. 281-291, Mar. 2006. |
International Application No. PCT/US2017/046602, International Search Report & Written Opinion, 8 pages, dated Oct. 19, 2017. |
International Application No. PCT/US2017/046607, International Search Report & Written Opinion, 7 pages, dated Oct. 23, 2017. |
International Application No. PCT/US2018/029496, International Search Report & Written Opinion, 10 pages, dated Jul. 10, 2018. |
Lu, Yingping et al., “Performance Study of iSCSI-Based Storage Subsystems,” IEEE Communications Magazine, pp. 76-82, Aug. 2003. |
Taiwanese Patent Application No. 107114399, Office Action, 14 pages, dated Jan. 18, 2019. |
Number | Date | Country | |
---|---|---|---|
20180314667 A1 | Nov 2018 | US |
Number | Date | Country | |
---|---|---|---|
62491006 | Apr 2017 | US |