1. Field of the Invention
The field of the invention relates to control systems generally, and more particularly to certain new and useful advances in network topologies connecting multiple devices within control systems for industrial applications, of which the following is a specification.
2. Description of Related Art
At a high level, controller devices are essentially specialized computers that contain most of the components found in a personal computer (hereinafter PC) today-including central processing units (hereinafter CPUs), memory, disk drives, and various input and output (hereinafter I/O) connections. Like computers, controller devices can be linked together in a network in order to communicate information and transfer data back and forth quickly and efficiently.
PCI™ (hereinafter PCI) and its successor PCI Express® (hereinafter PCIe) are serial bus standards that provide electrical, physical and logical interconnections for peripheral components of microprocessor-based systems. The native topology of connections supported by PCIe emulates the tree structure of its predecessor PCI. The native PCI tree topology allows only one master CPU in the system. This master CPU is known as a root complex. Other CPUs and similar compute devices can be connected to the PCI tree as a leaf node to the root complex. If the primary root complex fails, the CPU connected through the non-transparent (hereinafter NT) bridge can take over system control and become the new root complex.
Tree structures have some drawbacks for the needs of modern control systems that connect multiple controller devices in a network. For example, in the standard PCIe tree, all devices must be initialized by a common root complex in a process referred to as PCIe enumeration. The root complex must be aware of all PCIe devices in the network in order for the enumeration process and future communication to be successful. This limits the topology of devices to tree or star topologies and prevents the use of daisy-chained or ring topologies.
Thus, there is a need for devices, systems and methods that take advantage of the high-speed connection capabilities of the PCIe standard without the drawbacks and constraints of known network configurations for PCIe devices.
The devices, systems and methods of the subject invention are directed to connecting multiple devices in a daisy-chain topology. The daisy-chain or expansion topology of the subject invention is an extensible solution that allows for a variable number of nodes. Conventional PCIe devices are connected to a chassis with a fixed number of slots. In addition, PCIe devices are typically connected in a tree or star-topology with a single PCIe device serving as the PCIe root. In contrast, the devices, systems and methods of the subject invention do not require a chassis and do not require any fixed slots. In the subject invention, there is no single device or bridge designated as the PCIe root. Rather, the subject invention allows additional devices to be added to the daisy-chain topology without construction of a new chassis.
PCIe devices can be connected either through a cable or through direct module-to-module connectors. This allows flexibility in the connection of the devices which could also be simply stacked to provide a communication channel. In addition, if devices in the subject invention need to be some distance apart, then a connector, such as a PCIe cable can provide the connection as well. Alternately, the subject invention provides a hybrid connection that may be used to connect some groups of modules by physically stacking the modules or devices, and then linking the separate stacks with a connector, such as a PCIe cable. Once a device is connected on the daisy-chain topology, any device has the ability to communicate to any other device present on the network. The devices are thereby able to let their CPU or processors directly talk to each of the CPU and processors of other devices in the network. The subject invention therefore enables a master/master or peer-to-peer relationship between devices on the daisy-chain rather than any one device serving as the master and the remaining devices the slaves.
One embodiment of the present invention is a system comprising a first set of devices, each of the first set of devices having a central processing unit connected to an internal PCIe bridge, wherein each of the first set of devices are connected to each other in a peer-to-peer arrangement along an external PCIe bus in a daisy-chain topology. The PCIe bridge of each of the plurality of devices may have at least one NT port. The system of the PCIe bridge of each of the plurality of devices may have a first and a second port for connecting its respective device to the external bus, and may further comprise a third port for transmitting and receiving data to and from the respective device to one or more of the first set of devices in the daisy-chain topology. In another embodiment, the system of the present invention further comprises a second set of devices, each of the second set of devices having a central processing unit connected to a PCIe bridge. The second set of devices may be mechanically coupled or mated together. The at least one of the second set of devices may be connected along the external bus to the daisy-chain topology. In a further embodiment, the second set of devices can communicate or transfer data back and/or forth along a common communication channel when mechanically coupled.
A method of providing data transfer between an initiating device and a target device is also provided. In one embodiment, the method comprises the steps of providing a plurality of devices including an initiating device and a target device, each of the plurality of devices having a NT PCIe bridge; connecting the plurality of devices in a peer-to-peer daisy-chain topology; and performing transfer of data by traversing the daisy-chain topology starting from the initiating device and ending at the target device. The method may include a PCIe bridge that comprises at least a first port, a second port, and a third port. In another embodiment, the method comprises the step of connecting the plurality of devices in the daisy-chain topology, which includes cabling the first and second ports of each of the devices to an external expansion bus. The external expansion bus may be a PCIe connector. In yet another embodiment, the method further comprises the step of connecting the third port to an internal PCIe bus within each of the plurality of devices, respectively. In a further embodiment, the method further comprises the step of selecting one of the first port and the second port on the initiating device from which to begin transfer of the data based on a direction of the target device on the external expansion bus. The method may further comprise the step of reading or writing the data within an internal memory of the target device, after the daisy-chain topology has been traversed and transfer of data to and/or from the target device is completed.
The devices, systems and methods of the subject invention provide a networking solution for compute devices that eliminates the need for users to buy a chassis or external bridge switch to interconnect modules or devices, potentially saving space in an industrial setting, for example. The ability to add PCIe devices in differing topologies, other than the native PCIe tree, gives a user the flexibility to configure and implement a network in a unique way for whatever their needs may be. Furthermore, allowing multiple CPUs to work together increases performance and provides redundant more robust systems.
Other features and advantages of the disclosure will become apparent by reference to the following description taken in connection with the accompanying drawings.
Reference is now made briefly to the accompanying drawings, in which:
Like reference characters designate identical or corresponding components throughout the several views, which are not to scale unless otherwise indicated.
The devices, systems and methods of the subject invention are directed to an extensible topology for connecting compute devices. The subject invention is particularly useful for applications where high bandwidth, low latency, redundancy and ease of expansion are desired. The subject invention enables compute devices to be networked in an extensible daisy-chain or linear topology. In one embodiment, a bridge, such as a PCIe bridge or switch, is integrated into every node of the expansion. This PCIe bridge supports NT bridging and Spread-Spectrum Clocking The subject invention overcomes the native topology of communication bus standards, like PCIe, in order to achieve a number of benefits and advantages over known devices, systems and methods as described herein.
In a preferred embodiment, the bridge 30 is a PCIe bridge, or switch, and the physical connection 50 is a PCIe cable or connector. In one embodiment, there are multiple NT ports on each bridge. While one NT port is the minimal number to allow a PCIe daisy-chain topology, additional NT ports may also be used. Additional NT ports may allow for flexibility in the physical connector used to connect devices. For example, by reconfiguring NT ports a user can switch the physical connector from a PCIe cable to a proprietary stackable connector. In one embodiment, one or more of the compute devices 10 supports both a cable connector and a stacking connector to directly plug into a group of stacked compute devices, such as the stacked devices 10e-10g shown in
To support this relative addressing, the bridge window's NT address translations must be setup to shift the data window down by 1 for each hop through the next PCIe bridge. For example, suppose a device needs to write a single byte of data to a device that is four nodes down the daisy-chain. The device should write the byte into Window 3 of its expansion bridge. This access to Window 3 on the bridge must be translated to forward the transaction to Window 2 of the next bridge in the chain. Similarly, Window 2 translates to Window 1, Window 1 to Window 0, and finally Window 0 maps to internal memory on the target device. Window translations must be setup in both directions on every PCIe bridge to allow bi-directional communication between PCIe device nodes. Thus, any PCIe device can share memory and that shared memory can be accessed by any other PCIe device on the expansion bus 50.
RAM 22g and RAM 22f present on each device is the respective device's internal memory. RAM is the final destination of all transactions accessing a particular compute device on the ring (read or write of RAM). Direct Memory Access (hereinafter DMA) components, DMA 34g and DMA 32f are hardware components that may optionally be used by one or more compute devices to initiate transactions to another compute device on the daisy-chain. DMA can be programmed to transfer a set of data to or from a target device which allows the local device's CPU to concurrently perform other operations while DMA is in progress. The use of DMA improves performance especially for large data transfers.
While there is only one NT port required per PCIe bridge, any communication to an adjacent device in the system will go through the NT port of that device. In one direction, the CPU of a given compute device interfaces with the NT port windows of its own PCIe bridge. In the other direction, the CPU interfaces with the NT port windows of the adjacent device. Both the left and right ports in this embodiment are NT ports so an address translation can be made for each window. The Window 0 port address translation will be mapped to the internal CPU's memory of the “adjacent” CPU on the daisy-chain. The exact memory address can be different for each CPU. The other Windows (1-4) must have a memory translation to the next device's NT port and move the memory address down by 1 memory window (for example, 0xA0100000 translates to 0xA0000000 into the next NT port). Additional NT ports per device could be present. This includes an additional NT port on the other daisy-chain port or an NT port between the PCIe bridge and the local device's internal PCIe bus (port to local internal memory). Additional NT ports do not change the basic operation of the daisy-chain implementation described herein.
The address range of the memory windows determines which devices are accessed for any transaction (read or write of memory). The NT port address translations will direct any transaction to exactly one device in the daisy-chain. If a new device is added to the daisy-chain network and initialized, it will continuously listen for and process any transactions in its configured address range.
Devices in the daisy-chain topology of the subject invention may have heterogeneous operating systems. For example, in
The devices, systems and methods of the subject invention described herein allow for higher reliability of data transfer on a network. The extensible topology allows flexibility in connecting devices in a network with a high-speed interconnect through direct module-to-module connections or cabling. There can be a variable number of devices or nodes in the network, with easy addition or removal as necessary. Networked devices in the subject invention can be daisy-chained or ringed and are not limited to the conventional star topology or point-to-point connection of PCIe and similar standards. The network design does not need a fixed chassis or expansion bus root to allow processors to talk to each other. Additionally, the devices have a master/master relationship, and this multi-master mode allows all the devices to interact in a type of voting system. If a problem should arise, each device can come up with a solution and compare it with the solution of the other devices. This allows for more fail-safe and intelligent systems.
As used herein, an element or function recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural said elements or functions, unless such exclusion is explicitly recited. Furthermore, references to “one embodiment” of the claimed invention should not be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.
This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to make and use the invention. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.
Although specific features of the invention are shown in some drawings and not in others, this is for convenience only as each feature may be combined with any or all of the other features in accordance with the invention. The words “including”, “comprising”, “having”, and “with” as used herein are to be interpreted broadly and comprehensively and are not limited to any physical interconnection. Moreover, any embodiments disclosed in the subject application are not to be taken as the only possible embodiments. Other embodiments will occur to those skilled in the art and are within the scope of the following claims.