Patent Application
20230325183
Processing cores can cooperatively execute complex computations by executing component computations of that complex computations in distributed fashion across the processing cores. To do so, the processing cores need to share data required for the execution of those component computations as well as receive instructions regarding which component computations they have been assigned. The technology associated with how the complex computation is broken down into component computations and assigned to the various processing cores is associated with the field of parallel computing.
Processing cores can be connected via a network to facilitate the exchanges of data associated with parallel computing. Each processing core can be connected to the network via a dedicated router. When the processing cores are located on a single chip, the network can be referred to as a network-on-chip (NOC). Data can be transmitted amongst the cores using unicast, multicast, or broadcast transmissions. The transmissions can be single-hop or multi-hop depending upon how the cores are connected and the physical location of the source and destination cores within the network. For example, adjacent cores may be able to communicate using a single-hop transmission while distal cores might need to communicate via multi-hop transmissions as data is passed in series between adjacent routers and forwarded from each intermediate router on to the final destination router.
A rapidly developing computational application to which networks of processing cores have been applied is the field of artificial neural networks (ANNs). Modern ANNs are trained using a massive amount of computational power which is proportional to the number of parameters in the ANN. The current trend in the field of ANNs is one of ever-increasing parameters per ANN. As of the date of this application researchers have begun to train and deploy ANNs that are scaled to the order of over a trillion parameters. This trend creates a distinct challenge and opportunity for innovation.
There are various approaches being applied in the field of ANNs for dealing with large parameter ANN training. Certain approaches are centered on creating larger individual chips with an increasing number of processing cores to support the training of larger and larger ANNs. However, the physical size of a single chip cannot alone scale at the rate at which ANNs are increasing. Alternative approaches involve the creation of larger networks of chips to address the rapid increase in the size of ANNs. However, these approaches have likewise run into problems in that the larger networks tend to involve network hierarchies including chip-to-chip, server-to-server, and rack-to-rack hierarchical levels. Programming a given ANN to operate on these larger networks generally requires a different programming model for each level. The result is that a programmer is often forced to develop their ANN with a specific network in mind, or to adjust the code for implementing a completed ANN on a network by hand. The different programming models also make it difficult for compilers to compile an ANN for operating on a given network.
Methods and systems related to parallel computing using heterogenous networks of computational nodes are disclosed herein. Specific approaches disclosed herein allow for the automated place and route of a complex computation on a heterogenous network of computational nodes. As used herein, a heterogenous network of computational nodes refers to a network of computational nodes with at least one of nonuniform links between computational nodes and nonuniform computational nodes. As such, a heterogenous network of computational nodes can comprise a homogenous set of computational nodes that are linked together by a network with nonuniform links.
Specific approaches disclosed herein allow a programmer to specify a complex computation for execution on a network of computational nodes without concern for the underlying hardware configuration of the network. Accordingly, the programmer can instantiate additional computational nodes for the execution of portions of the complex computation in the source code representation of the complex computation, and the compiler will assign those computational nodes seamlessly to computational nodes in the heterogenous network. The programmer can effectively consider the network to be an infinite mesh of computational nodes and the compiler can engage in an automated place and route of the component computations and associated data involved in the execution of the complex computation amongst the computational nodes in the network to most efficiently execute the complex computation. These benefits are achieved through various means as disclosed herein.
A compiler and compilation process are disclosed herein which takes as its input a source code description of a complex computation for execution and a simple machine model of a heterogenous set of computational nodes linked together by a set of links in a network. The simple machine model can include a table of the bandwidths of the various links in the set of links in the network. The simple machine model can additionally include a table of the latencies of the various links in the set of links in the network.
Additionally disclosed herein is a network of computational nodes in the form of processing cores in a set of multicore processor chips where each multicore processor is a native ethernet device capable of packaging packets for the execution of a directed graph by the network of computational nodes such that data can be routed through the network between multicore processor chips without the need for external ethernet switches. In specific embodiments disclosed herein, the processing cores in the multicore processors are designed to communicate across their NOCs using packets of directed graph data such that the conversion of the NOC packets into Ethernet packets is nearly seamless and does not introduce significant latency to the flow of data through the network as local data is translated into data in ethernet compatible packets.
In specific embodiments of the invention, a method for executing a complex computation on a heterogenous set of computational nodes linked together by a set of links in a network is disclosed. The method comprises compiling, using a table of bandwidth values for the set of links in the network, a set of instructions for routing data for the execution of the complex computation. The method also comprises configuring a set of programmable controllers on the heterogenous set of computational nodes with the set of instructions. The method also comprises executing the set of instructions using the set of programmable controllers. The method also comprises routing data: (i) through the network; (ii) to facilitate the execution of the complex computation by the heterogenous set of computational nodes; and (iii) in response to the execution of the instructions.
In specific embodiments of the invention, a method for executing a complex computation on a heterogenous set of computational nodes linked together by a set of links in a network is disclosed. The method comprises compiling, using a machine model of the set of links, a set of instructions for routing data for the execution of the complex computation, wherein the machine model includes a bandwidth for each link in the set of links in the network. The method also comprises configuring a set of programmable controllers on the heterogenous set of computational nodes with the set of instructions. The method also comprises executing the set of instructions using the set of programmable controllers. The method also comprises routing data: (i) through the network; (ii) to facilitate the execution of the complex computation by the heterogenous set of computational nodes; and (iii) in response to the execution of the instructions.
In specific embodiments of the invention, a system for executing a complex computation is disclosed. The system comprises a heterogenous set of computational nodes and a set of links in a network. The set of links link the computational nodes in the heterogenous set of computational nodes. The system also comprises a compiler configured to compile, using a table of bandwidth values for the set of links in the network, a set of instructions for routing data for the execution of the complex computation. The system also comprises a set of programmable controllers on the heterogenous set of computational nodes configured with the set of instructions. Executing the set of instructions using the set of programmable controllers routes data through the network to facilitate the execution of the directed graph by the heterogenous set of computational nodes.
Methods and systems related to parallel computing using heterogenous networks of computational nodes in accordance with the summary above are disclosed in detail herein. The methods and systems disclosed in this section are nonlimiting embodiments of the invention, are provided for explanatory purposes only, and should not be used to constrict the full scope of the invention. Although the specific examples provided in this section are directed to a network of computational nodes in the form of a NOC connecting a set of processing cores, specific embodiments disclosed herein are broadly applicable to any form of network connecting any form of computational nodes.
Networks in accordance with this disclosure can be implemented on a single chip system, including wafer-scale single chip systems, in a multichip single package system, or in a multichip multipackage system in which the chips are commonly attached to a common substrate such as a printed circuit board (PCB), interposer, or silicon mesh. Networks in accordance with this disclosure can also include chips on multiple substrates linked together by a higher-level common substrate such as in the case of multiple PCBs each with a set of chips where the multiple PCBs are fixed to a common backplane. Networks in accordance with this disclosure can also be implemented in chiplet based systems. For example, in specific embodiments of the invention, one or more computational nodes could be housed or implemented by one or more chiplets, connected, for example, through an interposer. Networks in accordance with this disclosure can further be implemented to link computational nodes on a single server, a set of servers in a cabinet rack, in a set of cabinet racks, in an entire data center, or a collection of networked data centers.
The computational nodes in accordance with this disclosure can cooperate in the execution of a complex computation by sharing data using the links of a network. The computational nodes can be processing cores such as individual processing cores in a multicore processor, individual processing units, processing cores of a graphics processing units, neural network accelerator processing units, processing cores of a cryptographic accelerator processing units, processing cores of a tensor processing unit, or any other processing unit or processor. An application data flow graph may be executed for the data involved with the complex computation to be distributed through the system. The application data flow graph can describe how data needs to flow through the system between the computational nodes for the complex computation to be executed. The points at which data can enter or exit the application data flow graph can be referred to as endpoints of the network. The endpoints can be associated with computational nodes. The endpoints can be any component of the system that requires data from the network or that administrates the transfer of data to a system on the network. In keeping with the example above of a multicore processor, the endpoints could be the processing pipelines of the processing cores, CPUs located on the processing cores, the core controllers, or any other functional block that needs access to data from the network.
In specific embodiments of the invention, dedicated hardware components can be used to efficiently execute an application data flow graph on a set of computational nodes. The dedicated hardware components can take the form of the specialized circuit blocks, such as the programmable controllers. The application data flow graph can be generated by the compiler disclosed herein and the instructions to implement the application data flow graph, as generated by the compiler, can be used to configure the programmable controllers. The instructions can be provided by higher-level controllers such as a core controller on a processing core. The instructions can be provided by a computation layer of the system. The programmable controllers can then independently and asynchronously execute the instructions independently of any higher-level controllers. The execution of the instructions can allow the programmable controllers to execute an application data flow graph for the computation layer of the system.
The complex computations disclosed herein can be the execution of a directed graph. The directed graph can be described using application code (e.g., a source code description of an algorithm). The directed graph can implement a machine learning algorithm such as an artificial neural network (ANN) or support vector machine (SVM). The directed graph can also represent a hashing, encryption, decryption, or graphics rendering algorithm involving many component computations. In particular, the directed graph can implement algorithms requiring a high level of parallel processing such as a ray casting algorithm, cryptographic computations, recurrent neural networks, and others.
Flow chart 200 beings with a step 201 of compiling, using a table of bandwidth values for a set of links in a network, a set of instructions for routing data for the execution of the complex computation. The compiling can be conducted by a compiler. The set of instructions can be instructions for the network to route data between the computational nodes of the network such as instructions for executing an application data flow graph. The table of bandwidth values can be a data structure with a list of links in the network and an associated list of scalar values, or range of scalar values, for the bandwidth values for the associated links. The values could be provided in terms of bits or other data elements per unit time. The table could be derived from the network of computation nodes. As illustrated, table 215 is derived from network of processing cores 210. The table could be configured and made accessible to the compiler by the designer of the hardware of the network. In specific embodiments, the compiler could have a default program for exploring the topology and bandwidth of the various links of the network it was designed to provide instructions for. The table could also be a more complex data structure with more information regarding the links such as their latency values, peak frequency, quiescent and active state power consumption, and various other values.
In specific embodiments, step 201 can alternatively or in combination include compiling, using a machine model of the links of the network, the set of instructions for routing data for the execution of the complex computation. The machine model could be or include a model of the links of the network. The machine model could include a description of the topology and number of computational nodes in the network. The machine model could be or include a model of the computational nodes of the network. For example, the machine model could include an estimate of the operations per unit time that the computational nodes are able to execute. In specific embodiments, the machine model will include a table of bandwidth values and any of the additional values mentioned with respect to the tables in the prior paragraph. In embodiments where the computational nodes are homogenous, the simple machine model can solely include the information mentioned above. However, in embodiments where the computational nodes themselves are heterogenous, the simple machine model can additionally include a table of the computational resources of the computational nodes in the heterogenous network. The computational resources can include a computational speed such as a number of floating-point operations per second, a size of the main memory of the computational nodes, a size of the cache(s) of the computational nodes, and other high-level parameters that are indicative of the computational resources of the computational node.
Step 201 can be conducted using a compiler operating on a source code description of the complex computation. The compiler could automatically place and route the data for the component computations of the complex computation onto a given network using the description of the network provided by the tables or machine model described above. The instructions generated could include instructions for implementing an application data flow graph for the complex computation in a given network. The instructions could then be loaded into the network. As illustrated, instructions 216 are provided to network of processing cores 210 to allow the network of processing cores 210 to execute an application data flow graph for the complex computation. As the compiler is provided with data regarding the network and automatically places and routes the data for the component computations of a complex computation, the source code description of the complex computation can be written without concern for the idiosyncrasies of the network that will execute the complex computation.
The source language of the compiler and the target language of the compiler could take on various forms. The complex computation could be the execution of directed graph implementing a neural network and the source code description could be a PyTorch encoding of the neural network. The set of instructions for routing data for the execution of the complex computation could be one of the outputs of the compiler in the target language of the compiler. The target language could be machine code for the routing elements of the various computational nodes in the network of computational nodes. In implementations in which the heterogenous network includes different types of computational nodes, or different types of links, the compiler could have multiple target languages which are specialized for each of the various nodes or links. However, in alternative embodiments, the network could be such that the different computational nodes and different links were compatible with a given networking standard, such as Ethernet, and the target language would be Ethernet router instructions. In implementations in which the routing technology of the various links and computational nodes was homogenous, the target language of the compiler could be instructions for that routing technology.
In the specific embodiments of the invention, the compiled instructions are instructions for the execution of a directed graph which is used to implement an ANN. In these embodiments, a compiler in accordance with specific embodiments of the invention can engage in automated model parallelism by slicing the tensors defined in the source code description of the ANN into packets that are used to route the data between the computational nodes of the network. The compiler can thereby assure that the computational nodes are used efficiently and are not overloaded by data structures that are not able to efficiently process on an individual basis. The compiler can also engage in automated pipeline parallelism by allowing the layers of the ANN to be scaled out and distributed to different computational nodes such that the layers can be executed in a pipelined fashion.
Flow chart 200 continues with a step 220 of configuring a set of programmable controllers on the heterogenous set of computational nodes with the set of instructions. The instructions produced by the compiler in step 201 can be used to configure a set of programmable controllers which can execute the instructions to route data for the complex computation. The programmable controllers can execute the instructions asynchronously to the execution, by the computational nodes of the network, of the computations required for the execution of the complex computation. In specific embodiments, the application data flow graph can be efficiently executed using dedicated hardware components on the various computational nodes. The dedicated hardware components can include hardware subcomponents of varying degree of programmability, such as software-programmable general-purpose controllers or specialized programmable controllers. The programmable controllers on the various computational nodes can be configured using instructions provided to the controllers. The programmable controllers on the various computational nodes can then independently and asynchronously execute the instructions. These hardware subcomponents can each be configured to implement one transition of data in a set of transitions of data that combine to implement the data flow of the application data flow graph. The programmable controllers can be reconfigured using additional instructions to implement additional transfers of data during the execution of the application data flow graph. In specific embodiments, the programmable controllers can be native ethernet routers and be configured to route data for the execution of the complex computation using ethernet. In these embodiments, the computational nodes can include processing cores on multicore processors, where the multicore processors include integrated ethernet routers, so the multicore processor can communicate with other external computational nodes using ethernet, but without the need for an intermittent ethernet switch.
The programmable controllers can be configured in numerous ways. For example, the set of programmable controllers can be configured by a controller of a processing cores, such as a CPU of each processing core. The programmable controllers can be configured to read form one or more specific addresses in a memory of the system. Configuration instructions can be pre-stored in the memory addresses or can be dynamically written by a different component of the system, such as a computation layer controller, such that the programmable controllers can read the configuration instructions from the memory address and configure themselves in accordance with such instructions. In embodiments in which the configuration instructions are dynamically generated by the system, the dedicated programmable controllers will be able to read such configurations and provide a highly configurable set of functionalities in accordance with the system needs and current status. The programmable controllers can be configured to continuously read instructions from memory as they execute their instructions and administrate the transfer of data in the network layer. The controller/CPU can control the location in memory where the instructions are being read to thereby control the configurations of the programmable controllers. The controller/CPU can induce the programmable controllers to start reading from another location in memory to change the configuration of the programmable controllers. The controller/CPU can also write the instructions themselves to thereby configure the programmable controllers. The controller/CPU can also, by writing different sets of instructions and/or by passing different addresses for the instructions to be read from to the programmable controllers, control a schedule of configurations for the dedicated hardware components to execute. In other words, the controller/CPU can cause a programmable controller to execute different portions of an application data flow graph at different times during the execution of that application data flow graph.
Flow chart 200 continues with a step 230 of executing the set of instructions using a set of programmable controllers and routing data through the network. The step can be conducted in accordance with the approaches described in the preceding paragraphs. The data can be routed through the network to facilitate the execution of to complex computation by the heterogenous set of computational nodes and in response to the execution of the instructions.
In specific embodiments of the invention, the networks can include a network hierarchy with various levels and with various computational nodes in each level. The network can include a network hierarchy with at least a core level and a chip level. The network can include a core level and a chip level as well as a server level and a rack level. The links in the network can include computer system level networking links such as ethernet links. The ethernet links can connect computational nodes in the same level of the hierarchy or can connect computational nodes across hierarchy levels. The computational nodes in each level may be able to communicate directly or may need to communicate through a higher level. For example, the heterogenous set of computational nodes could include a set of processing cores in at least two multicore processors. The network would thereby by heterogenous in that the links between processing cores on the individual multicore processors would likely be much faster than the links between the chips themselves. The processor cores in the multicore processor could then communicate directly while the processor cores on different levels would need to communicate via a communication through the chip level.
In specific embodiments of the invention, the computational nodes communicate across levels of the hierarchy and within levels of the network hierarchy in various ways. In specific embodiments of the invention, each computational node in a heterogenous set of computational nodes includes a router from a set of routers. The routers can be distributed across the computational nodes in a one-to-one correspondence. The routers may be homogenous or heterogenous. In general, the routers in different levels of the hierarchy will be heterogenous. For example, a processing core could include a router configured to communicate with a NOC while a multichip processor could include a router configured to communicate with an ethernet network. The routing of data through the network could include the set of homogenous or heterogenous routers transitioning data through multiple levels of the network hierarchy. In the previous example, this would include transitioning data through the core level and the chip level of the hierarchy.
In specific embodiments of the invention, the same compiler could generate instructions for the different types of routers, in a heterogenous set of routers, to execute. In these embodiments, the compiler could have multiple target languages for generating the instructions for the different types of routers. The routers could include instructions to route data through the links of the network such as to route data through an ethernet link. However, the individual routers, such as routers on a processing core, may not need to know that the packet they are routing is destined for an ethernet link as the router may only know that it needs to pass the packet to another location, and not need to know that it will then be routed from that location through the ethernet link.
In specific embodiments of the invention, the data could be routed through the network using packets where the packets are identified using packet identifiers. The packets could include payloads and packet headers including an identifier for the packets. The routers of the network could route the packets using their instructions and the identifiers for the packets. In specific embodiments of the invention, the packets could be ethernet packets. In specific embodiments of the invention, different levels of the network hierarchy could be packet based using either the same or differing packet types. For example, the core level of the hierarchy could use proprietary NOC packets while the chip level of the hierarchy could use ethernet packets. However, the fact that both levels of the hierarchy used packets would be beneficial in that translating from one level to the next could be achieved simply by applying a wrapper to the underlying packets (with an optional step of gathering packets) as opposed to reformatting serial streaming data, or some other data format, into packets. In specific embodiments of the invention, packet routing can be conducted on the processing cores at the core level as is described in U.S. patent application Ser. No. 17/035,046 filed Sep. 28, 2020, which is incorporated by reference herein in its entirety for all purposes, with the exception that the terms of this application are to be interpreted in light of their usage in this specification and not in light of their usage in that specification of the incorporated reference.
Flow chart 310 illustrates how data can be routed through network hierarchy 300. As illustrated, the data passes from a processing core on chip 311, through chip 312, and on to a processing core on chip 313. As such, the data is transferred through the chip level of the network hierarchy. The instructions necessary to route the data could include instructions executed by the NOCs of chip 311 and 313. However, the transfer does not necessarily require any actions to be taken by the NOC of chip 312. In the illustrated case, each of the chips include integrated ethernet routers, such as integrated ethernet router 306. As such, chip 312 is simply a passthrough that receives the data and routes it on the chip level 302 but does not need to utilize any of the resources of the core level 301 on chip 312. Furthermore, in these embodiments the transfer of data does not require an external ethernet switch to route data on the chip, server, or rack level.
Flow chart 310 also illustrates the use of packets to route data through the network hierarchy. As illustrated, packet 320 includes a payload 322 and a packet header 321. The packet header could include a packet identifier for the packet. Flow chart 310 also shows packet 320 being routed through the chip level of the network hierarchy. As such, packet 320 is an ethernet packet. Accordingly, if the core level 301 is also packet based then the payload 322 of packet 320 may include the entire packet of the core level 301. Alternatively, packet 320 could be a packet routed by both the core layer and the chip level of the network hierarchy. In specific embodiments of the invention, the different levels of the network hierarchy can all conduct similar operations to simplify the operation of the compiler such as sharing a common library of primitives such as push, pop, copy, gather (N:1), scatter (1:N), shuffle N:N, and others. For example, the core level 301 and chip level 302 may operate on different types of packets but still conduct the same primitive operations on those packets.
In specific embodiments of the invention, the software level of source code description 410 and front end 403 can see network as a homogenous mesh of cores such that the source code description 410 does not need to include any information related to the network hierarchy of the network. Furthermore, while the source code description 410 may include a description of a set of processing cores that are intended to conduct specific operations in the complex computation, the set of cores identified in source code description 410 do not correlate with the set of computational nodes in the heterogenous set of computational nodes that will execute the set of instructions 420 because optimizer 404 can engage in layer and model parallelism to break both data and operations down into smaller pieces that can be executed by a larger number of cores. In the same manner, if the network utilized computational nodes with greater resources than required by specific computational nodes identified in source code description 410, optimizer 404 may condense the operations of multiple processing cores in source code description 410 into a single computational node. Optimizer 404, with visibility into the bandwidth of the links of the network, can determine if the benefit of multiple cores being applied to a computation is outweighed by the drawback of having to route data between them based on how many levels of the network hierarchy must be traversed and the bandwidth of individual inks in the network.
In specific embodiments of the invention, the disclosed network of computational nodes is in the form of a set of processing cores in a multicore processor connected by a NoC. The processing cores can each include a network interface unit (NIU) for interfacing with a router, a processing pipeline, and a memory. The memory could be a cache memory of the processing core such as a random-access volatile memory such as SRAM. The processing cores could also include additional or more specific elements such as a higher-level controller, serializer/de-serializer, nonvolatile memory for modifiable configuration information, and any number of arithmetic logic units and other fundamental computation units. The processing cores could also include one or more endpoints that can ultimately coordinate or execute operations within the core, such as a core controller. In specific embodiments, the core controller can be a dedicated Central Processing Unit (CPU). In either case, the core controller or CPU can administrate the transfer of data to a main processing pipeline of the processing core. The processing pipelines can be used to execute component computations required for the multicore processor to execute a complex computation. The NoC can include routers on each of the processing cores and a system of buses that either solely connect adjacent processing cores in the multicore processor for facilitating multi-hop communications or also connect distal processing cores for facilitating single-hop communications between those distal processing cores.
While the specification has been described in detail with respect to specific embodiments of the invention, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing, may readily conceive of alterations to, variations of, and equivalents to these embodiments. For example, the teachings disclosed herein should not be limited to a NOC connecting a network of processing cores as they can be applied to any network which operates to connect any set of computation units which are executing a complex computation. Furthermore, the disclosure should not be limited to any specific type of complex computation being executed on a multicore processor as the programming of and execution of any complex computation can be aided by specific embodiments disclosed herein including training or drawing an inference from an artificial neural network, or executing a hashing, encryption, decryption, or graphics rendering algorithm involving a large number of component computations. Furthermore, the examples of a network overlay layer isolating a computation layer from a network layer should not be interpreted to exclude systems which can be placed in a debugging mode, or similar mode of operation, in which the continuous execution of a complex computation is stalled and the hardware or software otherwise associated with the computation layer can have a direct impact on the network layer without communicating with the network overlay layer. These and other modifications and variations to the present invention may be practiced by those skilled in the art, without departing from the scope of the present invention, which is more particularly set forth in the appended claims.
