Patent Application
20220269534
The present technology relates to reconfigurable architectures, can be particularly applied to time-multiplexed use of reconfigurable hardware.
The following are incorporated by reference for all purposes as if fully set forth herein:
Reconfigurable processors can be configured to implement a variety of functions more efficiently or faster than might be achieved using a general-purpose processor executing a computer program. So called Coarse-Grain Reconfigurable Architectures (e.g., CGRAs) are being developed in which the configurable units in the array are more complex than those used in typical, more fine-grained Field-Programmable Gate Arrays (FPGAs), and may enable faster or more efficient execution of various classes of functions. For example, CGRAs have been proposed that can enable implementation of energy-efficient accelerators for machine learning and artificial intelligence workloads. See, Prabhakar, et al., “Plasticine: A Reconfigurable Architecture for Parallel Patterns,” ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada.
A reconfigurable architecture system may include general-purpose non-reconfigurable hardware, as well as reconfigurable hardware resources that can be reconfigured to suit a need of a specific application being executed in the reconfigurable hardware. In an example, certain portions of an application program are executed in the general-purpose hardware, and other portions of the application program are executed in the reconfigurable hardware. When portions of an application are being executed in the general-purpose hardware, the reconfigurable hardware can be idle. Similarly, when other portions of the application are being executed in the reconfigurable hardware, the general-purpose hardware can be idle. This may result in underutilization of hardware resources in the reconfigurable architecture system.
In order to maximize operating efficiency, it may be desirable to time-multiplex programs on the reconfigurable architecture system.
The following description will typically be with reference to specific structural embodiments and methods. It is to be understood that there is no intention to limit the technology to the specifically disclosed embodiments and methods but that the technology may be practiced using other features, elements, methods and embodiments. Preferred embodiments are described to illustrate the present technology, not to limit its scope, which is defined by the claims. Those of ordinary skill in the art will recognize a variety of equivalent variations on the description that follows.
Elements referred to herein with a common reference label followed by a particular number or alphabet may be collectively referred to by the reference label alone. For example, configuration files 613_1, . . . , 613_N (illustrated in
In contrast, the reconfigurable data processor 110 and one or more reconfigurable components therewithin (e.g., an array 190 of configurable units) are referred to as “reconfigurable hardware”, as the reconfigurable data processor 110 and the one or more components therewithin are configurable and reconfigurable to suit needs of a program being executed thereon, as will be discussed herein in further detail in turn.
As shown in the example of
As shown in the example of
The processor 110 includes an external I/O interface 130 connected to the host 120, and external I/O interface 150 connected to the memory 140. The I/O interfaces 130, 150 connect via a bus system 115 to the array 190 of configurable units and to the configuration load/unload controller 195. The bus system 115 may have a bus width of carrying one chunk of data, which can be for this example 128 bits (references to 128 bits throughout can be considered as an example chunk size more generally). In general, a chunk of the configuration file can have a number N of bits of data, and the bus system can be configured to transfer N bits of data in one bus cycle, where N is any practical bus width. A sub-file distributed in the distribution sequence can consist of one chunk, or other amounts of data as suits a particular embodiment. Procedures are described herein using sub-files consisting of one chunk of data each. Of course, the technology can be configured to distribute sub-files of different sizes, including sub-files that may consist of two chunks distributed in two bus cycles for example.
To configure configurable units in the array 190 of configurable units with a configuration file, the host 120 can send the configuration file to the memory 140 via the interface 130, the bus system 115, and the interface 150 in the reconfigurable data processor 110. The configuration file can be loaded in many ways, as suits a particular architecture, including in data paths outside the reconfigurable data processor 110. The configuration file can be retrieved from the memory 140 via the memory interface 150. Chunks of the configuration file can then be sent in a distribution sequence as described herein to configurable units in the array 190 of configurable units in the reconfigurable data processor 110.
The host 120 also executes a scheduler 126, which schedules interleaved sharing of general hardware (such as the host 120) and the reconfigurable hardware (such as reconfigurable data processor 110) among two or more applications, as will be discussed herein in further detail in turn.
The host 120 also executes processing logic 129, which performs operations when executing an application. For example, assume that an application to be executed in the system 100 has to pre-process data by the host 120, prior to the data being processed by the reconfigurable data processor 110. In an embodiment, the processing logic 129 within the host 120 preprocesses the data of the application.
In an example, the memory 140 is within a chip that is different from a chip comprising the reconfigurable data processor 110, and hence, the memory 140 is referred to herein as an off-chip memory. Similarly, the memory 128 is within a chip that is different from a chip comprising the reconfigurable data processor 110, and hence, the memory 128 is also referred to herein as an off-chip memory. Thus, off-chip memory refers to the memory 140 and/or the memory 128, in some examples. In contrast, the reconfigurable array of units 190 comprises configurable memory units (such as PMUs illustrated in
An external clock generator 170 or other clock signal sources can provide a clock signal 175 or clock signals to elements in the reconfigurable data processor 110, including the array 190 of configurable units, and the bus system 115, and the external data I/O interfaces.
Each of the four tiles has 4 AGCUs (Address Generation and Coalescing Units) (e.g., MAGCU1, AGCU12, AGCU13, AGCU14). The AGCUs are nodes on the top level network and nodes on the array level networks, and include resources for routing data among nodes on the top level network and nodes on the array level network in each tile.
Nodes on the top level network in this example include one or more external I/O interfaces, including interface 205. The interfaces to external devices include resources for routing data among nodes on the top level network and external devices, such as high-capacity memory, host processors, other CGRA processors, FPGA devices and so on, that are connected to the interfaces.
One of the AGCUs in a tile is configured in this example to be a master AGCU (MAGCU), which includes an array configuration load/unload controller for the tile. In other embodiments, more than one array configuration load/unload controller can be implemented and one array configuration load/unload controller may be implemented by logic distributed among more than one AGCU.
The MAGCU1 includes a configuration load/unload controller for Tile1, and MAGCU2 includes a configuration load/unload controller for Tile2. In other embodiments, a configuration load/unload controller can be designed for loading and unloading configuration of more than one tile. In other embodiments, more than one configuration controller can be designed for configuration of a single tile. Also, the configuration load/unload controller can be implemented in other portions of the system, including as a stand-alone node on the top level network and the array level network or networks.
The top level network is constructed using top level switches (211-216) connecting to each other as well as to other nodes on the top level network, including the AGCUs, and I/O interface 205. The top level network includes links (e.g., L11, L12, L21, L22) connecting the top level switches. Data travel in packets between the top level switches on the links, and from the switches to the nodes on the network connected to the switches. For example, top level switches 211 and 212 are connected by a link L11, top level switches 214 and 215 are connected by a link L12, top level switches 211 and 214 are connected by a link L13, and top level switches 212 and 213 are connected by a link L21. The links can include one or more buses and supporting control lines, including for example a chunk-wide bus (vector bus). For example, the top level network can include data, request and response channels operable in coordination for transfer of data in a manner analogous to an AXI compatible protocol. See, AMBA® AXI and ACE Protocol Specification, ARM, 2017.
Top level switches can be connected to AGCUs. For example, top level switches 211, 212, 214 and 215 are connected to MAGCU1, AGCU12, AGCU13 and AGCU14 in the tile Tile1, respectively. Top level switches 212, 213, 215 and 216 are connected to MAGCU2, AGCU22, AGCU23 and AGCU24 in the tile Tile2, respectively.
Top level switches can be connected to one or more external I/O interfaces (e.g., interface 205).
In this example, the array of configurable units 300 includes a plurality of types of configurable units. The types of configurable units in this example, include Pattern Compute Units (PCU), Pattern Memory Units (PMU), switch units (S), and Address Generation and Coalescing Units (each including two address generators AG and a shared CU). For an example of the functions of these types of configurable units, see, Prabhakar et al., “Plasticine: A Reconfigurable Architecture For Parallel Patterns”, ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada, which is incorporated by reference as if fully set forth herein. Each of these configurable units contains a configuration store comprising a set of registers or flip-flops that represent either the setup or the sequence to run a program, and can include the number of nested loops, the limits of each loop iterator, the instructions to be executed for each stage, the source of the operands, and the network parameters for the input and output interfaces.
Additionally, each of these configurable units contains a configuration store comprising a set of registers or flip-flops that store status usable to track progress in nested loops or otherwise. A configuration file contains a bit-stream representing the initial configuration, or starting state, of each of the components that execute the program. This bit-stream is referred to as a bit-file. Program load is the process of setting up the configuration stores in the array of configurable units based on the contents of the bit file to allow all the components to execute a program (i.e., a machine). Program Load may also require the load of all PMU memories.
The array level network includes links interconnecting configurable units in the array. The links in the array level network include one or more and, in this case three, kinds of physical buses: a chunk-level vector bus (e.g., 128 bits of data), a word-level scalar bus (e.g., 32 bits of data), and a multiple bit-level control bus. For instance, interconnect 321 between switch units 311 and 312 includes a vector bus interconnect with vector bus width of 128 bits, a scalar bus interconnect with a scalar bus width of 32 bits, and a control bus interconnect.
The three kinds of physical buses differ in the granularity of data being transferred. In one embodiment, the vector bus can carry a chunk that includes 16-Bytes (=128 bits) of data as its payload. The scalar bus can have a 32-bit payload, and carry scalar operands or control information. The control bus can carry control handshakes such as tokens and other signals. The vector and scalar buses can be packet switched, including headers that indicate a destination of each packet and other information such as sequence numbers that can be used to reassemble a file when the packets are received out of order. Each packet header can contain a destination identifier that identifies the geographical coordinates of the destination switch unit (e.g., the row and column in the array), and an interface identifier that identifies the interface on the destination switch (e.g., North, South, East, West, etc.) used to reach the destination unit. The control network can be circuit switched based on timing circuits in the device, for example. The configuration load/unload controller can generate a header for each chunk of configuration data of 128 bits. The header is transmitted on a header bus to each configurable unit in the array of configurable unit.
In one example, a chunk of data of 128 bits is transmitted on the vector bus that provides the chunk as vector inputs to a configurable unit. The vector bus can include 128 payload lines, and a set of header lines. The header can include a sequence ID for each chunk, which can include:
A bit to indicates if the chunk is scratchpad memory or configuration store data.
Bits that form a chunk number.
Bits that indicate a column identifier.
Bits that indicate a row identifier.
Bits that indicate a component identifier.
For a load operation, the configuration load controller can send the number N of chunks to a configurable unit in order from N−1 to 0. For this example, the 6 chunks are sent out in the most significant bit first order of Chunk 5->Chunk 4->Chunk 3->Chunk 2->Chunk 1->Chunk 0. (Note that this most significant bit first order results in Chunk 5 being distributed in round 0 of the distribution sequence from the array configuration load controller.) For an unload operation, the configuration unload controller can write the unload data out of order to the memory. For both load and unload operations, the shifting in the configuration serial chains in a configuration data store in a configurable unit is from LSB (least-significant-bit) to MSB (most-significant-bit), or MSB out first. Further detail of the load and unload process can be found in U.S. Non-provisional patent application Ser. No. 16/197,826, filed Nov. 21, 2018, entitled, “CONFIGURATION LOAD OF A RECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1001-1A), which is now issued as U.S. Pat. No. 10,831,507 issued on Nov. 10, 2020; and in U.S. Non-provisional patent application Ser. No. 16/198,086, filed Nov. 21, 2018, entitled, “CONFIGURATION UNLOAD OF A RECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1001-1B), each of which are incorporated by reference for all purposes as if fully set forth herein.
In an example, the switch unit is configurable. For example, when a first configuration file is being executed, the switch unit can interconnect a first PCU with a first PMU (e.g., such that the first PCU stores data in the first PMU). On the other hand, when a second configuration file is being executed, the same switch unit can interconnect the first PCU with a second PMU (e.g., such that the first PCU stores data in the second PMU).
A set of 2 switch units in each tile quadrant have connections to an Address Generation and Coalescing Unit (AGCU) that include multiple Address Generation (AG) units and a Coalescing Unit (CU) connected to the multiple address generation units. The Coalescing Unit (CU) arbitrates between the AGs and processes memory requests. Each of the 8 interfaces of a switch unit can include a vector interface, a scalar interface, and a control interface to communicate with the vector network, the scalar network, and the control network.
During execution of a machine after configuration, data can be sent via one or more unit switches and one or more links between the unit switches to the configurable units using the vector bus and vector interface(s) of the one or more switch units on the array level network.
In embodiments described herein, a configuration file or bit file, before configuration of the tile, can be sent from the configuration load controller using the same vector bus, via one or more unit switches and one or more links between the unit switches to the configurable unit using the vector bus and vector interface(s) of the one or more switch units on the array level network. For instance, a chunk of configuration data in a unit file particular to a configurable unit PMU 341 can be sent from the configuration load/unload controller 301 to the PMU 341, via a link 320 between the configuration load/unload controller 301 and the West (W) vector interface of the switch unit 311, the switch unit 312, and a link 331 between the Southeast (SE) vector interface of the switch unit 311 and the PMU 341.
In this example, one of the AGCUs is configured to be a master AGCU, which includes a configuration load/unload controller (e.g., 301). The master AGCU implements a register through which the host (120,
The configuration load controller in the master AGCU is responsible for reading the configuration file from the memory and sending the configuration data to every configurable unit of the tile. The master AGCU can read the configuration file from the memory at preferably the maximum throughput of the top level network. The data read from memory are transmitted by the master AGCU over the vector interface on the array level network to the corresponding configurable unit according to a distribution sequence described herein.
In one embodiment, in a way that can reduce the wiring requirements within a configurable unit, configuration and status registers holding unit files to be loaded in a configuration load process, or unloaded in a configuration unload process in a component are connected in a serial chain and can be loaded through a process of shifting bits through the serial chain. In some embodiments, there may be more than one serial chain arranged in parallel or in series. When a configurable unit receives for example 128 bits of configuration data from the master AGCU in one bus cycle, the configurable unit shifts this data through its serial chain at the rate of 1 bit per cycle, where shifter cycles can run at the same rate as the bus cycle. It will take 128 shifter cycles for a configurable unit to load 128 configuration bits with the 128 bits of data received over the vector interface. The 128 bits of configuration data are referred to as a chunk. A configurable unit can require multiple chunks of data to load all its configuration bits.
The configurable units interface with the memory through multiple memory interfaces (150,
The address generators AGs in the AGCUs can generate memory commands that are either dense or sparse. Dense requests can be used to bulk transfer contiguous off-chip memory regions, and can be used to read or write chunks of data from/to configurable units in the array of configurable units. Dense requests can be converted to multiple off-chip memory burst requests by the coalescing unit (CU) in the AGCUs. Sparse requests can enqueue a stream of addresses into the coalescing unit. The coalescing unit uses a coalescing cache to maintain metadata on issued off-chip memory requests and combines sparse addresses that belong to the same off-chip memory request to minimize the number of issued off-chip memory requests.
Configurable units in the array of configurable units include configuration data stores 420 (e.g., serial chains) to store unit files comprising a plurality of chunks (or sub-files of other sizes) of configuration data particular to the corresponding configurable units. Configurable units in the array of configurable units each include unit configuration load logic 440 connected to the configuration data store 420 via line 422, to execute a unit configuration load process. The unit configuration load process includes receiving via the bus system (e.g., the vector inputs), chunks of a unit file particular to the configurable unit, and loading the received chunks into the configuration data store 420 of the configurable unit.
The configuration data stores in configurable units in the plurality of configurable units in this example comprise serial chains of latches, where the latches store bits that control configuration of the resources in the configurable unit. A serial chain in a configuration data store can include a shift register chain for configuration data and a second shift register chain for state information and counter values connected in series.
A configurable unit can interface with the scalar, vector, and control buses using three corresponding sets of inputs and outputs (IO): scalar inputs/outputs, vector inputs/outputs, and control inputs/outputs. Scalar IOs can be used to communicate single words of data (e.g., 32 bits). Vector IOs can be used to communicate chunks of data (e.g., 128 bits), in cases such as receiving configuration data in a unit configuration load process, and transmitting and receiving data during operation after configuration across a long pipeline between multiple PCUs. Control IOs can be used to communicate control signals such as the start or end of execution of a configurable unit. Control inputs are received by control block 470, and control outputs are provided by the control block 470.
Each vector input is buffered using a vector FIFO in a vector FIFO block 460 which can include one or more vector FIFOs. Each scalar input is buffered using a scalar FIFO 450. Using input FIFOs decouples timing between data producers and consumers, and simplifies inter-configurable-unit control logic by making it robust to input delay mismatches.
Input configuration data 410 can be provided to a vector FIFO as vector inputs, and then be transferred to the configuration data store 420. Output configuration data 430 can be unloaded from the configuration data store 420 using the vector outputs.
The CGRA uses a daisy chained completion bus to indicate when a load/unload command has been completed. The master AGCU transmits the program load and unload commands to configurable units in the array of configurable units over a daisy-chained command bus. As shown in the example of
A configurable unit includes multiple reconfigurable datapaths in block 480. A datapath in a configurable unit can be organized as a multi-stage (Stage 1 . . . Stage N), reconfigurable SIMD (Single Instruction, Multiple Data) pipeline. Physical configuration of various stages and components of the SIMD is based on the configuration files loaded in the PCU, and they are reconfigurable based on the configuration files. The chunks of data pushed into the configuration serial chain in a configurable unit include configuration data for each stage of each datapath in the configurable unit. The configuration serial chain in the configuration data store 420 is connected to the multiple datapaths in block 480 via lines 421.
A Pattern Memory Unit (PMU) can contain scratchpad memory coupled with a reconfigurable scalar datapath intended for address calculation, along with the bus interfaces used in the PCU. PMUs can be used to distribute on-chip memory throughout the array of reconfigurable units. In one embodiment, address calculation within the memory in the PMUs is performed on the PMU datapath, while the core computation is performed within the PCU.
A PMU can contain scratchpad memory 530 coupled with a reconfigurable scalar data path 520 intended for address calculation (RA, WA) and control (WE, RE) of the scratchpad memory 530, along with the bus interfaces used in the PCU 400.
The bus interfaces can include scalar inputs, vector inputs, scalar outputs and vector outputs, usable to provide write data WD. The data path can be organized as a multi-stage reconfigurable pipeline, including stages of functional units FUs and associated pipeline registers PRs that register inputs and outputs of the functional units. PMUs can be used to store distributed on-chip memory throughout the array of reconfigurable units.
A scratchpad is built with multiple SRAM banks (e.g., 531, 532, 533, 534). Banking and buffering logic 535 for the SRAM banks in the scratchpad can be configured to operate in several banking modes to support various access patterns. A computation unit as described herein can include a Look-Up Table stored in the scratchpad memory 530, from a configuration file or from other sources. In a computation unit as described herein, the scalar data path 520 can translate a section of a raw input value I for addressing Look-Up Tables implementing a function f(I), into the addressing format utilized by the SRAM scratchpad memory 530, adding appropriate offsets and so on, to read the entries of the Look-Up Table stored in the scratchpad memory 530 using the sections of the input value I. Each PMU can include write address calculation logic and read address calculation logic that provide write address WA, write enable WE, read address RA and read enable RE to the banking buffering logic 535. Based on the state of the local FIFOs 511 and 512 and external control inputs, the control block 515 can be configured to trigger the write address computation, read address computation, or both, by enabling the appropriate counters 516. A programmable counter chain 516 (Control Inputs, Control Outputs) and control block 515 can trigger PMU execution.
This is one simplified example of a configuration of a configurable processor for implementing a computation unit as described herein. The configurable processor can be configured in other ways to implement a computation unit. Other types of configurable processors can implement the computation unit in other ways. Also, the computation unit can be implemented using dedicated logic in some examples, or a combination of dedicated logic and instruction-controlled processors.
At operations 608a and 608b, the compiler 122 executing on the host 120 receives the applications 604a and 604b, respectively, for compilation. Each of the applications 608a, 608b, for example, is an appropriate data flow graph of a corresponding neural network application.
At operation 610a, the compiler 122 compiles the application 604a, to generate one or more execution files 612a corresponding to the application 604a. At operation 610b, the compiler 122 compiles the application 604b, to generate one or more execution files 612b corresponding to the application 604b. The reception and compilation of the two applications can be performed by the compiler 122 in parallel, or sequentially. As discussed, although merely two applications are illustrated in
As discussed herein, certain sections of an application 604 are to be executed by the reconfigurable hardware, while other sections of the application 604 are to be executed by the general hardware. The execution files illustrated in
Referring to
The configuration files 613 implement computation graphs of the corresponding application 604 using the configurable units in the reconfigurable hardware. For example, a configuration file comprises a bit-stream representing an initial configuration, or starting state, of each of the configurable units of the reconfigurable hardware that execute the program. This bit-stream is referred to as a bit file, or herein as a configuration file.
The configurable files 613 further comprises a plurality of functions that are to be executed by the reconfigurable hardware, e.g., to execute the corresponding application 604. Examples of functions in the plurality of functions include, but are not limited to, non-linearities like Rectified Linear Units (ReLU) and its variants (e.g., leaky ReLU), convolutions, transpose convolutions, hyperbolic tangents, sigmoids, softmaxs, element-wise additions, matrix multiplications (e.g., General Matrix Multiply (GeMM)), layer normalizations (e.g., batch normalizations), loss functions like cross-entropy, and tensor shape modifiers like transpose.
For example, assume that the application 604a comprises a CNN having a forward pass and a backward pass (e.g., that are at least in part to be executed on the reconfigurable hardware), and a loss function determination path (e.g., that is to be executed on the general hardware). Then the plurality of functions included in the configurable files 613a corresponding to the application 604a would include, for example, convolutions, ReLUs and pooling functions for implementing the forward pass and the backward pass of the CNN.
The execution files 612 further include corresponding host execution files 619 (e.g., the execution files 612a include corresponding host execution files 619a, and the execution files 612b include corresponding host execution files 619b). For example, the execution files 612 of
The host execution files 619 include codes that are to be executed by the host 120 (e.g., by the processing logic 129 being executed by the host). For example, referring again to the above example where the application 604a is assumed to be a CNN having a forward pass and a backward pass (e.g., that are to be executed on the reconfigurable hardware) and a loss function determination path (e.g., that is to be executed on the general hardware), the host execution files 619a would include codes to implement the loss function determination path that is to be executed on the general hardware.
Thus, the reconfigurable hardware is to execute the configuration files 613 and the general hardware is to execute the host execution files 619. Execution of the configuration files 613 and the host execution files 619, in combination, result in execution of the corresponding application 604.
Referring to
As part of the metadata 615, the execution file 612 includes topology information 708 (see
The metadata 615 further comprises other information, e.g., as discussed in the co-pending U.S. Non-provisional patent application Ser. No. 16/922,975, filed Jul. 7, 2020, entitled, “RUNTIME VIRTUALIZATION OF RECONFIGURABLE DATA FLOW RESOURCES,” (Attorney Docket No. SBNV 1026-1), which is incorporated by reference.
Referring again to the process flow of
At operation 616, the scheduler 126 also accesses the execution files 612a, 612b. For example, the execution files 612a, 612b include corresponding priority indicators 617a, 617b, respectively. The scheduler 126 uses the priority indicators 617a, 617b and various other information (e.g., discussed herein later with respect to
At operations 620a and 620b, the runtime logic 124 loads the configuration files 612a and 612b, respectively, (or at least sections of the configuration files) and/or the data therefor (e.g., weights, coefficients, vectors, tensors (image data), audio data, natural language processing (NLP data), control data (e.g., control tokens)) on the reconfigurable hardware comprising the array of configurable units 190, based on the schedule provided by the scheduler 126. In an embodiment, the reconfigurable hardware (e.g., the array of configurable units 190) processes various functions included in the configuration files 612a, 612b, e.g., in accordance with the schedule provided by the scheduler 126. Similarly, the general hardware (e.g., the host 120 executing the processing logic 129) also executes portions of the applications 604a, 604b (e.g., the host execution files 619a and 619b) that are to be executed by the general hardware.
Assume that two applications are to be executed in a time-shared manner in the above discussed general hardware (e.g., comprising the host 120) and the reconfigurable hardware (e.g., comprising reconfigurable data processor 110). For example, assume that during a first time period, a first application is being executed in the reconfigurable hardware. Thus, during the first time period, one or more configuration files of the first application are loaded in the reconfigurable hardware, and are being executed by the reconfigurable hardware. For example, one or more configuration files of the first application are loaded and stored in one or more PMUs of the reconfigurable hardware, and at least sections of the one or more configuration files of the first application are loaded in one or more PCUs and executed by the PCUs. Now assume that the scheduler 126 wants to switch the usage of the reconfigurable hardware. That is, the scheduler 126 wants the second application to be executed by the reconfigurable hardware. To perform the switching of the reconfigurable hardware from the first application to the second application, the one or more configuration files of the first application already loaded in the PCUs/PMUs have to be invalidated, and one or more configuration files of the second application have to be loaded and executed in the reconfigurable hardware.
The transfer of the usage of the reconfigurable hardware from one application to another application is referred to as a “switching event.” In the above discussed use case example, the switching from the first application to the second application can be performed at one of a plurality of “switching points” pre-defined within the configuration files of the first application. For example, assume that the first application comprises execution of a plurality of convolution operations. Merely as an example, the switching event may not occur when the reconfigurable hardware is in the middle of a convolution operation—rather, the scheduler 126 waits until the current convolution operation is complete, and initiates the switching event at the end of the current convolution operation. Thus, switching of the usage of the reconfigurable hardware from the first application to the second application and/or switching of the usage of the reconfigurable hardware from the second application to the first application cannot occur at any arbitrary point in the configuration files of the first application. Rather, the switching of the usage of the reconfigurable hardware between the first application and the second application can occur at one of a plurality of “switching points” pre-defined within the configuration files of the first application. For example, when the compiler 122 generates the execution files from the application data flow graph of the corresponding application, the compiler 122 defines a plurality of switching points at which a switching event can possibly occur. Note that the reconfigurable hardware can initiate a switching event at a switching point, in response to receiving a switch command from the scheduler 126. In an example, a switching point can be an end of a corresponding execution fragment, as discussed herein later in further detail with respect to
As illustrated, a switching event may occur at a switching point defined between two consecutive or immediate adjacent configuration files, such as the example inter-configuration file switching point 804pq between the two consecutive or immediate adjacent configuration files 813p and 813q. Thus, an inter-configuration file switching point occurs between two configuration files that are to be executed sequentially. At this point (i.e., after execution of the configuration file 813p and prior to execution of the configuration file 813q), the scheduler 126 may schedule switching of the usage of the reconfigurable hardware from the application 801 to another application (or from another application to the application 801).
Note that there may not be an explicit switching point defined between a configuration file and a host execution file, such as between the configuration file 813q and the hose execution file 819q, as there is an implicit switching point between a configuration file and a host execution file. This is because at the end of execution of the configuration file 813q by the reconfigurable hardware, the application 801 is anyway relinquishing the usage of the reconfigurable hardware (e.g., as the host 120 will now execute the host execution file 819q), and the scheduler 126 can now schedule the reconfigurable hardware to be used by another application.
In an embodiment, one or more switching points (referred to herein as “intra-configuration file switching points”) can also exist within individual configuration files. For example,
When generating the configuration files (e.g., see
Referring to
Thus, referring to
In an embodiment, individual configuration file comprises a plurality of execution fragments. For example, an execution fragment (EF) is, in general, a portion of a program implementing a data processing operation, comprising a set of operations, such as a sequence of operations in the program that follow a similar control flow sequence. An EF may be loop bodies, or operations that are executed based on a conditional construct such as an if-else or switch statement. An EF can be hierarchical, and can fully contain other loop nests. One concrete example is an EF defined at the outermost loop level in loop-heavy programs such as deep learning training and inference workloads. Using a control flow graph representation suitable for some implementations, if the program control flow graph G is represented by a set of edges E and vertices V, each unique EF is a distinct subgraph g of G that partitions G into distinct edges e and vertices v of G. In an array of configurable units (such as the reconfigurable data processor 110), a data processing operation can be implemented by defining a plurality of execution fragments of the data processing operations. EFs may consist of a fixed or variable amount of work, as suits the program. Similarly, different EFs may contain different amounts of computation. EFs may represent parallel patterns, portions of parallel patterns or any other subset of operations in the program's control and data flow graph. EFs are discussed further in co-pending U.S. Non-provisional patent application Ser. No. 16/504,627, filed Jul. 8, 2019, entitled, “QUIESCE RECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1008-1), which is incorporated herein by reference. In an embodiment, an end of an EF is also a synchronization point, where a synchronization event occurs (e.g., data within the reconfigurable processor is synchronized with the off-chip memory). In an embodiment, the end of an EF is a switching point. Thus, switching of execution of an application in the reconfigurable hardware may occur after completion of an execution of an EF and prior to commencement of execution of an immediate subsequent EF, but may not occur while an EF is being executed.
Referring now to the leftmost section of the figure, initially, reconfigurable (labelled as “Re-config.” in the figure) hardware (referred to as “h/w” in the figure) operations 1 of App. 1 are to be executed in reconfigurable hardware, where the operations 1 of App. 1 are labelled as 902. These operations are included in one or more corresponding configuration files associated with this application.
Execution of the reconfigurable hardware operations 1 of App. 1 are to be followed by execution of general hardware operations 1 of App. 1 in the general hardware of system 100, also labelled as 904 in
This is to be followed by execution of the reconfigurable hardware operations 2 of App. 1, labelled as 906, which are included in one or more corresponding configuration files associated with this application. This is to be followed by execution of the general hardware operations 2 of App. 1, labelled as 908, which are included in one or more corresponding host execution files associated with this application. This is to be followed by execution of the reconfigurable hardware operations 3 of App. 1, labelled as 909, which are included in one or more corresponding configuration files associated with this application.
Thus, for the App. 1, operations of reconfigurable hardware and general hardware are interleaved, such as operations 1 in reconfigurable hardware, followed by operations 1 in general hardware, followed by operations 2 in reconfigurable hardware, followed by operations 2 in general hardware, followed by operations 3 in reconfigurable hardware.
Such interleaved sequence of reconfigurable and general hardware operations often occurs in neural network and machine learning applications. Merely as an example and without limiting the scope of this disclosure, when implementing a back propagation path of a CNN, input and/or weight gradients are generated, followed by updating of weight parameters. The input and/or weight gradients can be generated using the reconfigurable hardware, whereas weight update can be performed by general hardware. The process of generation of input and/or weight gradients and corresponding updating of the weights are repeated in sequence for various processing nodes of the CNN graph. Accordingly, in this example, reconfigurable hardware and general hardware are sequentially used by the application in an interleaved manner, as illustrated for App. 1 in leftmost section of
In another example, in a forward path of a CNN, tensors can be convolved with a corresponding kernel, tiled and zero-padded, and this process of convolution, tiling, and zero-padding is repeated for various sequential processing nodes of the CNN. In an example, the general hardware can perform the tiling and/or zero-padding operations, while the reconfigurable hardware can perform the convolutions. This is yet another example, where the reconfigurable hardware and the general hardware are sequentially used by the application in an interleaved manner, as illustrated for App. 1 in leftmost section of
Referring now to the rightmost section of the figure that illustrates sequence of operations for App. 2, initially, general hardware operations 1 of App. 2 are to be executed the general hardware of system 100, labelled as 910. This is to be followed by execution of reconfigurable hardware operations 1 of App. 2 in the reconfigurable hardware, labelled as 912. Note that the reconfigurable hardware operations 1 of App. 2 are broken into two sections—operations 1a and operations 1b, with a switching point between operations 1a and 1b. The switching point can be an inter-configuration file switching point, or an intra-configuration file switching point (see
Execution of the reconfigurable hardware operations 1a, 1b of App. 2 are to be followed by execution of general hardware operations 2 of App. 2 in the general hardware of system 100, also labelled as 914 in
Thus, for the App. 2, operations of general hardware and reconfigurable hardware are also interleaved, such as operations 1 in general hardware, followed by operations 1a, 1b in reconfigurable hardware, followed by operations 2 in general hardware, followed by operations 2 in reconfigurable hardware.
Such interleaved sequence of general and reconfigurable hardware operations also often occurs in neural network and machine learning applications. Merely as an example and without limiting the scope of this disclosure, when implementing a Bidirectional Encoder Representations from Transformers (BERT) model, data may be pre-processed in general hardware, followed by processing of the pre-processed data in reconfigurable hardware, followed by intermediate processing of the data in general hardware, and further processing of the data in reconfigurable hardware, e.g., similar to the sequence of operations illustrated for App. 2.
In another example, App. 2 may be a CNN, here general hardware operations 1 of 910 illustrate pre-processing of data in the general hardware. This is followed by reconfigurable hardware operations 1a, 1b of 912, where the reconfigurable hardware implements various sections of the forward path of the CNN. Subsequently, general hardware operations 2 of 914 are executed, where the general hardware calculates a loss function. Then the reconfigurable hardware operations 2 of 916 are executed, where reconfigurable hardware implements the backpropagation path of the CNN, e.g., similar to the sequence of operations illustrated for App. 2 in
It may be noted that as would be appreciated by those skilled in the art, the sequences and number of various operations illustrated in
A second column of
In the second column, some narrow boxes are dotted, and some other narrow boxes are greyed out. As illustrated in the drawing legend used in the bottom of
As also illustrated in the drawing legend used in the bottom of
Referring to the second and third columns of
After execution of the reconfigurable hardware operations 1 of App. 1, the reconfigurable hardware usage is switched and is now used to execute reconfigurable hardware operations 1a of App. 2 (e.g., 912a). Note that the reconfigurable hardware operations 1a of App. 2 are executed after completion of execution of general hardware operations 1 of App. 2 (illustrated by arrow 920). Also, while the reconfigurable hardware operations 1a of App. 2 are being executed by the reconfigurable hardware, general hardware operations 1 of App. 1 are being executed by the general hardware. Thus, there is at least in part a temporal overlap between execution of the reconfigurable hardware operations 1a of App. 2 and the general hardware operations 1 of App. 1, as illustrated.
Note that in the example use case of
Accordingly, at 940, after execution of the reconfigurable hardware operations 1a, the scheduler 126 issues a switch command, and commences a switching event. Accordingly, the reconfigurable hardware operations 1 of App. 2 is terminated, e.g., at the switching point after execution of the reconfigurable hardware operations 1a of App. 2. The reconfigurable hardware usage is switched to App. 1, and the reconfigurable hardware operations 2 of App. 1 is now being executed in the reconfigurable hardware (e.g., 906).
After the execution of the reconfigurable hardware operations 2 of App. 1, at switching event 941, the usage of the reconfigurable hardware is switched to App. 2, and the reconfigurable hardware operations 1b of App. 2 are then executed (e.g., 912b), e.g., at least in part simultaneously with execution of the general hardware operations 2 of App. 1 (e.g., 908).
This process of interleaved usage of the reconfigurable hardware and general hardware between Apps. 1 and 2 continues, until the reconfigurable and general hardware operations of both applications are fully executed. As illustrated, when reconfigurable hardware operations of App. 1 are being executed by the reconfigurable hardware, the general hardware may execute general hardware operations of App. 2. Similarly, when reconfigurable hardware operations of App. 2 are being executed by the reconfigurable hardware, the general hardware may execute general hardware operations of App. 1. In other examples, the reconfigurable hardware operations of App. 1 may overlap with the general hardware operations of App. 1 (e.g., where the reconfigurable hardware operations of App. 1 can be executed at least in part in parallel with execution of the general hardware applications of App. 1). Similarly, in some examples, the reconfigurable hardware operations of App. 2 may overlap with the general hardware operations of App. 2.
There may be some instances where the reconfigurable hardware or the general hardware is idle. For example, in
In an example, the QoS of the priority indicator 617a comprises a QoS identifier associated with the application 608a, and the QoS of the priority indicator 617b comprises a QoS identifier associated with the application 608b. Different applications can have different associated QoS. For example, assume that two Generative Pre-trained Transformer 3 (GPT-3) models are scheduled as application 608a and 608b in the system 100. Assume that a first GPT-3 model (which may be application 608a) is in a training phase, while a second GPT-3 model (which may be application 608b) is in an inference phase. In an example implementation, if inferencing from the second GPT-3 model is relatively more important than training the first GPT-3 model, then the second GPT-3 model can have a higher QoS than the first GPT-3 model. In another example implementation, if training the GPT-3 models is relatively more important than inferencing, then the first GPT-3 model can have a higher QoS than the second GPT-3 model.
In an example, the latency information of the priority indicator 617a comprises a latency sensitivity associated with the application 608a and/or a target maximum latency for the application 608a, and the latency information of the priority indicator 617b comprises similar latency information for the application 608b. For example, some applications may be able to tolerate higher latency than other applications. For example, a Natural Language Processing (NLP) machine learning model that provides real-time answers or results to users can have a relatively small latency sensitivity, such as in the range to a few milliseconds or less than a second. In another example, another NLP machine learning model that processes and categorizes user queries for future training of other models can have relatively large latency sensitivity.
In an example, the deadline information of the priority indicator 617a comprises a deadline timestamp by which the execution of the application 608a has to be completed, and the deadline information of the priority indicator 617b comprises similar information for the application 608b. For example, for a machine learning application that predicts weather information to be broadcast in a television show, there can be a deadline timestamp. For example, the weather prediction has to be completed at least 10 minutes prior to the prediction broadcast on television.
In an embodiment, the scheduler 126 takes into account the priority indicators 617a, 617b, while scheduling operations of the applications. For example, as discussed with respect to
In an embodiment, the scheduler 126 also takes into account length of various operations of the applications that are to be executed in the reconfigurable hardware and general hardware. For example, assume that a relatively large BERT model and a relatively small BERT model are App. 1 and App. 2, respectively, of
Referring to
In addition to taking into account information included in the execution files, the scheduler 126 can take into account other factors, when generating the schedule 905, such as current states 902a, 902b of applications 608a, 608b, respectively. At any point during execution, an application can be at one of a plurality of possible states, such as a ready state, a blocked state, a waiting state, etc.
A ready state of an application indicates that operations of the application are ready to be executed, and as soon as the operations are scheduled for execution, execution of the operations can commence.
A waiting state of the application indicates that the application is waiting for completion of a task, either by the application or by another application. For example, assume that the application needs data from an off-chip memory, and the application has issued a read request to the memory. Now the application is in a waiting state, waiting for the data to be fetched from the off-chip memory. A waiting state of an application indicates that if the scheduler schedules operations for the application, it may take some time for commencement of execution of the scheduled operations, e.g., depending on an amount of time it takes to resolve or exit the waiting state.
A blocked state of the application indicates that the application is currently blocked from being executed, and will commence execution once the blocking condition is resolved. For example, assume that the application needs data from a remote, cloud-based storage, and the application has issued a read request to the remote storage via a network. Now the application is blocked, as the application cannot execute until the requested data is received from the remote storage. The time taken to resolve the blocked state can be, for example, based on a speed of the network, such as the Internet. Similar to the waiting state, a blocked state of an application indicates that if the scheduler schedules operations for the application, it may take some time for the scheduled operations to execute, e.g., depending on an amount of time it takes to resolve the blocked state.
In an embodiment, the scheduler 126 takes into account various other factors, such as fairness indicator 903, workload 907 of the reconfigurable and general hardware, and/or the like. As discussed, while the execution files 612, including the priority indicators 617 are static in some examples, the current states 902, fairness indicators 903, and the workload 907 are dynamic and alter during execution of the applications.
In an example, if an application is stalled in the waiting state or the blocked state for a relatively long time period, it may not be fair for the application (e.g., the application will take relatively long time to complete). Accordingly, a fairness indicator 903 for that application will have a value that emphasizes a higher fairness factor for the application. Accordingly, once the stall condition is resolved, based on the higher fairness factor, the application may be given higher priority by the scheduler.
In an example, the scheduler 126 takes into account workload 907 of the reconfigurable and general hardware. For example, while scheduling, the scheduler 126 may aim to optimize the workload, and try to keep both the reconfigurable and general hardware engaged as much as possible. For example, referring to the switching event 940 of
Referring to
At time t2a, a switching event is initiated, and an invalidation phase 1020a commences from time t2a and lasts until time t3a. Note that the shadings of the loading and invalidation phase in
A second loading phase 1022a occurs between time t3a and time t4a, during which a configuration file 1013b for App. 2 is loaded into the PMU 1002 and the PCU 1001 from the off-chip memory 1004. The configuration file 1013b remains loaded and is executed by the PCU 1001 during a second execution phase 1024a that starts from time t4a.
Thus, in the example of
Although not illustrated in
Referring now to
At time t2b, a switching event is initiated, and an invalidation phase 1020b commences from time t2b and lasts until time t3b. During the invalidation phase 1020b, the configuration file 1013a is invalidated (e.g., deleted) in the PCU 1001 (but not from the PMU 1002).
A second loading phase 1022b occurs between time t3b and time t4b, during which the configuration file 1013b for App. 2 is loaded from the PMU 1002 to the PCU 1001. As the configuration file 1013b for App. 2 need not be loaded from the off-chip memory 1004 (and is to be loaded from the PMU to the PCU), the loading phase 1022b of
Thus, after the initial loading phase 1016b (when configuration files of both applications are loaded in the PMU 1002), each time there is a switching event between App. 1 and App. 2, the corresponding configuration file is loaded from the PMU 1002 to the PCU 1001. Because such loading avoids the off-chip memory 1004, all subsequent loading phases for the scenario discussed with respect to
Referring now to
In an example implementation, the vector FIFO block 460 (see
Furthermore, a multiplexer 1170 receives a selection signal 1174 from the scheduler 126, and provides one of the configuration files 1013a, 1013b to execution logic of the PCU 1001, depending on the application that is currently scheduled for execution by the scheduler 126. For example, during the loading phase 1016c, the multiplexer 1170 outputs the configuration file 1013 of App. 1 for processing by the PCU 1001 (e.g., as the subsequent execution phase 1018c is for App. 1).
During the first execution phase 1018c of App. 1, both the configuration files 1013a, 1013b remain loaded in the PCU 1001, and the PCU 1001 executes the output of the multiplexer 1170, e.g., executes the configuration file 1013a during the first execution phase 1018c.
At time t2c, a switching event is initiated, and an invalidation phase 1020c commences from time t2b and lasts until time t3b. During the invalidation phase 1020c, the configuration file 1013a output by the multiplexer 1170 and being executed by the PCU 1001 is invalidated (e.g., deleted), but the copy of the configuration files 1013a, 1013b are both retained in the buffer 1003a, 1003b, respectively. Because the configuration files need not be invalidated in the buffers, the invalidation phase 1020c of
A second loading phase 1022c occurs between time t3c and time t4c, during which the multiplexer 1170 outputs configuration file 1013b for App. 2 from the buffer 1013b, based on the selection signal 1174 generated by the scheduler 126. Thus, in the scenario of
Thus, each time there is a switching event between App. 1 and App. 2, the corresponding configuration file is output by the multiplexer 1170, based on the selection signal 1174 generated by the scheduler 126. Also, both the configuration files 1013a, 1013b remain loaded in the PCU 1001. Because such loading avoids the off-chip memory 1004 and the PMU 1002, all subsequent loading phases for the scenario discussed with respect to
While the present invention is disclosed by reference to the preferred embodiments and examples detailed above, it is to be understood that these examples are intended in an illustrative rather than in a limiting sense. It is contemplated that modifications and combinations will readily occur to those skilled in the art, which modifications and combinations will be within the spirit of the invention and the scope of the following claims.
