Patent Application
20210157550
The present invention relates to a technology for implementing a circuit to compute normalized elements of a batch.
The following are incorporated by reference for all purposes as if fully set forth herein:
Koeplinger et al., “Spatial: A Language And Compiler For Application Accelerators,” Proceedings Of The 39th ACM SIGPLAN Conference On Programming Language Design And Implementation (PLDI), Proceedings of the 43rd International Symposium on Computer Architecture, 2018;
Prabhakar et al., “Plasticine: A Reconfigurable Architecture for Parallel Patterns,” ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada;
U.S. Nonprovisional patent application Ser. No. 16/239,252, filed Jan. 3, 2019, entitled, “VIRTUALIZATION OF A RECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1000-1);
U.S. Nonprovisional patent application Ser. No. 16/197,826, filed Nov. 21, 2018, entitled, “CONFIGURATION LOAD OF A RECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1001-1A);
U.S. Nonprovisional patent application Ser. No. 16/198,086, filed Nov. 21, 2018, entitled, “CONFIGURATION UNLOAD OF A RECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1001-1B);
U.S. Nonprovisional patent application Ser. No. 16/260,548, filed Jan. 29, 2019, entitled, “MATRIX NORMAL/TRANSPOSE READ AND A RECONFIGURABLE DATA PROCESSOR INCLUDING SAME,” (Attorney Docket No. SBNV 1005-1);
U.S. Nonprovisional patent application Ser. No. 16/536,192, filed Aug. 8, 2019, entitled, “COMPILER FLOW LOGIC FOR RECONFIGURABLE ARCHITECTURES,” (Attorney Docket No. SBNV 1006-1);
U.S. Nonprovisional patent application Ser. No. 16/407,675, filed May 9, 2019, entitled, “CONTROL FLOW BARRIER AND RECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1007-1); and
U.S. Nonprovisional patent application Ser. No. 16/504,627, filed Jul. 8, 2019, entitled, “QUIESCE RECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1008-1).
The subject matter discussed in this section should not be assumed to be prior art merely as a result of its mention in this section. Similarly, a problem mentioned in this section or associated with the subject matter provided as background should not be assumed to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which in and of themselves can also correspond to implementations of the claimed technology.
Normalization is a process of adjusting the range of a set of elements, while preserving the relative magnitude of each element relative to other elements of the set. for example, normalization produces a distance-preserving, bijective map between elements of sets. For example, a set of numbers may be mapped to the unit interval (i.e., [0,1]), where the relative distance between any two normalized numbers is preserved. Normalization is useful in a variety of machine learning applications, such as gradient descent algorithms and multivariate optimization. Normalizing a set of numbers is a multi-step computationally-intensive process. In machine learning applications, batch normalization may be performed on matrices with large numbers of elements leading to increased computational loads, large power consumption, and extended computation times.
Current methods to improve computational throughput in batch normalization focuses on reducing the computational power expended on a per-element basis. For example, low-precision representations of matrix elements may be utilized in the batch normalization process, thereby reducing the per-element usage of memory during computation.
However, one drawback of utilizing low-precision representations of elements is that accumulated rounding errors in the multi-step normalization process may lead to large error rates. For example, large error rates may compromise the distance-preserving properties of batch normalized numbers leading to a variety of pathologies. For example, performing gradient descent on a pathological set of normalized matrix elements may lead to non-convergence, convergence to false minima, and/or the absence of true minima.
Accordingly, ways to reduce error rates in batch normalization, while preserving computational efficiency may be desirable.
In the drawings, like reference characters generally refer to like parts throughout the different views. Also, the drawings are not necessarily to scale, with an emphasis instead generally being placed upon illustrating the principles of the technology disclosed. In the following description, various implementations of the technology disclosed are described with reference to the following drawings, in which:
The following discussion is presented to enable any person skilled in the art to make and use the technology disclosed, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed implementations will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations and applications without departing from the spirit and scope of the technology disclosed. Thus, the technology disclosed is not intended to be limited to the implementations shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
In various embodiments, a composite circuit 100 may include a first circuit 110, a second circuit 120, and a third circuit 130 for producing normalized output elements for a batch of input elements. In operation, a first circuit 110 may receive one or more input elements in a first format. The first format may correspond to a reduced-memory and/or low-precision representation of a batch of input elements. The first circuit 110 may process the input elements to produce a batch mean, μ, in a second format. For example, the first circuit 110 may compute a product of each batch element, xi, and the inverse of the number of elements in the batch, m, the product being xi×(1/m). In various embodiments, the first circuit 110 may represent the product as a second-format number. The first circuit 110 may further accumulate the second-format products to obtain μ=Σi=1mxi×(1/m). Because the second format mean is computed by using second-format products, accumulated rounding errors may be reduced, thereby preserving accuracy. In further embodiments, the first circuit 110 may modify the second-format mean to produce a first-format representation of the mean. For example, the first circuit 110 may truncate a second-format batch mean to generate the first-format batch mean.
Furthermore, a second circuit 120 may utilize the first-format batch mean to compute a variance of the batch in the first format. For example, the second circuit 120 may compute the difference between each element xi and the first format batch mean, μ1, the difference being xi−μ1. The second circuit 120 may further compute the square of the difference (xi−μ1)2 and add a constant term, ϵ, to generate (xi−μ1)2+ϵ. The second circuit 120 may further multiply (xi−μ1)2+ϵ by (1/m) and accumulate the sum over all xi, the result being the variance: σ2+ϵΣi=1m[(xi−μ1)2+ϵ]×(1/m). The second circuit 120 may further compute the inverse square root of the variance to obtain the inverse standard deviation: 1/√(σ2+ϵ), where ϵ is a constant term. Because a variance is an average across all xi in the batch, accumulated rounded errors are, on average, canceled out in the variance computation. Accordingly, utilizing the first-format batch mean to compute the variance preserves computational throughput during the computationally-intensive multiplication steps, without losing computational accuracy.
Furthermore, the third circuit 130 may compute normalized values for each element of the input batch. For example, the third circuit 130 may compute the product of a second-format scale factor, γ, and the inverse standard deviation, the product being: γ×1/√(σ2+ϵ). The third circuit 130 may further convert a first-format element, xi, into two second-format elements, x0 and x1. For example, the third circuit 130 may multiply each first-format element with a first-format representation of the identity (i.e., 1) to represent the product of the two first-format as a second-format number. The third circuit 130 may further compute the difference between the second-format elements and the second-format mean, μ2, the differences being: x0−μ2 and x1−μ2. Furthermore, the third circuit 130 may compute the product of the differences and γ×1/√(σ2+ϵ), while also summing by a second-format shift factor, β, the results being: y0=[(x0−μ2)×γ×1/√(σ2+ϵ)]+β, and y1=[(x1−μ2)×γ×1/√(σ2+ϵ)]+β, where y0 and y1 are the scaled and shifted batch normalized values. The third circuit 130 may further perform stochastic rounding of the normalized elements, y0 and y1, to produce first-format representations of the normalized batch elements. The third circuit 130 may further package the normalized elements in a data structure (e.g., as a data structure Y (y0, y1)). These first format normalized elements may be used in subsequent computations by various applications (e.g., gradient descent algorithms). The third circuit 130 may further output the data structure for use by various systems in subsequent calculations.
Accordingly, the composition of a first circuit 110 for computing the batch mean, the second circuit 120 for computing the batch variance, and the third circuit 130 for computing normalized values for batch elements may be implemented in normalizing batch elements. For example, the first circuit 110 generates a batch mean in a first format and a second format. The second circuit 120 may compute the batch variance using the batch mean in the first format. Accuracy may be preserved because accumulation errors in computing the variance and standard deviation cancel each other out. Additionally, the third circuit 130 maintains computational accuracy by using the second-format mean to compute the differences in the normalization step. These computed differences are not averaged, thus, using second-format elements provides improvements to computational accuracy relative to using first format numbers. Similarly, a second-format scale factor and a second-format shift factor are used to improve computational accuracy. Finally, stochastic rounding is performed on the normalized output to generate a first-format data format based on the normalized second-format number for use in further computations by various other applications.
In various embodiments, the first format may be a low-precision and/or reduced-memory representation of an input element. For example, the first format may be a 16-bit Bfloat number. In addition, the second format may be a high-precision and/or full-memory representation of an input element. For example, the second format may be a 32-bit floating-point number. In various embodiments, a batch of input values may enter the composite circuit 100 in the first format. However, as the composite circuit 100 computes the normalized elements in mixed-format region 140, the composite circuit 100 may utilize one or more of the first format and the second format.
At step 220, a second circuit 120 may traverse the batch of input elements xi to produce a standard deviation σ for the batch using the mean μ1 in the first format. For example, the second circuit 120 may traverse the batch of input elements to compute [(xi−μ1)2+ϵ]×(1/m). The second circuit 120 may further sum over the computed values to obtain the variance: σ2+ϵ=Σi=1m[(xi−μ1)2+ϵ]×(1/m). The second circuit 120 may compute the inverse square root of the variance to obtain the inverse standard deviation: 1/√(σ2+ϵ), where E is a constant term.
At Step 230, the third circuit 130 may traverse the batch of input elements xi to produce a normalized set of values yi using the mean μ2 in the second format and the standard deviation σ. For example, the third circuit 130 may convert a first-format element, xi, into two second-format elements, x0 and x1. The third circuit 130 may further compute the difference between the second-format elements and the second-format mean, μ2, the differences being: x0−μ2 and x2−μ2. Furthermore, the third circuit 130 may compute the product of the differences and γ×1/√(σ2+ϵ), while also summing by a second-format shift factor, β, the results being: y0=[(x0−μ2)×γ×1/√(σ2+ϵ)]+β, and y1=[(x1−μ2)×γ×1/√(σ2+ϵ)]+β, where y0 and y1 are the scaled and shifted batch normalized values.
Configurable units in an array 290 of configurable units are further described in reference to
The processor 210 includes an external I/O interface 230 connected to the host 220 by line 225, and an external I/O interface 250 connected to the memory 240 by line 245. The I/O interfaces 230, 250 connect via a bus system 215 to the array 290 of configurable units and to the configuration load/unload controller 295. The bus system 215 may have a bus width of carrying one chunk of data, which can be for this example 128 bits (references to 128 bits throughout can be considered as an example chunk size more generally).
To configure configurable units in the array 290 of configurable units with a configuration file, the host 220 can send the configuration file to the memory 240 via the interface 230, the bus system 215, and the interface 250 in the reconfigurable data processor 210. The configuration file can be loaded in many ways, as suits a particular architecture, including in data paths outside the configurable processor 210. The configuration file can be retrieved from the memory 240 via the memory interface 250. Chunks of the configuration file can then be sent in a distribution sequence to configurable units in the array 290 of configurable units in the reconfigurable data processor 210.
An external clock generator 270 or other clock line sources can provide a clock line 275 or clock lines to elements in the reconfigurable data processor 210, including the array 290 of configurable units, and the bus system 215, and the external data I/O interfaces. The bus system 215 can communicate data at a processor clock rate via a clock line 275 or clock lines.
Each of the tiles has 4 AGCUs (Address Generation and Coalescing Units) (e.g. MAGCU1, AGCU12, AGCU13, AGCU14). The AGCUs are nodes on the top-level network and nodes on the array level networks, and include resources for routing data among nodes on the top-level network and nodes on the array level network in each tile.
Nodes on the top-level network in this example include one or more external I/Os, including interface 305. The interfaces to external devices include resources for routing data among nodes on the top-level network and external devices, such as high-capacity memory, host processors, other CGRA processors, FPGA devices and so on, that are connected to the interfaces.
One of the AGCUs in a tile is configured in this example to be a master AGCU, which includes an array configuration load/unload controller for the tile. In other embodiments, more than one array configuration load/unload controller can be implemented and one array configuration load/unload controller may be implemented by logic distributed among more than one AGCU.
The MAGCU1 includes a configuration load/unload controller for Tile1, and MAGCU2 includes a configuration load/unload controller for Tile2. In other embodiments, a configuration load/unload controller can be designed for loading and unloading configuration of more than one tile. In other embodiments, more than one configuration controller can be designed for configuration of a single tile. Also, the configuration load/unload controller can be implemented in other portions of the system, including as a stand-alone node on the top-level network and the array level network or networks.
The top-level network is constructed using top-level switches (311-316) connecting to each other as well as to other nodes on the top-level network, including the AGCUs, and I/O interface 305. The top-level network includes links (e.g. L11, L12, L21, L22) connecting the top-level switches. Data travel in packets between the top-level switches on the links, and from the switches to the nodes on the network connected to the switches. For example, top-level switches 311 and 312 are connected by a link L11, top-level switches 314 and 315 are connected by a link L12, top-level switches 311 and 314 are connected by a link L13, and top-level switches 312 and 313 are connected by a link L21. The links can include one or more buses and supporting control lines, including for example a chunk-wide bus (vector bus). For example, the top-level network can include data, request and response channels operable in coordination for transfer of data in a manner analogous to an AXI compatible protocol. See, AMBA® AXI and ACE Protocol Specification, ARM, 2017.
Top-level switches can be connected to AGCUs. For example, top-level switches 311, 312, 314 and 315 are connected to MAGCU1, AGCU12, AGCU13 and AGCU14 in the tile Tile1, respectively. Top-level switches 312, 313, 315 and 316 are connected to MAGCU2, AGCU22, AGCU23 and AGCU24 in the tile Tile2, respectively.
Top-level switches can be connected to one or more external I/O interfaces (e.g. interface 305).
In this example, the array of configurable units 500 includes a plurality of types of configurable units. The types of configurable units in this example, include Pattern Compute Units (PCU), Pattern Memory Units (PMU), switch units (S), and Address Generation and Coalescing Units (each including two address generators AG and a shared CU). For an example of the functions of these types of configurable units, see, Prabhakar et al., “Plasticine: A Reconfigurable Architecture For Parallel Patterns”, ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada, which is incorporated by reference as if fully set forth herein. In this example, the PCUs (e.g. 442) and PMUs (e.g. 443) in the array of configurable units 500 can include resources configurable for implementation of a computation unit, an example configuration of which is described herein (e.g.
Additionally, each of these configurable units contains a configuration store comprising a set of registers or flip-flops that store status usable to track progress in nested loops or otherwise. A configuration file in the configuration store contains a bit-stream representing the initial configuration, or starting state, of each of the components that execute the program. This bit-stream is referred to as a bit file. Program load is the process of setting up the configuration stores in the array of configurable units based on the contents of the bit file to allow the components to execute a program (i.e., a machine), including programs that utilize the lookup table with a sub-table. Program Load may also require the load of all PMU memories.
The array level network includes links interconnecting configurable units in the array. The links in the array level network include one or more and, in this case, three kinds of physical buses: a chunk-level vector bus (e.g. 128 bits of data), a word-level scalar bus (e.g. 32 bits of data), and a multiple bit-level control bus. For instance, interconnect 421 between switch units 411 and 412 includes a vector bus interconnect with a vector bus width of 128 bits, a scalar bus interconnect with a scalar bus width of 32 bits, and a control bus interconnect.
The three kinds of physical buses differ in the granularity of data being transferred. In one embodiment, the vector bus can carry a chunk that includes 16-Bytes (=128 bits) of data as its payload. The scalar bus can have a 32-bit payload, and carry scalar operands or control information. In some machines implemented using this system, data can be represented using floating point data formats, including standard or non-standard formats. Example formats include FP32 and BF16, among others. It can be understood that the number of data values carried on the scalar and vector buses is a function of the encoding format of the data values, with FP32 utilizing 32 bits per value and BF16 using 16 bits per value.
The control bus can carry control handshakes such as tokens and other lines. The vector and scalar buses can be packet switched, including headers that indicate a destination of each packet and other information such as sequence numbers that can be used to reassemble a file when the packets are received out of order. Each packet header can contain a destination identifier that identifies the geographical coordinates of the destination switch unit (e.g. the row and column in the array), and an interface identifier that identifies the interface on the destination switch (e.g. North, South, East, West, etc.) used to reach the destination unit. The control network can be circuit switched based on timing circuits in the device, for example. The configuration load/unload controller can generate a header for each chunk of configuration data of 128 bits. The header is transmitted on a header bus to each configurable unit in the array of configurable unit.
In one example, a chunk of data of 128 bits is transmitted on the vector bus that provides the chunk as vector inputs to a configurable unit. The vector bus can include 128 payload lines, and a set of header lines. The header can include a sequence ID for each chunk, which can include:
For a load operation, the configuration load controller can send the number N of chunks to a configurable unit in order from N−1 to 0. For this example, the 6 chunks are sent out in most-significant-bit-first order of Chunk 5->Chunk 4->Chunk 3->Chunk 2->Chunk 1->Chunk 0. (Note that this most-significant-bit-first order results in Chunk 5 being distributed in round 0 of the distribution sequence from the array configuration load controller.) For an unload operation, the configuration unload controller can write out the unload data of order to the memory. For both load and unload operations, the shifting in the configuration serial chains in a configuration data store in a configurable unit is from LSB (least-significant-bit) to MSB (most-significant-bit), or MSB out first.
During execution of a machine after configuration, data can be sent via one or more unit switches and one or more links between the unit switches to the configurable units using the vector bus and vector interface(s) of the one or more switch units on the array level network.
In embodiments described herein, a configuration file or bit file, before configuration of the tile, can be sent from the configuration load controller using the same vector bus, via one or more unit switches and one or more links between the unit switches to the configurable unit using the vector bus and vector interface(s) of the one or more switch units on the array level network. For instance, a chunk of configuration data in a unit file particular to a configurable unit PMU 441 can be sent from the configuration load/unload controller 401 to the PMU 441, via a link 420 between the configuration load/unload controller 401 and the West (W) vector interface of the switch unit 411, the switch unit 411, and a link 431 between the Southeast (SE) vector interface of the switch unit 411 and the PMU 441.
In this example, one of the AGCUs is configured to be a master AGCU, which includes a configuration load/unload controller (e.g. 401). The master AGCU implements a register through which the host (220,
The configuration load controller in the master AGCU is responsible for reading the configuration file from the memory and sending the configuration data to every configurable unit of the tile. The master AGCU can read the configuration file from the memory at preferably the maximum throughput of the top-level network. The data read from memory are transmitted by the master AGCU over the vector interface on the array level network to the corresponding configurable unit according to a distribution sequence described herein.
In one embodiment, in a way that can reduce the wiring requirements within a configurable unit, configuration and status registers holding unit files to be loaded in a configuration load process, or unloaded in a configuration unload process, in a component are connected in a serial chain and can be loaded through a process of shifting bits through the serial chain. In some embodiments, there may be more than one serial chain arranged in parallel or in series. When a configurable unit receives the for example 128 bits of configuration data from the master AGCU in one bus cycle, the configurable unit shifts this data through its serial chain at the rate of 1 bit per cycle, where shifter cycles can run at the same rate as the bus cycle. It will take 128 shifter cycles for a configurable unit to load 128 configuration bits with the 128 bits of data received over the vector interface. The 128 bits of configuration data are referred to as a chunk. A configurable unit can require multiple chunks of data to load all its configuration bits.
The configurable units interface with the memory through multiple memory interfaces (250,
Each vector input is buffered in this example using a vector FIFO in a vector FIFO block 560 which can include one or more vector FIFOs. Likewise, in this example, each scalar input is buffered using a scalar FIFO 550. Using input FIFOs decouples timing between data producers and consumers, and simplifies inter-configurable-unit control logic by making it robust to input delay mismatches.
A configurable unit includes multiple reconfigurable data paths in block 580. A data path in a configurable unit can be organized as a multi-stage (Stage 1 . . . Stage N), reconfigurable SIMD (Single Instruction, Multiple Data) pipeline. The chunks of data pushed into the configuration serial chain in a configurable unit include configuration data for each stage of each data path in the configurable unit. The configuration serial chain in the configuration data store 520 is connected to the multiple data paths in block 580 via lines 521.
A configurable data path organized as a multi-stage pipeline can include multiple functional units (e.g. 581, 582, 583; 584, 585, 586) at respective stages. A computation unit or parts of a computation unit can be implemented in multiple functional units at respective stages in a multi-stage pipeline or in multiple multi-stage pipelines, as described in
Configurable units in the array of configurable units include configuration data stores 520 (e.g. serial chains) to store unit files comprising a plurality of chunks (or sub-files of other sizes) of configuration data particular to the corresponding configurable units. Configurable units in the array of configurable units each include unit configuration load logic 540 connected to the configuration data store 520 via line 522, to execute a unit configuration load process. The unit configuration load process includes receiving, via the bus system (e.g. the vector inputs), chunks of a unit file particular to the configurable unit, and loading the received chunks into the configuration data store 520 of the configurable unit. The unit file loaded into the configuration data store 520 can include configuration data, including opcodes and routing configuration, for circuits (e.g. module) implementing a lookup table with a sub-table as described herein.
The configuration data stores in configurable units in the plurality of configurable units in this example comprise serial chains of latches, where the latches store bits that control configuration of the resources in the configurable unit. A serial chain in a configuration data store can include a shift register chain for configuration data and a second shift register chain for state information and counter values connected in series.
Input configuration data 510 can be provided to a vector FIFO as vector inputs, and then be transferred to the configuration data store 520. Output configuration data 530 can be unloaded from the configuration data store 520 using the vector outputs.
The CGRA uses a daisy-chained completion bus to indicate when a load/unload command has been completed. The master AGCU transmits the program load and unload commands to configurable units in the array of configurable units over a daisy-chained command bus. As shown in the example of
The bus interfaces can include scalar inputs, vector inputs, scalar outputs and vector outputs, usable to provide write data WD. The data path can be organized as a multi-stage reconfigurable pipeline, including stages of functional units FUs and associated pipeline registers PRs that register inputs and outputs of the functional units. PMUs can be used to store distributed on-chip memory throughout the array of reconfigurable units.
A scratchpad is built with multiple SRAM banks (e.g. 631-634). Banking and buffering logic 635 for the SRAM banks in the scratchpad can be configured to operate in several banking modes to support various access patterns. A computation unit as described herein can include PMUs to store intermediary values written by functional units of the PCUs, the intermediary values stored in the scratchpad memory 630. In a computation unit as described herein, the scalar data path 620 can be used for reading data from PMUs and/or writing data to PMUs utilizing the addressing format of the SRAM scratchpad memory 630, adding appropriate offsets and so on, to read the data entries stored in the scratchpad memory 630. Additionally, or alternatively, the vector data path may be utilized for storing batch elements for input to a PCU. Each PMU can include write address calculation logic and read address calculation logic that provide write address WA, write enable WE, read address RA and read enable RE to the banking buffering logic 635. Based on the state of the local FIFOs 611 and 612 and external control inputs, the control block 615 can be configured to trigger the write address computation, read address computation, or both, by enabling the appropriate counters 616.
A programmable counter chain (Control Inputs, 616, Control Outputs) and control block 615 can trigger PMU execution.
In various embodiments, a computation unit may include an array of configurable units in a reconfigurable data processor, like that represented by
In the configuration, a floating-point input I may be supplied to a circuit. The input I has a sign bit, an E bit exponent and an M bit mantissa. For instance, the floating point input I can be in the FP32 floating point format, which includes a sign bit, an 8-bit exponent (E=8) and a 23 bit mantissa (M=23), for a total of 32 bits.
In addition, the computational unit may convert data from a first data format to a second data format. For example, and not by way of limitation, the computational unit 100 may convert an element from the FP32 data format to the BF16 data format by truncating bits [0:15] from the FP32 number. Additionally, or alternatively, the computational unit may convert an element from the FP32 data format to the BF16 data format by performing stochastic rounding based on properties of the batch of elements.
Similarly, the computational unit 100 may convert an element from the BF16 data format to the FP32 data format by performing a multiplication operation between the BF16 number and a representation of the identity operator (i.e., 1) in the BF16 data format. The computational unit may further store the product of the multiplication operation in the FP32 data format by storing 32-bits of significant digits of the product.
In various embodiments, each stage may operate in series. For example, an input Xi may receive at state S0 by functional unit FU0. Functional unit FU0 may perform one or more floating-point operations to generate a result. The result of FU0 may be an input to functional units in any or all of stages S1, S2, S3, S4, and S5. For example, and not by way of limitation, FU0 may write the result to a pipeline register PR0. HA in stage S1 may perform a read operation on PR0 to obtain the FU0 result. Functional unit HA may utilize the FU0 result as an input for performing floating-point operations. In further embodiments, multiple pipeline registers may be utilized. For example, a PCU 900 may be connected via a bus 160 to a pipeline register 1 (PR1), a pipeline register 2 (PR2), a pipeline register 3 (PR3), etc. Functional units in each stage may perform read operations from one or more of the pipeline registers. Similarly, functional units may perform write operations to one or more of the functional units.
In further embodiments, the PCU 900 may enable computational processes to operate in parallel. For example, each stage may include multiple functional units. Each functional unit within a stage may be assigned to a data lane (e.g., lane 0, lane 1, lane 2, . . . , lane 14, lane 15, etc.). Each functional unit may read data in parallel from one or more pipeline registers. Similarly, each functional unit may write data in parallel to one or more functional units.
(1/m)Σi=1mxi, (1)
where xi is the ith element of the mini-batch and m is the total number of elements in the mini-batch. In various embodiments, the mini-batch mean may be computed with a six-stage PCU 1000: S0, S1, S2, S3, S4, and S5. Each stage may support 16 data lanes: Lane 0 through Lane 15. A pipeline register (e.g., PR0) may store data between stages of the PCU 1000.
In various embodiments, stage S0 may receive input elements xi. Input elements xi may include numbers in a low-precision representation (e.g., BF16 numbers). In some embodiments, stage S0 may include 16 multiplication-accumulation (MACC) units that process input data in parallel. Each MACC unit may correspond to a lane of the PCU 800. In operation, Each MACC unit may access the pipeline register PR0 to read a portion of the input elements. For example, and not by way of limitation, input elements may be stored in pipeline register 0 in sixteen portions. Each MACC unit may perform a read operation on a different portion of the input elements.
The MACC units may further perform one or more multiplication steps and one or more accumulation steps. For example, and not by way of limitation, each MACC unit may input an element xi as the multiplicand. Each MACC unit may be configured to set the multiplier to be (1/m), where m is the total number of elements in the mini-batch. For each xi, a MACC unit may compute the product of xi and 1/m. The accumulator operator in the MACC unit may further sum the product xi×(1/m) for each input xi to generate a result. Each MACC unit may further write the result to the pipeline register 0. Although the above description is written with the example of 16 lanes and 16 MACC units, it is to be understood that any combination of lanes and MACC units are within the scope of the present disclosure.
In various embodiments, stages S1, S2, S3, S4, and S5 may provide a reduction tree. For example, stage S1 may include one or more functional units that perform read operations on data stored in the pipeline register PR0. Each of the functional units may read a portion of the results written by the MACC units to the PR0 in stage S0. Each functional of the units may further perform accumulation operations by summing the obtained results to generate an accumulated result. The functional unit may further write the accumulated result to the pipeline register PR0. Accordingly, the reduction tree compresses stage S0 resultants from two or more lanes to generate a single accumulated result.
In subsequent stages (e.g., S2, S3, S4, and S5), the functional units may continue to perform read operations from the pipeline register PR0, accumulation operations on the read data, and write operations to write the accumulated result to the pipeline register PR0. At the final stage (e.g., stage 5), a final accumulated result is written to the pipeline register PR0. The final accumulated result is the mini-batch mean μ.
In various embodiments, the mini-batch mean μ may be stored as a high-precision number, e.g., in the FP32 data format. In further embodiments, the computational unit 100 may generate a low-precision mini-batch mean, μL, based on the high-precision mini-batch mean μ. For example, and not by way of limitation, a PMU may read the high-precision mini-batch mean from a pipeline register (e.g., PR0). The PMU may further generate a low-precision mini-batch mean by truncating the high-precision mini-batch mean. For example, and not by way of limitation, the PMU may read a high-precision mini-batch mean in a FP32 data format. The PMU may write the first 16 bits of the high-precision mini-batch mean to a pipeline register, thereby generating a low-precision mini-batch mean in the BF16 data format. Thus, the low-precision mini-batch mean utilizes less memory relative to the high-precision mini-batch mean. In addition, generating the low-precision mini-batch mean from the high-precision mini-batch mean, instead of by direct calculation, reduces the effect of accumulated rounding errors.
(1/m)/Σi=1m(xi−μL)2, (2)
where xi is an ith element of the mini-batch, m is a total number of elements in the mini-batch, and μL is a low-precision mini-batch mean.
In various embodiments, a mini-batch variance may be computed with two six-stage computational units (e.g., PCUs). Each computation unit may include stages: S0, S1, S2, S3, S4, and S5. Each stage of the computational unit may access one or more pipeline registers, e.g., PR0 and PR1. For example, one or more pipeline registers (e.g., PR0 and PR1) may store data between stages of the computation unit.
In some embodiments, the subtraction unit may input the xi as a first input and the mini-batch mean, μ, as the second input to a subtraction operator. Accordingly, the subtraction operator may compute a difference xi−μ and write the difference to a pipeline register (e.g., PR0). Although one subtraction unit is illustrated, it is to be understood that any number of subtraction units and any number of pipeline registers may be used in computing differences between mini-batch elements and mini-batch means.
In further embodiments, stage S1 may include a multiplication-addition (MADD) unit. The MADD unit may perform a read operation on a pipeline register (e.g., PR0) to obtain a computed difference, xi−μ. In various embodiments, the MADD unit may input the difference as both the multiplicand input and the multiplier input to a multiplication operator. Accordingly, the MADD unit may compute the square of xi−μ. In some embodiments, the MADD unit may further include an error correction during the addition operation step. Specifically, the MADD unit may perform an addition operation by inputting the product (xi−μ)2 and a constant term ϵ into an addition operator. The MADD unit may perform an addition operation using the inputted data and write the sum of the addition operation to a pipeline register (e.g., PR0). The sum will herein be denoted as Tmp. In various embodiments, the constant term may be pre-set in the MADD unit. Additionally, or alternatively, the constant term may be determined based on a stochastic probability measurement of accumulation errors in low-precision floating-point numbers (e.g., error bounds on BF16 numbers).
In additional embodiments, stage S2 may include a multiplication-accumulation (MACC) unit. In various embodiments, the MACC unit may perform a read operation on a pipeline register (PR0) to obtain the Tmp. The MACC unit may input the Tmp as the multiplicand to a multiplication operator. The MACC unit may further utilize a stored value of the inverse number of mini-batch elements (i.e., 1/m) as the multiplier to the multiplication operator. The multiplication operator may generate a product and the MACC unit may pass the product as an input to an accumulate operator. In various embodiments, each MACC unit may read a portion of the Tmp values stored in pipeline register PR0. Accordingly, each MACC unit may perform a multiplication operation and an accumulate operation for each Tmp value read by the MACC unit. The MACC unit may further output the resultant to a pipeline register (e.g., PR0).
In various embodiments, functional units in stages S3, S4, and S5 may be bypassed. For example, and not by way of limitation, these functional units may be configured to remain in an off state. Accordingly, these functional units may not perform read operations from pipeline registers, computational operations, or write operations to pipeline registers.
In various embodiments, stage S0 may be bypassed. Accordingly, functional units in stages S0 may be configured to remain in an off state. Thus, these functional units may not perform read operations from pipeline registers, computational operations, or write operations to pipeline registers.
In various embodiments, stages S1, S2, S3, S4, and S5 may provide a reduction tree. For example, in stage S1, one or more functional units may perform read operations on data stored in the pipeline register PR0. Each functional unit may read a portion of the accumulated results in pipeline register PR0. Each functional unit may further perform accumulation operations by adding two or more inputted resultants to produce an accumulated resultant. The functional unit may further write the accumulated resultant to the pipeline register (e.g., PR0). In each stage of the reduction tree, one or more functional units accumulates two or more resultants stored in a pipeline register (e.g., PR0) to form a single accumulated result. The functional unit may further perform a write operation to write the single accumulated result to a pipeline register (e.g., PR0). Accordingly, the reduction tree compresses two or more obtained resultants into a single accumulated resultant.
In subsequent stages (e.g., S2, S3, S4, and S5), the functional units may continue to perform read operations from the pipeline register PR0, accumulation operations on the read data, and write operations to write the accumulated resultant to the pipeline register (e.g., PR0). At the final stage (e.g., stage 5), a final accumulated resultant is written to the pipeline register (e.g., PR0). The final accumulated resultant is denoted as the mini-batch variance, σ2.
In various embodiments, a further stage of the computational unit 1110 may include a Tail stage. The Tail stage may include one or more special functional units (e.g., an inverse square root unit). The special functional unit may perform a read operation on data stored in the pipeline register (e.g., a the mini-batch variance stored in pipeline register PR0). The special functional unit may additionally perform one or more computational operations on the inputted data to generate an operational result. In addition to performing one or more operations, the special functional unit may further perform a write operation to write the operational result to a pipeline register. For example, an inverse square root unit may perform a read operation to obtain the mini-batch variance from a pipeline register (e.g., PR0). The inverse square root unit may further perform an inverse square root operation to compute the mini-batch inverse standard deviation 1/√(σ2+ϵ). For example, and not by way of limitation, the inverse square root unit 600 may include an inverse square root operator that utilizes an iterative root-finding algorithm to compute the inverse square root of the mini-batch variance (e.g., by utilizing Newton's method). For example, and not by way of limitation, the inverse square root unit may perform a write operation to write the mini-batch inverse standard deviation to the pipeline register PR0.
x
i{circumflex over ( )}=(xi−μH)√(σ2+ϵ), (3)
where xi is an element of the mini-batch, xi{circumflex over ( )} is a normalized element of the mini-batch, μ is a mini-batch average, σ2 is the mini-batch variance, and ϵ is a constant term.
y
i
=γ×x
i{circumflex over ( )}+β, (4)
where xi{circumflex over ( )} is a normalized element of the mini-batch, yi is a scaled and shifted normalized element, γ is a scale factor, and β is a shifting factor.
In various embodiments, the normalized mini-batch element, xi{circumflex over ( )}, and the scaled and shifted mini-batch element, yi, may be computed by a seven-stage computational unit 1000 (e.g., a PCU). The computation unit 1200 may include stages: S0, S1, S2, S3, S4, S5 and a Tail stage. Each stage of the computational unit may access, via a bus, one or more pipeline registers, e.g., PR0, PR1, PR2, PR3, and PR4. In various embodiments, pipeline registers (e.g., PR0 through PR4) may store data between stages of the computation unit 1200.
In some embodiments, a pipeline register PR0 may store an ith mini-batch element xi. The pipeline register PR1 may store an inverse standard deviation, 1/√(σ2+ϵ). The pipeline register PR2 may store the scale factor, γ. The pipeline register PR3 may store the high-precision mini-batch mean, μ. Further, the pipeline register PR4 may store the shift factor β. In some embodiments, the mini-batch element xi may be a low-precision number (e.g., BF16). The inverse standard deviation may be a low-precision number (e.g., BF16). The scale factor, γ, may be high-precision number (e.g., FP32). The mini-batch mean, μ, may be high-precision number (e.g., FP32). Further, the shift factor, β, may be high-precision number (e.g., FP32).
In some embodiments, stage S0 may include one or more multiplication units. The multiplication unit may perform a read operation on pipeline register PR1 to obtain the inverse standard deviation, 1/√(σ2+ϵ). The multiplication unit may further read the pipeline register PR2 to obtain the scale factor, γ. The multiplication units may input the inverse standard deviation as the multiplicand and the scale factor as the multiplier to a multiplication operator. The multiplication operator may compute the product, γ×1/√(σ2+ϵ). The multiplication unit may further write the product to the pipeline register PR2.
In additional embodiments, stage S1 may also include one or more multiplication units. A multiplication unit may perform a read operation on pipeline register PR0 to obtain the ith mini-batch element xi. The multiplication unit may further input the mini-batch element xi as the multiplicand to a multiplication operator. In addition, the multiplication unit may be configured to set the multiplier to be of unit value (e.g., 1). In various embodiments, value 1 may be a low-precision number (e.g., a BF16 unit value). The multiplication operator may compute a product, xi×1. As is well-known in the art, the product of two low-precision numbers may be written as a high-precision number. In the instance case, the product of a BF16 number (i.e., xi) and a second BF16 number (i.e., 1) may be stored as an FP32 number (i.e., xi×1). In addition, the multiplication operator may output two products, herein designated as x0 and x1. The multiplication unit may perform a write operation to write each product to a pipeline register. For example, the multiplication unit may write the x0 result in pipeline register PR0 and further write the xi result in pipeline register PR1.
In further embodiments, stage S2 may include one or more subtraction units. The subtraction unit may perform a read operation on one or more pipeline registers to obtain two inputs. For example, and not by way of limitation, the subtraction unit may perform a read operation on pipeline register PR0 to obtain the mini-batch element x0. Additionally, the subtraction unit may perform a read operation on a pipeline register PR3 to obtain the high-precision mini-batch mean, μH. The subtraction unit may configure a subtraction operator to perform a subtraction operation to compute the difference between the mini-batch element and the high-precision mean (i.e., x0−μH). The subtraction unit may store the difference, x0−μH, in a pipeline register (e.g., pipeline register PR0).
In addition, the stage S3 may also include one or more subtraction units. A subtraction unit may perform a read operation on one or more pipeline registers to receive two inputs. For example, and not by way of limitation, the subtraction unit may perform a read operation on pipeline register PR1 to obtain the mini-batch element x1. Additionally, the subtraction unit may perform a read operation on a pipeline register PR3 to obtain the high-precision mini-batch mean, μH. The subtraction unit may configure the subtraction operator to compute the difference between the mini-batch element and the high-precision mean (i.e., x1−μH). The subtraction unit may store the the difference, x1−μH, in a pipeline register (e.g., pipeline register PR1). Although one subtraction unit is illustrated, it is to be understood that any number of subtraction units and any number of pipeline registers computing differences between mini-batch elements and mini-batch means is envisioned by the present disclosure.
Furthermore, stage S4 may include on or more multiplication-addition (MADD) units. A MADD unit may perform a read operation on a pipeline register (e.g., PR0) to obtain an input multiplicand (i.e., x0−μH). In addition, the MADD unit may perform a read operation on a second pipeline register (e.g., PR2) to obtain an input multiplier (i.e., γ×1/√(σ2+ϵ)). The MADD unit may further perform configure a multiplication operator to compute a product (x0−μH)×(γ×1/√(σ2+ϵ)).
The MADD unit may further input the product to an addition operator. Also, the MADD unit may perform a read operation on a pipeline register (e.g., PR4) to obtain a second additive element (i.e., the shift factor, β). The MADD unit may further compute the result [(x0−μH)×(γ×1/√(σ2+ϵ))]+β, which is herein referred to as y0, the scaled and shifted value of the normalized mini-batch element x0. The MADD unit may write the resultant, y0, to a pipeline register (e.g., pipeline register PR0).
Similarly, stage S5 may include one or more multiplication-addition (MADD) units. A MADD unit may perform a read operation on a pipeline register (e.g., PR1) to obtain a input multiplicand (i.e., xi−μH). In addition, the MADD unit may perform a read operation on a second pipeline register (e.g., PR2) to obtain an input multiplier (i.e., γ×1/√(σ2+ϵ)). The MADD unit may further configure a multiplication operator to compute a product (x1−μH)×(γ×1/√(σ2+ϵ)).
The MADD unit may utilize the computed product as an input an addition operator. Also, the MADD unit may perform a read operation on a pipeline register (e.g., PR4) to obtain a second additive element (i.e., the shift factor, β). The MADD unit may further compute the result (x0−μH)×(γ×1/√(σ2+ϵ))+β, which is herein referred to as yi, the scaled and shifted value of the normalized mini-batch element x1. The MADD unit may further write the resultant, y1, to a pipeline register, (e.g., pipeline register PR1).
In various embodiments, a further stage of the computational unit may include a tail stage. The tail stage may include one or more special functional units (e.g., a stochastic rounding unit, a packing unit, etc.). The special functional unit may perform a read operation on data stored in a pipeline register (e.g., the result, y0, and the result, y1). The special functional unit may perform one or more computational operations on the inputted data to generate an operational result. In addition to performing one or more operations, the special functional unit may further perform a write operation to write the operational result to a pipeline register. For example, the stochastic rounding unit may convert the results y0 and y1 into low-precision floating-point numbers. For example, the results may be FP32 numbers. Accordingly, the stochastic rounding unit may convert the FP32 results y0 and y1 into low-precision BF16 numbers w0 and w1. In addition, the packaging unit may store the low-precision BF16 numbers as a single data structure (e.g., a vector). The packaging unit may perform a write operation to write the generated data structure to a pipeline register.
In sum, in various embodiments, a computation unit may include a first circuit to traverse a batch of input elements xi having a first format, to produce a mean μ1 in the first format and a mean μ2 in a second format, the second format having more bits than the first format. The computation unit may further include a second circuit operatively coupled to the first circuit to traverse the batch of input elements xi to produce a standard deviation σ for the batch using the mean μ1 in the first format. The computation unit may also include a third circuit operatively coupled to the second circuit to traverse the batch of input elements xi to produce a normalized set of values yi using the mean μ2 in the second format and the standard deviation σ.
In the computation unit, the third circuit may also compute differences xi−μ2, and may further multiply the differences by a factor which is a function of the variance σ in the first format. In the computation unit, the factor may be an inverse of (σ+ϵ), where ϵ is a constant.
In the computation unit, the third circuit may convert input elements xi from the first format to the second format by computing the product of each input element xi in the first format and the identity operator in the first format. In the computation unit, the third circuit may produce two normalized elements y0 and y1 for each input element xi. Furthermore, the third circuit may compress the two normalized elements to produce a single output. In addition the third circuit may produce a scaled and shifted normalized set of elements in the second format. Also, the third circuit may perform stochastic rounding to convert the normalized set of elements to the first format.
In various embodiments, in the computation unit, the second circuit may traverse the batch of input elements xi to compute, for each xi: a difference xi−μ1; a square of the difference, (xi−μ1)2; and a sum of the square of the difference and a constant term ϵ, (xi−μ1)2+ϵ. In the computation unit, the second circuit may further traverse the batch of input elements xi to compute: an accumulated sum of the sums, (xi−μ1)2+ϵ; and an inverse square root of the accumulated sum.
In some embodiments, in the computation unit, the first circuit may modify the mean μ2 in the second format to produce the mean μ1 in the first format. The first circuit may modify the mean μ2 by truncating at least one bit from the mean μ2 in the second format.
In various embodiments, a re-configurable processor may include a first circuit to traverse a batch of input elements xi having a first format, to produce a mean μ1 in the first format and a mean μ2 in a second format, the second format having more bits than the first format. The re-configurable processor may further include a second circuit operatively coupled to the first circuit to traverse the batch of input elements xi to produce a standard deviation σ for the batch using the mean μ1 in the first format. The re-configurable processor may also include a third circuit operatively coupled to the second circuit to traverse the batch of input elements xi to produce a normalized set of values yi using the mean μ2 in the second format and the standard deviation σ.
In the re-configurable processor, the third circuit may also compute differences xi−μ2, and may further multiply the differences by a factor which is a function of the variance σ in the first format. In the re-configurable processor, the factor may be an inverse of (σ+ϵ), where ϵ is a constant.
In the re-configurable processor, the third circuit may convert input elements xi from the first format to the second format by computing the product of each input element xi in the first format and the identity operator in the first format. In the re-configurable processor, the third circuit may produce two normalized elements y0 and y1 for each input element xi. Furthermore, the third circuit may compress the two normalized elements to produce a single output. In addition, the third circuit may produce a scaled and shifted normalized set of elements in the second format. Also, the third circuit may perform stochastic rounding to convert the normalized set of elements to the first format.
In various embodiments, in the re-configurable processor, the second circuit may traverse the batch of input elements xi to compute, for each xi: a difference xi−μ1; a square of the difference, (xi−μ1)2; and a sum of the square of the difference and a constant term ϵ, (xi−μ1)2+ϵ. In the re-configurable processor, the second circuit may further traverse the batch of input elements xi to compute: an accumulated sum of the sums, (xi−μ2)2+ϵ; and an inverse square root of the accumulated sum.
In some embodiments, in the re-configurable processor, the first circuit may modify the mean μ2 in the second format to produce the mean μ1 in the first format. The first circuit may modify the mean μ2 by truncating at least one bit from the mean μ2 in the second format.
In various embodiments, a method may include traversing, by a first circuit, a batch of input elements xi having a first format, to produce a mean μ1 in the first format and a mean μ2 in a second format, the second format having more bits than the first format. The method may further include traversing, by a second circuit, the batch of input elements xi to produce a standard deviation σ for the batch using the mean μ1 in the first format. The method may also include traversing, by a third circuit, the batch of input elements xi to produce a normalized set of values yi using the mean μ2 in the second format and the standard deviation σ.
In the method, the third circuit may also compute differences x2−μ2, and may further multiply the differences by a factor which is a function of the variance σ in the first format. In the method, the factor may be an inverse of (σ+ϵ), where ϵ is a constant.
In the method, the third circuit may convert input elements xi from the first format to the second format by computing the product of each input element xi in the first format and the identity operator in the first format. In the method, the third circuit may produce two normalized elements y0 and y1 for each input element xi. Furthermore, the third circuit may compress the two normalized elements to produce a single output. In addition, the third circuit may produce a scaled and shifted normalized set of elements in the second format. Also, the third circuit may perform stochastic rounding to convert the normalized set of elements to the first format.
In various embodiments, in the method, the second circuit may traverse the batch of input elements xi to compute, for each xi: a difference xi−μ1; a square of the difference, (xi−μ1)2; and a sum of the square of the difference and a constant term ϵ, (xi−μ1)2+ϵ. In the method, the second circuit may further traverse the batch of input elements xi to compute: an accumulated sum of the sums, (xi−μ1)2+ϵ; and an inverse square root of the accumulated sum.
In some embodiments, in the method, the first circuit may modify the mean μ2 in the second format to produce the mean μ1 in the first format. The first circuit may modify the mean μ2 by truncating at least one bit from the mean μ2 in the second format.
Although examples are described herein based on the use of single precision floating point FP32 format floating point numbers, the technique is applicable to other data encoding formats, including, for example, double-precision floating-point FP64 format numbers, binary format numbers, and other encoding formats.
While the present disclosure is disclosed by reference to the preferred embodiments and examples detailed above, it is to be understood that these examples are intended in an illustrative rather than in a limiting sense. It is contemplated that modifications and combinations will readily occur to those skilled in the art, which modifications and combinations will be within the spirit of the invention and the scope of the following claims.
This application is related to U.S. Nonprovisional patent application Ser. No. ______ entitled “COMPUTATIONAL UNITS FOR ELEMENT APPROXIMATION,” (Attorney Docket No. SBNV 1014-1) filed contemporaneously. The related application is incorporated by reference for all purposes.
