Patent Application
20240134930
N/A
The present invention relates generally to integrated circuit (IC) devices and artificial intelligence (AI). More specifically, the present invention relates to methods and device structures for accelerating computing workloads, such as those in transformer-based models (a.k.a. transformers).
The transformer has been the dominant neural network architecture in the natural language processing (NLP) field, and its use continues to expand into other machine learning applications. The original Transformer was introduced in the paper “Attention is all you need” (Vaswani et al., 2017), which sparked the development of many transformer model variations, such as the generative pre-trained transformer (GPT) and the bidirectional encoder representations from transformers (BERT) models. Such transformers have significantly outperformed other models in inference tasks by their use of a self-attention mechanism that avoids recursion and allows for easy parallelism. On the other hand, the transformer workloads are very computationally intensive and have high memory requirements, and have been plagued as being time-intensive and inefficient.
Most recently, NLP models have grown by a thousand times in both model size and compute requirements. For example, it can take about 4 months for 1024 graphics processing units (GPUs) to train a model like GPT-3 with 175 billion parameters. New NLP models having a trillion parameters are already being developed, and multi-trillion parameter models are on the horizon. Such rapid growth has made it increasingly difficult to serve NLP models at scale.
From the above, it can be seen that improved devices and methods to accelerate compute workloads for AI are highly desirable.
The present invention relates generally to integrated circuit (IC) devices and artificial intelligence (AI) systems. More particularly, the present invention relates to methods and device structures for accelerating computing workloads, such as those in transformer-based neural network models (a.k.a. transformers) and the like. These methods and structures can be used in machine/deep learning applications such as natural language processing (NLP), computer vision (CV), and the like. Merely by way of example, the invention has been applied to AI accelerator apparatuses and chiplet devices configured in a PCIe card.
According to an example, the present invention relates to data compression and decompression in a matrix compute apparatus. In certain applications, it is desirable to improve the handling of large data sizes. For example, transformer-based modeling networks typically involve an enormous number of elements (e.g., weights, activations, etc.) that cannot all be stored in on-chip memory. Thus, accessing these elements requires frequent transfers from a memory storage device (e.g., DDR), which can cause the processing of these elements to become memory bound due to the large latency of such memory operations.
In an example, the present invention provides for a matrix multiply compute apparatus and method of operation therefor. The apparatus includes a memory device configured to store a plurality of weight matrix elements in a first format including a plurality of matrix weight columns, each of which includes a plurality of scale factors and a plurality of mantissa blocks. A crossbar device is coupled to the memory device and to one or more compute paths including a weight buffer (WB) device, a compute device, and an output buffer device. A first register device coupled to the crossbar device is configured to receive the plurality of scale factors of each weight matrix column, and the converter device is configured to determine a max exponent for each column using the plurality of scale factors of the column. A second register device coupled to the crossbar device is configured to store the resulting plurality of max exponents. Also, the first register device is configured to receive the plurality of mantissa blocks of each column, and the converter device is configured to determine a plurality of converted mantissa blocks using all of the plurality of scale factors and the plurality of mantissa blocks. The WB device is configured to receive the plurality of converted mantissa blocks and the plurality of max exponents, which results in the plurality of weight matrix elements in a second format. The compute device then determines a plurality of matrix multiply outputs using the plurality of matrix weight elements in the second format and the OB device stores the plurality of matrix multiply outputs.
In an example, the first register device, the second register device, and the converter device may be configured separately or together, and may be configured within the crossbar device. Each of the mantissa blocks can include one or more mantissas, and each of the plurality of scale factors is associated with one of the mantissa blocks. The first and second formats can include block floating point formats (e.g., 36×64 bytes, 65×64 bytes, etc.) and the scale factors can be characterized by floating point (FP) scale factors (e.g., unsigned 8-bit FP scale factor having 4-bit exponent field and 4-bit fraction field). Further, the converter device can be configured to determine the plurality of converted mantissa blocks by multiplying each mantissa with its associated scale factor, shifting each scaled mantissa, and rounding each shifted mantissa. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives.
The compression/decompression techniques described previous can also be implemented in one or more chiplet devices coupled to a memory device (e.g., DDR memory) within an AI accelerator apparatus. In this case, each chiplet device includes a CPU coupled to a plurality of slice devices, and each slice device includes at least a memory device and a compute device. Similar to the previous example, a first register coupled to the CPU is configured to receive the plurality of scale factors and the plurality of mantissa blocks. A converter device coupled to the CPU is configured to determine the plurality of max exponents from scale factors of each column and to determine the plurality of converted mantissa blocks from the scale factors and mantissa blocks from memory. A second register device coupled to the CPU stores the plurality of max exponents, which is sent along with the plurality of converted mantissa blocks to form the plurality of matrix weight elements in a second format within the memory devices of the plurality of slices. Then, the compute devices of the slices are configured to determine a plurality of matrix multiply outputs using these weight matrix elements in the second format. As in the previous example, there can be variations, modifications, and alternatives.
Although previous examples discuss weight matrix elements, the present compression/decompression implementation can also be applied to other matrix elements, such as matrix activations. In this case, the crossbar converter device is coupled to crossbar device and the IB device, and the decompression method can be applied to a plurality of activation matrix elements or input matrix elements stored in the memory device. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives.
Embodiments of this matrix compute apparatus and its related methods can provide many benefits. The present method and apparatus enables the storage of a large number of matrix elements in a compressed format that can be decompressed upon retrieval for matrix computations. Also, this compression-decompression capability can be accomplished without requiring entirely separate hardware and compute pathways. Further, these benefits can be realized in IC chips and chiplet devices with minimal added cost of silicon area.
A further understanding of the nature and advantages of the invention may be realized by reference to the latter portions of the specification and attached drawings.
In order to more fully understand the present invention, reference is made to the accompanying drawings. Understanding that these drawings are not to be considered limitations in the scope of the invention, the presently described embodiments and the presently understood best mode of the invention are described with additional detail through use of the accompanying drawings in which:
The present invention relates generally to integrated circuit (IC) devices and artificial intelligence (AI) systems. More particularly, the present invention relates to methods and device structures for accelerating computing workloads in transformer-based neural network models (a.k.a. transformers). These methods and structures can be used in machine/deep learning applications such as natural language processing (NLP), computer vision (CV), and the like. Merely by way of example, the invention has been applied to AI accelerator apparatuses and chiplet devices configured to perform high throughput operations for NLP.
Currently, the vast majority of NLP models are based on the transformer model, such as the bidirectional encoder representations from transformers (BERT) model, BERT Large model, and generative pre-trained transformer (GPT) models such as GPT-2 and GPT-3, etc. However, these transformers have very high compute and memory requirements. According to an example, the present invention provides for an apparatus using chiplet devices that are configured to accelerate transformer computations for AI applications. Examples of the AI accelerator apparatus are shown in
As shown, the AI accelerator apparatuses 101 and 102 are embodied in peripheral component interconnect express (PCIe) card form factors, but the AI accelerator apparatus can be configured in other form factors as well. These PCIe card form factors can be configured in a variety of dimensions (e.g., full height, full length (FHFL); half height, half length (HHHL), etc.) and mechanical sizes (e.g., 1×, 2×, 4×, 16×, etc.). In an example, one or more substrate members 140, each having one or more chiplets, are coupled to a PCIe card. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to these elements and configurations of the AI accelerator apparatus.
Embodiments of the AI accelerator apparatus can implement several techniques to improve performance (e.g., computational efficiency) in various AI applications. The AI accelerator apparatus can include digital in-memory-compute (DIMC) to integrate computational functions and memory fabric. Algorithms for the mapper, numerics, and sparsity can be optimized within the compute fabric. And, use of chiplets and interconnects configured on organic interposers can provide modularity and scalability.
According to an example, the present invention implements chiplets with in-memory-compute (IMC) functionality, which can be used to accelerate the computations required by the workloads of transformers. The computations for training these models can include performing a scaled dot-product attention function to determine a probability distribution associated with a desired result in a particular AI application. In the case of training NLP models, the desired result can include predicting subsequent words, determining contextual word meaning, translating to another language, etc.
The chiplet architecture can include a plurality of slice devices (or slices) controlled by a central processing unit (CPU) to perform the transformer computations in parallel. Each slice is a modular IC device that can process a portion of these computations. The plurality of slices can be divided into tiles/gangs (i.e., subsets) of one or more slices with a CPU coupled to each of the slices within the tile. This tile CPU can be configured to perform transformer computations in parallel via each of the slices within the tile. A global CPU can be coupled to each of these tile CPUs and be configured to perform transformer computations in parallel via all of the slices in one or more chiplets using the tile CPUs. Further details of the chiplets are discussed in reference to
The CPUs 221 of each tile 210 can be coupled to a global CPU via a global CPU interface 230 (e.g., buses, connectors, sockets, etc.). This global CPU can be configured to coordinate the processing of all chiplet devices in an AI accelerator apparatus, such as apparatuses 101 and 102 of
Further, the chiplet 201 includes a PCIe interface/bus 260 coupled to each of the CPUs 221 in each of the tiles. The PCIe interface 260 can be configured to communicate with a server or other communication system. In the case of a plurality of chiplet devices, a main bus device is coupled to the PCIe bus 260 of each chiplet device using a master chiplet device (e.g., main bus device also coupled to the master chiplet device). This master chiplet device is coupled to each other chiplet device using at least the D2D interconnects 240. The master chiplet device and the main bus device can be configured overlying a substrate member (e.g., same substrate as chiplets or separate substrate). An apparatus integrating one or more chiplets can also be coupled to a power source (e.g., configured on-chip, configured in a system, or coupled externally) and can be configured and operable to a server, network switch, or host system using the main bus device. The server apparatus can also be one of a plurality of server apparatuses configured for a server farm within a data center, or other similar configuration.
In a specific example, an AI accelerator apparatus configured for GPT-3 can incorporate eight chiplets (similar to apparatus 102 of
In an example, the DIMC is coupled to a clock and is configured within one or more portions of each of the plurality of slices of the chiplet to allow for high throughput of one or more matrix computations provided in the DIMC such that the high throughput is characterized by 512 multiply accumulates per a clock cycle. In a specific example, the clock coupled to the DIMC is a second clock derived from a first clock (e.g., chiplet clock generator, AI accelerator apparatus clock generator, etc.) configured to output a clock signal of about 0.5 GHz to 4 GHz; the second clock can be configured at an output rate of about one half of the rate of the first clock. The DIMC can also be configured to support a block structured sparsity (e.g., imposing structural constraints on weight patterns of a neural networks like a transformer).
In an example, the SIMD device 350 is a SIMD processor coupled to an output of the DIMC. The SIMD 350 can be configured to process one or more non-linear operations and one or more linear operations on a vector process. The SIMD 350 can be a programmable vector unit or the like. The SIMD 350 can also include one or more random-access memory (RAM) modules, such as a data RAM module, an instruction RAM module, and the like.
In an example, the slice controller 360 is coupled to all blocks of each compute path 312 and also includes a control/status register (CSR) 362 coupled to each compute path. The slice controller 360 is also coupled to a memory bank 370 and a data reshape engine (DRE) 380. The slice controller 360 can be configured to feed data from the memory bank 370 to the blocks in each of the compute paths 312 and to coordinate these compute paths 312 by a processor interface (PIF) 364. In a specific example, the PIF 364 is coupled to the SIMD 350 of each compute path 312.
Further details for the compute core 310 are shown in
These IMC modules 332 can also be coupled to a block floating point alignment module 334 and a partial products reduction module 336 for further processing before outputting the DIMC results to the output buffer 540. In an example, the input buffer 320 receives input data (e.g., data vectors) from the memory bank 370 (shown in
In addition to the details discussed previously, the SIMD 350 can be configured as an element-wise vector unit. The SIMD 350 can include a computation unit 352 (e.g., add, subtract, multiply, max, etc.), a look-up table (LUT) 354, and a state machine (SM) module 356 configured to receive one or more outputs from the output buffer 340.
The NoC device 342 is coupled to the output buffer 340 configured in a feedforward loop via shortcut connection 344. Also, the NoC device 342 is coupled to each of the slices and is configured for multicast and unicast processes. More particularly, the NoC device 342 can be configured to connect all of the slices and all of the tiles, multi-cast input activations to all of the slices/tiles, and collect the partial computations to be unicast for a specially distributed accumulation.
Considering the previous eight-chiplet AI accelerator apparatus example, the input buffer can have a capacity of 64 KB with 16 banks and the output buffer can have a capacity of 128 KB with 16 banks. The DIMC can be an 8-bit block have dimensions 64×64 (eight 64×64 IMC modules) and the NoC can have a size of 512 bits. The computation block in the SIMD can be configured for 8-bit and 32-bit integer (int) and unsigned integer (uint) computations, as well as floating point computations, such as IEEE 854 float16 or float32. These slice components can vary depending on which transformer the AI accelerator apparatus will serve.
As shown in close-up 401, each of the memory-select units 422, 424 includes a memory cell 430 (e.g., SRAM cell, or the like) and a select multiplexer 432. Each of the memory-select units 422, 424 is coupled to a read-write controller 440, which is also coupled to a memory bank/driver block 442. In an example, the read-write controller 440 can be configured with column write drivers and column read sense amplifiers, while the memory bank/driver block 432 can configured with sequential row select drivers.
An input activation controller 450 can be coupled to the activation multiplexer 426 each of the read-write blocks 420. The input activation controller 450 can include precision and sparsity aware input activation register and drivers. The operator unit 428 receives the output of the first memory-select unit 422 and receives the output of this block 450 through the activation multiplexer 426, which is controlled by the output of the second memory-select unit 424. The output of the operator unit 428 is then fed into the computation tree block 410.
The input activation block 450 is also coupled to a clock source/generator 460. As discussed previously, the clock generator 460 can produce a second clock derived from a first clock configured to output a clock signal of about 0.5 GHz to 4 GHz; the second clock can be configured at an output rate of about one half of the rate of the first clock. The clock generator 460 is coupled to one or more sign and precision aware accumulators 470, which are configured to receive the output of the computation tree blocks 410. In an example, an accumulator 470 is configured to receive the outputs of two computation tree blocks 410. Example output readings of the IMC are shown in
Referring back to the eight-chiplet AI accelerator apparatus example, the memory cell can be a dual bank 2×6T SRAM cell, and the select multiplexer can be an 8T bank select multiplexer. In this case, the memory bank/driver block 442 includes a dual-bank SRAM bank. Also, the read/write controller can include 64 bytes of write drivers and 64 bytes of read sense amplifiers. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to these IMC module components and their configurations.
Transformer model variations include those based on just the decoder stack (e.g., transformer language models such as GPT-2, GPT-3, etc.) and those based on just the encoder stack (e.g., masked language models such as BERT, BERT Large, etc.). Transformers are based on four parameters: sequence length (S) (i.e., number of tokens), number of attention heads (A), number of layers (L), and embedding length (H). Variations of these parameters are used to build practically all transformer-based models today. Embodiments of the present invention can be configured for any similar model types.
A transformer starts as untrained and is pre-trained by exposure to a desired data set for a desired learning application. Transformer-based language models are exposed to large volumes of text (e.g., Wikipedia) to train language processing functions such as predicting the next word in a text sequence, translating the text to another language, etc. This training process involves converting the text (e.g., words or parts of words) into token IDs, evaluating the context of the tokens by a self-attention layer, and predicting the result by a feed forward neural network.
The self-attention process includes (1) determining query (Q), key (K), and value (V) vectors for the embedding of each word in an input sentence, (2) calculating a score for from the dot product of Q and K for each word of the input sentence against a target word, (3) dividing the scores by the square root of the dimension of K, (4) passing the result through a softmax operation to normalize the scores, (5) multiplying each V by the softmax score, and (6) summing up the weighted V vectors to produce the output. Note that the value matrix V becomes the weight matrix for matrix multiplication with softmax attention matrix; in the context of block floating point numerics, this requires a column blocking converter for V as described below.
Many things impact the performance of such transformer architectures. The softmax function tends to be the critical path of the transformer layers (and has been difficult to accelerate in hardware). Requirements for overlapping the compute, SIMD operations and NoC transfers also impacts performance. Further, efficiency of NoC, SIMD, and memory bandwidth utilization is important as well.
Different techniques can be applied in conjunction with the AI accelerator apparatus and chiplet device examples to improve performance, such as quantization, sparsity, knowledge distillation, efficient tokenization, and software optimizations. Supporting variable sequence length (i.e., not requiring padding to the highest sequence lengths) can also reduce memory requirements. Other techniques can include optimizations of how to split self-attention among slices and chips, moving layers and tensors between the slices and chips, and data movement between layers and FC matrices.
According to an example, the present invention provides for an AI accelerator apparatus (such as shown in
In an example, each of the transformers is configured within one or more DIMCs such that each of the transformers comprises a plurality of matrix multipliers including QKV matrices configured for an attention layer of a transformer followed by three fully-connected matrices (FC). In this configuration, the DIMC is configured to accelerate the transformer and further comprises a dot product of Q KT followed by a softmax (Q KT/square root (dk))V. In an example, the AI accelerator apparatus also includes a SIMD device (as shown in
Using a transformer like BERT Large, NLP requires very high compute (e.g., five orders of magnitude higher than CV). For example, BERT Large requires 5.6 giga-multiply-accumulate operations per second (“GMACs”) per transformer layer. Thus, the NLP inference challenge is to deliver this performance at the lowest energy consumption.
Although the present invention is discussed in the context of a BERT Large transformer for NLP applications, those of ordinary skill in the art will recognize variations, modifications, and alternatives. The particular embodiments shown can also be adapted to other transformer-based models and other AI/machine learning applications.
As discussed previously, block floating point (BFP) formats are important for efficient hardware acceleration of matrix multiplication operations in deep neural network inference. Matrix weights are often blocked along the columns, while the activations are often blocked along the rows. Thus, BFP numerics enable an efficient integer arithmetic implementation of matrix multiplication while maintaining a large dynamic range. After a matrix multiplication, the activation row vector dot product with the weight column vector is accumulated into a floating point (FP) format (e.g., FP32, FP16, etc.) and stored into an output buffer as a matrix tile (e.g., 64×64 tile of FP16). We may also use BFP32-1, with 24 bit mantissa in 2's complement, and 8 bit exponent in 2's complement, as an equivalent format to FP32 for accumulation of partial products.
The output buffer memory load/store is typically implemented row-wise, which is convenient for the typical case of row-wise BFP blocking to generate the activations for the next matrix multiplication. However, there are cases in which the output of a matrix multiplication is used as a weight matrix for a subsequent matrix multiplication (e.g., matrix multiplication with a value matrix for an attention function in a BERT encoder model), which requires storing the data from the output buffer in a column blocking configuration. In such cases, blocking across the columns poses a challenge when the memory load/store is characterized by a row-wise storage configuration because the output converter can only read data one row at a time.
According to an example, the present invention provides a column blocking converter apparatus and method for converting data from a first format in a row blocking configuration to a second format in a column blocking configuration. The column blocking apparatus can be configured as an IC for an AI accelerator apparatus, such as the examples AI accelerator ICs described previously.
The apparatus 701 of
An OB converter device 710 can be coupled between the compute device 330 and the OB device 340. This OB converter device 710 can be configured to store the plurality of matrix outputs in the first format within the OB device 340. As shown, the OB converter device 710 is configured separately from the OB device 340, however, the OB converter device 710 can also be configured within the OB device 340. These configurations can be implemented as an inline column blocking converter apparatus.
A crossbar device 360 is coupled to the IB device 320, the compute device 330, and the OB device 340. A crossbar converter device 720 is also coupled to the OB device 340 and is configured to convert the plurality of matrix outputs from the first format to a second format using a max exponent value and a mantissa value determined for each of the plurality of matrix outputs, resulting in a plurality of converted matrix outputs. As shown, the crossbar converter device 720 is configured within the compute path 312; however, the crossbar converter device 720 can also be configured within the crossbar device 360.
Further, a memory device 370 is coupled to the crossbar device 360. This memory device is configured to store the plurality of converted matrix outputs, in the second format and in a column blocking configuration, using the max exponent values and the mantissa values. The first format can be a floating point (FP) format, while the second format can be a block floating point (BFP) format. In a specific example, the first format is an FP16 format, the second format is a BFP format with a block size of 64 elements, mantissa bit width of 8 bits, and a shared exponent of 8 bits (BFP16-64 format). In this case, the plurality of matrix outputs can be characterized by a 64.64 byte tile of mantissas and a 64 byte row of shared exponents. This embodiment of the invention encompasses an efficient algorithm and hardware architecture for a column blocking converter for converting an FP16 tile of 64×64 elements stored in an output buffer to a BFP16-64 tile blocked along the columns.
In an example, the crossbar converter device 720 includes a max exponent register 722 configured to store the max exponent values of each of the plurality of matrix outputs. The OB device 340 and the converter device can be configured together to determine the max exponent value of each of the plurality of matrix outputs in a first row-by-row process, to determine the mantissa value of each of the plurality of matrix outputs in a second row-by-row process, and to store the max exponent values and the mantissa values in the memory device. The max exponent register 722 can be used in the first row-by-row process to store the max exponent values.
In a specific example, the crossbar converter device 720 is configured to perform a shifting process and a rounding process on the mantissa value for each of the plurality of matrix outputs during the second row-by-row process. The crossbar device 360 can be configured to write the mantissa values to the memory device 370 after each row in the second row-by-row process. Further, the crossbar device 360 can be configured to write the max exponent values to the memory device 370 after the second row-by-row process. The crossbar converter device 720 can be coupled to the OB device 340 in a feedback configuration to perform the first and second row-by-row processes.
Compared to
Similar to the first architecture, the crossbar converter device 720 can be configured to perform the shifting process and the rounding process on the mantissa value for each of the plurality of matrix outputs. And, the crossbar device 360 can be configured to write the max exponent values to the memory device 370 after the second row-by-row process. Further details of the processes performed by the OB converter device 710 and the crossbar converter device 720 are discussed with reference to
Here, the crossbar converter device 720 reads the OB bank 340 to perform the first and second row-by-row processes to determine the max exponent and mantissa values, respectively. The crossbar converter device 720 reads each row of the data stored in the OB bank 340 one row at a time to determine the max exponent value of each entry and to update the max exponent register 722 (e.g., if expi<reg_exp[i], then reg_exp[i]=expi). In the 64×64 byte example, the converter device 720 reads in a row of 64 FP16 elements at a time. After all rows are processed, the max exponent register 722 contains the max exponent (denoted by “Exp1” to “ExpM”) for each column of the tile of data stored in the OB bank 340.
Then, the converter device 720 reads each row from the OB bank 340 again in the second row-by-row process to determine the mantissa values. For each of the OB bank entries, the converter device 720 can perform a shifting process and a rounding process to convert the mantissa values to a desired format (e.g., integer format or other numerical format). In the 64×64 byte example, the shifting and rounding processes can result in converting the mantissa values into an 8-bit integer (int8) format. After processing a row of mantissas (denoted by “Man1” to “ManN”), the processed data is sent to be written to the memory device 370. Once all rows are processed, the conversion of the mantissas to the second format (denoted by “DN,M-F2”) in the column blocking configuration is complete. With the max exponent register data sent afterwards, the memory device 370 will contain a contiguous block of data in which each column is in the second format. In the 64×64 byte matrix data example, the contiguous block is characterized by 65×64 bytes and each column is in a BFP16-64 format.
Then, the crossbar converter device 720 reads the max exponent data from the OB converter register 712 to its own max exponent register 722. Similar to method 801, the crossbar converter device 720 reads each row from the OB bank 340 in the second row-by-row process to determine the mantissa values. The converter device 720 also performs the shifting process and the rounding process to convert the mantissa values to a desired format (e.g., integer format or other numerical format). After processing a row of mantissas, the processed data is sent to be written to the memory device 370. Once all rows are processed, the conversion of the mantissas to the second format in the column blocking configuration is complete. With the max exponent register data sent afterwards, the memory device 370 will contain a contiguous block of data in which each column is in the second format. In the 64×64 byte matrix data example, the contiguous block is characterized by 65×64 bytes and each column is in a BFP16-64 format.
Although these examples are discussed with respect to the FP and BFP numerical formats, the column blocking converter apparatus and its method can be applied to the conversion of data from any first format to any second format that can be determined by corresponding exponent and mantissa values. There are also variations on computing the shared block exponent; for example, instead of the max exponent, it possible to use a percentile value. Also, in cases that buffer memory load/store is implemented column-wise, the same techniques described herein can be used convert from a column-wise storage configuration to a row-wise storage configuration. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives these blocking conversion methods and structures.
According to an example, the present invention provides a method and device for data conversion in a matrix compute apparatus. In a specific example, the matrix compute apparatus can be configured as a multiply and accumulate (MAC) unit that serves as a key building block of the dot product and matrix multiplication hardware used to accelerate deep neural network inference applications, including the NLP workloads discussed previously. In such applications, there may be a need to handle more than one kind of data format. For example, efficient MAC implementations are often based on integer arithmetic, which support fixed point or block floating point (BFP) numerical formats. However, in certain applications, it is desirable for the MAC unit, or other matrix compute apparatus, to have the capability of handling floating point (FP) or brain floating point (Bfloat) numerical formats.
Thus, the present invention provides for a method and device to enable a matrix compute apparatus configured to process matrix data in a target format by segmenting the data and parallel processing the segmented data portions in the native format of the matrix compute apparatus. Merely by way of example, the present invention discusses the native format as an 8-bit integer (int8) format and the target format as a 16-bit floating point (FP16) format. Embodiments of the present matrix compute apparatus can be configured as an IC for an AI accelerator IC, such as the AI accelerator systems discussed previously. Further details are discussed below with reference to
In an example, the input buffer (TB) device 1010 is configured to receive one or more matrix inputs (e.g., from a memory device or the like). This IB device 1010 can be configured similarly to the IB devices shown previously (e.g.,
The IB device 1010 can receive a first matrix input or a plurality of matrix inputs in the first format from an input converter device configured to convert the matrix input(s) to the first format. This input converter device, such as a CPU (e.g., tile CPU 221 shown in
Merely by way of example, the matrix compute apparatus can be configured to perform matrix computations in an integer numerical format. In such cases, the compute apparatus can be configured to process the matrix input in data portions that can fit within the integer format. For example, each of the plurality of compute units can be configured for matrix computations in an int8 format and the matrix inputs can be in an FP16 format in a 64×64 byte tile configuration. In this case, the input converter device (e.g., tile CPU, inline input converter 1012, etc.) converts the FP16 matrix input to a 24-bit block floating point (BFP24) format, with a 16-bit mantissa and an 8-bit exponent. The mantissa can then be split into the two 8-bit portions, a most significant byte (MSB) portion and a least significant byte (LSB) portion, to be processed in parallel by the compute device 1020.
In an example, the compute device 1020 includes a plurality of compute units 1022 having at least a first compute unit 1022 and a second compute unit 1022. This pair of compute units can be configured to perform matrix computations for matrix inputs in a non-native format. More specifically, the first compute unit 1022 can be configured to determine a first matrix output using at least the first input portion, and the second compute unit 1022 can be configured to determine a second matrix output using at least the second input portion. Then, the compute device 1020 can be configured to determine a combined matrix output in a second format using the first matrix output and the second matrix output. In a specific example, the compute device 1020 determines the combined matrix output by shifting the first matrix output and adding the shifted first matrix output to the second matrix output.
In an example, each of the matrix inputs includes a matrix weight and a matrix activation. Each of the matrix weight inputs can include a matrix weight exponent and a matrix weight mantissa. Referring back to the FP16 example, the matrix weight exponent includes 8 bits and the matrix weight mantissa includes 16 bits that can be separated into an 8-bit MSB portion and an 8-bit LSB portion. Similarly, the matrix activation exponent also includes 8 bits and the matrix activation mantissa also includes 16 bits that can be separated into an 8-bit MSB portion and an 8-bit LSB portion. In this case, the compute device determines the first matrix output by performing a dot product process using the matrix activation and MSB portion of the matrix weight mantissa. Similarly, the compute device determines the second matrix output by performing a dot product process using the matrix activation and the LSB portion of the matrix weight mantissa.
Although the previous example only discusses segmenting the matrix input data into two portions, other examples may segment the data in to a plurality of data portions that are processed in a parallel by a plurality of compute units. In such cases, the compute device 1020 will determine a plurality of matrix outputs using similar shifting and addition processes to combine these matrix outputs into the combined matrix output with each data portion positioned in the appropriate order. These portions can also be stored in the segmented portions that match the native format of the compute device. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to the choices of data formats and data segmentation.
Considering the FP16 example, the first input portion is the MSB weight portion, while the second input portion is the LSB weight portion. The first compute unit 1022 would be configured to determine the first matrix output using the MSB portion, while the second compute unit 1022 would be configured to determine the second matrix output using the LSB portion. The matrix outputs are combined as shown in
In an example, the compute device includes an alignment device 1024 coupled to the plurality of compute units 1022. The alignment device 1024 can be configured to determine a rounded matrix output in a third format using the combined output. This rounding process may be used to prepare the matrix output for a subsequent partial products reduction (PPR) process. In the FP16 example, the combined matrix output in the BFP46-1 format can be rounded down to a matrix output in a BFP32-1 format. In another example, the BFP46-1 combined matrix output can be converted to an FP32 matrix output by the alignment device 1024 or a data converter coupled to the alignment device 1024.
In an example, a PPR device 1026 is coupled to the alignment device 1024. The PPR device 1026 can be configured to determine a reduced matrix output using the rounded matrix output. The PPR process may be used to prepare the matrix output for subsequent conversion the original data format (e.g., FP16) to be stored in the OB device 1030.
In an example, the compute device 1020 also includes a compute converter 1028 configured to determine a first converted matrix output in a converted output format using the previous matrix outputs. In the FP16 example, the compute converter 1028 converts the reduced matrix output in the BFP32-1 format to an FP16 matrix output. In the case that the combined matrix output is converted to an FP32 format, the compute converter 1028 converts the reduced matrix output in the FP32 format to an FP16 matrix output.
In an example, the OB device 1030 is configured to store the resulting converted matrix output. This OB device 1030 can be configured similarly to the OB devices shown previously (e.g.,
Embodiments of this matrix compute apparatus and its related methods can provide many benefits. The present method and apparatus enables the computational handling of matrix inputs in different data formats that can be segmented into portions that are compatible with a native format. Also, this multi-format capability can be accomplished without requiring entirely separate hardware and compute pathways. Further, these benefits can be realized in IC chips and chiplet devices with minimal added cost of silicon area.
As discussed in a previous example, each of the matrix inputs can include a matrix weight and a matrix activation. Each of the matrix weight inputs can include a matrix weight exponent and a matrix weight mantissa. Referring back to the FP16 example, the matrix weight exponent includes 8 bits and the matrix weight mantissa includes 16 bits that can be separated into an 8-bit MSB portion and an 8-bit LSB portion. In this case, the matrix activation exponent also includes 8 bits and the matrix activation mantissa also includes 16 bits that can be separated into an 8-bit MSB portion and an 8-bit LSB portion.
In this case, the first portion of the matrix weight (e.g., MSB) is stored in a first compute unit 1022-0 (shown as IMC0) while the second portion of the matrix weight (e.g., LSB) is stored in a second compute unit 1022-4 (shown as IMC4). The compute device 1020 determines the first matrix output by performing a dot product process using the matrix activation and the first portion of the matrix weight, and determines the second matrix output by performing a dot product process using the matrix activation and the second portion of the matrix weight. Then, the first matrix output is shifted (by 8 bits in the FP16 example) and added to the second matrix output to determine the combined matrix output.
Subsequently, the alignment device 1024 can determine the rounded matrix output from the combined matrix output, and the PPR device 1026 can determine the reduced matrix output from the rounded matrix output. Further, the compute converter 1028 can determine a converted matrix output from the reduced matrix output. A flow diagram of the matrix outputs is shown in
As discussed previously, other examples may segment the data in to a plurality of data portions that are processed in a parallel by a plurality of compute units. In such cases, the compute device 1020 will determine a plurality of matrix outputs using similar shifting and addition processes to combine these matrix outputs into the combined matrix output with each data portion positioned in the appropriate order. These portions can also be stored in the segmented portions that match the native format of the compute device (e.g., int8 compute unit configured to process FP16 matrix input). Further, steps for processing the matrix outputs, along with their respective hardware components, may be added, removed, or rearranged depending upon the application. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to the choices of data formats and data segmentation.
According to an example, the present invention provides a method and device for data compression and decompression in a matrix compute apparatus. In a specific example, the matrix compute apparatus can be configured as a matrix multiply compute apparatus (e.g., a MAC unit) configured to accelerate deep neural network inference applications, including the NLP workloads discussed previously. In such applications, it is desirable to improve the handling of large data sizes. For example, transformer-based modeling networks typically involve an enormous number of elements (e.g., weights, activations, etc.) that cannot all be stored in on-chip memory. Thus, accessing these elements requires frequent transfers from a memory storage device (e.g., DDR), which can cause the processing of these elements to become memory bound due to the large latency of such memory operations.
Thus, the present invention provides for a method and device to enable a matrix compute apparatus configured to process matrix data stored in a compressed format and to decompress this data for matrix computations. Merely by way of example, the present invention discusses data blocks in a 36×64 byte compressed block floating point (BFP) format, denoted as SBFP-12-16, that decompress into a 65×64 byte BFP format, denoted as BFP16-64. Embodiments of the present matrix compute apparatus can be configured as an IC for an AI accelerator IC, such as the AI accelerator systems discussed previously. Further details are discussed below with reference to
In an example, the WB device 1120 can be configured together with the input buffer (TB) device 320 as one buffer device. Also, the crossbar converter device 1110, the first register device 1112, the second register device 1114, and any other register devices can be configured together or separately within each compute path 312. Alternatively, the crossbar converter device 1110 any registers can also configured within the crossbar device 360 and be coupled to each compute path 312. Further details of the method of operating this apparatus are discussed below.
According to an example, the present invention provides a method of operating a matrix multiply compute apparatus using block compression/decompression. This apparatus includes at least a memory device configured to store a plurality of weight matrix elements in a first format, which includes a plurality of weight matrix columns. Each such column includes a plurality of scale factors and a plurality of mantissa blocks. In this case, the apparatus is configured similarly to apparatus 1101 shown in
The above sequence of steps is used to operating a matrix multiply compute apparatus configured for an AI accelerator apparatus according to one or more embodiments of the present invention. Depending upon the embodiment, one or more of these steps can be combined, or removed, or other steps may be added without departing from the scope of the claims herein. One of ordinary skill in the art will recognize other variations, modifications, and alternatives. Further details of this method are provided throughout the present specification and more particularly below.
In an example, each of the mantissa blocks includes one or more mantissa values, and each of the plurality of scale factors is associated with one of the plurality of mantissa bocks. The step of determining the plurality of converted mantissa blocks can include multiplying each mantissa of each mantissa block with the associated scale factor resulting in a scaled mantissa. This step can also include shifting each scaled mantissa of each mantissa block resulting in a shifted mantissa. Further, this step can include rounding each shifted mantissa of each mantissa block resulting in a rounded mantissa.
In an example, the plurality of weight matrix elements in the first format can be characterized by a 36×64 byte storage configuration, such as the SBFP12-16 format, and the plurality of weight matrix elements in the second format can be can be characterized by a 65×64 byte storage configuration, such as the BFP16-64. In the SBFP12-16 case, the plurality of weight matrix elements are configured in 64 weight matrix columns such that each weight matrix column includes four 8-byte mantissa blocks and four 1-byte scale factors (totaling 36 bytes). In a specific example, each of the plurality of scale factors (always positive numbers in SBFP12-16 format) are represented by an unsigned 8-bit floating point (FP8) scale factor, which includes a 4-bit exponent field and a 4-bit fraction field. An optional programmable exponent bias can be used to optimize the scale factor dynamic range. For each FP8 scale factor, there correspond 8-bytes of mantissas, each byte storing two 4-bit integer (int4) mantissa values (so a total of 16 4-bit mantissas elements for each FP8 scale factor). Thus, a BFP16-64 block of 65 bytes can be compressed to four blocks of SBFP12-16 totaling 36 bytes, a compression ratio of 1.8056.
Referring again to the SBFP12-16, the previous method can include reading in 4×64 bytes scale factors into the first register device, which requires the first register device to have a capacity of at least 256 bytes. The converter device can than compute the max exponent over the exponent fields of the four scale factors, and save the result in the second register device, which requires the second register device to have a capacity of at least 64 bytes. Then, each 64 byte row of mantissa blocks (each with two 4-bit mantissas) is read into the first register device to determine the converted mantissa blocks. The conversion process includes multiplying each 4-bit mantissa by its respective FP8 scale factor mantissa value (a 5 bit integer, including the 4 fraction bits and implicit 1), then shifting and rounding the result to 8-bits. After multiplication we have a 9 bit mantissa. The shifting and rounding process is characterized by first shifting each 9 bit mantissa by an amount computed by subtracting the block exponent for the mantissa from the max exponent over 4 blocks, and then rounding the result to 8 bits. One or more rows of converted mantissa blocks can be sent to the WB device, and after all rows of mantissa blocks are processed, the 64 bytes of exponents can be sent to the WB device to complete the decompression to the BFP16-64 format. Here, we assume the matrix multiply unit is designed for BFP16-64 numerical format, hence requires de-compression from SBFP12-16. Of course, there can be other variations, modifications, and alternatives. For example, the SBFP block size can be increased or decreased from 16, to tradeoff compression ratio with compression accuracy. Similarly, we may consider different bit widths for the SBFP mantissas, and different FP formats for the scale factors.
Although previous examples discuss weight matrix elements, the present compression/decompression implementation can also be applied to other matrix elements, such as matrix activations. In this case (see
In an example, the blocks 1132 are configured in an N×M array, denoted by BN,M, and the factors 1134 are configured in an N×M array, denoted by SN,M. Here, the overall array includes the array of blocks 1132 configured above the array of factors 1134, but this order may be flipped. In row-wise configurations, the blocks 1132 and factors 1134 may be configured on the right and left sides of the overall array. Depending on the application, other configurations may be used as well.
Referring back to the SBFP12-16 format, each column is configured as a weight matrix column including four 8-byte mantissa blocks and four 1-byte scale factors. In this case, each mantissa block row includes 64 mantissa blocks, and each scale factor row includes 64 scale factors, which means that the matrix data format includes 4×64 mantissa blocks and 4×64 scale factors in total. As discussed previously, each mantissa block includes 2 int4 mantissas and each scale factor is configured as an FP8 scale factor. Of course, there can be other variations, modifications, and alternatives.
Embodiments of this matrix compute apparatus and its related methods can provide many benefits. The present method and apparatus enables the storage of a large number of matrix elements in a compressed format that can be decompressed upon retrieval for matrix computations. Also, this compression-decompression capability can be accomplished without requiring entirely separate hardware and compute pathways. Further, these benefits can be realized in IC chips and chiplet devices with minimal added cost of silicon area.
According to an example, the present invention provides for an AI accelerator apparatus configured for block compression/decompression. This apparatus includes at least a memory device (e.g., DDR memory) configured to store a plurality of weight matrix elements in a first format, which includes a plurality of weight matrix columns. Each such column includes a plurality of scale factors and a plurality of mantissa blocks. The apparatus is also configured with one or more chiplet devices coupled to the memory device, and each chiplet device having at least a CPU coupled to a plurality of slice devices.
In this case, the apparatus is configured similarly to apparatus 201 shown in
The above sequence of steps is used to operating an AI accelerator apparatus configured for block compression/decompression according to one or more embodiments of the present invention. Depending upon the embodiment, one or more of these steps can be combined, or removed, or other steps may be added without departing from the scope of the claims herein. One of ordinary skill in the art will recognize other variations, modifications, and alternatives. Further details of this method are provided throughout the present specification.
While the above is a full description of the specific embodiments, various modifications, alternative constructions and equivalents may be used. As an example, the AI accelerator apparatus and chiplet devices can include any combination of elements described above, as well as outside of the present specification. Therefore, the above description and illustrations should not be taken as limiting the scope of the present invention which is defined by the appended claims.
