Patent Application
20250209017
Conventional Network Interface Cards (NICs) that enable Ethernet connectivity face significant challenges when scaling across distributed AI accelerators for applications such as natural language processing (NLP), computer vision (CV), generative AI, agentic AI, autonomous reasoning/decision-making, dataset analysis, and the like. The challenges become especially pronounced in environments where multiple nodes are involved. A solution, RoCEv2 (commonly known as “RDMA over Converged Ethernet”) combined with RDMA IB (“InfiniBand”) fabric, can facilitate multi-node GPU accelerator communication
Various limitations, however, exist with RDMA over Converged Ethernet combined with InfiniBand. Such a solution often requires a complex shared address space to support one-sided communication, making deployment and management more complex. Additionally, both software and hardware fabric solutions face constraints when attempting to meet the low-latency demands required for Generative AI (GenAI) inferences. Such limitation hinders their ability to fully capitalize on the performance potential of modern accelerators.
Other conventional techniques involve the use of PCIe (Peripheral Component Interface Express) topologies. However, PCIe fabric topologies present scalability limitations. While such PCIe can support intra-switch communication, PCIe is restricted by a limited number of PCIe lanes provided by CPU sockets and PCIe switches. Accordingly, PCIe further complicates achieving efficient multi-CPU socket Peer-to-Peer (P2P) connectivity across nodes.
Certain PCIe switch vendors offer synthetic fabric models that enable cross-switch x16 link communication through the use of custom firmware. Unfortunately, such synthetic fabric models remain highly specialized and not yet broadly adopted.
From the above, it is seen that techniques for scaling across distributed accelerators are highly desirable.
The present invention relates generally to integrated circuit (IC) devices and artificial intelligence (AI) systems. More particularly, the present invention relates to methods and device structures for accelerating computing workloads of neural network models (e.g., transformer models, convolution neural network models, etc.). These methods and structures can be used in applications such as natural language processing (NLP), computer vision (CV), generative AI, agentic AI, autonomous reasoning/decision-making, and the like. Merely by way of example, the invention has been applied to AI accelerator apparatuses and chiplet devices configured in a PCIe card.
According to an example, the present invention provides for a digital in-memory compute (DIMC) accelerator system using a chiplet architecture. The system includes a host device is configured to compile computational workload data for a target application obtained from data gathering devices into an instruction set architecture (ISA) graph to be executed by a plurality of accelerator apparatuses. Each such accelerator includes a plurality of chiplets, each of which includes a plurality of tiles, and each such tile includes a plurality of slices, a central processing unit (CPU), and a DIMC device configured to perform high throughput computations using the ISA graph to process the computational workload. The target application can include natural language processing (NLP), autonomous reasoning/decision-making, video/image processing, cybersecurity/fraud detection, manufacturing/industrial processes, agentic artificial intelligence (AI), or smart cities/Internet of Things (IoT). And the data gathering devices can include a web-scrapers, a dataset readers, a crowdsourcing devices, a sensors, a simulation devices, an IoT network, and others.
In an example, the DIMC architecture and high memory bandwidth can significantly speed up the processing of target computational workloads of a particular application, such as those mentioned previously. The DIMC accelerator system can perform precise and efficient computations of data in a block floating point (BFP) format and can also switch to a lower precision floating point (FP) during runtime. By dynamically switching between precision levels based on real-time analysis of the target workload, the DIMC system can optimize computational efficiency while maintaining the necessary level of accuracy for each step of the workload computation. And with a high memory bandwidth, the DIMC architecture enables a high throughput of workload computations.
The accelerator and chiplet architecture and its related methods can provide many benefits. With modular chiplets, the accelerator apparatus can be easily scaled to accelerate the workloads for transformers of different sizes. The DIMC configuration within the chiplet slices also improves computational performance and reduces power consumption by integrating computational functions and memory fabric. Further, embodiments of the accelerator apparatus can allow for quick and efficient mapping from computational workload data to enable effective implementation of AI applications, and the like.
A further understanding of the nature and advantages of the invention may be realized by reference to the latter portions of the specification and attached drawings.
In order to more fully understand the present invention, reference is made to the accompanying drawings. Understanding that these drawings are not to be considered limitations in the scope of the invention, the presently described embodiments and the presently understood best mode of the invention are described with additional detail through use of the accompanying drawings in which:
The present invention relates generally to integrated circuit (IC) devices and artificial intelligence (AI) systems. More particularly, the present invention relates to methods and device structures for accelerating computing workloads of neural network models (e.g., transformer models, convolution neural network models, etc.). These methods and structures can be used in applications such as natural language processing (NLP), computer vision (CV), and the like. Merely by way of example, the invention has been applied to AI accelerator apparatuses and chiplet devices configured in a PCIe card.
Currently, the vast majority of NLP models are based on the transformer model, such as the bidirectional encoder representations from transformers (BERT) model, BERT Large model, and generative pre-trained transformer (GPT) models such as GPT-2 and GPT-3, etc. However, these transformers have very high compute and memory requirements. According to an example, the present invention provides for an apparatus using chiplet devices that are configured to accelerate transformer computations for AI applications. Examples of the AI accelerator apparatus are shown in
As shown, the AI accelerator apparatuses 101 and 102 are embodied in peripheral component interconnect express (PCIe) card form factors, but the AI accelerator apparatus can be configured in other form factors as well. These PCIe card form factors can be configured in a variety of dimensions (e.g., full height, full length (FHFL); half height, half length (HHHL), etc.) and mechanical sizes (e.g., 1×, 2×, 4×, 16×, etc.). In an example, one or more substrate members 140, each having one or more chiplets, are coupled to a PCIe card. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to these elements and configurations of the AI accelerator apparatus.
Embodiments of the AI accelerator apparatus can implement several techniques to improve performance (e.g., computational efficiency) in various AI applications. The AI accelerator apparatus can include digital in-memory-compute (DIMC) to integrate computational functions and memory fabric. Algorithms for the mapper, numerics, and sparsity can be optimized within the compute fabric. And, use of chiplets and interconnects configured on organic interposers can provide modularity and scalability.
According to an example, the present invention implements chiplets with in-memory-compute (IMC) functionality, which can be used to accelerate the computations required by the workloads of transformers. The computations for training these models can include performing a scaled dot-product attention function to determine a probability distribution associated with a desired result in a particular AI application. In the case of training NLP models, the desired result can include predicting subsequent words, determining contextual word meaning, translating to another language, etc.
The chiplet architecture can include a plurality of slice devices (or slices)
controlled by a central processing unit (CPU) to perform the transformer computations in parallel. Each slice is a modular IC device that can process a portion of these computations. The plurality of slices can be divided into tiles/gangs (i.e., subsets) of one or more slices with a CPU coupled to each of the slices within the tile. This tile CPU can be configured to perform transformer computations in parallel via each of the slices within the tile. A global CPU can be coupled to each of these tile CPUs and be configured to perform transformer computations in parallel via all of the slices in one or more chiplets using the tile CPUs. Further details of the chiplets are discussed in reference to
The CPUs 221 of each tile 210 can be coupled to a global CPU via a global CPU interface 230 (e.g., buses, connectors, sockets, etc.). This global CPU can be configured to coordinate the processing of all chiplet devices in an AI accelerator apparatus, such as apparatuses 101 and 102 of
Further, the chiplet 201 includes a PCIe interface/bus 260 coupled to each of the CPUs 221 in each of the tiles. The PCIe interface 260 can be configured to communicate with a server or other communication system. In the case of a plurality of chiplet devices, a main bus device is coupled to the PCIe bus 260 of each chiplet device using a master chiplet device (e.g., main bus device also coupled to the master chiplet device). This master chiplet device is coupled to each other chiplet device using at least the D2D interconnects 240. The master chiplet device and the main bus device can be configured overlying a substrate member (e.g., same substrate as chiplets or separate substrate). An apparatus integrating one or more chiplets can also be coupled to a power source (e.g., configured on-chip, configured in a system, or coupled externally) and can be configured and operable to a server, network switch, or host system using the main bus device. The server apparatus can also be one of a plurality of server apparatuses configured for a server farm within a data center, or other similar configuration.
In a specific example, an AI accelerator apparatus configured for GPT-3 can incorporate eight chiplets (similar to apparatus 102 of
In an example, the DIMC is coupled to a clock and is configured within one or more portions of each of the plurality of slices of the chiplet to allow for high throughput of one or more matrix computations provided in the DIMC such that the high throughput is characterized by 512 multiply accumulates per a clock cycle. In a specific example, the clock coupled to the DIMC is a second clock derived from a first clock (e.g., chiplet clock generator, AI accelerator apparatus clock generator, etc.) configured to output a clock signal of about 0.5 GHz to 4 GHz; the second clock can be configured at an output rate of about one half of the rate of the first clock. The DIMC can also be configured to support a block structured sparsity (e.g., imposing structural constraints on weight patterns of a neural networks like a transformer).
In an example, the SIMD device 350 is a SIMD processor coupled to an output of the DIMC. The SIMD 350 can be configured to process one or more non-linear operations and one or more linear operations on a vector process. The SIMD 350 can be a programmable vector unit or the like. The SIMD 350 can also include one or more random-access memory (RAM) modules, such as a data RAM module, an instruction RAM module, and the like.
In an example, the slice controller 360 is coupled to all blocks of each compute path 312 and also includes a control/status register (CSR) 362 coupled to each compute path. The slice controller 360 is also coupled to a memory bank 370 and a data reshape engine (DRE) 380. The slice controller 360 can be configured to feed data from the memory bank 370 to the blocks in each of the compute paths 312 and to coordinate these compute paths 312 by a processor interface (PIF) 364. In a specific example, the PIF 364 is coupled to the SIMD 350 of each compute path 312.
Further details for the compute core 310 are shown in
These IMC modules 332 can also be coupled to a block floating point alignment module 334 and a partial products reduction module 336 for further processing before outputting the DIMC results to the output buffer 540. In an example, the input buffer 320 receives input data (e.g., data vectors) from the memory bank 370 (shown in
In addition to the details discussed previously, the SIMD 350 can be configured as an element-wise vector unit. The SIMD 350 can include a computation unit 352 (e.g., add, subtract, multiply, max, etc.), a look-up table (LUT) 354, and a state machine (SM) module 356 configured to receive one or more outputs from the output buffer 340.
The NoC device 342 is coupled to the output buffer 340 configured in a feedforward loop via shortcut connection 344. Also, the NoC device 342 is coupled to each of the slices and is configured for multicast and unicast processes. More particularly, the NoC device 342 can be configured to connect all of the slices and all of the tiles, multi-cast input activations to all of the slices/tiles, and collect the partial computations to be unicast for a specially distributed accumulation.
Considering the previous eight-chiplet AI accelerator apparatus example, the input buffer can have a capacity of 64 KB with 16 banks and the output buffer can have a capacity of 128 KB with 16 banks. The DIMC can be an 8-bit block have dimensions 64×64 (eight 64×64 IMC modules) and the NoC can have a size of 512 bits. The computation block in the SIMD can be configured for 8-bit and 32-bit integer (int) and unsigned integer (uint) computations. These slice components can vary depending on which transformer the AI accelerator apparatus will serve.
As shown in close-up 401, each of the memory-select units 422, 424 includes a memory cell 430 (e.g., SRAM cell, or the like) and a select multiplexer 432. Each of the memory-select units 422, 424 is coupled to a read-write controller 440, which is also coupled to a memory bank/driver block 442. In an example, the read-write controller 440 can be configured with column write drivers and column read sense amplifiers, while the memory bank/driver block 432 can configured with sequential row select drivers.
An input activation controller 450 can be coupled to the activation multiplexer 426 each of the read-write blocks 420. The input activation controller 450 can include precision and sparsity aware input activation register and drivers. The operator unit 428 receives the output of the first memory-select unit 422 and receives the output of this block 450 through the activation multiplexer 426, which is controlled by the output of the second memory-select unit 424. The output of the operator unit 428 is then fed into the computation tree block 410.
The input activation block 450 is also coupled to a clock source/generator 460. As discussed previously, the clock generator 460 can produce a second clock derived from a first clock configured to output a clock signal of about 0.5 GHz to 4 GHz; the second clock can be configured at an output rate of about one half of the rate of the first clock. The clock generator 460 is coupled to one or more sign and precision aware accumulators 470, which are configured to receive the output of the computation tree blocks 410. In an example, an accumulator 470 is configured to receive the outputs of two computation tree blocks 410. Example output readings of the IMC are shown in
Referring back to the eight-chiplet AI accelerator apparatus example, the memory cell can be a dual bank 2×6T SRAM cell, and the select multiplexer can be an 8T bank select multiplexer. In this case, the memory bank/driver block 442 includes a dual-bank SRAM bank. Also, the read/write controller can include 64 bytes of write drivers and 64 bytes of read sense amplifiers. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to these IMC module components and their configurations.
Transformer model variations include those based on just the decoder stack (e.g., transformer language models such as GPT-2, GPT-3, etc.) and those based on just the encoder stack (e.g., masked language models such as BERT, BERT Large, etc.). Transformers are based on four parameters: sequence length(S) (i.e., number of tokens), number of attention heads (A), number of layers (L), and embedding length (H). Variations of these parameters are used to build practically all transformer-based models today. Embodiments of the present invention can be configured for any similar model types.
A transformer starts as untrained and is pre-trained by exposure to a desired data set for a desired learning application. Transformer-based language models are exposed to large volumes of text (e.g., Wikipedia) to train language processing functions such as predicting the next word in a text sequence, translating the text to another language, etc. This training process involves converting the text (e.g., words or parts of words) into token IDs, evaluating the context of the tokens by a self-attention layer, and predicting the result by a feed forward neural network.
The self-attention process includes (1) determining query (Q), key (K), and value (V) vectors for the embedding of each word in an input sentence, (2) calculating a score for from the dot product of Q and K for each word of the input sentence against a target word, (3) dividing the scores by the square root of the dimension of K, (4) passing the result through a softmax operation to normalize the scores, (5) multiplying each V by the softmax score, and (6) summing up the weighted V vectors to produce the output. An example self-attention process 700 is shown in
As shown, process 700 shows the evaluation of the sentence “the beetle drove off” at the bottom to determine the meaning of the word “beetle” (e.g., insect or automobile). The first step is to determine the qbeetle, kbeetle, and vbeetle vectors for the embedding vector ebeetle. This is done by multiplying ebeetle by three different pre-trained weight matrices Wq, Wk, and Wv. The second step is to calculate the dot products of qbeetle with the K vector of each word in the sentence (i.e., kthe, kbeetle, kdrove, and koff), shown by the arrows between qbeetle and each K vector. The third step is to divide the scores by the square root of the dimension dk, and the fourth step is to normalize the scores using a softmax function, resulting in λi. The fifth step is to multiply the V vectors by the softmax score (λivi) in preparation for the final step of summing up all the weight value vectors, shown by v′ at the top.
Process 700 only shows the self-attention process for the word “beetle”, but the self-attention process can be performed for each word in the sentence in parallel. The same steps apply for word prediction, interpretation, translation, and other inference tasks. Further details of the self-attention process in the BERT Large model are shown in
A simplified block diagram of the BERT Large model (S=384, A=16, L=34, and H=1024) is shown in
Further details of the attention head 810 are provided in
The function is implemented by several matrix multipliers and function blocks. An input matrix multiplier 910 obtains the Q, K, and V vectors from the embeddings. The transpose function block 920 computes KT, and a first matrix multiplier 931 computes the scaled dot product QKT/√dk. The softmax block 940 performs the softmax function on the output from the first matrix multiplier 931, and a second matrix multiplier 932 computes the dot product of the softmax result and V.
For BERT Large, 16 such independent attention heads run in parallel on 16 AI slices. These independent results are concatenated and projected once again to determine the final values. The multi-head attention approach can be used by transformers for (1) “encoder-decoder attention” layers that allow every position in the decoder to attend over all positions of the input sequence, (2) self-attention layers that allows each position in the encoder to attend to all positions in the previous encoder layer, and (3) self-attention layers that allow each position in the decoder to attend to all positions in the decoder up to and including that position. Of course, there can be variations, modifications, and alternatives in other transformers.
Returning to
Using a transformer like BERT Large, NLP requires very high compute (e.g., five orders of magnitude higher than CV). For example, BERT Large requires 5.6 giga-multiply-accumulate operations per second (“GMACs”) per transformer layer. Thus, the NLP inference challenge is to deliver this performance at the lowest energy consumption.
Although the present invention is discussed in the context of a BERT Large transformer for NLP applications, those of ordinary skill in the art will recognize variations, modifications, and alternatives. The particular embodiments shown can also be adapted to other transformer-based models and other AI/machine learning applications.
Many things impact the performance of such transformer architectures. softmax function tends to be the critical path of the transformer layers (and has been difficult to accelerate in hardware). Requirements for overlapping the compute, SIMD operations and NoC transfers also impacts performance. Further, efficiency of NoC, SIMD, and memory bandwidth utilization is important as well.
Different techniques can be applied in conjunction with the AI accelerator apparatus and chiplet device examples to improve performance, such as quantization, sparsity, knowledge distillation, efficient tokenization, and software optimizations. Supporting variable sequence length (i.e., not requiring padding to the highest sequence lengths) can also reduce memory requirements. Other techniques can include optimizations of how to split self-attention among slices and chips, moving layers and tensors between the slices and chips, and data movement between layers and FC matrices.
According to an example, the present invention provides for an AI accelerator apparatus (such as shown in
In an example, each of the transformers is configured within one or more DIMCs such that each of the transformers comprises a plurality of matrix multipliers including QKV matrices configured for an attention layer of a transformer followed by three fully-connected matrices (FC). In this configuration, the DIMC is configured to accelerate the transformer and further comprises a dot product of Q KT followed by a softmax (Q KT/square root (dk))V. In an example, the AI accelerator apparatus also includes a SIMD device (as shown in
According to an example, the present invention provides for methods of compiling the data representations related to transformer-based models mapping them to an AI accelerator apparatus in a spatial array. These methods can use the previously discussed numerical formats as well as sparsity patterns. Using a compile algorithm, the data can be configured to a dependency graph, which the global CPU can use to map the data to the tiles and slices of the chiplets. Example mapping methods are shown in
In an example, the embedding E is a [64L, 1024] matrix (L=6 for sentence length of 384), and Ei is a [64, 1024] submatrix of E, which is determined as Ei=E(64i-63):(64i), 1:1024, where i=1 . . . L. Each of the K and Q matrices can be allocated to two slices (e.g., @[SL1:AC3,4]: Kj←Ei×K1 . . . 1024, 1 . . . 64; and @[SL1:AC1,2]: Qi←ei×Q1 . . . 1024, 1 . . . 64). An example data flows through IMC and SIMD modules are shown in the simplified tables of
According to an example, the advancement of artificial intelligence (AI) and large language models (LLMs) facilitated by high-speed inference engines can be improved or even optimized with the present chiplet-based processor and memory hardware. The use of advanced hardware accelerators configured with the chiplets and high-bandwidth memory, for example, enhances computational efficiency, reducing latency and power consumption while increasing throughput. The advancements enable real-time decision-making and automation across a wide range of industries. This disclosure describes applications of high-speed inference engines in domains such as natural language processing (NLP), reasoning, video and image processing, cybersecurity, manufacturing, drug discovery, and autonomous AI agents. By integrating AI-driven capabilities into these sectors, we can achieve improvements in efficiency, automation, and decision-making. This disclosure provides an example of these engines across multiple applications.
In a specific example, the present accelerator apparatuses can use the DIMC architecture and high memory bandwidth to significantly speed up the processing of target computational workloads of a particular application, such as those mentioned previously. The DIMC accelerator system can perform precise and efficient computations of data in a block floating point (BFP) format and can also switch to a lower precision floating point (FP) during runtime. By dynamically switching between precision levels based on real-time analysis of the target workload, the DIMC system can optimize computational efficiency while maintaining the necessary level of accuracy for each step of the workload computation. And with a high memory bandwidth, the DIMC architecture enables a high throughput of workload computations. Specific applications are discussed in further detail below.
As discussed previously, the present techniques can be configured for NLP, such as shown in
In an example, the present techniques are configured for reasoning and decision-making. That is, high-speed inference engines enhance AI-driven decision-making, enabling autonomous reasoning in applications such as financial analysis, legal research, scientific hypothesis generation, and others. Financial markets benefit from AI-based trading systems capable of making real-time risk assessments, while the legal industry leverages AI for contract analysis and regulatory compliance. Scientific research is further accelerated by AI's ability to analyze datasets and simulate experiments, driving innovation in fields like genomics and materials science. The integration of the present chiplet-based architectures within AI hardware allows for distributed processing, increasing computational efficiency and supporting more various models.
In an example, the present techniques are configured for video and image processing, such as for computer vision (CV) applications. In an example, AI-powered video and image processing applications include surveillance, healthcare, and entertainment. In an example, the present high-speed inference engines facilitate real-time video analytics (e.g., facial), enabling security systems to detect anomalies and track individuals. In healthcare, AI-driven medical imaging provides diagnostic accuracy in radiology and pathology. Additionally, generative AI is content creation by automating video editing and generating realistic visual media for entertainment and advertising. The incorporation of high-bandwidth memory in AI accelerators according to the present technique provides data access, reducing bottlenecks in processing large-scale image, video datasets, and others.
In an example, the present techniques are configured for cybersecurity and fraud detection. In an example, cybersecurity using chiplet-based AI-powered inference engines contribute to real-time threat detection and risk mitigation. In example, the present AI systems analyze network traffic patterns to identify cyber threats, while fraud detection models in banking and e-commerce prevent financial crimes. Automated penetration testing further enhances security by identifying system vulnerabilities before exploitation occurs. In an example, the utilization of AI hardware with chiplets enables rapid (e.g., nearly real-time) anomaly detection and real-time response, ensuring cybersecurity defenses.
In an example, the present techniques are configured for manufacturing and advanced Industry 4.0 (a.k.a. the Fourth Industrial Revolution or 4IR). Industry 4.0 can be defined as the integration of intelligent digital technologies into manufacturing and industrial processes and includes a set of technologies that include industrial IoT (Internet of Things) networks, AI (Artificial Intelligence), Big Data (e.g., large datacenters), robotics, automation, among other technologies based upon compute. In an example, AI-driven manufacturing is provided with industrial processes through predictive maintenance, robotic automation, and supply chain optimization. In an example, the present chiplet-based high-speed inference engines power intelligent robotics for assembly and quality control, and reducing or event minimizing human intervention. In an example, predictive maintenance systems analyze sensor data to prevent equipment failures, reducing downtime and operational costs. AI-powered logistics and inventory management enhance supply chain efficiency by dynamically adjusting production schedules. The present chiplet-based accelerators in industrial AI systems provide execution of complex manufacturing workflows by efficiently managing parallel computations in an example.
In an example, the present techniques are configured for drug discovery and healthcare. In an example, the present techniques can be applied to AI-driven drug discovery, where high-speed chiplet-based inference engines facilitate molecular simulations and compound screening. In an example, AI algorithms predict drug efficacy and improve and/or optimize clinical trial processes, accelerating the development of new treatments and discoveries. In an example, personalized medicine, enabled by AI's analysis of patient genomics, allows for tailored therapies with improved outcomes. In healthcare, for example, AI-powered diagnostic tools assist medical professionals by analyzing patient data and identifying potential health risks. In an example, the present AI accelerators provide for bioinformatics processing, enabling faster analysis of large-scale genomic datasets.
In an example, the present techniques are configured for agentic AI applications. In an example, the present agentic AI systems provide for AI-driven automation, characterized by autonomous decision-making and task execution with no or minimal human oversight. The intelligent agents autonomously conduct research, manage business operations, and optimize supply chains, etc. In an example, AI-powered legal agents provide in contract negotiation and compliance monitoring, while autonomous AI-driven customer service agents handle inquiries and support requests without human intervention. In an example, an emergence of AI-powered CEOs and business decision-makers are enabled with agentic AI in strategic planning, enterprise management, and other applications. In an example, the present hardware accelerators including chiplets and high-bandwidth memory further enhance these applications by enabling real-time learning and adaptation.
In an example, the present techniques are configured for smart cities and IoT. In an example, smart cities are configured using the present AI-powered inference engines to optimize urban infrastructure, traffic management, energy distribution, among other applications. In an example, AI-driven traffic control systems reduce congestion by analyzing real-time data, while smart grid solutions enhance energy efficiency through predictive analytics. In an example, environmental monitoring applications utilize AI to track pollution levels and model climate change impacts, enabling data-driven policy decisions. In an example, the present AI hardware accelerators provide for low-latency processing, making real-time optimizations feasible for large-scale IoT deployments.
The compiler stack 1520 includes at least a handles layer 1522 and an instruction set architecture (ISA) graph layer 1524. The host runtime 1512 can use the handles layer 1522 to determine references to resources for program, workload, or model; and the host runtime 1512 can use the ISA graph layer 1524 to translate the program, workload, or model into an ISA graph. For example, the ISA graph layer 1524 can translate a computation graph representing a target neural network model workload in machine code.
The workload preprocessor 1530 can be configured to determine a plurality of workload parameters using the translated computation graph from the ISA graph layer 1524. Afterwards, the host runtime 1512 can use the compiler stack 1520 to issue commands for the workload parameters and instructions to the execute stack 1540, which sends these commands to a target hardware. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to configuration of the host computing device 1510 and the associated software system.
The host computing device 1510 is also coupled to one or more data gathering devices 1580. Each such data gathering device 1580 can be configured to obtain data for one or more target applications (e.g., for a program, workload, or model) and to send the data to the host computing device 1510 to be translated as an ISA graph to be processed via the plurality of accelerator apparatuses 1512. Depending on the application, these data gathering devices 1520 can take many forms and implement different methods. To gather text data for NLP or image/video data for CV, the data gathering device 1520 can include a web-scraping device, a dataset reader device, a crowdsourcing device, and the like. Similar methods can be used to gather data for financial analysis, legal research, scientific research, and other fields as well.
For gathering real-time data, the data gathering device 1520 can include a variety of sensor devices, such as medical imaging devices used to capture patient data, equipment monitoring devices used to determine manufacturing data, network scanning devices used to detect abnormal transactions, and others. The data gathering device 1520 can also accumulate data from a network of devices, vehicles, appliances or other physical objects embedded with sensors, software and network connectivity (i.e., IoT). Further, the scale of such a network of devices can be expanded to smart city networks that manage traffic flow, energy systems, waste collection, and the like.
In addition, the data gathering device 1520 can be configured to analyze collected data to determine fraudulent activity or run simulations using collected data to drive autonomous reasoning and decision-making. For example, traffic data can be analyzed and simulated to assist in autonomous vehicle pathing, supply chain data can be analyzed and simulated to assist in delivery routes and manage inventory, medical data can be analyzed and simulated to assist in personalized treatment schedules or even assist in new discoveries, etc. Of course, there can be other variations, modifications, and alternatives the devices and methods used to gather data for the present DIMC accelerator system.
In an example, the target hardware includes a plurality of accelerator apparatus 1550 with a plurality of chiplet devices 1560 coupled to a CPU 1562, which can include a global CPU and a plurality of local CPUs. The chiplet CPU 1562 is coupled to a plurality of matrix compute apparatuses 1570 via their crossbar devices 1572, each of which is coupled to at least a compute device 1574 (e.g., DIMC device) and a Single Input, Multiple Data (SIMD) device 1576. In an example, the compiler commands are sent to accelerator apparatuses 1550, which can be used to program the CPU 1532 (or CPUs) and connected elements of matrix compute apparatus 1570 via the crossbar device 1572. The AI accelerator apparatus 1550, the chiplet devices 1560, and the matrix compute apparatus 1570 can be configured similarly to any of the previously discussed examples.
In a specific example, the host device 1510 and the plurality of accelerator apparatuses 1512 can be configured within a server device or within multiple connected server devices of a server system. In such cases, the host device 1510 can be configured to coordinate the operations of the accelerator apparatuses 1512 in the server device or a host server can be configured to coordinate the operations of the server devices in the server system.
Although the matrix compute apparatus 1570 is configured within a chiplet device 1560 in an AI accelerator apparatus 1550 in this example, the host computing device 1510 can also be configured send the compiler commands to an independent chiplet device with matrix compute apparatuses or a server system having a plurality of AI accelerator apparatuses. For example, the server system can include a plurality of AI accelerator PCIe card devices coupled to a plurality of switches, each of with is coupled to one or more server CPUs. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to this workload transfer configuration.
The integration of high-speed inference engines into AI applications is driving advancements across various industries. The adoption of advanced hardware accelerators including the present chiplets and, e.g., high-bandwidth memory, enhances computational efficiency, supporting larger models and reducing inference latency. In an example, by enhancing decision-making, automation, and predictive analytics, these systems enable organizations to operate more efficiently and intelligently. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to these applications.
While the above is a full description of the specific embodiments, various modifications, alternative constructions and equivalents may be used. As an example, the AI accelerator apparatus and chiplet devices can include any combination of elements described above, as well as outside of the present specification. Therefore, the above description and illustrations should not be taken as limiting the scope of the present invention which is defined by the appended claims.
The present application is a continuation-in-part of U.S. patent application Ser. No. 18/493,616, filed Oct. 24, 2023; which is a continuation of U.S. patent application Ser. No. 17/538,923, filed Nov. 30, 2021 (now U.S. Pat. 11,847,072).
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17538923 | Nov 2021 | US |
| Child | 18493616 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18493616 | Oct 2023 | US |
| Child | 19076153 | US |
