Patent Grant
12190086
Use and implementations of machine learning (ML) and artificial intelligence (AI) methods on electronic devices has become ubiquitous. The design of a hardware architecture of the electronic devices, whether a processor, a programmable logic, a dedicated hardware such as application specific integrated circuit (ASIC), or a dedicated ML hardware, often goes through various optimization and compilation processes.
A compilation process or a compiler generates low-level executable instructions (in binary) from one or more high-level code and identifies hardware resources to execute the low-level executable instructions. The compilation process may include quantization, reduction in mathematical precision, mapping of the application (e.g., a neural network) to a specific number of processing tiles of the hardware. In general, the compiler maps data, e.g., the network tensor weight, the network tensor bias constants, the network tensor input and output for each network layer, etc., to particular memories and generates the executable code associated therewith. For example, the compiler decides on which processing tile and which processing unit (e.g., POD and/or PE) of the tile of a multi-core system will be processing certain data. As another example, the compiler may decide that certain data is to be processed by a central processing unit as opposed to a tile within a ML hardware.
In order to perform an inference run of an ML model on an ML-specific hardware (e.g., a hardware-based ML/AI accelerator) and/or a general-purposed CPU, a binary file (e.g., a set of target specific low-level instructions and/or model-specific data sections) has to be generated. This is typically done in an ahead-of-time (AOT) fashion by a compiler. State of the art ML inference models are often large and complex, containing many different operators. In some embodiments, these models may be represented as (model) graphs containing many nodes (i.e. layers) which are operating on large multi-dimensional tensors.
In order to achieve best performance (i.e. low latency and/or high throughput) an inference may be performed on a group of compute devices, which might be a set of general-purpose CPUs, ML-specific hardware accelerators, or a combination of both. A common compiler approach is to generate code for the CPUs and/or the accelerators by analyzing and processing each individual operator (i.e. layer) of a graph. As such, a tidily-coupled data and control flow with significant overhead may be generated for each layer. Unfortunately, analyzing and processing each individual layer is not only time consuming but also power and process intensive given the complexity and large number of layers. In some conventional systems, a model graph may be partitioned manually to generate two or more operator-based sub-graphs with the control and data flow. Unfortunately, manually determining the data and control flow is not only laborious but also unpractical given the complexity and large number of layers.
Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. It is noted that, in accordance with the standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.
The following disclosure provides many different embodiments, or examples, for implementing different features of the subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. In addition, the present disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed.
Before various embodiments are described in greater detail, it should be understood that the embodiments are not limiting, as elements in such embodiments may vary. It should likewise be understood that a particular embodiment described and/or illustrated herein has elements which may be readily separated from the particular embodiment and optionally combined with any of several other embodiments or substituted for elements in any of several other embodiments described herein. It should also be understood that the terminology used herein is for the purpose of describing the certain concepts, and the terminology is not intended to be limiting. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood in the art to which the embodiments pertain.
In general, a compiler is configured to go through multiple levels or stages during compilation of high-level code into low-level executable instructions on a hardware. At each level (i.e. stage), the compiler needs to make one or more decisions on compilation, e.g., how to map the data to be processed and to which memory blocks, decision on a particular processing tile to execute the executable code for a particular data, etc. It is appreciated that references to level of backend compiler (discussed later in the application) refers to stages of compilation by the backend compiler. At each level, the compiler in addition to generating the low-level executable code may also generate multi-layered structured metadata for that stage that reflects the action(s)/decision(s) being made by the compiler, e.g., mapping of data to memory blocks, precision, quantization, processing tile to perform a particular task/instruction, dimension reordering, copying across processing tiles, etc. It is appreciated that the compiler action(s)/decision(s) occur first in order for the high-level code to be compiled into low-level executable instructions.
It is appreciated that in order to support inference runs, the ML model may be split across multiple devices, e.g., an accelerator (hereinafter ML hardware) and a general processor such as a central processing unit (CPU) or graphics pipeline unit (GPU), etc. In other words, the original model graph may be partitioned into sub-graphs such that certain sub-graphs are executed by an ML hardware and certain sub-graphs are executed by a general processor. The sub-graphs are mapped to specific target hardware. In general, the type of operations within a sub-graph and/or the amount of processing/computation may be used to determine a hardware target selection, e.g., ML hardware as opposed to general processor. Optimized code may then be generated for each sub-graph for its specific target hardware and the control and data flow between multi-operator sub-graphs is determine.
A new approach is proposed for splitting/dividing/partitioning the original model graph into sub-graphs based on the type of operation and/or latency, as nonlimiting examples. In some embodiments, the compiler may recognize operators (i.e., network layers) of the graph and whether the recognized operators are supported by the ML hardware or not. Any operator of the graph that is unsupported by the ML hardware may be partitioned into a sub-graph for execution by a general processor. Additionally, operators of the graph that are supported by the ML hardware may still be partitioned and split into a sub-graph for execution by a general processor to reduce latency, data movement between two sub-graphs, etc. Other operators are formed into a sub-graph for execution by the ML hardware.
In some embodiments, once the sub-graphs are identified by the compiler, the input output node pairs of the sub-graphs are generated, e.g., output of one sub-graph from a general processor to input of one sub-graph of an ML hardware for example. In other words, the generated input/output node pairs to connect the sub-graphs is a representation of the original model graph.
The compiler performs target specific transformations and optimizations on each sub-graph. It is appreciated that because the target associated with each sub-graph may be different, e.g., ML hardware, ML emulator, general processor, etc., their resources and/or architecture are also different, e.g., memory, processing units, etc. As such, each sub-graph may undergo a different transformation and/or optimization depending on the target that will be executing the code generated for the sub-graph.
Partitioning a graph into sub-graphs in an automated fashion and in a more intelligent fashion, as described, based on respective target capabilities and resources improves the processing speed, reduces the required resource, and/or reduces latency. In other words, the original model graph is partitioned based on analysis of optimization/transformation that closely maps to the target hardware, e.g., ML hardware, ML emulator, general processor, etc., thereby enabling detailed computation and communication estimates, which is based on resources such as processing power and data movement, etc. Expressing the original model as sub-graphs, as described, enables the compilation process to be automated such that the model can be executed (i.e., inferred) in a time efficient and resource efficient manner. In some embodiments, a self-tuning capability to perform graph partitioning, compiling, and performing inference run analysis. In yet other embodiments, manual interference by a user may be provide in addition to the automated compilation and partitioning. As such, partitioning the original model graph into sub-graphs, as described, reduces latency by reducing unnecessary memory (or hierarchy) access, reducing unnecessary data movement, etc., in comparison to the conventional system.
It is appreciated that one or more components of the system may run on one or more computing units or devices (not shown) each with software instructions stored in a storage unit such as a non-volatile memory of the computing unit for practicing one or more processes. When the software instructions are executed, at least a subset of the software instructions is loaded into memory by one of the computing units, which becomes a special purposed one for practicing the processes. The processes may also be at least partially embodied in the computing units into which computer program code is loaded and/or executed, such that, the computing units become special purpose computing units for practicing the processes. For non-limiting examples, the compiler may take certain actions and make certain decisions to reduce one or more of data movement, data conversions, storage usage, computation (or duplication of computation), and communication (by duplicating compute if beneficial), etc. The ML hardware may be a dedicated hardware including one or more microprocessors and/or on-chip memory (OCM) units storing the data and/or the set of low-level instructions compiled from the high-level code by the compiler to perform one or more ML operations. At runtime, the ML hardware is configured to retrieve the set of low-level instructions and/or data from the compiler and execute the set of low-level instructions to perform the one or more ML operations according to the set of low-level instructions. For a non-limiting example, the ML-specific hardware can be but is not limited to an inference engine, which is configured to infer and identify a subject via an inference operation from data input according to the ML network model.
In some embodiments, the multi-layered structured metadata may include comments in a generated code that is human readable. It is appreciated that the multi-layered structured metadata may be readable or executable by the compiler or another software in some embodiments. In some embodiments, the multi-layered structured metadata may be stored in one or more files or it may be included as part of the assembly code.
In some ML applications, the multi-layered structured metadata may be generated by the compiler automatically and it may include information such as location of data, e.g., tensor, which is a nested data structure widely used for ML applications, in various memory blocks within the layer. It is appreciated that the multi-layered structured metadata may also provide information regarding the memory location (e.g., host memory, device memory, chip memory, etc.) for each tensor at any given stage in the network execution. Accordingly, expected memory dumps may be generated based on the original tensor that can be used for comparison to memory dumps of the actual hardware, software emulator or hardware emulator runs. As such, the low-level code/instructions can be verified and debugged based on the metadata generated by the compiler.
The multi-layered structured metadata at each layer may also include information regarding certain actions (i.e. decisions) by the compiler, e.g., precision, orientation, split across distributed blocks, quantization, processing tile to perform a certain operation, etc. In some embodiments, the multi-layered structured metadata may describe transformation associated with data being processed, e.g., transformation associated with tensors such as quantization, reducing precision, dimension reordering (e.g., conversion to/from width/height/channel (WHC) from/to channel/height/width (CHW)), splitting or copying across processing tiles, or other compile time optimizations that may result in reduced execution time of the compiled code. It is appreciated that references to tensors are provided for illustrative purposes throughout the application and should not be construed as limiting the scope of the embodiments.
In some embodiments, the multi-layered structured metadata at each layer may be used for optimization purposes, e.g., reducing data movement, reducing storage, reducing duplicate computations, reducing communication by duplicating computing if beneficial, reducing data conversions, etc. In some embodiments, the multi-layered structured metadata generated from one layer may be input into a subsequent layer and it may be relied upon by the compiler itself in order to optimize the compilation and decisions on how to process data and perform operations at the subsequent layer in an optimized fashion, e.g., by reducing data movement, reducing storage, reducing duplicate computations, reducing communications, reducing data conversions, etc.
It is appreciated that the compiler automatically generates the multi-layered structured metadata because the compiler is aware of the system requirements, e.g., channel/height/width as opposed to height/width/channel and number of bytes needs due to alignment needed in hardware. Moreover, the compiler is aware of the hardware architecture, e.g., ML hardware (number of processing tiles, etc.), and as a result automatically generates the multi-layered structured metadata for each layer and decisions that the compiler is making with respect to how to process/map processing and data to the hardware. As such, the multi-layered structured metadata once generated can be used for debugging, verification, or optimization purposes.
Since the overall number of low-level instructions to be executed on the ML hardware remains the same and no additional instructions are introduced because the multi-layered structured metadata is generated as comments that are not executed or stored in one or more files, the instruction flow and the executables of the application are not adversely affected or disturbed for performance profiling purposes. As a result, accurate performance profiling and debugging of the application can be achieved as well as optimization if desired.
Although an instruction set architecture (ISA) is used as a non-limiting example of the low-level instruction format to illustrate the proposed approach in the embodiments described below, it is appreciated that the same or similar approach is equally applicable to other types of low-level instructions. It is also appreciated that an ML hardware (e.g., inference engine) is used as a non-limiting example of the hardware where the low-level instructions are executed to illustrate the proposed approach in the embodiments described below, it is appreciated that the same or similar approach is equally applicable to other types of hardware or hardware simulator to support generating a metadata using a compiler that can ultimately be used for verification, debugging, and optimization purposes. Moreover, it is appreciated that an ML-related operation or function is used as a non-limiting example of the application of the high-level code to illustrate the proposed approach in the embodiments described below, it is appreciated that the same or similar approach is equally applicable to other types of software applications including but not limited to firmware, hardware simulation software, or register transfer level (RTL) simulation software, to support the compiler generating a metadata.
In the example of
In the example of
Here, the high-level code is a software code written through a commonly-used high-level programming language. For a non-limiting example, the high-level functions of the application or ML operation can be a dense and/or regular operation, e.g., a matrix operation such as multiplication, matrix manipulation, tanh, sigmoid, etc. For another non-limiting example, the high-level functions of the application or ML operation can be a sparse or irregular operation, e.g., memory transpose, addition operation, operations on irregular data structures (such as trees, graphs, and priority queues), etc. In some embodiments, the high-level code of the application may include one or more library function calls to an ML library 180. For a non-limiting example, the compiler 120 may call a library function to perform a matrix-matrix-multiplication of two matrices of given sizes and the ML library 180 returns the set of low-level instructions that are needed to perform this library function, wherein the set of low-level instructions includes one or more of loading data from a memory (e.g., OCM) into registers, executing dot-product, and storing the data back into the memory.
In some embodiments, the set of low-level instructions are in the format of ISA designed for efficient data processing covering, for non-limiting examples, one or more of different addressing modes, native data types, registers, memory architectures, and interrupts. In some embodiments, the ISA is a predominantly asynchronous instruction set, wherein each instruction in the ISA format programs a state-machine, which then runs asynchronously with respect to other state machines. It is appreciated that a series of instructions in the ISA format do not necessarily imply sequential execution. In some embodiments, the ISA provides separate synchronizing instructions to ensure order between instructions where needed. In some embodiments, when being executed on the ML hardware 160, the set of low-level instructions in the ISA format program the ML hardware 160 by one or more of: (i) programming one or more input data streams to the ML hardware 160; (ii) programming one or more operations to be performed on the input data streams; and (iii) programming one or more output data streams from the ML hardware 160.
In some embodiments, the compiler 120 is configured to generate additional information to further correlate the high-level function to one or more layers of a neural network used for machine learning applications. For non-limiting examples, the neural network can be but is not limited to one of a convolution neural network (CNN), a recurrent neural network (RNN), a gradient boosting machine (GBM), and a generative adversarial neural network. For non-limiting examples, the additional information includes but is not limited to which tasks of the high-level function belong to a specific neural network layer as well as which neural network layer the high-level function belongs to.
Once the set of low-level instructions has been compiled from each high-level function, the compiler 120 is configured to stream the set of low-level instructions as well as data received from the host for the application to the ML hardware 160 for execution. In the example of
In order to generate the low-level instructions from high-level functions/code, the compiler 120 having knowledge of the ML hardware 160 architecture and software/system requirements makes certain decisions and performs certain operations in order to generate low-level instructions that are as efficient and as optimized as possible (e.g., from hardware perspective and/or software perspective). For example, the compiler 120 may take certain actions and make certain decisions to reduce data movement, to reduce data conversions, to reduce storage usage, to reduce computation (or duplication of computation), to reduce communication (by duplicating compute if beneficial), etc. A nonlimiting and non-exhaustive list of decisions being made by the compiler 120 in addition to the above includes but is not limited to:
Referring now to
while the bias data (channel=2 data of the weight tensor) may be a matrix
The memory layout when stored is illustrated in
It is appreciated that, in some embodiments, the compiler 120 has knowledge of the architecture of the ML hardware 160 and its requirements, e.g., determining that conversion to HWC format is needed. Referring now to
It is appreciated that the conversion and the information regarding the memory layout for example is encapsulated within the multi-level structured metadata 122 being generated by the compiler 120. It is similarly appreciated that other decisions or operations performed by the compiler 120 is captured within the multi-level structured metadata 122 that can be used to optimize the operation of the compiler, debug the code, and/or verify the code.
Referring now to
It is appreciated that the backend compiler 220 may include multiple levels according to some embodiments. For example, the backend compiler 220 may include a first level backend compiler 222, a second level backend compiler 224, a third level backend compiler 226, and Nth level backend compiler 228, as illustrated in
It is appreciated that at each level backend compiler one or more structure metadata is generated in addition to the specific tasks/operations being performed by the backend compiler. For example, the first level backend compiler 222 receives the intermediate data 212 and performs transformation/optimization, e.g., target specific fusing/composition, specific data/weigh/output layout format adjustment (an example of the data/weight/output layout format adjustment is illustrated in
In some embodiments, the second level backend compiler 224 in some nonlimiting examples performs a specific multi-layer based optimization (as an example and described in greater detail in
In some nonlimiting examples, the computation and data are moved by the compiler 120 from inference time to compiler time once in compilation in order to reduce computations and data movements at inference runtime. It is appreciated that the second level backend compiler 224 may use a model, e.g., roofline model, given the target hardware configuration (i.e. ML hardware 160) and data layouts, at compilation time to estimate specific runtime performance. It is appreciated that the second level backend compiler 224 similar to the first level backend compiler 222 also generates a structured metadata 225. The structured metadata 225 provides information regarding the operations/decisions performed/made by the second level backend compiler 224. It is appreciated that the output of the second level backend compiler 224 is input to the third level backend compiler 226.
In some embodiments, the third level backend compiler 226 may transform the layer subgraph from the structured metadata 225 to primitive subgraph where each of the primitives may describe a certain algorithmic procedures. In some embodiments, the primitives may perform only computational tasks, only communication tasks between tiles or between tiles and double data rate (DDR), only synchronization tasks, or any combination thereof. For example, the matrix-matrix-multiply primitives LAMM and SAMM are two different computational primitives that are optimized for different matrix shapes. While “all to all” is a communication primitive, as are halo, rebalance and forward gather which are primitives that perform data movements on distributed tensor data. An example of a combined communication and computation primitive is the flattening overlap. Examples of other algorithmic procedures may include MAXPOOL, direct convolution, padding, scratch, etc. The third level backend compiler 226 determines mapping, resource allocation, and parallelism that may be applied on a layer by layer case. For example, the third level backend compiler 226 may determine whether to split input/output on tiles, split weight/bias on tiles, combination of split input/output and weight/bias and serialization on tiles, overlap primitives on tiles, use LAMM as opposed to SAMM1/SAMM2 (described in
It is appreciated that in some embodiments the third level backend compiler 226 may receive data from a source other than other backend compilers. For example, the third level backend compiler 226 may also receive a strategy indicated by a user (i.e. user strategy) in addition to receiving the output from the second level backend compiler 224, as illustrated below. It is appreciated that the strategy may be an external strategy generated by an analysis/profiling tool which is run external to the compiler flow. It is appreciated that in the following strategy, information for each layer of the fused graph is give. Details such as the type of operation, e.g., convolution or maxpool, the corresponding first and last ONNX operator of the original ONNX graph, the selected strategy and the externally provided strategy hints are given. For the first layer, in this example, the strategy of splitting the input and output among the tiles is applied while the weights and bias tensors are being duplicated. For this example, the hints are matching the applied strategy, but it does not need to be.
It is appreciated that the third level backend compiler 226 similar to the first and second level backend compilers 222 and 224 also generates a structured metadata 227. The structured metadata 227 provides information regarding the operations/decisions performed/made by the third level backend compiler 226. It is appreciated that the output of the third level backend compiler 226 is input to the subsequent level backend compiler(s). It is appreciated that the structured metadata 227 generated by the third level backend compiler 226 may be fed back into the third level backend compiler 226 in order to reduce the number of primitives (an example is described in
A nonlimiting example of the structured metadata 227 is shown below for illustration purposes. Below the structured metadata for DDR or OCM regions and inputs, flattening addresses, weights and bias outputs, etc., is shown. The following is a nonlimiting example of a structured metadata for convolution layer. The first part of the structured metadata provides information regarding the input, weight, bias, and outputs tensors regarding shape, format, name in original network graph and local memory allocation. Moreover, pertinent information regarding the convolution operation is given such as stride in this example. The second section of the structure metadata here provides the sequential call list of the calls to the ML-library and the specific arguments.
Another example of a structured metadata 227 is shown below for illustration purposes. In the example below the strategy to map to 8 tiles is illustrated and illustrates how the input tensor is split among the tiles, rebalanced, haloed, and how the output tensors are split after computation. In this nonlimiting example, the maxpool layer is executed in parallel on 8 tiles. Here, the structured metadata provides information regarding the applied strategy and the mapping information of the data across 8 tiles when a row-wise split is applied. Moreover, the information includes the number of rows including padded rows as well as the number of halo rows on each tile.
Other level backend compilers may perform other operations and make other decisions. For example, other backend level compilers may perform functions based on specified attributes for the primitives, e.g., forming a set of common ML library and application peripheral interface (APIs), in order to generate ISA tasks codes to fulfill the need for all primitives for the ML hardware 160. In some nonlimiting examples, based on specified ML library APIs with their arguments, the particular level backend compiler may generate the appropriate ISA task codes to utilize the ML hardware 160 in a streaming fashion, as an example. It is appreciated that for each ML library API with its arguments, a per ML library API roofline model is used, at the time that the code is being generated, to estimate the target specific runtime performance and to monitor and check performance with respect to each ISA instruction, and/or to determine boundary violations (attributes that lead to memory wrap around or data hazard ISA instructions being produced due to memory address reuse). It is appreciated that at the time that the compiler calls the ML library API, the arguments to the library call have all the pertinent information regarding tensors and the arithmetical operations to be performed. Thus, a roofline model can be computed for this specific API call which will provide an estimate target specific runtime of these arithmetical operations. Accordingly, the compiler can iteratively decide on which API to call in cases where multiple different APIs are available to perform the same arithmetical operations. In some nonlimiting examples, other operations/decisions may include a model binary analyzer subcomponent that performs an overall analysis to identify potential problems in the low-level instructions (i.e. generate model binary), e.g., ill-formed OCM memory overlapping between ISA tasks/instructions, data hazard between consumer-producer tasks, etc. It is appreciated that these other level backend compilers may also generate their respective structured metadata that provide information regarding the operations/decisions performed/made by their respective level backend compiler. It is appreciated that the generated structured metadata and other output from other backend compilers are input to the Nth level backend compiler 228 as an input.
The Nth level backend compiler 228 in some nonlimiting examples performs ahead of time (AOT) inference on the ML hardware 160 accelerators and/or other processing units (e.g., CPU). In some examples, the Nth level backend compiler 228 generates performance statistics for the inference run associated with the ML hardware 160. The Nth level backend compiler 228 may decide on whether to perform AOT on the ML hardware 160, on its software emulator, or on a full machine emulator with the ML hardware 160 submodules. Based on the performance statistics, certain aspects of the system may be optimized, e.g., calibrate and optimize the generated code, the primitives, etc. It is appreciated that the Nth level backend compiler 228 also generates the low-level instructions for execution by the ML hardware 160. The Nth level backend compiler 228 also generates a structured metadata 229. The structured metadata 229 provides information regarding the operations/decisions performed/made by the Nth level backend compiler 228.
It is appreciated that even though not shown, one or more outputs of a given level backend compiler may be fed as a feedback loop into itself or to a preceding level backend compiler. For example, in some nonlimiting examples the output of the third level backend compiler 226 may be fed back into itself for optimization while the output of the Nth level backend compiler 228 may be fed back into the first level backend compiler 222, the third level backend compiler 226, etc. It is also appreciated that one or more of the level backend compilers may receive additional data from a source other than other level backend compilers.
Referring now to
In this nonlimiting example, the ML hardware (i.e., accelerator) may be integrated with a ML compiler framework such as TVM that supports Bring Your Own Codegen (BYOC), thereby enabling the TVM ecosystem to become available to users of the ML hardware. In this nonlimiting example, the ML hardware may be an ML/AI inference accelerator (MLIP) and may be embedded in a processor, e.g., CPU, GPU, field programmable gate array (FPGA), etc. In other words, the ML model, e.g., pre-trained network, that is received may be split across multiple devices, e.g., an accelerator (hereinafter ML hardware) and a general processor such as a CPU or GPU, etc. In one nonlimiting example, the ML model may be received (e.g., loaded) and processed by the frontend compilation and code-gen AOT.
The pre-trained network of the ML model, mentioned above, is shown in
The example of the pre-trained network of the ML model in
In some embodiments, the compiler may process the received ML network model and identify a plurality of well-defined boundaries for input and output in the ML network model based on a set of primitives. It is appreciated that the set of primitives may refer to a set of functions, units, and/or operators that are basic, generic, and essential (in contrast to specialized) to the ML operations of the ML network model. It is appreciated that each of the primitives may invoke one or more library function calls to an ML library to generate low-level instructions to be executed on a hardware. For a non-limiting example, a library function may be called to perform a matrix-matrix-multiplication of two matrices of given sizes and the ML library returns the set of low-level instructions that are needed to perform this library function, wherein the set of low-level instructions includes one or more of loading data from a memory, e.g., OCM, into registers, executing dot-product, and storing the data back into the memory.
Once the plurality of well-defined boundaries are identified, the compiler partitions the ML network model into a plurality of units/layers/graph/sub-graphs based on the plurality of well-defined boundaries. In some embodiments, the boundaries are defined by one or more leaf nodes of the graphs, e.g., graph 252, where each leaf node corresponds to an ending edge of a layer (which corresponds to one or more nodes) created by the compiler by executing one or more primitive functions/operators on one or more hardware components. In some embodiments, the well-defined boundary of the layer corresponds to executing last primitive function/operator in a graph on the hardware components for the layer. In some embodiments, the functionality of this last primitive function/operator can also be mapped back to its corresponding one or more ML operations in the ML network model.
The compiler then generates an internal/interim representation for each of the plurality of units/nodes of the graph 252. In this nonlimiting example nodes 1-9 and a-p are executable nodes of an ML layer. The compiler has knowledge of the architecture of the ML hardware, architecture of general processing units such as CPU, GPU, FPGA, etc., respective configurations, and software/system requirements etc. In some embodiments, the type of operations within a graph and/or the amount of processing/computation may be used to determine a hardware target selection, e.g., ML hardware as opposed to a general processor. It is appreciated that the compiler may split the original model graph into sub-graphs based on the type of operation and/or latency, as nonlimiting examples. In some embodiments, the compiler may recognize operators (i.e., network layers) of the graph and whether the recognized operators are supported by the ML hardware or not. Any operator of the graph that is unsupported by the ML hardware may be flagged by the compiler and partitioned into a sub-graph for execution by a general processor. In this nonlimiting example, the graph 252 shows executable nodes where nodes (i.e. e, j, k, n, o) that are not supported or unsuited for execution on the ML hardware 160 are shaded for illustration purposes. According to some embodiments, operators of the graph that are supported by the ML hardware may still be partitioned and split into a sub-graph for execution by a general processor to reduce latency, data movement between two sub-graphs, etc. In other words, the compiler may determine that unsupported operators/nodes that have been flagged along with some unflagged nodes should be split into a sub-graph for execution by a general processor to improve processing and achieve certain efficiencies, e.g., reduce data movement, reduce latency, etc. In some embodiments, the compiler is configured to estimate the computing cost of each node (e.g., when executed on the ML hardware as opposed to a general processor) and communication cost for data movement (e.g., between the ML hardware and the general processor). The compiler may split the graph into sub-graphs based on the estimated computing cost, etc., in order to achieve certain efficiencies in processing the ML model. Operators that are supported by the ML hardware and that can be executed efficiently by the ML hardware are formed into a different sub-graph for execution by the ML hardware. It is appreciated that it may be desirable to split the graph into the least number of sub-graphs, e.g., 2 sub-graphs.
In this nonlimiting example, the first subgraph 254 is created based on nodes that are suited for execution on the ML hardware 160 layer (e.g., not only supported but also efficient to be executed by the ML hardware 160). The second subgraph 256 is created based on nodes that contains nodes that are better suited for execution on a processor other than the ML hardware 160. In this example, the first level backend compiler 222 has determined that even though nodes d, f, h, m, l, and p are executable on the ML hardware 160 (e.g., native to the ML hardware 160), it is more efficient for them to be executed on a non-ML hardware component, e.g., CPU, along with other nodes, e.g., e, j, k, n, o. It is appreciated that the nodes e, j, k, n, and o may be nonnative to the ML hardware 160 and as such better suited to be executed by a different component. Efficiency on the node being executed on the ML hardware or a general processor may be determined based on latency, amount of data movement, etc. Also, since nodes f, h, l, m, and p are intertwined in such a way with nodes e, j, k, n, and o that increases the overhead (e.g., storage, data movement, processing, etc.) the first level compiler 222 may determine that it is more efficient for those node layers to be executed with nodes e, j, k, n, and o. As such, nodes f, l, h, m, and p are defused back to original intermediate data (i.e. intermediate representation) operations. The defused nodes are illustrated as f″, l″, h″, m″, and p″. Moreover, it is appreciated that the output from nodes c, i, and d from subgraph 254 is input to subgraph 256.
As described above, the first level backend compiler 222 may make a determination to split the graph of nodes 252 to two subgraph nodes 254 and 256, as illustrated on the right hand side, the input output node pairs of the sub-graphs are generated, e.g., output of one sub-graph from a general processor to input of one sub-graph of an ML hardware for example. In other words, the generated input/output node pairs to connect the sub-graphs is a representation of the original model graph. In some embodiments, the subgraph nodes 254 will be executed by the ML hardware 160 and the subgraph nodes 256 will be executed by a processing component other than the ML hardware 160, e.g., a CPU. As such, the internal representation of the sub-graph 254 is mapped to the ML hardware or ML software emulator and the internal representation of the sub-graph 256 is mapped to a general processor. The ML model that is split into sub-graphs is shown in
As described above, the ML hardware is a dedicated hardware including one or more microprocessors and/or OCM units storing the data and/or the first set of low-level instructions to perform the plurality of ML operations. The internal representation of sub-graph 256 is mapped to one or more components in a general-purposed computing device (e.g., a general CPU or GPU), a special-purposed hardware (e.g., another (second) ML hardware that is different from the (first) ML-specific hardware), or a software simulator or emulator of a hardware, or a combination of the ML hardware and ML hardware emulator. In some embodiments, the ML hardware and the general-purposed computing device may be separate devices even though they may be integrated on a same physical device.
It is appreciated that each sub-graph may be optimized. For example, the compiler may perform target specific transformations and optimizations on each sub-graph. It is appreciated that because the target associated with each sub-graph may be different, e.g., ML hardware, ML emulator, general processor, etc., their resources and/or architecture are also different, e.g., memory, processing units, etc. As such, each sub-graph may undergo a different transformation and/or optimization depending on the target that will be executing the code generated for the sub-graph. As such, the pre-trained ML model is processed in an efficient and optimized fashion.
In some embodiments, partitioning a graph into sub-graphs in an automated fashion and in a more intelligent fashion, as described, based on respective target capabilities and resources improves the processing speed, reduces the required resource, and/or reduces latency. In other words, the original model graph is partitioned based on analysis of optimization/transformation that closely maps to the target hardware, e.g., ML hardware, ML emulator, general processor, etc., thereby enabling detailed computation and communication estimates, which is based on resources such as processing power and data movement, etc. Expressing the original model as sub-graphs, as described, enables the compilation process to be automated such that the model can be executed (i.e., inferred) in a time efficient and resource efficient manner. In some embodiments, a self-tuning capability to perform graph partitioning, compiling, and performing inference run analysis. In yet other embodiments, manual interference by a user may be provide in addition to the automated compilation and partitioning. As such, partitioning the original model graph into sub-graphs, as described, reduces latency by reducing unnecessary memory (or hierarchy) access, reducing unnecessary data movement, etc., in comparison to the conventional system.
It is appreciated that the embodiments for splitting the graph into subgraphs such that one subgraph is executed by the ML hardware and one subgraph is executed by a general processor is for illustrative purposes and should not be construed as limiting the scope of the embodiments. For example, the embodiments are equally applicable to splitting the graph into subgraphs where one subgraph is executed by a software emulator (emulation of ML hardware) and where the other subgraph is executed by a general processor. As such, discussions with respect to the subgraph being executed by the ML hardware is for illustrative purposes and should not be construed as limiting the scope of the embodiments. It is appreciated that in some embodiments, the subgraphs created for execution by ML hardware and the general processor may be compiled using the same compiler or by using different compilers.
Referring now to
As shown by the example of
In some embodiments, the compiler 292 may receive the ML model 291, e.g., pre-trained ML model, as described above. For example, the pre-trained ML model may be loaded. The compiler 292 processes the ML model 291 to generate internal representation graph, e.g., graph 252 as described above. The graph 252 may include meta data representation and as such not only capture the structure and attributes but also functionality and usage of the ML model 291. It is appreciated that the compiler 292 may perform hardware specific layout conversion in order to transform the graph to meet specific requirements of the hardware, e.g., ML hardware, general processor, etc. It is appreciated that in some embodiments, the compiler 292 may perform certain optimizations, e.g., merging/fusing, additional transformation, etc.
According to some embodiments, the compiler 292 may partition the graph into subgraphs, e.g., subgraph 254 and 256, as described above. In this nonlimiting example, the compiler 292 partitions the graph into subgraphs based on whether the node/operation is supported by the ML hardware or not and even if it is supported whether the data movement and latency can be improved by including it in a subgraph for the general processor as opposed to the subgraph for the ML hardware. Accordingly, the compiler 292 generates subgraphs, one subgraph for the ML hardware and one subgraph for the general processor to be compiled by the TVM compiler 298. As described above, at least a pair of input/output nodes are added to the subgraphs when the graph is being partitioned. Accordingly, the subgraphs cover the entire ML model 291. In this nonlimiting example, the subgraph 293A is created to be processed by the ML hardware, e.g., inference engine/emulator 296, whereas the subgraph 293B is created to be compiled by the TVM compiler 298 and processed by the general processor of the local host 280. It is appreciated that in one nonlimiting example, the ML hardware may only use input tensors of a given pre-trained network as its subgraph input tensor. ML hardware specific attributes may be introduced for each node of the subgraph 293A, thereby generating files and constants, e.g., JavaScript Objection Notation (JSON) format.
In some embodiments, the subgraph 293A and the ML hardware specific attributes for each node, in this nonlimiting example as JSON file and constants, are used by the compiler distribution 294 unit to distribute the compiler, e.g., backend compiler, as an executable in the ML/AI tar ball. Accordingly, the nodes JSON and constants JSON files for each node of the subgraph 293A along with meta data may be used by the compiler distribution 294 to generate the final instruction, e.g., in binary.
The binary model 295 is generated and transmitted to the inference engine/emulator 296. It is appreciated that the inference engine may be the ML hardware, as described above that executes the binary model 295. In some nonlimiting example, instead of using the ML hardware an emulator may be used. The emulator may be executed by the processor of the local host 280 whereas the ML hardware may execute the binary model 295 independent from the local host 280. In this nonlimiting example, the inference engine/emulator 296 runs inference in float 16 mode or int8 quantization mode.
The inference engine/emulator 296 generates the ML hardware/emulation subgraph output 297 which is input into the TVM compiler 298. The TVM compiler 298 also receives the subgraph 293B and compiles that into binary. The compiled binary of the subgraph 293B and the ML hardware/emulation subgraph output 297 are used to generate the output 299.
Referring now to
Referring now to
Referring now to
Referring now to
As described above, the backend compiler may generate one or more structure metadata. Below is another example of a code followed by the backend compiler generated metadata that illustrates the input, the weight, and the bias constants and output for a fp16 network for illustration purposes. In this nonlimiting example, a convolution layer in a network that is reduced to fp16 precision is illustrated. The structured metadata first describes the tensors involved in this operation in addition to operator specific arguments such as padding and stride information. The total number of multiply and accumulate (MAC) instructions are given. The second part of the structured metadata describes the memory layout of the tensors.
Below is yet another example of a code followed by the backend compiler generated metadata that illustrates quantized network for illustration purposes. In this nonlimiting example, the same convolution layer as in the previous example is shown except that in this example a network is quantized to int8. The structured metadata first describes the tensors involved in this operation in addition to operator specific arguments such as padding and stride information. The total number of MAC instructions are given. The second part of the structured metadata describes the memory layout of the tensors.
As yet another example, the code as generated by the backend compiler and its structured metadata is shown below for illustration purposes. In this nonlimiting example a low-level ISA instructions to be executed on ML hardware 160 is shown which is augmented with the structured metadata that are provided as comments that are excluded at runtime by the hardware.
In the example of
It is appreciated that in some embodiments a structured metadata may be selected and fed back into one of the backend compilers in order to optimize its operation. It is further appreciated that in some embodiments the one or more processing operations is one of changing precision, quantization, dimension reordering, or splitting or copying data across one or more processing tiles of the hardware. In one nonlimiting example, the method may further include reducing data movement by using the one or more processing operations. According to some embodiments, the hardware may be a dedicated hardware block including one or more microprocessors and/or OCM units storing the data and/or the set of low-level instructions compiled from the high-level function. According to some embodiments, the method further includes reducing storage by using the one or more processing operations. In one alternative example, the method may further include reducing computations by using the one or more processing operations or reducing data conversion by using the one or more processing operations. It is appreciated that as described above, the method may further include comparing data stored in a memory location as identified by a structured metadata of the plurality of structured metadata to expected data for verification of the high-level function. In some nonlimiting examples, the method may further include comparing data stored in a memory location as identified by a structured metadata of the plurality of structured metadata to expected data and debugging of the high-level function based on the comparison. According to some embodiments, the method may further include determining resources in the hardware and mapping operations and data to one or more tiles of the hardware to execute the set of low-level instructions. It is appreciated that the method may further include optimizing the high-level function of the high-level code based on the plurality of structured metadata.
The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to the practitioner skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical application, thereby enabling others skilled in the relevant art to understand the claimed subject matter, the various embodiments and the various modifications that are suited to the particular use contemplated.
This application is a Continuation-in-Part nonprovisional application that claims the benefit and priority to the U.S. application Ser. No. 17/390,143 filed on Jul. 30, 2021, which claims the benefit and priority to a U.S. Provisional Application No. 63/214,651 that was filed on Jun. 24, 2021, which are incorporated herein by reference in their entirety. The instant application also claims the benefit and priority to the U.S. Provisional Application No. 63/285,733 filed on Dec. 3, 2021, and to the U.S. Provisional Application No. 63/282,600 filed on Nov. 23, 2021, which are incorporated herein by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 10977018 | Hwang | Apr 2021 | B1 |
| 11797876 | Wang | Oct 2023 | B1 |
| 20200285788 | Brebner | Sep 2020 | A1 |
| 20220101194 | Liu | Mar 2022 | A1 |
| 20220113972 | Nakamura | Apr 2022 | A1 |
| 20220197616 | Drepper | Jun 2022 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 63214651 | Jun 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17390143 | Jul 2021 | US |
| Child | 17747813 | US |
