Patent Grant
11636569
Artificial neural networks are computing systems with an architecture based on biological neural networks. An artificial neural network can include a set of weights. Through computations, the weights can be combined with input data to perform an inferencing operation. For example, for a computer vision application to detect an object, the artificial neural network can combine the weights with an input image to extract certain features of the object from the image. Based on the extracted features, the artificial neural network can generate a decision of whether the object is in the image. Artificial neural networks can be trained, using training data in a training operation, to learn about how to perform the inferencing operation for an application.
An artificial neural network typically performs computations on data in the form of matrices to perform the inferencing operations and the training operations. For example, input data and weights can be organized into matrices. The artificial neural network can perform various matrix computations, such as dot-product computations between the input data and weight matrices, at each neural network layer for a computing task. In addition, the data and weights matrices can be pre-processed to support the matrix computations.
Various embodiments in accordance with the present disclosure will be described with reference to the drawings, in which:
Examples of the present disclosure relate to neural network processing, and more specifically, to techniques of accelerating matrix transpose operations to support neural network computations.
An artificial neural network (hereinafter, neural network) is typically implemented in a computing system, such as a neural network hardware accelerator, to have an architecture based on biological neural networks, and to process input data in an analogous fashion as biological neural networks. A neural network typically includes a number of cascading neural network layers, with each layer including a set of weights. In an inferencing operation, each layer can combine input data (e.g., inputs to the neural network layer, intermediate output from a previous layer, etc.) with the weights to generate intermediate outputs, and propagate the intermediate outputs to a higher neural network layer. The forward propagation operations can start at the first neural network layer and end at the highest neural network layer. The forward propagation operation at each neural network layer can represent different stages of extraction and processing of the inputs to the neural network. The neural network can then generate a decision based on the output of the highest neural network layer.
The set of weights of the neural network can be generated and/or updated by a training operation to improve the likelihood of the neural network generating a correct decision. The training operation can be performed by the same computing system (e.g., a neural network hardware accelerator) that performs the inferencing operation. An example training operation can use a gradient decent scheme. Specifically, as part of the training operation, the aforementioned forward propagation operations can be performed on a training input data set, using the set of weights at each neural network layer, to generate a training output data set at the highest level neural network layer. The training output data, as well as target output data, can be input to a loss gradient operation to compute loss gradients representing a partial derivative of the output errors (between the training output data and the target output data) with respect to the training output data. The loss gradients can be propagated back to the highest neural network layer.
A set of backward propagation operations can then be performed at each neural network layer. At the highest neural network layer, a first backward propagation operation can be performed, in which the loss gradients can be combined with the intermediate outputs of the previous neural network layer (the second highest neural network layer) to generate first weight gradients for the highest neural network layer. The first weight gradients can represent a partial derivative of the output errors with respect to the weights of the highest neural network layer. The first weight gradients can be used to update the weights of the highest neural network layer to minimize the output errors. Moreover, the loss gradients can be combined with the original weights of the highest neural network layer to obtain first error gradients. The first error gradients can then be propagated backward to the second highest neural network layer, which can then perform a second backward propagation operation to generate second weight gradients and second error gradients, and propagate the second data gradients to the preceding layer. The backward propagation operation can be performed from the highest neural network layer backward to the lowest neural network layer to compute weight gradients to update the weights at each layer. The training operation can be repeated for the same input data set for a number of iterations until a loss objective (e.g., a threshold first output error from the highest neural network layer) is achieved.
A neural network hardware accelerator can be programmed to implement an artificial neural network to perform computing tasks. A neural network hardware accelerator typically performs computations of data in the form of matrices for the inferencing operations and the training operation. For example, the combination of inputs and weights in the forward propagation operations, the combination of intermediate outputs and error gradients, the combination of weights and weight gradients in the backward propagation operations, etc., can be in the form dot-product computations between matrices. To accelerate the dot-product computations, the neural network hardware accelerator may include dedicated circuits, such as arithmetic circuits, to perform the arithmetic operations involved in the dot-product computations.
In addition, the data matrices (input and intermediate outputs, weights, etc.) can be pre-processed to support the matrix computations. One example of a pre-processing operation may include a matrix transpose operation, in which the indices of two dimensions of a matrix are swapped. For example, for a two-dimensional matrix organized into rows and columns, a transpose operation can be performed by swapping the matrix row indices and matrix column indices, which can result in flipping the matrix over its diagonal. Matrix transpose operations can be performed during the training operation of a neural network. For example, the weight matrix of a neural network layer is combined with inputs to the neural network layer in a forward propagation operation, but in the backward propagation operation of that layer, a transposed version of the weight matrix is combined with an input error gradient matrix (received from the next layer) to compute an output error gradient matrix to be propagated to the preceding layer. The transposed version of the weight matrix is also combined with the weight gradients to adjust the weights. In addition, in a case where the neural network is configured as a convolutional neural network (CNN), the intermediate outputs matrix from a forward propagation operation of a neural network layer can also be transposed, and combined with the input error gradients matrix (received from the next layer) to generate weight gradients in the backward propagation of that neural network layer.
The matrix transpose operation can be performed by a software application which controls the neural network hardware accelerator to perform the inferencing and training operations. But given that the software application is typically executed on a general purpose computer which lacks dedicated hardware to perform the transpose operation, using the application software to perform matrix transpose operation can introduce substantial latency to the matrix transpose operations, which in turn can reduce the throughput of the inferencing operations and training operations that use the transposed matrices. The latency of the matrix transpose operation is further increased when multi-dimensional matrices (e.g., four dimensional (4D) matrices) are involved in the matrix transpose operation, which increases the complexity of the matrix transpose operation and makes it even more costly, in terms of time and performance, to perform the matrix transpose operations using software.
Examples of the present disclosure relate to neural network processing, and more specifically, to techniques of accelerating matrix transpose operations to support neural network computations. In some examples, a memory system, which includes a buffer memory and a memory access circuit, can be configured to perform read and write operation at the buffer memory to transpose a first matrix stored in a source memory to form a second matrix, and to store the second matrix at a destination memory. The source and destination memories can include dynamic random access memory (DRAM) devices, static random access memory (SRAM) devices, high bandwidth memory (HBM) devices, etc., whereas the buffer memory can include various on-chip devices such as, for example, SRAM devices, flip-flops, and registers. In some example, the memory system can be part of a neural network hardware accelerator.
Specifically, the first matrix can be multi-dimensional, with each data element associated with indices in a first dimension, a second dimension, a third dimension, a fourth dimension, etc. The first dimension can be of the lowest dimension, the second dimension can be the next level dimension above the first dimension, the third dimension can be the next level dimension above the second dimension, while the fourth dimension can be of the highest dimension. Data elements associated with consecutive indices in the first dimension can be stored at a range of consecutive addresses at the source memory to form first groups. Pairs of the first groups associated with consecutive indices in the second dimension can be stored adjacent to each other at the source memory to form a second group. As used herein, the storage of two groups as being adjacent to each other at a memory can include, for example, assigning the two groups to two contiguous memory regions in the memory, or storing a memory pointer to indicate that a second one of the two groups immediately follow a first one of the two groups. Pairs of the second groups associated with consecutive indices in the third dimension can be stored adjacent to each other at the source memory to form a third group. Finally, pairs of the third groups associated with consecutive indices in the fourth dimension can be stored adjacent to each other at the source memory.
As part of the write operation, the memory access circuit can fetch each first group of data elements, and store each first group at a pattern of write addresses at the buffer memory. The pattern of write addresses can be defined based on whether a non-x-striding operation or an x-striding operation is to be performed. In the non-x-striding operation, the memory access circuit can store each first group, each comprising data elements associated with consecutive indices at the first dimension, at consecutive addresses at the buffer memory. In some examples, the buffer memory can include a plurality of memory banks, and each first group can be stored at consecutive addresses at each memory bank, while adjacent first groups can be stored at adjacent memory banks. As data elements of consecutive indices at the first dimension are still stored at consecutive addresses at the buffer memory, the shape of the first matrix can be retained.
On the other hand, an x-striding operation is to stride across lowest dimension elements with a certain x-striding distance so that elements in the lowest dimension (e.g., the first dimension) are store adjacent to corresponding elements in a higher dimension (e.g., the second dimension, the third dimension, etc.) instead of adjacent to each other. To perform the x-striding operation, the memory access circuit can stride, depending on the x-striding distance, across each stride group which can be a first group, a second group, or a third group, for example, to select corresponding data elements having the same index in the first dimension (the lowest dimension) from each stride group, and store the corresponding data elements at consecutive addresses at the buffer memory to form a new first group, and multiple new first groups can be formed for each index in the first dimension. For example, the memory access circuit can obtain each data element (e.g., a first data element, a second data element, a third data element, etc.) from each first group, each second group, or each third group depending on the x-striding distance. The memory access circuit can then store each first data elements at a first memory address at each memory bank, each second data element at a second memory address at each memory bank, and each third data element at a third memory address at each memory bank. As a result of the x-striding operation, data elements having consecutive indices in a higher dimension (e.g., second, third, or fourth dimension), rather than in the first dimension, are stored in consecutive addresses in the memory buffer, which can result in a reshaping and a transpose of the first matrix.
In addition, as part of the read operation, the memory access circuit can fetch each first group, or each new first group, from the buffer memory, and store the groups at the destination memory to form a second matrix. The fetching of the first groups can follow a memory read address pattern to perform a non-y-striding operation, or to perform a y-striding operation. In a non-y-striding operation, adjacent first groups (e.g., first groups stored in adjacent memory banks) are stored as adjacent groups in the destination memory, so pairs of first groups associated with consecutive indices in the second dimension remain adjacent to each other to form a second group, and the shape of the first matrix can be retained. On the other hand, y-striding is to stride across the first groups with a certain y-striding distance so that first groups associated with consecutive indices in a higher dimension, rather than in the second dimension, are put adjacent each other to form new second groups. To perform a y-striding operation, the memory access circuit can fetch a pair of first groups from non-neighboring memory banks by skipping a number of memory banks based on the y-striding distance, and store the pair of first groups as adjacent first groups in the destination memory. As a result of the y-striding operation, first groups having consecutive indices in a higher dimension than the second dimension (e.g., third, or fourth dimension) are stored as adjacent groups in the destination memory, which can also result in a reshaping and a transpose of the first matrix.
In some examples, the memory access circuit can be configured based on programming information to perform at least one of the non-x-striding operation, the x-striding operation, the non-y-striding operation, and the y-striding operation to transpose the multi-dimensional first matrix. The programming information can also define the x-striding distance and y-striding distance. As to be described below, the memory access circuit can perform transpose operations between different dimensions based on, for example, an x-striding operation, a y-striding operation, or an x-striding operation followed by one or more y-striding operations.
With the disclosed techniques, matrix transpose operations can be performed based on read and write operations at a memory buffer of the neural network hardware accelerator, rather than by a software application executing on a general purpose computer, which can improve the throughput of the transpose operation. Moreover, by providing the capability to program the memory access circuit to perform different combinations of x-striding and y-striding operations as well as non-striding operations, different reshaping/transpose operations between different dimensions for a multi-dimensional matrix (e.g., a 4D matrix) can be achieved to support a wide range of transpose operations for different neural network computations.
In the following description, various examples will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the examples. However, it will also be apparent to one skilled in the art that the example may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the embodiments being described.
In some examples, the image recognition service can be provided in a multi-tenant compute service system. The multi-tenant compute service system may typically include a plurality of servers that can host data and be used by multiple clients or organizations to run instances, such as virtual machine instances or bare-metal instances (e.g., operating systems that run directly on the server hardware). In most instances, such as bare-metal or virtual machine instances, a multi-tenant compute service system may be allocated to a client when the client needs them and decommissioned when they are no longer needed, such that the resources can be reallocated to other clients. In the present disclosure, the terms “tenant,” “client,” and “customer” may be used interchangeably, although such terms do not necessarily imply the existence of any particular business arrangement. The term “instance” may refer to, for example, an instance that is executed directly on server hardware or as a virtual machine. Different types of instances generally correspond to different hardware functions and/or arrangements of hardware (e.g., different amounts of available memory and/or processing hardware). In the example of
In the example of
Prediction model 103 can be in the form of an artificial neural network. The artificial neural network may include a plurality of processing nodes, with each processing node configured to process part of the input pixel data, or to further process the intermediate outputs from other processing nodes.
Layer 207 may process pixel data representing different portions of image 104. For example, in the example of
Layer 209 may process the scaled outputs from layer 207 to generate a set of intermediate outputs. For example, assuming processing node 210a of layer 209 is connected to n processing nodes in layer 207, processing node 210a may generate a sum of the scaled outputs received from layer 207 based on the following equation:
Here, sum —210a represents an intermediate output generated by processing node 210a. W1i×xi represents a scaling of a particular pixel value (e.g., x0) with the associated weight (e.g., W10) by a processing node of layer 207. In a case where prediction model 103 is a DNN, each processing node of layer 209 may generate the sum based on the scaling of pixel values from each processing node of layer 207, and then generate a sum (e.g., Sum210a) by summing the scaled pixel values. The sum may also represent a dot-product between an input vector comprising a number of elements (e.g., pixel values) and a weight vector (e.g., W1). In some examples, a bias can also be added the scaled outputs to generate the intermediate output.
In a case where prediction model 103 is a CNN, each processing node of layer 209 may generate the intermediate output based on the scaling of pixel values from a group of processing nodes of layers 207. The intermediate output may represent a convolution result between a group of pixel values and a filter comprising the weight values.
As shown in
Referring back to
In addition to ReLU, other forms of activation function can also be used including, for example, a softplus function (which can be a smooth approximation of a ReLU function), a hyperbolic tangent function (tan h), an arc tangent function (arctan), a sigmoid function, a Gaussian function, etc.
A processing node of layer 209 (e.g., processing node 210a) may process the sum with the ReLU function to generate a first output of layer 209 based on the following equation:
first_output210a=ReLU(Sum210a) (Equation 3)
Optionally, prediction model 103 may include a pooling layer to reduce the number of intermediate outputs (e.g., sum210a) of layer 209. The pooling layer may group the intermediate outputs and perform a pooling operation on each group. The pooling operation may include such as max pooling (e.g., selecting a maximum intermediate output within the group), min pooling (e.g., selecting a minimum intermediate output), average pooling (e.g., finding an average of each group), summation pooling (finding a sum of each group), etc., and the reduced intermediate outputs can be processed by the activation function to generate first outputs of layer 209. The pooling operation can be performed to reduce the computation cost associated with activation function processing.
Layer 211 may further process the scaled intermediate outputs from layer 209 by, for example performing additional convolution operations based on different sets of filters. The outputs from each processing node of layer 211 may be forwarded to other higher intermediate layers, or to an output layer (not shown in
Although
The weights and filter coefficients described in
A loss gradient operation 304 can compare the intermediate output data outn of layer n against reference output data ref outn to generate output error gradients din. The error gradient din can measure a partial derivative of the output error, represented by the difference between outn and ref outn, with respect to each data element of output data outn. In some examples, an objective of the training is to minimize the difference between outn and ref outn such that the error gradients din can become close to zero.
Following the generation of loss gradients din by loss gradient operation 304, a backward propagation operation 306 can be performed for each neural network layer. For example, a backward propagation operation 306n can be performed at highest layer n, a backward propagation operation 306b can be performed at layer 2, a backward propagation operation 306a can be performed at layer 1. A backward propagation operation at a neural network layer can be based on the weights of that neural network layer, the error gradient input to that neural network layer, as well as the input to the forward propagation operation of that layer. For example, for layer n, backward propagation operation 306n can receive, as inputs, weights wn, intermediate output data outn−1 (from forward propagation operation at neural network layer n−1), and loss gradients 1grad. The backward propagation operation can perform multiplication and summation computations similar to those of Equations 1 and 2 on the inputs to generate weight gradients wgrad (dwn, dw2, dw1, etc. in
Weight gradients wgrad of a neural network layer can represent a partial derivative of the output error with respect to each weight of the neural network layer, and can be used to update the weights of the neural network layer. The weight gradients can be generated based on the intermediate outputs of a previous layer and the input error gradients to that neural network layer. For example, at layer n, weight gradients dwn can be generated based on the intermediate outputs outn−1 and loss gradients 1grad, whereas at layer n−1, weight gradients dwn−1 can be generated based on the intermediate outputs outn−2 and output error gradients of layer n, dn.
The weights at layer n can be updated by an update operation 308 (e.g., update operation 308n for layer n) based on the weight gradients dwn based on the following equation:
wn′=wn−α×dwn (Equation 4)
In Equation 4, wn′ can refer to the updated weights wn, whereas a can include a set of pre-determined constants.
In addition, the output data gradients of a neural network layer can represent a partial derivative of the output error with respect to the product sums at the neural network layer. The output data gradients can be forwarded to the next lower neural network layer as inputs to the backward propagation operation in that layer to compute the weight gradients in that layer. For example, as described above, the output data gradients of layer n, dn, can be propagated to layer n−1 to compute weight gradients dwn−1. The output data gradients of a neural network layer can be generated based on the input data gradients to that neural network layer, as well as the original weights (prior to being updated with the weight gradients) of that neural network layer. For example, at layer n, the output data gradients dn can be generated based on weights wn as well as loss gradients 1grad in backward propagation operation 306n.
The output data gradients dn generated by layer n can then propagate to the next lower neural network layer n−1 as input to the backward propagation operation at that layer. Backward propagation operation 306b of layer 2 can operate on data gradients d3, weights w2, and intermediate output data outs to generate output data gradients d1 as well as weight gradients dw2. Weight gradients dw2 can be used by update operation 308b to update w2 weights based on Equation 4. Data gradients d2 can propagate to layer 1. Backward propagation operation 306a of layer 1 can operate on data gradients d2, weights w1, and training input data to generate weight gradients dw1. Weight gradients dw1 can be used by update operation 308a to update w1 weights based on Equation 4.
The training operation of
Referring to the right of
The transpose operation of weight matrix 320 in
To perform an x-striding operation, a data element from each of the first group and the second group can be selected and stored at consecutive addresses forming a new group, and the new groups can be assembled to form a matrix 410 having the order of dimensions XY as a transposed version of matrix 400. The order of dimension XY indicates that the y-dimension is the lowest dimension, whereas the x-dimension is the highest dimension. The selection can be based on striding across the first group and the second group based on an x-striding distance equal to the size of each group. For example, data elements I0,0 of group 402 and I0,1 of group 404, Separated by an x-striding distance corresponding to the size of each group (6), can be selected to form a new group 412, while data elements I1,0 of group 402 and I1,1 of group 404 can be selected to form a new group 414. As in matrix 410 data elements having consecutive indices in the y-dimension (e.g., I0,0 and I0,1, I1,0 and rather than in the x-dimension, are stored in consecutive addresses, matrix 410 can become a transposed version of matrix 400.
The x-striding operation of
Furthermore, third groups 426a and 426b can be stored adjacent to each other in the memory based on them having consecutive indices in the highest w dimension (w0, w1).
An x-striding operation can be performed on matrix 420 to change its shape.
In addition, referring to
In addition to x-striding, a y-striding operation can also be performed to change the shape of the matrix. A y-striding operation is to stride across groups of data elements of consecutive indices in the first/lowest dimension (e.g., first groups 422a-h in
Referring to
In some examples, the transpose/reshaping operation of a matrix can include a sequence of striding operations, such as a sequence of x-striding and y-striding operations, a sequence of y-striding operations, etc.
With the x-striding operation with an x-striding distance of 6, each second group includes data elements associated with a particular x-dimension index. Specifically, second group 524a is associated with x-dimension index x0, second group 524b is associated with x-dimension index x1, second group 524c is associated with x-dimension index x2, second group 524d is associated with x-dimension index x3, second group 524e is associated with x-dimension index x4, whereas second group 524f is associated with x-dimension index x5. As the second groups having consecutive x-dimension indices are stored adjacent to each other to form intermediate matrix 505, the x-dimension becomes the highest dimension, giving rise to intermediate matrix 505 having the order of dimensions XWZY.
Y-striding operation 506 can then be performed to transpose intermediate matrix 505 to matrix 502. Referring back to
With the y-striding operation, one subgroup 526 can be selected from each second group 524. Referring to
The following table illustrates the x-striding and/or y-striding operations and their configurations to transpose matrix 420 (having order of dimensions WZYX) to matrices of different orders of dimensions. In the first row, WZYX and WZXY refer to the order of the dimensions of the target matrix as explained above in
Specifically, first matrix 620 can be multi-dimensional, with each data elements associated with indices in a first dimension, a second dimension, a third dimension, a fourth dimension, etc. First matrix 620 can be, for example, matrix 420 of
A pair of first groups that are adjacent to each other can be stored in memory banks of consecutive indices. For example, first group 422a can be stored in memory bank 640a associated with an index of 1, whereas first group 422b can be stored in memory bank 640b associated with an index of 2. First groups can be fetched following the order of the indices of the memory banks, starting with the index of 1 and followed by the indices of 2, 3, 4, etc. First groups fetched from memory banks 640 of consecutive indices can then be stored as adjacent groups in destination memory 612, as shown in
In some cases, the number of first groups may exceed the number of the memory banks 640a. For example, in the example of
As described above, memory access circuit 602 can read and write operation at buffer memory 608 to implement an x-striding operation, a y-striding operation, or any combination thereof, to transpose first matrix 620. Specifically, programming modules 610a and 610b can receive a matrix reshape command 622, which can be part of an instruction set involving matrix computations (e.g., instructions for a neural network hardware accelerator). Matrix reshape command 622 may specify, for example, which of the matrix reshaping operation of Table 1 above to be performed by memory access circuit. Each of programming modules 601a and 601b may store a mapping table that maps each matrix reshaping operation and one or more x-striding and/or y-striding operations to be performed for each matrix reshaping operation as described in Table 1. The mapping table may also stores the x-striding distance and x_num_group parameters of the x-striding operation and the y-striding distance and y_num_group parameters for the y-striding operations. In a case where an x-striding operation is not be performed, the x-striding distance can set to one. Likewise, in a case where a y-striding operation is not to be performed, the y-striding distance can also be set to one. Based on the mapping table and matrix reshaping command 622, programming module 601a can generate programming information 624a specifying the x-striding distance and x_num_groups parameters for x-striding circuit 604, whereas programming module 601b can generate programming information 624b specifying the y-striding distance and y_num_groups parameters for y-striding circuit 606.
The x-striding circuit 604 can be configured, based on programming information 624a, the first groups of data elements (data elements associated with consecutive indices in the lowest dimension), and store each first group at a pattern of write addresses at the buffer memory. The pattern of write addresses can be defined based on whether a non-x-striding operation or an x-striding operation is to be performed, and an x-striding distance, all of which can be defined according to programming information 624a. Specifically, in a case where programming information 624a indicate that x-striding is disabled (based on an x-striding distance of one), x-striding circuit 604 can perform a non-x-striding operation and store each first group at consecutive addresses at buffer memory 608. For example, based on the x_num_groups parameter in programming information 624a, x-striding circuit 604 can determine there are 24 stride groups, such that each stride group corresponds to a first group of matrix 620, and determine that each stride group comprises the first group which includes 6 data elements. X-striding circuit 604 can then fetch 6 data elements from 6 consecutive memory addresses for a first group (e.g., first group 422a), store the first group at consecutive memory addresses at a memory bank (e.g., memory bank 640a), and then fetch the next 6 data elements for another first group and store that first group at the next memory bank (e.g., memory bank 640b), until the 24 first groups are fetched, as described in
On the other hand, in a case where programming information 624a indicate that x-striding is enabled, x-striding circuit 604 can perform an x-striding operation with an x-striding distance over a number of the first groups, based on the x-striding distance and x_num_groups parameters in programming information 624a.
For example, as shown in
Referring back to
In operation 802, x-striding circuit 604 fetches, from a first memory (e.g., source memory 610), a set of first groups of data elements of a first matrix, each first group of data elements being stored at consecutive memory addresses at the first memory. An example of first matrix is matrix 420 as shown in
In operation 804, based on a first configuration that defines first parameters for an x-striding operation, x-striding circuit 604 stores the data elements of each first group at a pattern of write addresses at the buffer memory.
Specifically, the first parameters can include an indication of whether to perform an x-striding operation, a first number of the first groups to be stridden by the x-striding operation, and an x-striding distance. In a case where the first parameters indicate no x-striding operation is to performed, the pattern of write addresses can include consecutive addresses at the buffer memory to store the data elements of a first group. In some examples, the buffer memory can include a plurality of memory banks, and the data elements of the first group can be stored at consecutive addresses at a memory bank, and each memory bank can store the data elements of each first group. The x-striding circuit 604 can identify each first group based on the first number of the first groups defined in the first parameters, and store the data elements of each first group at each memory bank.
On the other hand, in a case where the first parameters indicate an x-striding operation is to performed, the pattern of write addresses can include non-consecutive addresses, such as addresses at different memory banks to store the data elements of a first group, such that the data elements of the first group is distributed across the memory banks. The x-striding circuit 604 can identify a first data element from each first group based on the x-striding distance, and store the first data elements at consecutive addresses at a first memory bank to form a first one of a second group. The x-striding circuit 604 can identify a second data element from each first group based on the x-striding distance, and store the second data elements at consecutive addresses at a second memory bank to form a second one of a second group. As data elements associated with consecutive indices at a higher dimension, other than the lowest dimension in the first matrix, are stored at consecutive addresses, the first matrix can be transposed by the x-striding operation.
In operation 806, based on a second configuration that defines parameters for a y-striding operation, y-striding circuit 606 fetches a set of second groups of the data elements at a pattern of read addresses at the buffer memory.
Specifically, each second group of data element can be stored at consecutive memory addresses at each memory bank of the buffer memory. Each second group can include a first group of the first matrix (in a case where no x-striding operation is performed), or data elements from each first group of the first matrix (in a case where x-striding operation is performed).
The second parameters comprise a second number of the second groups to be stridden by a y-striding operation, and a y-striding distance. The y-striding circuit 606 can identify each second group based on, for example, determining a number of data elements in each second group based on the second number of the second groups. Moreover, the y-striding operation can define whether y-striding circuit 606 fetches a second group from each adjacent memory bank, or based on skipping a number of memory banks based on the y-striding distance.
In operation 808, the second groups fetched from the y-striding operation can be stored in the destination memory as adjacent second groups. As a result of the y-striding operations, data elements of each second group remains stored at consecutive addresses at the destination memory such that the lowest dimension remain the same, but second groups associated with consecutive indices at a different/higher dimension are stored as adjacent groups, which can result in a transpose/reshaping of the matrix between the buffer memory and the destination memory.
In various implementations, the memory subsystem 904 can include multiple memory banks 914. In these implementations, each memory bank 914 can be independently accessible, meaning that the read of one memory bank is not dependent on the read of another memory bank. Similarly, writing to one memory bank does not affect or limit writing to a different memory bank. In some cases, each memory bank can be read and written at the same time. Various techniques can be used to have independently accessible memory banks 914. For example, each memory bank can be a physically separate memory component that has an address space that is separate and independent of the address spaces of each other memory bank. In this example, each memory bank may have at least one read channel and may have at least one separate write channel that can be used at the same time. In these examples, the memory subsystem 904 can permit simultaneous access to the read or write channels of multiple memory banks. As another example, the memory subsystem 904 can include arbitration logic such that arbitration between, for example, the outputs of multiple memory banks 914 can result in more than one memory bank's output being used. In these and other examples, though globally managed by the memory subsystem 904, each memory bank can be operated independently of any other. In some examples, memory subsystem 904 can be coupled with memory system 600 of
Having the memory banks 914 be independently accessible can increase the efficiency of the accelerator 902. For example, values can be simultaneously read and provided to each row of the processing engine array 910, so that the entire processing engine array 910 can be in use in one clock cycle. As another example, the memory banks 914 can be read at the same time that results computed by the processing engine array 910 are written to the memory subsystem 904. In contrast, a single memory may be able to service only one read or write at a time. With a single memory, multiple clock cycles can be required, for example, to read input data for each row of the processing engine array 910 before the processing engine array 910 can be started.
In various implementations, the memory subsystem 904 can be configured to simultaneously service multiple clients, including the processing engine array 910, the activation engine 916, the pooling engine 918, and any external clients that access the memory subsystem 904 over a communication fabric 920. In some implementations, being able to service multiple clients can mean that the memory subsystem 904 has at least as many memory banks as there are clients. In some cases, each row of the processing engine array 910 can count as a separate client. In some cases, each column of the processing engine array 910 can output a result, such that each column can count as a separate write client. In some cases, output from the processing engine array 910 can be written into the memory banks 914 that can then subsequently provide input data for the processing engine array 910. As another example, the activation engine 916 and the pooling engine 918 can include multiple execution channels, each of which can be separate memory clients. The memory banks 914 can be implemented, for example, using static random access memory (SRAM).
In various implementations, the memory subsystem 904 can include control logic. The control logic can, for example, keep track of the address spaces of each of the memory banks 914, identify memory banks 914 to read from or write to, and/or move data between the memory banks 914. In some implementations, memory banks 914 can be hardwired to particular clients. For example, a set of memory banks 914 can be hardwired to provide values to the rows of the processing engine array 910, with one memory bank servicing each row. As another example, a set of memory banks can be hard wired to receive values from columns of the processing engine array 910, with one memory bank receiving data for each column.
The processing engine array 910 is the computation matrix of the example accelerator 902. The processing engine array 910 can, for example, execute parallel integration, convolution, correlation, and/or matrix multiplication, among other things. The processing engine array 910 includes multiple processing engines 911, arranged in rows and columns, such that results output by one processing engine 911 can be input directly into another processing engine 911. Processing engines 911 that are not on the outside edges of the processing engine array 910 thus can receive data to operate on from other processing engines 911, rather than from the memory subsystem 904.
In various examples, the processing engine array 910 uses systolic execution, in which data arrives at each processing engine 911 from different directions at regular intervals. In some examples, input data can flow into the processing engine array 910 from the left and weight values can be loaded at the top. In some examples weights and input data can flow from the left and partial sums can flow from top to bottom. In these and other examples, a multiply-and-accumulate operation moves through the processing engine array 910 as a diagonal wave front, with data moving to the right and down across the array. Control signals can be input at the left at the same time as weights, and can flow across and down along with the computation.
In various implementations, the number of columns in the processing engine array 910 determines the computational capacity of the processing engine array 910, and the number of rows determines the required memory bandwidth for achieving maximum utilization of the processing engine array 910. The processing engine array 910 can have, for example, 64 columns and 428 rows, or some other number of columns and rows.
An example of a processing engine 911 is illustrated in
In the illustrated example, an input from above can include a partial sum, pin, provided either from another processing engine 911 or from a previous round of computation by the processing engine array 910. When starting a computation for a new set of input data, the top row of the processing engine array 910 can receive a fixed value for p_in, such as zero. As illustrated by this example, i and w are multiplied together and the result is summed with p_in to produce a new partial sum, p_out, which can be input into another processing engine 911. Various other implementations of the processing engine 911 are possible.
Outputs from the last row in the processing engine array 910 can be temporarily stored in the results buffer 912. The results can be intermediate results, which can be written to the memory banks 914 to be provided to the processing engine array 910 for additional computation. Alternatively, the results can be final results, which, once written to the memory banks 914 can be read from the memory subsystem 904 over the communication fabric 920, to be output by the system.
In some implementations, the accelerator 902 includes an activation engine 916. In these implementations, the activation engine 916 can combine the results from the processing engine array 910 into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce an output activation for a single channel. In other examples, accumulating results from one or more columns in the processing engine array 910 may be needed to produce an output activation for a single node in the neural network. In some examples, activation engine 916 can be bypassed.
In various examples, the activation engine 916 can include multiple separate execution channels. In these examples, the execution channels can correspond to the columns of the processing engine array 910, and can perform an operation on the outputs of a column, the result of which can be stored in the memory subsystem 904. In these examples, the activation engine 916 may be able to perform between 1 and n parallel computations, where n is equal to the number of columns in the processing engine array 910. In some cases, one or more of the computations can be performed simultaneously. Examples of computations that each execution channel can perform include exponentials, squares, square roots, identities, binary steps, bipolar steps, sigmoidals, and ramps, among other examples.
In some implementations, the accelerator 902 can include a pooling engine 918. Pooling is the combining of outputs of the columns of the processing engine array 910. Combining can include for example, computing a maximum value, a minimum value, an average value, a median value, a summation, a multiplication, or another logical or mathematical combination. In various examples, the pooling engine 918 can include multiple execution channels that can operating on values from corresponding columns of the processing engine array 910. In these examples, the pooling engine 918 may be able to perform between 1 and n parallel computations, where n is equal to the number of columns in the processing engine array 910. In various examples, execution channels of the pooling engine 918 can operate in parallel and/or simultaneously. In some examples, the pooling engine 918 can be bypassed.
Herein, the activation engine 916 and the pooling engine 918 may be referred to collectively as execution engines. The processing engine array 910 is another example of an execution engine. Another example of an execution engine is a Direct Memory Access (DMA) engine, which may be located outside the accelerator 902.
Input data 950 can arrive over the communication fabric 920. The communication fabric 920 can connect the accelerator 902 to other components of a processor, such as a DMA engine that can obtain input data 950 from an Input/Output (I/O) device, a storage drive, or a network interface. The input data 950 can be, for example one-dimensional data, such as a character string or numerical sequence, or two-dimensional data, such as an array of pixel values for an image or frequency and amplitude values over time for an audio signal. In some examples, the input data 950 can be three-dimensional, as may be the case with, for example, the situational information used by a self-driving car or virtual reality data. In some implementations, the memory subsystem 904 can include a separate buffer for the input data 950. In some implementations, the input data 950 can be stored in the memory banks 914 when the accelerator 902 receives the input data 950.
In some examples, the accelerator 902 can implement a neural network processing engine. In these examples, the accelerator 902, for a set of input data 950, can execute a neural network to perform a task for which the neural network was trained. Executing a neural network on a set of input data can be referred to as inference or performing inference.
The weights for the neural network can be stored in the memory subsystem 904, along with input data 950 on which the neural network will operate. The neural network can also include instructions, which can program the processing engine array 910 to perform various computations on the weights and the input data. The instructions can also be stored in the memory subsystem 904, in the memory banks 914 or in a separate instruction buffer. The processing engine array 910 can output intermediate results, which represent the outputs of individual layers of the neural network. In some cases, the activation engine 916 and/or pooling engine 918 may be enabled for computations called for by certain layers of the neural network. The accelerator 902 can store the intermediate results in the memory subsystem 904 for inputting into the processing engine array 910 to compute results for the next layer of the neural network. The processing engine array 910 can further output final results from a last layer of the neural network. The final results can be stored in the memory subsystem 904 and then be copied out to host processor memory or to another location.
The host processor 1072 is a general purpose integrated circuit that is capable of executing program instructions. In some examples, the host processor 1072 can include multiple processing cores. A multi-core processor may include multiple processing units within the same processor. In some examples, the host system 1000 can include more than one host processor 1072. In some examples, the host processor 1072 and the acceleration engine 1060 can be one chip, such as, one or more integrated circuits within the same package.
In various examples, the host processor 1072 can communicate with other components in the host system 1000 over one or more communication channels. For example, the host system 1000 can include a host processor bus, which the host processor 1072 can use to communicate with the DRAM 1030, for example. As another example, the host system 1000 can include an I/O bus, such as a PCI-based bus, over which the host processor 1072 can communicate with the acceleration engine 1060 and/or the I/O devices 1032, for example. In various examples, the host system 1000 can, alternatively or additionally, include other communication channels or busses, such as serial busses, power management busses, storage device busses, and so on.
In some examples, software programs executing on the host processor 1072 can receive or generate input for processing by the acceleration engine 1060. In some examples, the programs can select an appropriate neural network to execute for a given input. For example, a program may be for language translation, and can select one or more neural networks capable of speech recognition and/or machine translation. In these and other examples, the programs can configure the acceleration engine 1060 with the neural network to execute, and/or can select a neural network processing engine on the acceleration engine 1060 that has previously been configured to execute the desired neural network. In some examples, once the acceleration engine 1060 has started an inference on input data, the host processor 1072 can manage the movement of data (such as weights, instructions, intermediate results, results of conditional layers, and/or final results) into or out of the acceleration engine 1060.
In some examples, a software program that is using the acceleration engine 1060 to conduct an inference can read the result from a conditional layer from the acceleration engine 1060 and/or from a storage location, such as in DRAM 1030. In these examples, the program can determine what action the neural network should take next. For example, the program can determine to terminate the inference. As another example, the program can determine to change the direction of the inference, which can be translated by lower level code and/or the neural network processor to a next layer to execute. In these and other examples, the execution flow of the neural network can be coordinated by software. In some examples, a software program can also use acceleration engine 1060 to perform a training operation.
The DRAM 1030 is memory that is used by the host processor 1072 for storage of program code that the host processor 1072 is in the process of executing, as well as values that are being operated on. In some examples, the data for a neural network (e.g., weight values, instructions, and other data) can be all or partially stored in the DRAM 1030. DRAM is a common term for processor memory, and though DRAM is volatile memory, processor memory can be volatile and/or non-volatile. Though not illustrated here, the host system 1000 can include other volatile and non-volatile memories for other purposes. For example, the host system 1000 can include a Read-Only Memory (ROM) that stores boot code for booting the host system 1000 at power on, and/or Basic Input/Output System (BIOS) code. In some examples, DRAM 1030 can be coupled with memory system 600 of
Though not illustrated here, the DRAM 1030 can store instructions for various programs, which can be loaded into and be executed by the host processor 1072. For example, the DRAM 1030 can be storing instructions for an operating system, one or more data stores, one or more application programs, one or more drivers, and/or services for implementing the features disclosed herein.
The operating system can manage and orchestrate the overall operation of the host system 1000, such as scheduling tasks, executing applications, and/or controller peripheral devices, among other operations. In some examples, a host system 1000 may host one or more virtual machines. In these examples, each virtual machine may be configured to execute its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, and the like. The operating system may, alternatively or additionally, be a proprietary operating system.
The data stores can include permanent or transitory data used and/or operated on by the operating system, application programs, or drivers. Examples of such data include web pages, video data, audio data, images, user data, and so on. The information in the data stores may, in some examples, be provided over the network(s) to user devices. In some cases, the data stores may additionally or alternatively include stored application programs and/or drivers. Alternatively or additionally, the data stores may store standard and/or proprietary software libraries, and/or standard and/or proprietary application user interface (API) libraries. Information stored in the data stores may be machine-readable object code, source code, interpreted code, or intermediate code.
The drivers can include programs that provide communication between components in the host system 1000. For example, some drivers can provide communication between the operating system and peripheral devices or I/O devices 1032. Alternatively or additionally, some drivers may provide communication between application programs and the operating system, and/or application programs and peripheral devices accessible to the host system 1000. In many cases, the drivers can include drivers that provide well-understood functionality (e.g., printer drivers, display drivers, hard disk drivers, Solid State Device drivers, etc.). In other cases, the drivers may provide proprietary or specialized functionality.
The I/O devices 1032 can include hardware for connecting to user input and output devices, such as keyboards, mice, pens, tablets, voice input devices, touch input devices, displays or monitors, speakers, and printers, among other devices. The I/O devices 1032 can also include storage drives and/or network interfaces for connecting to a network 1080. For example, the host system 1000 can use a network interface to communicate with storage devices, user terminals, other computing devices or servers, and/or other networks, among various examples.
In various examples, one or more of the I/O devices 1032 can be storage devices. In these examples, the storage devices include non-volatile memory and can store program instructions and/or data. Examples of storage devices include magnetic storage, optical disks, solid state disks, flash memory, and/or tape storage, among others. The storage device can be housed in the same chassis as the host system 1000 or may be in an external enclosure. A storage device can be fixed (e.g., attached by screws) or removable (e.g., having a physical release mechanism and possibly a hot-plug mechanism).
Storage devices, the DRAM 1030, and any other memory component in the host system 1000 are examples of computer-readable storage media. Computer-readable storage media are physical mediums that are capable of storing data in a format that can be read by a device such as the host processor 1072. Computer-readable storage media can be non-transitory. Non-transitory computer-readable media can retain the data stored thereon when no power is applied to the media. Examples of non-transitory computer-readable media include ROM devices, magnetic disks, magnetic tape, optical disks, flash devices, and solid state drives, among others. As used herein, computer-readable storage media does not include computer-readable communication media.
In various examples, the data stored on computer-readable storage media can include program instructions, data structures, program modules, libraries, other software program components, and/or other data that can be transmitted within a data signal, such as a carrier wave or other transmission. The computer-readable storage media can, additionally or alternatively, include documents, images, video, audio, and other data that can be operated on or manipulated through the use of a software program.
In various implementations, the support systems 1074 can include hardware for coordinating the operations of the acceleration engine 1060. For example, the support systems 1074 can include a microprocessor that coordinates the activities of the acceleration engine 1060, including moving data around on the acceleration engine 1060. In this example, the microprocessor can be an integrated circuit that can execute microcode. Microcode is program code that can enable an integrated circuit to have some flexibility in the operations that the integrated circuit can execute, but because the program code uses a limited instruction set, the microprocessor may have more limited capability than the host processor 1072. In some examples, the program executed by the microprocessor is stored on the hardware of microprocessor, or on a non-volatile memory chip in the host system 1000. In some examples, the microprocessor and the acceleration engine 1060 can be on chip, such as one integrated circuit on the same die and in the same package.
In some examples, the support systems 1074 can be responsible for taking instructions from the host processor 1072 when programs executing on the host processor 1072 request the execution of a neural network. For example, the host processor 1072 can provide the support systems 1074 with a set of input data and a task that is to be performed on the set of input data. In this example, the support systems 1074 can identify a neural network that can perform the task, and can program the acceleration engine 1060 to execute the neural network on the set of input data. In some examples, the support systems 1074 only needs to select an appropriate neural network processing engine of the neural network processor. In some examples, the support systems 1074 may need to load the data for the neural network onto the acceleration engine 1060 before the acceleration engine 1060 can start executing the neural network. In these and other examples, the support systems 1074 can further receive the output of executing the neural network, and provide the output back to the host processor 1072.
In some examples, the operations of the support systems 1074 can be handled by the host processor 1072. In these examples, the support systems 1074 may not be needed and can be omitted from the host system 1000.
In various examples, the host system 1000 can include a combination of host systems, processor nodes, storage subsystems, and I/O chassis that represent user devices, service provider computers or third party computers.
User devices can include computing devices to access an application (e.g., a web browser or mobile device application). In some examples, the application may be hosted, managed, and/or provided by a computing resources service or service provider. The application may enable a user to interact with the service provider computer to, for example, access web content (e.g., web pages, music, video, etc.). The user device may be a computing device such as, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a netbook computer, a desktop computer, a thin-client device, a tablet computer, an electronic book (e-book) reader, a gaming console, etc. In some examples, the user device may be in communication with the service provider computer over one or more networks. Additionally, the user device may be part of the distributed system managed by, controlled by, or otherwise part of the service provider computer (e.g., a console device integrated with the service provider computers).
The host system 1000 can also represent one or more service provider computers. A service provider computer may provide a native application that is configured to run on user devices, which users may interact with. The service provider computer may, in some examples, provide computing resources such as, but not limited to, client entities, low latency data storage, durable data storage, data access, management, virtualization, cloud-based software solutions, electronic content performance management, and so on. The service provider computer may also be operable to provide web hosting, databasing, computer application development and/or implementation platforms, combinations of the foregoing or the like. In some examples, the service provider computer may be provided as one or more virtual machines implemented in a hosted computing environment. The hosted computing environment can include one or more rapidly provisioned and released computing resources. These computing resources can include computing, networking and/or storage devices. A hosted computing environment may also be referred to as a cloud computing environment. The service provider computer may include one or more servers, perhaps arranged in a cluster, as a server farm, or as individual servers not associated with one another, and may host application and/or cloud-based software services. These servers may be configured as part of an integrated, distributed computing environment. In some examples, the service provider computer may, additionally or alternatively, include computing devices such as for example a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a desktop computer, a netbook computer, a server computer, a thin-client device, a tablet computer, a gaming console, etc. In some instances, the service provider computer may communicate with one or more third party computers.
The modules described herein may be software modules, hardware modules or a suitable combination thereof. If the modules are software modules, the modules can be embodied on a non-transitory computer readable medium and processed by a processor in any of the computer systems described herein. It should be noted that the described processes and architectures can be performed either in real-time or in an asynchronous mode prior to any user interaction. The modules may be configured in the manner suggested in the preceding figures, and/or functions described herein can be provided by one or more modules that exist as separate modules and/or module functions described herein can be spread over multiple modules.
The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the disclosure as set forth in the claims.
Other variations are within the spirit of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated examples thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the disclosure to the specific form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure, as defined in the appended claims.
The use of the terms “a” and “an” and “the” and similar referents in the context of describing the disclosed examples (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including, but not limited to,”) unless otherwise noted. The term “connected” is to be construed as partly or wholly contained within, attached to, or joined together, even if there is something intervening. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate examples of the disclosure and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.
Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is intended to be understood within the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain examples require at least one of X, at least one of Y, or at least one of Z to each be present.
Various examples of this disclosure are described herein, including the best mode known to the inventors for carrying out the disclosure. Variations of those examples may become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate and the inventors intend for the disclosure to be practiced otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20200310803 | Gradstein | Oct 2020 | A1 |
| 20200356367 | Van Lunteren | Nov 2020 | A1 |
