Patent Application
20220188628
This disclosure relates generally to analog resistive processing systems for neuromorphic computing, and techniques for controlling peripheral circuitry of an analog resistive processing system to perform various operations on an array of resistive processing unit (RPU) cells of the analog resistive processing system. Information processing systems such as Neuromorphic computing systems and artificial neural network (ANN) systems are utilized in various applications such as machine learning and inference processing for cognitive recognition and computing. Such systems are hardware-based systems that generally include a large number of highly interconnected processing elements (referred to as “artificial neurons”) that operate in parallel to perform various types of computations. The artificial neurons (e.g., pre-synaptic neurons and post-synaptic neurons) are connected using artificial synaptic devices which provide synaptic weights that represent connection strengths between the artificial neurons. The synaptic weights can be implemented using an array of RPU cells having tunable resistive memory devices, the conductance states of the RPU cells are encoded or otherwise mapped to the synaptic weights.
Embodiments of the disclosure include analog resistive processing systems for neuromorphic computing, and techniques for dynamically configuring a hardware configuration of peripherical circuitry, such as readout circuitry, of an analog resistive processing system when performing different operations on an array of RPU cells of the analog resistive processing system.
An exemplary embodiment includes a device which comprises an array of RPU, first control lines extending in a first direction across the array of RPU cells, and second control lines extending in a second direction across the array of RPU cells. Each RPU cell is connected at an intersection of one of the first control lines and one of the second control lines. Peripheral circuitry is coupled to the first control lines and to the second control lines, wherein the peripheral circuitry comprises readout circuitry. A control system is operatively coupled to the peripheral circuitry. The control system generates control signals to control the peripheral circuitry to perform a first operation on the array of RPU cells and to perform a second operation on the array of RPU cells. The control signals comprise a first configuration control signal to configure the readout circuitry to have a first hardware configuration when the first operation is performed on the array of RPU cells, and a second configuration control signal to configure the readout circuitry to have a second hardware configuration, which is different from the first hardware configuration, when the second operation is performed on the array of RPU cells.
Other embodiments will be described in the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying figures.
Embodiments of the invention will now be described in further detail with regard to analog resistive processing systems for neuromorphic computing, and techniques for dynamically configuring peripherical circuitry, such as readout circuitry, of an analog resistive processing system to perform different operations on an array of resistive processing unit (RPU) cells of the analog resistive processing system. For example, as explained in further detail below, in some embodiments, the hardware configuration of the readout circuitry of an analog resistive processing system is dynamically configured to have to have different operating signal ranges for different operations (e.g., forward and backward pass operations of a neural network training process) that are performed on array of RPU cells of the analog resistive processing system.
It is to be understood that the various features as shown in the accompanying drawings are schematic illustrations that are not drawn to scale. Moreover, the same or similar reference numbers are used throughout the drawings to denote the same or similar features, elements, or structures, and thus, a detailed explanation of the same or similar features, elements, or structures will not be repeated for each of the drawings. Further, the term “exemplary” as used herein means “serving as an example, instance, or illustration”. Any embodiment or design described herein as “exemplary” is not to be construed as preferred or advantageous over other embodiments or designs.
Exemplary embodiments of the disclosure include in-memory computing systems or computational memory systems, which utilize an array of RPU cells for a dual purpose of storing data and processing the data to perform some computational tasks. In some embodiments, the RPU cells implement resistive memory devices such as resistive random-access memory (ReRAM) devices, phase-change memory (PCM) devices, etc., which have a tunable conductance (G) with variable conductance states over a range from a min conductance (Gmin) to a maximum conductance (Gmax). As noted above, neuromorphic computing systems and ANN systems are types of in-memory computing systems in which artificial neurons are connected using artificial synaptic devices to provide synaptic weights which represent the strength of connection between two artificial neurons. The synaptic weights can be implemented using tunable resistive memory devices, wherein the variable conductance states are used to represent the synaptic weights and to perform computations (e.g., vector-matrix multiplication). The conductance states of the analog resistive memory devices are encoded or otherwise mapped to synaptic weights.
Various types of artificial neural networks, such as deep neural networks (DNNs) and convolutional neural networks (CNNs) implement neuromorphic computing architectures for machine learning applications such as image recognition, object recognition, speech recognition, etc. The in-memory computations associated with such neural networks include, e.g., training computations in which the synaptic weights of the resistive memory cells are optimized by processing a training dataset, and forward inference computations in which the trained neural networks are used for to process input data for purposes of, e.g., classifying the input data, predicting events based on the input data, etc.
DNN training generally relies on a backpropagation algorithm which includes three repeating cycles: forward, backward and weight update, which are repeated many times until a convergence criterion is met. The forward and backward cycles mainly involve computing vector-matrix multiplication in forward and backward directions. This operation can be performed on a 2D array of analog resistive memory cells. In forward cycle, stored conductance values of the resistive memory devise in the 2D array form a matrix, and an input vector is transmitted as voltage pulses through each input rows of the 2D array. In a backward cycle, voltage pulses are supplied from columns as an input, and a vector-matrix product is computed on the transpose of a matrix. The weight update involves calculating a vector-vector outer product which consists of a multiplication operation and an incremental weight update to be performed locally in each resistive memory cell within the 2D array.
A stochastically trained DNN comprising arrays of RPU cells can have synaptic weights implemented using tunable resistive devices. In some embodiments, each RPU cell 110 in the computing system 100 comprises a resistive element with a conductance value that represents a matrix element or weight of the RPU cell 110. In some embodiments, the resistive elements of the RPU cells 110 are implemented using resistive devices such as resistive switching devices (interfacial or filamentary switching devices), ReRAM devices, memristor devices, PCM devices, etc., and other types of devices which have a tunable conductance (or tunable resistance level) which can be programmatically adjusted within a range of a plurality of different conductance levels to tune the weight of the RPU cell 110. In some embodiments, the variable conductance elements of the RPU cells 110 can be implemented using ferroelectric devices such as ferroelectric field-effect transistor devices. Furthermore, in some embodiments, the RPU cells 110 can be implemented using an analog CMOS-based framework in which each RPU cell 110 comprises a capacitor and a read transistor. With the framework, the capacitor serves as a memory element of the RPU cell 110 and stores a weight value in the form a capacitor voltage, wherein the capacitor voltage is applied to a gate terminal of the read transistor to modulate a channel resistance of the read transistor based on the level of the capacitor voltage, and wherein the channel resistance of the read transistor represents the conductance of the RPU cell and is correlated to a level of a read current that is generated based on the channel resistance.
To properly train a DNN and achieve high-accuracy, the operating characteristics of the tunable resistive devices should meet a stringent set of specifications of acceptable RPU device parameters that a given DNN algorithm can tolerate without significant error penalty. These specifications include, for example, variations in the switching characteristics of the resistive memory device, such as, minimum incremental conductance change (+Δgmin) due to a single potentiation pulse, symmetry in up and down conductance changes, tunable range of the conductance values, etc.
The computing system 100 further comprises peripheral circuitry 120 connected to the row control lines RL1, RL2, RL3, . . . , RLm, as well peripheral circuitry 130 connected to the column control lines CL1, CL2, CL3, . . . , CLn. Further, the peripheral circuitry 120 is connected to a data input/output (I/O) interface block 125, and the peripheral circuitry 130 is connected to a data I/O interface block 135. The computing system 100 further comprises control signal circuitry 140 which comprises various types of circuit blocks such as power, clock, bias and timing circuitry to provide power distribution and control signals and clocking signals for operation of the peripheral circuity 120 and 130 of the computing system 100.
In a neuromorphic computing application, the RPU cells 110 comprise artificial synapses that provide weighted connections between pre-neurons and post-neurons. Multiple pre-neurons and post-neurons are connected through the 2D crossbar array of RPU cells 110, which naturally expresses a fully-connected neural network. In some embodiments, the computing system 100 is configured to perform DNN or CNN computations wherein a conductance of each RPU cell 110 represents a matrix element or weight wij, which can be updated or accessed through operations of the peripheral circuitry 120 and 130 (wherein wij, denotes a weight value for the ith row and the jth column in the array of RPU cells 110). As noted above, DNN training generally relies on a backpropagation process which comprises three repeating cycles: a forward cycle, a backward cycle, and a weight update cycle. The computing system 100 can be configured to perform all three cycles of the backpropagation process in parallel, thus potentially providing significant acceleration in DNN training with lower power and reduced computation resources. The computing system 100 can be configured to perform vector-matrix multiplication operations in the analog domain in a parallel manner.
While the row control lines RL and column control lines CL are each shown in FIG. 1 as a single line for ease of illustration, it is to be understood that each row and column control line can include two or more control lines connected to the RPU cells 110 in the respective rows and columns, depending on the implementation and the specific architecture of the RPU cells 110. For example, in some embodiments, each row control line RL can include a complementary pair of word lines for a given RPU cell 110. Moreover, each column control line CL may comprise multiple control lines including, e.g., one or more source lines (SL) and one or more bit lines (BL).
The peripheral circuitry 120 and 130 comprises various circuit blocks which are connected to the respective rows and columns in the 2D array of RPU cells 110, and which are configured to perform vector-matrix multiply functions, matrix-vector multiply functions, and outer product update operations to implement the forward, backward and weight update operations of a backpropagation process (for neural network training), as well inference processing using a trained neural network. For example, in some embodiments, to support RPU cell read/sensing operations (e.g., read a weight value of given RPU cell 110), the peripheral circuitry 120 and 130 comprises pulse-width modulation (PWM) circuitry and read pulse driver circuitry to generate and apply PWM read pulses to the RPU cells 110, in response to input vector values (read input values) received during forward/backward cycles.
More specifically, in some embodiments, the peripheral circuitry 120 and 130 comprises digital-to-analog (D/A) converter circuitry that is configured to receive digital input vectors (to be applied to rows or columns) and convert the digital input vector into analog input vector values that are represented by input voltage voltages of varying pulse width. In some embodiments, a time-encoding scheme is used when input vectors are represented by fixed amplitude Vin=1 V pulses with a tunable duration (e.g., pulse duration is a multiple of 1 ns and is proportional to the value of the input vector). The input voltages applied to rows (or columns) generate output vector values which are represented by output currents, wherein the weights of the RPU cells 110 are read out by measuring the output currents.
The peripheral circuitry 120 and 130 further comprises current integrator circuitry and analog-to-digital (A/D) converter circuitry to integrate read currents (IREAD) which are output and accumulated from the connected RPU cells 110 and convert the integrated currents into digital values (read output values) for subsequent computation. In particular, the currents generated by the RPU cells 110 are summed on the columns (or rows) and this total current is integrated over a measurement time, tmeas, by current readout circuitry of the peripheral circuitry 120 and 130. The current readout circuitry comprises current integrators and analog-to-digital (A/D) converters. In some embodiments, each current integrator comprises an operational amplifier that integrates the current output from a given column (or row) (or differential currents from pairs of RPU cells implementing negative and positive weights) on a capacitor, and an analog-to-digital (A/D) converter converts the integrated current (e.g., an analog value) to a digital value.
The data I/O interfaces 125 and 135 are configured to interface with a digital processing core, wherein the digital processing core is configured to process input/outputs to the computing system 100 (neural core) and route data between different RPU arrays. The data I/O interfaces 125 and 135 are configured to (i) receive external control signals and data from a digital processing core and provide the received control signals and data to the peripheral circuitry 120 and 130, and (ii) receive digital read output values from peripheral circuitry 120 and 130, and send the digital read output values to a digital processing core for processing. In some embodiments, the digital processing core implements a non-linear function circuitry which calculates activation functions (e.g., sigmoid neuron function, softmax, etc.) and other arithmetical operations on data that is to be provided to a next or previous layer of a neural network.
As is known in the art, fully connected DNNs comprise stacks of fully connected layers such that a signal propagates from an input layer to an output layer by going through series of linear and non-linear transformations. The entire DNN expresses a single differentiable error function that maps the input data to class scores at the output layer. Typically, a DNN is trained using a simple stochastic gradient decent (SGD) scheme, in which an error gradient with respect to each parameter is calculated using the backpropagation algorithm. The backpropagation algorithm is composed of three cycles, forward, backward and weight update that are repeated many times until a convergence criterion is met. The forward and backward cycles mainly involve computing vector-matrix multiplication operations in forward and backward directions using the 2D crossbar array of RPU device cells 110 of the computing system shown in
In the computing system 100 of
For a single fully connected layer where N input neurons are connected to M output (or hidden) neurons, the forward pass (
The backward cycle (
Finally, in an update cycle (
In summary, all operations on the weight matrix W can be implemented using the 2D crossbar array of two-terminal RPU device with M rows and N columns where the stored conductance values in the crossbar array form the matrix W. In the forward cycle, input vector x is transmitted as voltage pulses through each of the rows and the resulting vector y can be read as current signals from the columns. Similarly, when voltage pulses are supplied from the columns as an input in the backward cycle, then a vector-matrix product is computed on the transpose of the weight matrix WT. Finally, in the update cycle, voltage pulses representing vectors x and δ are simultaneously supplied from the rows and the columns. In the update cycle, each RPU cell 110 performs a local multiplication and summation operation by processing the voltage pulses coming from the column and the row and hence achieving an incremental weight update.
To determine the product of the xi and δj vectors for the weight update cycle, stochastic translator circuitry in the peripheral circuitry 120 and 130 is utilized to generate stochastic bit streams that represent the input vectors xi and δj. The stochastic bits streams for the vectors xi and δj are fed through rows and columns in the 2D crossbar array of RPU cells, wherein the conductance of a given RPU cell will change depending on the coincidence of the xi and δj stochastic pulse streams input to the given RPU cell. The vector cross product operations for the weight update operation are implemented based on the known concept that that coincidence detection (using an AND logic gate operation) of stochastic streams representing real numbers is equivalent to a multiplication operation. All three operating modes described above allow the RPU cells forming the neural network to be active in all three cycles and, thus, enable a very efficient implementation of the backpropagation algorithm to compute updated weight values of the RPU cells during a DNN training process.
During the forward and backward pass operations where vector-matrix multiplications are performed on the RPU array, the digital input vectors (x or δ) are converted to analog input vectors that are transmitted as voltage pulses with a fixed amplitude and tunable durations over the rows and columns. In some embodiments, the maximal pulse duration represents unity with regard to an integration time (Tmeas→1), wherein all pulse durations are scaled accordingly depending on the values of xi or δj. This scheme works optimally for the forward cycle where all xi in x are with a given range, e.g., [−1,1]. However, this scheme becomes problematic for the backward cycle, as there are no guarantees for the range of the error signal values in δ. For instance, all δj in δ may become significantly smaller than unity (δ«1) as a training process progresses and the classification errors become increasingly smaller. In this instance, the signal strength of the output signals, which are generated during a backward pass operation as a result of low signal strength of the input error signals δj of an input vector δ, may be too small and not compatible with the full scale operating ranges of the readout circuity.
In some embodiments, a noise management system can be implemented in the RPU array to scale the input error signals δj of an input error vector δ based on the maximum input error signal δj of the input error vector δ, so that the input error signals δj fall between e.g., [−1,1], similar to the full input range of the input signals xi of an input vector that is applied to the RPU array during a forward pass operation. However, in some instances, the process of matching the input signals ranges for the forward and backward pass operations may be insufficient to match the resulting output signals ranges for the forward and backward pass operations due to, e.g., the manner in which neural networks (e.g., DNNs) are trained.
In particular, for a forward pass operation, the distribution of Wx is large because the values of the input vector x and the weight values of the weight matrix W are typically correlated (e.g., achieved through the DNN learning which is meant to increase such correlations), whereas for WTδ for the backward pass operation, the error vector δ and the matrix WT are typically uncorrelated vectors and thus have a result close to zero. Consequently, the output signal strengths of the forward and backward pass operations are significantly different. Furthermore, another reasons that weak output signals are generated during a backward pass operation is that many elements δj of an input error vector δ for many DNNs are typically zero or very close to zero. In this instance, scaling the values δj of the input error vector δ may not significantly increase the signal strength of the output signals generated by the backward pass operation, even if the input range of the values δj of the input error vector δ are matched to the input range (e.g., [−1,1]) of the input signals xi of the input vector x for the forward pass operation.
For at least the above-noted reasons, the output signals that are generated as a result of vector-matrix multiplication operations performed during the backward pass operations can be too small and not readily detectable or quantizable in instances where the readout circuitry (which is shared for the forward and backward pass operations) is configured to have a fixed output signal bound b (e.g., operating signal range) which is more optimal for the range of output signals that are generated during the forward pass operations. For instance, the fixed output signal bound b is a result of the current integrator circuits of the shared readout circuitry having a fixed size integration capacitor, or the ADC circuit of the shared readout circuitry having a fixed ADC resolution, etc. In such instances, the analog output signals that relatively small (e.g., close to zero) will be quantized to zero because of the finite ADC resolution.
This effect is particularly severe, when the shared ADC circuit is configured to have a relatively small resolution (e.g., 3-bit, 4-bit, 5-bit resolution), where the ADC bin size (i.e., least significant bit (LSB) voltage) is relatively large. The use of shared readout circuitry (for forward and backward pass operations) having a fixed output bound b (e.g., same size integration capacitor and/or same ADC resolution, etc.) may sufficient to effectively readout and quantize the voltage signals generated during the forward pass operations, but insufficient to effectively readout and quantize the voltage signals that are generated during the backward pass operations (which represent the error signals that are propagated through the neural network layers during the backward pass operations). Because the backward propagating error signals have much lower signal strength than the forward propagating signals, the fixed signal output bound b of the shared readout circuitry can result lead to insufficient quantization of the backward propagating signals, which results in an effect referred to as “vanishing gradients.”
As is known in the art, an ADC circuit converts an analog signal (continuous-time or continuous-amplitude analog signal) to a digital signal through a quantization process. The resolution of an ADC refers to a number of discrete values that the ADC can generate over an allowed range of analog input values. For example, an ADC with an 8-bit resolution can encode an analog input signal to one of 256 different levels (28=256), providing a dynamic range of 256:1. The values can represent the ranges from [0 to 255] (i.e., as unsigned integers) or from [−128 to 127] (i.e., as signed integer), depending on the application. The resolution of the ADC is also denoted by a least significant bit (LSB) voltage (also referred to as “voltage resolution”). The LSB of the ADC represents the smallest interval that can be detected and in the case of an 8-bit ADC, the LSB is approximately 1/256 or 3.9×10−3. The LSB voltage (or voltage resolution) refers to a change in voltage that is needed to guarantee a change in the output code level. The voltage resolution of an ADC is equal to the allowed range of analog input voltage values (e.g., full scale operating voltage range) divided by the number of discrete intervals.
For example, assuming that the full-scale operating voltage range of the ADC is from −10V to +10V (where V can be microvolts (μV), millivolts (mV), etc.), the voltage resolution of an 8-bit ADC would be 20/256, which is approximately 0.078V. If the analog voltage values for a given operation (e.g., a backward pass) fall within a much smaller voltage range, e.g., [−0.1V to +0.1V] as compared to the full-scale operating voltage range [−10V to +10V] of the ADC, the effective voltage resolution for the smaller voltage range would be significantly limited due to the LSB of 0.078V which would cause any analog voltage signal with an absolute magnitude of less than 0.078V to be quantized as zero. In other words, with the smaller voltage range [−0.1V to +0.1V], the 8-bit ADC would not be able to ideally resolve voltage differences smaller than 0.078V. In this scenario, for the 8-bit ADC, it would be more desirable to enhance the voltage resolution of the 8-bit ADC to a greater number of discrete levels (e.g., 256 levels) to more effectively quantize the voltage values within the smaller voltage range (−0.1V to +0.1V). In some embodiments, a lower bit resolution with a smaller full-scale range can be implemented to measure smaller voltages.
Exemplary embodiments of the disclosure implement bound management techniques that are implemented in the analog domain by dynamically changing a configuration (e.g., hardware configuration) of the shared readout circuitry to provide different signal bounds for different operational modes, e.g., forward pass and backward pass operations, of the analog RPU crossbar array. In order to enhance the output signal for the backward pass operation when (δ«1), various techniques can used to dynamically configure the shared readout circuitry to have a first configuration during the forward pass operation and to have a second configuration during the backward pass operation. In particular, in the forward pass operation, the shared readout circuitry is configured to have a first configuration in which the readout circuitry operates with a first output signal bound b1 (e.g., a first operating signal range), and in the backward pass operation, the shared readout circuitry is configured to operate with a second output signal bound b2 (e.g., a second operating signal range), wherein b2<b1.
For the forward pass operation, multiplexers in the peripheral circuitry of the computing system 300 are activated to selectively connect row line driver circuitry 320 to the row lines R1, R2, . . . , Rm. The row line driver circuitry 320 comprises plurality of DAC circuit blocks 322-1, 322-2, . . . , 322-m (collectively DAC circuit blocks 322) which are connected to respective row lines R1, R2, . . . , Rm. In addition, multiplexers in the peripheral circuitry of the computing system 300 are activated to selectively connect readout circuitry 330 to the column lines C1, C2, . . . , Cn. The readout circuitry 330 comprises a plurality of readout circuit blocks 330-1, 330-2, . . . , 330-n, which are connected to respective column lines C1, C2, . . . , Cn. The readout circuit blocks 330-1, 330-2, . . . , 330-n comprise respective current integrator circuitry 332-1, 332-2, . . . , 332-n, and respective ADC circuitry 334-1, 334-2, . . . , 334-n.
The forward pass operation in a neural network is performed to calculate neuron activations of a downstream layer (e.g., hidden layer or output layer) based on neuron activations of an upstream layer (e.g., input layer or hidden layer) and the synaptic weights that connect the neurons of the upstream layer to the neurons of the downstream layer. In
In some embodiments, the DAC circuit blocks 322-1, 322-2, . . . , 322-m each comprise a pulse-width modulation circuitry and driver circuitry which is configured to generate pulse-width modulated (PWM) read pulses V1, V2, . . . , Vm that applied to the respective row lines R1, R2, . . . , Rm. More specifically, in some embodiments, the DAC circuit blocks 322-1, 322-2, . . . , 322-m are configured to perform a digital-to-analog conversion process using a time-encoding scheme where the input vectors are represented by fixed amplitude pulses (e.g., V=1V) with a tunable duration, wherein the pulse duration is a multiple of a prespecified time period (e.g., 1 nanosecond) and is proportional to the value of the input vector. For example, a given digital input value of 0.5 can be represented by a voltage pulse of 4 ns, while a digital input value of 1 can be represented by a voltage pulse of 80 ns (e.g., a digital input value of 1 can be encoded to an analog voltage pulse with a pulse duration that is equal to the integration time Tmeas). As shown in
During a forward pass operation, the analog input voltages V1, V2, . . . , Vm (e.g., read pulses), are applied to the row lines R1, R2, . . . , Rm, wherein each RPU 310 generates a corresponding read current IREAD=Vi×Gij (based on Ohm's law), wherein V, denotes the analog input voltage applied to the given RPU cell 310 on the given row i and wherein Gij denotes the conductance value of the given RPU cell 310 (at the given row i and column j). As shown in
The resulting aggregate read currents I1, I2, . . . , In at the output of the respective columns C1, C2, . . . , Cn are input to respective readout circuit blocks 330-1, 330-2, . . . , 330-n of the readout circuitry 330. The aggregate read currents I1, I2, . . . , In are integrated by the respective current integrator circuits 332-1, 332-2, . . . , 332-n to generate respective output voltages, which are quantized by the respective ADC circuits 334-1, 334-2, . . . , 334-n to generate respective digital output signals y1, y2, . . . , yn of an output vector y. The digital output signals y1, y2, . . . , yn are processed and transmitted to the next downstream layer to continue the forward propagation operation. As data propagates forward through the neural network, vector-matrix multiplications are performed, wherein the hidden neurons/nodes take the inputs, perform a non-linear transformation, and then send the results to the next weight matrix. This process continues until the data reaches the output layer comprising output neurons/nodes. The output neurons/nodes evaluate classification errors, and generate classification error signals δ which are propagated back through the neural network using backward pass operations. The error signals δ can be determined as a difference between the results of the forward inference classification (estimated labels) and the correct labels at the output layer of the neural network.
In some embodiments, as noted above, the rows and columns do not share the DAC circuitry such that each row and column has a dedicated DAC circuit block, as illustrated in
As shown in
During the backward pass operation, the analog voltage signals V1, V2, . . . , Vn (e.g., read pulses which represent the error signals), are applied to the column lines C1, C2, . . . , Cn, wherein each RPU 310 generates a corresponding read current IREAD=Vj×Gij (based on Ohm's law) wherein Vj denotes the analog input voltage applied to the given RPU cell 310 on the given column j and wherein Gij denotes the conductance value of the given RPU cell 310 (at the given row i and column j). As shown in
The resulting aggregate read currents I1, I2, . . . , Im at the output of the respective rows R1, R2, . . . , Rm are input to the respective readout circuit blocks 330-1, 330-2, . . . , 330-m of the shared readout circuitry 330. The aggregate read currents I1, I2, . . . , Im are integrated by the respective current integrator circuits 332-1, 332-2, . . . , 332-m to generate respective output voltages, which are quantized by the respective ADC circuits 334-1, 334-2, . . . , 334-m to generate respective digital output signals z1, z2, . . . , zn of an output vector z. The digital output signals z1, z2, . . . , zn are then processed and transmitted to the next upstream layer to continue the backward propagation operation. This process continues until the error signals reach the input layer.
After the backward pass operation is completed on the array of RPU cells 305 of the computing system 300, a weight update process is performed to tune conductance values of the RPU cells 310 (which represent the conductance matrix G of the array of RPU cells 305) based on the forward-propagated digital signals x1, x2, . . . , xm and the backward-propagated digital error signals δ1, δ2, . . . , δn, that were received by the array of RPU cells 305 during the forward and backward pass operations. Once the error signal values (or delta values) have been integrated for a given neuron layer, that layer is ready for weight update. The update process that is performed on the array of RPU cells 305 of the computing system 300 can be pipelined with the backward propagation of the error vector δ through additional upstream layers of the computing system 300. In some embodiments, the backward propagation from the first hidden layer back to the input layer neurons is performed, but not required as the input neurons have no upstream synapses, so the highest layer that uses the δ error values is a first hidden layer.
As further illustrated in
Furthermore, as shown in
It is to be understood that the mode control system 350 shown in
While
More specifically,
Next,
In other words, since the conductance values of RPU devices can only be positive, differential scheme in
A shown in
Next,
A shown in
Exemplary embodiments will now be discussed in further detail in conjunction with
For ease of illustration,
As schematically illustrated in
As further shown in
In this configuration, the multiplexer circuit 630 is configured to control the sharing of the readout circuit block 630 for forward and backward pass operations performed by the RPU crossbar array. For example, in the exemplary embodiment of
The current integrator circuit 640 performs an integration function over an integration period (Tmeas) to convert an input current at the input node N1 of the current integrator circuit 640 to an analog voltage VOUT at the output node N2 of the current integrator circuit 640. At the end of the integration period, the ADC circuit 650 latches in the output voltage VOUT, and quantizes the output voltage VOUT to generate a digital signal which corresponds to the analog output voltage VOUT. The input current can be (i) an aggregate column current which is output from the column line COL(i) that is selectively connected (via operation of the multiplexer circuit 630) to the input node N1 of the current integrator circuit 640 during a forward pass operation or (ii) an aggregate row current which is output from the row line ROW (i) that is selectively connected (via operation of the multiplexer circuit 630) to the input node N1 of the current integrator circuit 640 during a backward pass operation.
The current integrator circuit 640 is configured as an operational transconductance amplifier (OTA) with a selectable capacitive feedback provided by one of the first and second integrating capacitors 644-1 and 644-2 to convert the input current (aggregate row current or aggregate column current) to an output voltage VOUT on output node N2 of the current integrator circuit 640. In the exemplary configuration of
The first and second switches 646-1 and 646-2 are controlled by the respective control signals CAP_Select and
In this configuration, when the control signal CAP_Select is asserted to logic “1”, the first switch 646-1 is activated (closed) and the first integration capacitor 644-1 is selectively connected in the feedback path between the input node N1 and output node N2 of the operational amplifier 642. In this instance, the complementary control signal
In the exemplary embodiment of
In general, the output voltage VOUT generated by the current integrator circuit 640 is determined as:
wherein R denotes the respective resistance values (or conductance values) of the RPU cells that contribute to the aggregate current (row current or column) current at the input node N1 of the current integrator circuit 640, wherein CINT denotes the capacitance value of the integrating capacitor selectively connected in the feedback path between the input and output nodes N1 and N2 of the operational amplifier 642, wherein VIN denotes the respective input voltage pulses that contribute to the generation of the aggregated row or column current, and wherein TMEAS denotes the integration time. As shown, the magnitude of the output voltage VOUT is indirectly proportional to the capacitance value CINT of the integrating capacitor. As such, for given values of R, TMEAS, and VIN, the output voltage VOUT increases as the capacitance CINT of the feedback integrating capacitor decreases.
In an exemplary embodiment of the system configuration shown in
For ease of illustration,
As schematically illustrated in
The current integrator circuit 740 performs integration function over an integration period (TMEAS) to convert an input current at the input node N1 of the current integrator circuit 740 to an analog voltage VOUT at the output node N2 of the current integrator circuit 740. The input current can be (i) an aggregate column current which is output from the column line COL(i) that is selectively connected (via operation of the multiplexer circuit 730) to the input node N1 of the current integrator circuit 740 during a forward pass operation or (ii) an aggregate row current which is output from the row line ROW (i) that is selectively connected (via operation of the multiplexer circuit 730) to the input node N1 of the current integrator circuit 740 during a backward pass operation. In the exemplary embodiment of
As further shown in
The control signal ADC_L comprises control signal that is output from the mode control system 710 when the RPU crossbar array is performing a forward pass operation, and the control signal ADC_H comprises control signal that is output from the mode control system 710 when the RPU crossbar array is performing a backward pass operation. In some embodiments, selection circuitry 760 is configured to (i) selectively connect the output node N2 of the current integrator circuit 740 to the input of the first ADC circuit 750-1 in response to an assertion of the control signal ADC_H, and (ii) selectively connect the output node N2 of the current integrator circuit 740 to the input of the second ADC circuit 750-2 in response to an assertion of the control signal ADC_L.
In some embodiments, the first ADC circuit 750-1 comprises a first resolution and the second ADC circuit 750-2 comprises a second resolution, wherein the first resolution is higher than the second resolution. In some embodiments, the first and second ADC circuits 750-1 and 750-2 comprise the same bit resolution wherein the first and second ADC circuits 750-1 and 750-2 each comprise an n-bit ADC resolution where, e.g., n=3, 4, 5, 6, 7, 8, but where the first and second ADC circuits 750-1 and 750-2 are configured to have different least significant bit (LSB) voltage resolutions for the given n-bit resolution.
For example, assume that the first and second ADC circuits 750-1 and 750-1 are 8-bit resolution ADCs, but where the first ADC circuit 750-1 is configured to have a full-scale operating voltage range of, e.g., [−0.1V to +0.1V], and the second ADC circuit 750-2 is configured to have a full-scale operating voltage range of, e.g., [−10V to +10V] (where V can be microvolts (μ,V), millivolts (mV), etc.). In this instance, the voltage resolution (first resolution) of the first ADC circuit 750-1 (8-bit ADC) would be 0.2/256, which is approximately 0.000078V, while the voltage resolution (second resolution) of the second ADC circuit 750-2 (8-bit ADC) would be 20/256, which is approximately 0.078V. In the example, the first voltage resolution (0.000078V) of the first ADC circuit 750-1 provides a higher voltage resolution than the second voltage resolution (0.078V) of the second ADC circuit 750-2 (8-bit ADC). The enhanced voltage resolution of the first ADC circuit 750-1 would be more effective to digitize the lower level output voltages VOUT generated at the output node N2 of the current integrator circuit 740 during backward pass operations, while voltage resolution of the second ADC circuit 750-2 would be effective to digitize the higher level output voltages VOUT generated at the output node N2 of the current integrator circuit 740 during forward pass operations.
In other embodiments, the first and second ADC circuits 750-1 and 750-2 can be configured to have different gains, wherein a first gain of the first ADC circuit 750-1 is greater than a second gain of the second ADC circuit 750-2. For example, in some embodiments, the first and second ADC circuits 750-1 and 750-2 can be configured to have ADC conversion circuitry with the same bit resolution, same voltage resolution, and same operating voltage input range designed for the expected voltage range of the output voltages VOUT generated by forward pass operations, but where the first ADC circuit 750-1 comprises an analog front end comprising an amplifier and/or or level-shifting circuitry that is configured to provide a proper gain and level shifting of the low level output voltages VOUT (generated during backward pass operations) to match the operating input range of the ADC conversion circuitry.
Next, 7B schematically illustrates a system 701 for dynamically configuring the shared readout circuitry to select between two different ADC circuits which have different resolutions depending on the operating mode (forward pass or backward pass) of RPU crossbar array, according to an exemplary embodiment of the disclosure. As noted above, the system 701 of
For ease of illustration,
As schematically illustrated in
The selection circuitry 731 is configured to operate in a manner similar to the multiplexer circuits 630 and 730 of
In an alternative embodiment of
Again, for ease of illustration,
As schematically illustrated in
In some embodiments, the configurable ADC circuit 850 comprises a configurable LSB voltage resolution which is dynamically adjusted in response to the control signals ADC_H and ADC_L. For example, in response to an assertion of the control signal ADC_H during a backward pass operation, the ADC circuit 850 is dynamically configured to increase the voltage resolution to a level which is sufficient to effectively quantize the low level output voltage VOUT that is generated on the output node N2 of the current integrator circuit 840 during a backward pass operation. Further, in response to an assertion of the control signal ADC_L during a forward pass operation, the configurable ADC circuit 850 is dynamically configured to decrease the voltage resolution to a level which is sufficient to effectively quantize the high level output voltage VOUT that is generated on the output node N2 of the current integrator circuit 840 during the forward pass operation. In this regard, the configurable ADC circuit 850 is dynamically configured to have a first resolution for a backward pass operation, and dynamically configured to have a second resolution for a forward pass operation, wherein the first resolution is greater than the second resolution.
The ADC circuit 850 with a configurable ADC resolution can be implemented using suitable time-based ADC conversion circuits and techniques. For example, the configuration ADC circuit 850 can be implemented using a single-slope or dual-slope integration ADC architecture, wherein the ADC conversion is based on the integration of the output voltage VOUT, a reference voltage, or the output voltage VOUT and a reference voltage. In this exemplary embodiment, the LSB voltage resolution can be dynamically modified by changing one or more operating parameters of the integrating ADC including, but not limited to, the level of the reference voltage, the integration time of the integration ADC, etc.
In other embodiments, the configurable ADC circuit 850 comprises a configurable gain which is dynamically adjusted in response to the control signals ADC_H and ADC_L. For example, the configurable ADC circuit 850 may have a fixed (non-configurable) resolution which is designed for the output signal bound for forward pass operations, but yet implement a programmable gain amplifier and level-shifting circuitry in an analog front end of the ADC circuit 850 to dynamically adjust a gain of the programmable in amplifier depending on the operation mode (e.g., forward pass or backward pass operations).
For example, in some embodiments, in response to an assertion of the control signal ADC_H during a backward pass operation, the front end analog circuitry of the ADC circuit 850 is dynamically configured to have a higher gain to amplify the output voltage VOUT to a level which falls within the higher operating input voltage range of the conversion circuitry of the ADC circuit 850, thereby allowing the low level output voltage VOUT to be more accurately quantized based on the fixed resolution of the ADC circuit 850. Further, in response to an assertion of the control signal ADC_H during a forward pass operation, the frontend analog circuitry of the ADC circuit 850 is dynamically configured to have a lower gain (e.g., unity gain of 1) so that the output voltage VOUT (which is latched into the ADC circuit 850 from the output node of the current integrator circuit 840) is maintained at a level which falls within the operating input voltage range of the conversion circuitry of the ADC circuit 850.
Referring to
As schematically illustrated in
The current integrator circuit 940 comprises an operational amplifier 942, an integrating capacitor 944 which is connected between input and output nodes N1 and N2 of the operational amplifier 942, and control circuitry 946 including an integration time counter and reset control circuitry. The current integrator circuit 940 performs an integration function to convert an input current at the input node N1 of the current integrator circuit 940 to an analog voltage VOUT at the output node N2 of the current integrator circuit 940, wherein the current integration operation is performed over configurable integration period which can be dynamically adjusted to have a first integration time T1MEAS or a second integration time T2MEAS depending on an operating mode (e.g., forward pass or backward pass operation) of the RPU crossbar array. The second integration time T2MEAS is greater than the first integration time T1MEAS. For example, in some embodiments first integration time T1MEAS is 80 nanoseconds (ns), while second integration time T2MEAS is greater than 80 ns (e.g., T2MEAS=2×T1MEAS).
In the exemplary embodiment of
On the other hand, during a backward pass operation, the current integrator circuit 940 is dynamically configured to integrate the input current over the second integration period T2MEAS to enable a larger output voltage VOUT to be generated on the output node N2 of the current integrator circuit 940 while preventing or otherwise minimizing the possibility of saturating the operational amplifier 942 during the current integration operation for the backward pass operation. As schematically illustrated in
While the control circuitry 946 is generically depicted in
In an exemplary embodiment, the control circuitry 946 comprises an integration time counter which counts a number of clock pulses that are input to the integration time counter, wherein the integration time is correlated to a specific count of received clock pulses, as is understood by those of ordinary skill in the art. In this regard, in some embodiments, the integration time of the current integrator circuit 940 is adjusted by configuring the integration time counter circuitry of the control circuitry 946 to perform one of (i) a first counting process which correlates to the first integration time T1MEAS and (ii) a second counting process which correlates to the second integration time T2MEAS, in response to the integration time control signals T1MEAS_Mode, and T1MEAS_Mode.
In addition to increasing the integration time (e.g., T2MEAS) of the current integrator circuit 946 for the backward pass operation, in some embodiments, the DAC circuit blocks of the line driver circuitry (e.g., DAC circuit blocks 342 of the column line driver circuitry 340,
In the exemplary embodiment of
The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
