MEMORY-BASED DEVICE

Abstract

A device includes first and second wires, resistors, and a processor. Input signals are transmitted from the first wires through the resistors to the second wires. The processor receives a sum value of the input signals from one of the second wires, and shifts the sum value by a nonlinear activation function to generate a shifted sum value. The processor calculates a backpropagation value based on the shifted sum value and a target value related to a corresponding input signal of the input signals, and generates a pulse number based on the corresponding input signal of the input signal and the backpropagation value. The processor applies a voltage pulse to one of the resistors related to the corresponding input signal based on the pulse number, in order to modify a value of the corresponding input signal in the input signals to be the same as the target value.

Description

BACKGROUND

In software-defined neural networks (NN), the weight update is computed precisely through mathematical equations, and the value of the weight update is then stored in digital memories. In hardware neural networks (HNNs), the weight update is directly computed and stored in synapses according to the effective number of pulses that one synapse receives during weight update. The weight state of an RRAM-based (Resistive Random Access Memory, RRAM) synapse has limited precision and is bounded. Furthermore, the weight update is asymmetric and nonlinear.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. It is noted that, in accordance with the standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.

FIG. 1 is a diagram of perceptron network implemented in a hardware neural network device, in accordance with various embodiments.

FIG. 2A is a diagram of a hardware neural network device including, for illustration, the perceptron network in FIG. 1, in accordance with various embodiments.

FIG. 2B is a diagram of a hardware neural network device including, for illustration, the perceptron network in FIG. 1, in accordance with various embodiments.

FIG. 3 is an operation flow of the hardware neural network device in FIG. 2A, in accordance with various embodiments.

FIG. 4 is a diagram of a waveform of a nonlinear activation function, in accordance with various embodiments.

FIG. 5 is a diagram of a waveform of a nonlinear activation function, in accordance with various embodiments.

FIG. 6 is a diagram of an experimental data chart of the hardware neural network device in FIGS. 2A, 2B, in accordance with various embodiments.

FIG. 7 is a diagram of an experimental data chart of the hardware neural network device in FIGS. 2A, 2B, in accordance with various embodiments.

FIG. 8 is a diagram of an experimental data chart of the hardware neural network device in FIGS. 2A, 2B, in accordance with various embodiments.

FIG. 9 is a diagram of a calculating circuit for implementing at least a portion of the processor in FIG. 2A or FIG. 2B in accordance with various embodiments.

FIG. 10 is a diagram of a calculating circuit for implementing at least a portion of the processor in FIG. 2A or FIG. 2B in accordance with various embodiments.

DETAILED DESCRIPTION

The following disclosure provides many different embodiments, or examples, for implementing different features of the provided subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. For example, the formation of a first feature over or on a second feature in the description that follows may include embodiments in which the first and second features are formed in direct contact, and may also include embodiments in which additional features may be formed between the first and second features, such that the first and second features may not be in direct contact. In addition, the present disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed.

Further, spatially relative terms, such as “beneath,” “below,” “lower,” “above,” “upper” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. The spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. The apparatus may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein may likewise be interpreted accordingly.

Reference is now made to FIG. 1. FIG. 1 is a diagram of perceptron network implemented in a hardware neural network device, in accordance with various embodiments of the present disclosure.

As illustratively shown in FIG. 1, the perceptron network 100 of the present disclosure includes an input neuron layer 110, a hidden neuron layer 120, and an output neuron layer 130. The input neuron layer 110 includes a plurality of neurons X₁-X_I, the hidden neuron layer 120 includes a plurality of neurons H₁-H_J, and the output neuron layer 130 includes a plurality of neurons O₁-O_K. In some embodiments, two adjacent neuron layers are referred to as pre-neuron and post-neuron layers along a forward propagation direction, and are fully connected to each other. For illustration, the neuron X₁ in the input neuron layer 110 is connected to each of the neurons H₁-H_J, the neuron X₂ in the input neuron layer 110 is also connected to each of the neurons and each of the rest neurons X₃-X₁ in the input neuron layer 110 is also connected to all of the neurons H₁-H_J. The above-mentioned connection manner is regarded as each one neuron X₁ being fully connected to the neurons H₁-H_J. In some embodiments, two adjacent neuron layers are fully connected using RRAM (Resistive Random Access Memory) synapses. The RRAM can be made of single-layer or combination of the following insulators, including, for example, TiOx, TaOx, HfOx, AlOx, ZrOx, WOx, NiO, which are sandwiched between two metal electrodes including, for example, TiN, TaN, Ti, Ta, Pt, Pd, Cu, Ag, Ni, Au, Hf, Zr, Al, W. In various embodiments, the insulator films can be deposited by, for example, physical vapor deposition (PVD), chemical vapor deposition (CVD), atomic layer deposition (ALD), thermal oxidation, or the like.

In some embodiments, the input neuron layer 110 and the hidden neuron layer 120 are two adjacent neuron layers, and input data are inputted from the input neuron layer 110 to the hidden neuron layer 120. The input data includes, for example, a symbol such as “8” or other handwritten numbers/characters, and the symbol is transformed into a binary number or other suitable digital type. Subsequently, the binary number is inputted into the neurons X₁-X_I of the input neuron layer 110. The input neuron layer 110 and the hidden neuron layer 120 are fully connected with each other, and two connected neurons in the input neuron layer 110 and the hidden neuron layer 120 have a weight W_i,j. For instance, the neuron X₁ in the input neuron layer 110 and the neuron H₁ the hidden neuron layer 120 are connected to each other, and there is a weight W_1,1 between the neuron X₁ and the neuron H₁. Each of the neurons H₁-H_J in the hidden neuron layer 120 receives products of every input data and the weight W_i,j, and the product is referred to as a weight sum in some embodiments.

In various embodiments, the hidden neuron layer 120 and the output neuron layer 130 are two adjacent neuron layers, and the input data are inputted from the hidden neuron layer 120 to the output neuron layer 130. The hidden neuron layer 120 and the output neuron layer 130 are fully connected with each other, and two connected neurons in the hidden neuron layer 120 and the output neuron layer 130 have a weight W_j,k. For instance, the neuron H₂ in the hidden neuron layer 120 and the neuron O₂ in the output neuron layer 130 are connected to each other, and there is a weight W_2,2 between the neuron H₂ and the neuron O₂. The weight sum from each of the neurons H₁-H_J of the hidden neuron layer 120 is regarded as an input of the output neuron layer 130. Each of the neurons O₁-O_K in the output neuron layer 130 receives products of every weight sum and the weight W_j,k.

As illustratively shown in FIG. 1, the weight sum outputted from each of the neurons O₁-O_K of the output neuron layer 130 is regarded as an output of the perceptron network 100. The outputs from the neurons O₁-O_K are compared with target values T₁-T_K, respectively. If one of the outputs from the neurons O₁-O_K is different from the corresponding target value of the target values T₁-T_K, the weight W_i,j; between the input neuron layer 110 and the hidden neuron layer 120 and the weight W_j,k between the hidden neuron layer 120 and the output neuron layer 130 are adjusted until the output and the corresponding target value are the same. In some embodiments, the target value is a predetermined value set to be corresponding to the input data, such that the weight between two adjacent neurons may be trained so as to find out the suitable weight value. In some embodiments, it is referred to as a supervised learning through repeatedly training the weight between two adjacent neuron layers, such that the output and the target are the same, so as to increase classification accuracy. Explained in a different way, the supervised learning is performed through repeatedly training the weight between two adjacent neuron layers, such that the input and the target are the same, so as to increase classification accuracy.

Reference is now made to FIG. 2A. FIG. 2A is a diagram of a hardware neural network device including, for illustration, the perceptron network 100 in FIG. 1, in accordance with various embodiments of the present disclosure.

In some embodiments, the perceptron network 100 in FIG. 1 is implemented by the hardware neural network (HNN) device as shown in FIG. 2A of the present disclosure. In some embodiments, the hardware neural network device in FIG. 2A illustrates the relationship between the input neuron layer 110 and the hidden neuron layer 120 of the perceptron network 100 in FIG. 1, which is elaborated in the following descriptions.

In some embodiments, the hardware neural network device in FIG. 2A includes a plurality of wires W1, a plurality of wires W2, a plurality of resistors G₁₁-G_IJ, a processor 210, and a processor 220. For simplicity of illustration, FIG. 2A illustrates only three wires W1 and three wires W2. Various numbers of the wires W1 and W2 are within the contemplated scope of the present disclosure.

As illustratively shown in FIG. 2A, the wires W1 are coupled to the processor 210, and the wires W2 are coupled to the processor 220. The wires W1 are arranged across the wires W2. Explained in a different way, each of the wires W1 and each of the wires W2 are intersected with each other, to form an array as illustratively shown in FIG. 2A. In various embodiments, each of the wires W1 and each of the wires W2 are not coupled to each other.

As illustratively shown in FIG. 2A, the direction of each of the wires W1 is perpendicular to the direction of each of the wires W2. In various embodiments, the wires W1 are parallel to each other, and the wires W2 are parallel to each other. In some embodiments, each of the resistors G₁₁-G_IJ is correspondingly coupled to one of the wires W1 and one of the wires W2. For illustration, one terminal of the resistor G11 is coupled to the upmost wire W1, and the other terminal of the resistor G₁₁ is coupled to the leftmost wire W2, as shown in FIG. 2A. In some embodiments, the wires W1 are coupled to the wires W2 by the resistors G₁₁-G_IJ, respectively. The configuration as illustratively shown in FIG. 2A is referred to as a crossbar array in some embodiments. In various embodiments, the fully connected hardware neural network device in FIG. 2A is implemented by, for illustration, a crossbar RRAM array.

In various embodiments, the wires W1 and the wires W2 are arranged to cross each other for forming an array. Each of the resistors located at the same column of the array has a first terminal and a second terminal. For illustration, the resistors G₁₁, G₂₁, . . . , G_I1 are located at the same column of the array, and each of the resistors G₁₁, G₂₁, . . . , G_I1 has a upper terminal and a lower terminal. As illustratively shown in FIG. 2A, the upper terminals of the resistors G₁₁, G₂₁, . . . , G_I1 are coupled to the wires W1, respectively, and the lower terminals of the resistors G₁₁, G₂₁, . . . , G_I1 are coupled to the same wire W2 located at the leftmost side.

As illustratively shown in FIG. 2A, the processor 210 is configured to provide input data through the wires W1. For illustration, the processor 210 provides input data X₁-X_I through the wires W1 to the array as shown in FIG. 2A, and the processor 210 together with the wires W1 are therefore configured to implement the input neuron layer 110 in FIG. 1. The input data X₁-X_I correspond to the neurons X₁-X_I, and are designated with the same reference numbers as the neurons X₁-X_I as discussed above, for ease of understanding. The wires W2 are fully connected with the wires W1 via the resistors G₁₁-G_IJ. The input data from the wires W1 are thus transmitted to the wires W2 and then the processor 220 for calculating. As such, the wires W2 and the processor 220 are therefore configured to implement the hidden neuron layer 120 in FIG. 1. As mentioned above in FIG. 1, the two connected neurons in the input neuron layer 110 and the hidden neuron layer 120 have a weight W_k,j therebetween, and such a weight is generated based on the input data transmitted through the resistors G₁₁-G_IJ as illustratively shown in FIG. 2A, which will be explained in detail below.

In some embodiments, the weight values are both positive and negative numbers in the perceptron network 100 of FIG. 1. In such embodiments, a dummy wire (for illustration, a wire L2 as discussed below) is able to be introduced in the hardware neural network device of FIG. 2A, to provide a reference value for subtracting, such that negative numbers are able to be generated according to actual requirements, which is elaborated in the following descriptions.

In some embodiments, the hardware neural network device in FIG. 2A further includes a wire L1, a wire L2, a plurality of resistors b₁-b_J, and a plurality of reference resistors G_r. Each of the resistors b₁-b_J is coupled to the wire L1 and corresponding one of the wires W2. For illustration, one terminal of the resistor b₁ is coupled to the leftmost wire W2, and the other terminal of the resistor b₁ is coupled to the wire L1. In addition, each of the reference resistors G_r is coupled to the wire L2 and one of the wires W1. For illustration, one terminal of the upmost reference resistor G_r is coupled to the upmost wire W1, and the other terminal of the upmost reference resistor G_r is coupled to the wire L2. When the input data transmitted through the reference resistors G_r, reference values are generated and transmitted through the wire L2. With the foregoing configuration, the wire L2 provides the reference value for subtracting, such that negative numbers are generated. Based on the above, the perceptron network 100 of FIG. 1 can be implemented by the hardware neural network device of FIG. 2A.

Reference is now made to FIG. 3. FIG. 3 is an operation flow of the hardware neural network device as shown in FIG. 2A, in accordance with various embodiments of the present disclosure.

As illustratively shown in FIG. 3, in the operation 310, the input signals X₁-X^I are transmitted from the wires W1 through the resistors G₁₁-G_IJ to the wires W2, respectively, the input signals X₁-X_I are transmitted from the wires W1 through the reference resistors G_r, respectively, to the wire L2, and a bias signal BS is transmitted from the wire L1 through the resistors b₁-b_J to the wires W2.

In some embodiments, the input signals X₁-X_I are voltage signals, and the processor 210 provides the voltage signals X₁-X_I to input terminals of the wires W1. As illustratively shown in FIG. 2A, the input terminals of the wires W1 are leftmost terminals that are coupled to the processor 210, of the wires W1, and the processor 210 provides the voltage signals X₁-X_I to the leftmost terminals of the wires W1. Subsequently, the voltage signals X₁-X_I are transformed into current signals through the resistors G₁₁-G_IJ, and the current signals are transmitted to the wires W2. In some embodiments, the above-mentioned current signals indicate data including the weight sum values as discussed above, and the processor 220 receives the current signals to calculate the weight sum values.

As discussed above, weight W_i,j is generated based on the input data transmitted through the resistors G₁₁-G_IJ as illustratively shown in FIG. 2A. For illustration, the voltage signal X₁ is transmitted through the resistors G₁₁ and is transformed into a corresponding current signal, and this current signal is then transmitted for the processor 220 to calculate, in order to obtain the weight. Accordingly, the weight is generated based on the voltage signal X₁.

Correspondingly, the bias signal BS is a voltage signal in some embodiments, and the processor 210 provides the voltage signal BS to the input terminal of the wire L1. For illustration in FIG. 2A, the voltage signal BS is transformed into current signals through the resistors b₁-b_J, and the current signals are transmitted to the wires W2. In some embodiments, the above-mentioned current signals transformed from the bias signal BS indicate data including bias values, and the processor 220 receives the current signals to calculate the bias values.

In addition, the processor 210 provides the voltage signals X₁-X_I to the input terminals of the wires W1. For illustration in FIG. 2A, the voltage signals X₁-X_I are transformed into current signals through the resistors G_r, and the current signals are transmitted to the wire L2. In some embodiments, the above-mentioned current signals transformed from the resistors G_r indicate data including reference value for subtracting, and the processor 220 receives the current signals to calculate the reference values.

As illustratively shown in FIG. 3, in the operation 320, the processor 220 is configured to receive, from one of the wires W2, a weight sum value corresponding to the input signals X₁-X_I and a bias value corresponding to the bias signal BS, and receive, from the wire L2, a reference value corresponding to the input signals X₁-X_I. In some embodiments, the weight sum value is given by:

$\begin{array}{cccc}{H}_j& =& f_a\left(S_j^H\right)=f_a\left[b_j+\sum _{i=1}^I\left(X_i\times W_{\mathrm{ij}}\right)\right]& \left(1\right)\\ =& f_a\left\{b_j+\sum _{i=1}^I\left[X_i\times \left(G_{\mathrm{ij}}-G_r\right)\right]\right\}& \left(2\right)\end{array}$

As shown in formula (1) and (2), f_a is a nonlinear activation function, and X_i is the value of the input signal. S_J^H is the weighted sum before activation. b_j is the bias value. W_ij is the weight value. G_ij is a conductance value of one of the resistors G₁₁-G_IJ in the hardware neural network device in FIG. 2A. The parameter G_r in formula (2) will be described in the following discussions.

Reference is now made to FIG. 2B. FIG. 2B is a diagram of a hardware neural network device including, for illustration, the perceptron network 100 in FIG. 1, in accordance with various embodiments of the present disclosure.

As illustratively shown in FIG. 2B, the input neuron layer 110, the hidden neuron layer 120, and the output neuron layer 130 in FIG. 1 are implemented by the hardware neural network device in FIG. 2B. For illustration, the hardware neural network device in FIG. 2B shows a relationship between the input neuron layer 110 and the hidden neuron layer 120 and a relationship between the hidden neuron layer 120 and the output neuron layer 130. In some embodiments, the relationship between the input neuron layer 110 and the hidden neuron layer 120 is configured such that the input neuron layer 110 is the input neuron layer and the hidden neuron layer 120 is the output neuron layer. In some embodiments, the relationship between the hidden neuron layer 120 and the output neuron layer 130 is configured such that the hidden neuron layer 120 is the input neuron layer and the output neuron layer 130 is the output neuron layer.

In some embodiments, the hardware neural network device in FIG. 2B includes a plurality of wires W1, a plurality of wires W2, a plurality of wires W3, a plurality of wires W4, a plurality of resistors M₁₁-M_JK, a processor 210, a processor 220, and a processor 230. With respect to the embodiments of FIG. 2A, like elements in FIG. 2B are designated with the same reference for ease of understanding. The arrangement of the wires W1, the wires W2, the processor 210, and the processor 220 in FIG. 2B are similar to the arrangements of those elements in FIG. 2A, and thus details of such arrangements are therefore omitted herein for brevity. For simplicity of illustration, FIG. 2B illustrates only three wires W3 and three wires W4. Various numbers of the wires W3 and W4 are within the contemplated scope of the present disclosure.

As illustratively shown in FIG. 2B, the wires W3 are coupled to the processor 220, and the wires W4 are coupled to the processor 230. The wires W3 are arranged across the wires W4. Explained in a different way, each of the wires W3 and each of the wires W4 are intersected with each other, to form an additional array as illustratively shown in FIG. 2B. In various embodiments, each of the wires W3 and each of the wires W4 are not coupled to each other.

In various embodiments, the direction of each of the wires W3 is perpendicular to the direction of each of the wires W4. In various embodiments, the wires W3 are parallel to each other, and the wires W4 are parallel to each other. In some embodiments, each of the resistors M₁₁-M_JK is correspondingly coupled to one of the wires W3 and one of the wires W4. For illustration, one terminal of the resistor M₁₁ is coupled to the upmost wire W3 in the corresponding array, and the other terminal of the resistor G₁₁ is coupled to the leftmost wire W4 in the corresponding array, as shown in FIG. 2B. In some embodiments, the wires W3 are coupled to the wires W4 by the resistors M₁₁-M_JK, respectively. The configuration as illustratively shown in FIG. 2B is referred to as a crossbar array in some embodiments. In various embodiments, the fully connected hardware neural network device in FIG. 2B is implemented by, for illustration, a crossbar RRAM array.

As illustratively shown in FIG. 2B, the wires W3 and the wires W4 are arranged to cross each other for forming an array. Each of the resistors located at the same column of the array has a first terminal and a second terminal. For illustration, the resistors M₁₁, M₂₁, . . . , M_J1 located at the same column of the array, and each of the resistors M₁₁, M₂₁, . . . , M_J1 has a upper terminal and a lower terminal. As illustratively shown in FIG. 2B, the upper terminals of the resistors M₁₁, M₂₁, . . . , M_J1 are coupled to the wires W3, respectively, and the lower terminals of the resistors M₁₁, M₂₁, . . . , M_J1 are coupled to the same wire W4 located at the leftmost side.

In some embodiments, the processor 220 is configured to provide weight sums through the wires W3. For illustration, the processor 220 provides weight sums H₁-H_J through the wires W3 to the additional array as shown in FIG. 2B, and the processor 220 together with the wires W3 are therefore configured to implement the hidden neuron layer 120 in FIG. 1. The wires W4 are fully connected with the wires W3 via the resistors M₁₁-M_JK. The weight sums H₁-H_j from the wires W3 are thus transmitted to the wires W4 and then the processor 230 for calculating. As such, the wires W4 and the processor 230 are therefore configured to implement the output neuron layer 130 in FIG. 1. As mentioned above in FIG. 1, the two connected neurons in the hidden neuron layer 120 and the output neuron layer 130 have a weight W_j,k therebetween, and such a weight is generated based on the weight sums transmitted through the resistors M₁₁-M_JK as illustratively shown in FIG. 2B, as discussed above with respect to the embodiments in FIG. 2A. In various embodiments, the number of the hidden neuron layer 120 in FIG. 1 is plural. Therefore, in some embodiments, the number of the processor 220 and the wires W3 implement the hidden neuron layer 120 in FIG. 1 is plural as well, and the plural processors 220 and the plural wires W3 are cascaded to each other to form multilayer of the hidden neuron layer 120 in FIG. 1 according to actual requirements.

Reference is now made to both FIG. 1 and FIG. 2B, the processor 230 is configured to receive the weight sum from each of the neurons H₁-H_J of the hidden neuron layer 120, and the weight sum values are given by:

$\begin{array}{cccc}{O}_k& =& f_a\left(S_k^0\right)=f_a\left[b_k+\sum _{j=1}^J\left(H_j\times W_{\mathrm{jk}}\right)\right]& \left(3\right)\\ =& f_a\left\{b_k+\sum _{i=1}^I\left[H_j\times \left(M_{\mathrm{jk}}-G_r\right)\right]\right\}& \left(4\right)\end{array}$

As shown in formula (3) and (4), f_a is a nonlinear activation function, and H_j is the value of the input signal. S_k⁰ is the weighted sum before activation. b_k is the bias value. W_jk is the weight value. M_jk is a conductance value of the resistors in a hardware neural network device. The parameter G_r in formula (4) will be described in the following discussions.

As shown in formula (2) and (4), G_r is a reference value given by:

G _r=(G_max−G_min)/2   (5)

As shown in formula (5), the conductance values of the resistors G₁₁-G_IJ in the hardware neural network device of FIG. 2A are randomized in the range between a minimum resistor value G_min and a maximum resistor value G_max. The reference value G_r associated with the wire L2 is a value of the difference between the maximum resistor value G_max and the minimum resistor value G_min to be divided by two. Correspondingly, the conductance values of the resistors M₁₁-M_JK in the hardware neural network device of FIG. 2B are randomized in the range between the minimum resistor value G_min and the maximum resistor value G_max. The reference value G_r associated with the wire L4 is a value of the difference between the maximum resistor value G_max and the minimum resistor value G_min to be divided by two. In some embodiments, the reference value G_r associated with the wire L2 is different from the reference value G_r associated with the wire L4. In some other embodiments, the reference value G_r associated with the wire L2 is the same as the reference value G_r associated with the wire L4.

As illustratively shown in FIG. 3, in the operation 330, the processor 220 is configured to shift the weight sum value by a nonlinear activation function to generate a shifted weight sum value. For illustration, after the weight sum value is calculated and obtained, the weight sum value is shifted by one of the following equations:

$\begin{array}{cc}{f}_a\left(X\right)=\frac{1}{1+e^{-\left(x-s\right)}}& \left(6\right)\\ f_a\left(X\right)=\max \left[0,\left(x-s\right)\right]& \left(7\right)\end{array}$

As shown in formula (6) and (7), s is a parallel shifted value of the functions. Reference is now made to FIG. 4. FIG. 4 is a diagram of a waveform of a nonlinear activation function, in accordance with various embodiments of the present disclosure. As illustratively shown in FIG. 4, the waveform S1 is a waveform of the formula (6) with shifting, and the waveform S2 is a waveform without shifting.

FIG. 5 is a diagram of a waveform of a nonlinear activation function, in accordance with various embodiments of the present disclosure. As illustratively shown in FIG. 5, the waveform S1 is a waveform of the formula (7) with shifting, and the waveform S2 is a waveform without shifting. In some embodiments, the activation function in the formula (6) is a sigmoid function. In various embodiments, the activation function in the formula (7) is a rectified linear unit (ReLU) function. The classification accuracy and the immunity to asymmetric nonlinearity of the hardware neural network device as shown in FIGS. 2A and 2B are improved by adopting the nonlinear activation function with shifting, which will be described in the following discussions.

As illustratively shown in FIG. 3, in the operation 340, the processor 220 is configured to calculate a backpropagation value based on the weight shifted sum value and a target value, and to generate a pulse number based on a corresponding input signal and the backpropagation value. Each of a value of the corresponding input signal and the backpropagation value is higher than or equal to a threshold value. As illustratively shown in FIG. 1, the backpropagation value is generated from the output neuron layer 130 and/or the hidden neuron layer 120 and is given by:

$\begin{array}{cc}{O}_k^{\mathrm{BP}}=\left(T_k-O_k\right)f_a^{\prime }\left(S_k^O\right)& \left(8\right)\\ H_j^{\mathrm{BP}}=\sum _{k=1}^K\left(O_k^{\mathrm{BP}}{W}_{\mathrm{jk}}\right)f_a^{\prime }\left(S_j^H\right)& \left(9\right)\end{array}$

As shown in formula (8) and (9), O_k^BP is the backpropagation value of the output. T_k is the target value of each output neuron as discussed above in FIG. 1, f_a′ is the derivative of the activation function, and H_j^BP is the backpropagation value of the hidden neurons.

In some embodiments, in operation 340, the processor 230 calculates a backpropagation value O_k^BP based on the difference between the target value T_k and the output value O_k as well as the weighted sum being calculated using derivative of activation, i.e., f_a′(S_k^O). Substantially, the processor 220 calculates a backpropagation value H_j^BP based on the sum of the backpropagation value O_k^BP and the weight value W_jk as well as the weighted sum being calculated using derivative of activation, i.e., f_a′(S_j^H).

In addition, the pulse numbers applied on the resistors G₁₁-G_IJ, G₁₁-G_JK are given by:

Δn_ij=round(X_i×ηH_j^BP)   (10)

Δn_jk=round(H_j×ηO_k^BP)   (11)

As shown in formula (10) and (11), η is a learning rate that is a scaling factor to regulate training. Rounding is performed considering only an integer of the effective pulse number. Positive and negative Δn induce potentiation and depression of the device conductance, respectively.

As illustratively shown in FIG. 3, in the operation 350, the processor 220 is configured to apply a voltage pulse to one of the resistors G₁₁-M_JK related to the corresponding input signal based on the pulse number. In some embodiments, one corresponding resistor of the resistors G₁₁-M_JK has a new conductance value G^new when the voltage pulse is applied thereto. In some embodiments, the new conductance value G^new generated by applying the voltage pulse to corresponding one of the resistors G₁₁-M_JK is given by:

$\begin{array}{cccc}{G}^{\mathrm{new}}& =& G_P\left(n_P^{\mathrm{old}}\times \Delta n\right)& \left(12\right)\\ =& G_D\left(n_D^{\mathrm{old}}\times \Delta n\right)& \left(13\right)\end{array}$

As shown above in formula (12) and (13), n_P^old is a cumulative number of potentiation pulse corresponding to the potentiation of the device conductance as discussed above, which is applied to the corresponding resistor, before the weight is updated. n_D^old is a cumulative number of depression pulse corresponding to the depression of the device conductance as discussed above, which is applied to the corresponding resistor, before the weight is updated.

As illustratively shown in FIG. 1, if the output of the neuron O₁ is different from the target value T₁, the weight W_j,k is to be adjusted. As described above, the weight W_j,k is generated based on the input signals, including, for illustration, the weight sums H₁-H_J that are transmitted through the resistors M₁₁-M_JK in FIG. 2B. Accordingly, the weight W_j,k can be adjusted by changing the conductance of the resistors M₁₁-M_JK in FIG. 2B. The way of changing the conductance of the resistors M₁₁-M_JK in FIG. 2B is achieved by providing corresponding voltage pulses. Alternatively stated, a voltage pulse is calculated and applied to the corresponding resistor M_JK based on the pulse number, such that the weight W_j,k is adjusted.

Reference is now made to FIG. 1. For illustration, when the output of the neuron O₁ is different from the target value T₁, not only the weight W_j,k is to be adjusted, but the weight W_i,j is also to be adjusted. Therefore, a voltage pulse is calculated and applied to the resistor G₁₁-G_IJ related to the target value T₁ based on the pulse number, such that the weight W_i,j and the weight W_j,k can be adjusted, so as to adjust the output of the neuron O1 to be the same as the target value T₁.

Reference is now made to FIG. 6. FIG. 6 is a diagram of an experimental data chart of the hardware neural network device as shown in FIGS. 2A and 2B, in accordance with various embodiments of the present disclosure.

As illustratively shown in FIG. 6, the curve C1 relates to an experimental data chart adopting an activation function with shifting, and the curve C2 relates to an experimental data chart adopting an activation function with shifting. For illustration, the activation function adopted for the curve C1 is different from the activation function adopted for the curve C2. In some embodiments, the activation function adopted for the curve C1 is, for illustration, a sigmoid function. In various embodiments, the activation function adopted for the curve C2 is, for illustration, a ReLU function. The curve C3 relates to an experimental data chart adopting an activation function without shifting. As illustratively shown in FIG. 6, it is clear that the accuracy of each of the curve C1 and the curve C2 is higher than that of the curve C3. As discussed above, the wires W2 and the processor 220 are configured to implement the hidden neuron layer 120 in FIG. 1. In some embodiments, the processor 220 is configured to receive and shift the weight sum value by a nonlinear activation function from the wire W2 to generate a shifted weight sum value. The configuration as illustratively shown in FIG. 2A adopts a nonlinear activation function with shifting the weight sum value, and the accuracy of experimental data chart is therefore improved. Accordingly, adopting an activation function with shifting improves the immunity to asymmetric nonlinearity.

Reference is now made to FIG. 7. FIG. 7 is a diagram of an experimental data chart of the hardware neural network device in FIGS. 2A and 2B, in accordance with various embodiments of the present disclosure.

According to the experiment results, if the input signal and the feedback are higher than a threshold value, the accuracy is improved. The threshold value is set to be compared with the input signal and the feedback, and the input signal and the feedback will be adopted if they are higher than the threshold value. As illustratively shown in FIG. 7, the curve C1 relates to an experimental data chart adopting the threshold value, and the curve C2 relates to an experimental data chart which does not adopt the threshold value. As illustrated in FIG. 7, it is clear that the accuracy of the curve C1 is higher than that of the curve C2. Accordingly, adopting the threshold value improves the immunity to asymmetric nonlinearity. In some embodiments, the activation function adopted in the embodiments in FIG. 7 is, for illustration, a sigmoid function.

Reference is now made to FIG. 8. FIG. 8 is a diagram of an experimental data chart of the hardware neural network device in FIGS. 2A and 2B, in accordance with various embodiments of the present disclosure.

Correspondingly, according to the experiment results, if the input signal and the feedback are higher than a threshold value, the accuracy is improved. The threshold value is set to be compared with the input signal and the feedback, and the input signal and the feedback will be adopted if they are higher than the threshold value. As illustratively shown in FIG. 8, the curve C1 relates to an experimental data chart adopting the threshold value, and the curve C2 relates to an experimental data chart that does not adopt the threshold value. As illustrated in FIG. 8, it is clear that the accuracy of the curve C1 is higher than that of the curve C2. Accordingly, adopting the threshold value improves the immunity to asymmetric nonlinearity. In some embodiments, the activation function adopted in the embodiment in FIG. 8 is, for illustration, a ReLU function.

FIG. 9 is a diagram of a calculating circuit for implementing at least a portion of the processor in FIG. 2A or FIG. 2B, in accordance with various embodiments. For illustration in FIG. 9, the calculating circuit 900 is, for example, a 2-bit calculating circuit. The calculating circuit 900 is configured to receive 2-bit input data X1, X0 and threshold values th1, th2, and to calculate the input data X1, X0 with the threshold values th1, th2 to generate output data F(x1), F(x2). In some embodiments, the calculating circuit 900 is implemented by a digital circuit.

In some embodiments, a truth table implemented by the calculating circuit 900 is shown below. For convenience of illustration, values of the input data X1, X0 shown in the below truth table are referred to as (X1, X0), the threshold values th1, th2 shown in the below truth table are referred to as (th1, th2), and values of the output data F(x1), F(x2) shown in the below truth table are referred to as (F(x1), F(x2)).

	x1	x0	th1	th2	F(x1)	F(x2)
1	0	0	0	0	0	0
2	0	1	0	0	0	1
3	1	0	0	0	1	0
4	1	1	0	0	1	1
5	0	0	0	1	0	0
6	0	1	0	1	0	1
7	1	0	0	1	1	0
8	1	1	0	1	1	1
9	0	0	1	0	0	0
10	0	1	1	0	0	0
11	1	0	1	0	1	0
12	1	1	1	0	1	1
13	0	0	1	1	0	0
14	0	1	1	1	0	0
15	1	0	1	1	0	0
16	1	1	1	1	1	1

For example, as illustrated in the above mentioned truth table, the input data X1, X0 in item 3 are (1, 0), and the threshold values th1, th2 set in item 3 are (0, 0). Because the bit value of the input data (1, 0) is larger than the bit value of the threshold values (0,0) in item 3, the output data F(x1), F(x2) in item 3 are also (1, 0). For another example, referring to item 10, the input data X1, X0 are (0, 1), and the threshold values set in item 10 are (1, 0). Because the bit value of the input data X1, X0 (0,1) is smaller than the bit value of the threshold values (1, 0) in item 10, the input data X1, X0 (0, 1) is blocked by the calculating circuit 900, such that the output data F(x1), F(x2) in item 10 are (0, 0). Alternatively stated, the input data X1, X0 (0, 1) having values that are smaller than the threshold values thl, th2 are filtered out by the calculating circuit 900 in some embodiments, in order to generate corresponding output data F(x1), F(x2). Other output data F(x1), F(x2) in the above truth table are calculated in the same manner. The rest of output data F(x1), F(x2) may be deduced by analogy, and thus detailed descriptions regarding the calculations are omitted herein.

In some embodiments, the formula implemented in the calculating circuit 900 is given by:

F(x1)=x1x0+x1bar(th1)+x1bar(th2)   (14)

F(x2)=1x0+x0bar(th1)   (15)

FIG. 10 is a diagram of a calculating circuit for implementing for implementing at least a portion of the processor in FIG. 2A or FIG. 2B, in accordance with various embodiments. For illustration in FIG. 10, the calculating circuit 1000 is, for example, a 2-bit calculating circuit. The calculating circuit 1000 is configured to receive 2-bit input data X1, X0 and shifted values s1, s2, and to calculate the input data X1, X0 with the shifted values s1, s2 to generate output data F(x1), F(x2). In some embodiments, the calculating circuit 1000 is implemented by a digital circuit.

In some embodiments, a truth table implemented by the calculating circuit 900 is shown below. For convenience of illustration, values of the input data X1, X0 shown in the below truth table are referred to as (X1, X0), the shifted values s1, s2 shown in the below truth table are referred to as (s1, s2), and values of the output data F(x1), F(x2) shown in the below truth table are referred to as (F(x1), F(x2)).

	x1	x0	s1	s2	F(x1)	F(x2)
1	0	0	0	0	0	0
2	0	1	0	0	0	1
3	1	0	0	0	1	0
4	1	1	0	0	1	1
5	0	0	0	1	0	0
6	0	1	0	1	0	0
7	1	0	0	1	0	1
8	1	1	0	1	1	0
9	0	0	1	0	0	0
10	0	1	1	0	0	0
11	1	0	1	0	0	0
12	1	1	1	0	0	1
13	0	0	1	1	0	0
14	0	1	1	1	0	0
15	1	0	1	1	0	0
16	1	1	1	1	0	0

For example, as illustrated in the above-mentioned truth table, the input data X1, X0 in item 3 are (1, 0), and the shifted values s1, s2 set in item 3 are (0, 0) which indicates no shift. Accordingly, the input data X1, X0 are not shifted in item 3, and the output data F(x1), F(x2) in item 3 are therefore (1, 0) as same as the input data X1, X0. In item 10, the input data X1, X0 are (0, 1), and the shifted values s1, s2 set in item 10 are (1, 0). Because the bit value of the input data (0, 1) is smaller than the shifted values (1, 0) and is shifted therewith, the bit value of the input data (0, 1) is shifted more than the bit value of the input data (0, 1) itself. Accordingly, the output data F(x1), F(x2) in item 10 become (0, 0). Alternatively stated, the input data X1, X0 (0, 1) are shifted based on the shifted values s1, s2 by the calculating circuit 1000 in some embodiments, in order to generate corresponding output data F(x1), F(x2). Other output data F(x1), F(x2) in the above truth table are calculated in the same manner. The rest of output data F(x1), F(x2) may be deduced by analogy, and thus detailed descriptions regarding the calculations are omitted herein.

In some embodiments, the formula of the calculating circuit 1000 is given by:

F(x1)=x1bar(s1)bar(s2)+x1x0bar(s1)   (16)

F(x2)=x0bar(s1)bar(s2)+x1x0bar(s1+x1bar(x0)bar(s2)s1   (17)

Also disclosed is a device that includes first wires, second wires, resistors, and a processor. The second wires are arranged across the first wires. Each of the resistors is coupled to one of the first wires and one of the second wires. Input signals are transmitted from the first wires through the resistors to the second wires. The processor is configured to receive a sum value of the input signals from one of the second wires, and shift the sum value by a nonlinear activation function to generate a shifted sum value. The processor is configured to calculate a backpropagation value based on the shifted sum value and a target value related to a corresponding input signal of the input signals, and generate a pulse number based on the corresponding input signal of the input signal and the backpropagation value. Each of a value of the corresponding input signal and the backpropagation value is higher than or equal to a threshold value. The processor is configured to apply a voltage pulse to one of the resistors related to the corresponding input signal based on the pulse number, in order to modify a value of the corresponding input signal in the input signals to be the same as the target value.

Also disclosed is a device that includes a first net, a second net, and a processor. The first net includes first wires and first resistors. The second net includes second wires and second resistors. The first wires are arranged to form a first array. Each of the first resistors is disposed at one of areas formed between the first wires, and coupled to two of the first wires being across each other. The second wires are arranged to form a second array. Each of the second resistors is disposed at one of areas formed between the second wires, and coupled to two of the second wires being across each other. The processor is configured to obtain first input signals transmitted through the first resistors to calculate first sum values, provide second signals related to the first sum values to the second net, and obtain second input signals transmitted through the second resistors to calculate second sum values. The processor is configured to shift the first sum values and the second sum values by a nonlinear activation function to generate first shifted sum values and second shifted sum values. The processor is configured to calculate first feedbacks of the first net and second feedbacks of the second net based on the second shifted sum values and target values related to a corresponding input signal of the first input signals. The processor is configured to generate first pulse numbers based on the first input signals and the first feedbacks, and generate second pulse numbers based on the second input signals and the second feedbacks. The processor is configured to apply a first voltage pulse to one of the first resistors based on the first pulse numbers, and configured to apply a second voltage pulse to one of the second resistors based on the second pulse numbers, in order to modify a value of the corresponding input signal of the first input signals to be the same as a corresponding target value of the target values.

Also disclosed is a device. The device includes first wires, second wires, resistors and a processor. The first wires are configured to receive input signals. The resistors configured to transmit the input signals to one of the second wires to generate a sum value. The processor is configured to shift the sum value by a nonlinear activation function to generate a shifted sum value, calculate a feedback based on the shifted sum value and a target value related to a corresponding input signal of the input signals, generate a pulse number based on the corresponding input signal and the feedback, apply a voltage pulse to one of the resistors related to the corresponding input signal based on the pulse number, and modify a value of the corresponding input signal to be the same as the target value.

The foregoing outlines features of several embodiments so that those skilled in the art may better understand the aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and/or achieving the same advantages of the embodiments introduced herein. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that they may make various changes, substitutions, and alterations herein without departing from the spirit and scope of the present disclosure.

Claims

1. A memory-based device, comprising: a plurality of first wires;a plurality of second wires arranged across the plurality of first wires;a plurality of resistors, wherein each of the plurality of resistors is coupled to one of the plurality of first wires and one of the plurality of second wires, wherein a plurality of input signals are transmitted from the plurality of first wires through the plurality of resistors to the plurality of second wires; anda processor configured to receive a sum value of the input signals from one of the plurality of second wires, and shift the sum value by a nonlinear activation function to generate a shifted sum value, wherein the processor is configured to calculate a backpropagation value based on the shifted sum value and a target value related to a corresponding input signal of the plurality of input signals, and generate a pulse number based on the corresponding input signal of the plurality of input signals and the backpropagation value, wherein each of a value of the corresponding input signal and the backpropagation value is higher than or equal to a threshold value, wherein the processor is configured to apply a voltage pulse to one of the plurality of resistors related to the corresponding input signal based on the pulse number, in order to modify the value of the corresponding input signal in the plurality of input signals to be the same as the target value.

2. The device of claim 1, wherein the plurality of first wires and the plurality of second wires are arranged to form an array, wherein each of the plurality of resistors located at the same column of the array has a first terminal and a second terminal, the first terminals of the plurality of resistors are coupled to different wires of the plurality of first wires, and the second terminals of the plurality of resistor are coupled to the same wire of the plurality of second wires.

3. The device of claim 2, wherein the processor is configured to respectively shift sum values from the plurality of second wires located in columns of the array by the nonlinear activation function to generate shifted sum values.

4. The device of claim 3, wherein the processor is configured to calculate backpropagation values based on the shifted sum values and a plurality of target values, wherein each of the plurality of target values corresponds to one of the plurality of input signals.

5. The device of claim 4, wherein the processor is configured to compare each of the values of the plurality of input signals with the threshold value, and compare each of the backpropagation values with the threshold value, wherein the processor is configured to generate pulse numbers based on the plurality of input signals and the backpropagation values if each of the plurality of input signals and the backpropagation values is higher than or equal to the threshold value.

6. The device of claim 5, wherein the processor is configured to correspondingly apply voltage pulses to the plurality of resistors, through one of the plurality of first wires and one of the plurality of second wires, based on the pulse numbers, so as to change resistance of the plurality of resistors.

7. The device of claim 1, wherein the nonlinear activation function comprises at least one of a sigmoid function and a rectified linear unit (ReLU) function.

8. A device, comprising: a first net, comprising: a plurality of first wires arranged to form a first array; anda plurality of first resistors, wherein each of the plurality of first resistors is disposed at one of areas formed between the plurality of first wires, and coupled to two of the plurality of first wires being across each other;a second net comprising: a plurality of second wires arranged to form a second array; anda plurality of second resistors, wherein each of the plurality of second resistors is disposed at one of areas formed between the second wires, and coupled to two of the plurality of second wires being across each other; anda processor configured to obtain first input signals transmitted through the plurality of first resistors to calculate first sum values, provide second signals related to the first sum values to the second net, and obtain second input signals transmitted through the plurality of second resistors to calculate second sum values;wherein the processor is configured to shift the first sum values and the second sum values by a nonlinear activation function to generate first shifted sum values and second shifted sum values;wherein the processor is configured to calculate first feedbacks of the first net and second feedbacks of the second net based on the second shifted sum values and a plurality of target values related to a corresponding input signal of the first input signals;wherein the processor is configured to generate a plurality of first pulse numbers based on the first input signals and the first feedbacks, and generate a plurality of second pulse numbers based on the second input signals and the second feedbacks;wherein the processor is configured to apply a first voltage pulse to one of the plurality of first resistors based on the plurality of first pulse numbers, and configured to apply a second voltage pulse to one of the plurality of second resistors based on the plurality of second pulse numbers, in order to modify a value of the corresponding input signal of the first input signals to be the same as a corresponding target value of the plurality of target values.

9. The device of claim 8, wherein the processor is configured to compare each of values of the first input signals with a threshold value, compare each of values of the first feedbacks with the threshold value, and generate the first pulse numbers based on the first input signals and the first feedbacks if each of values of the first input signals and the first feedbacks is higher than or equal to the threshold value.

10. The device of claim 9, wherein the processor is configured to compare each of values of the second input signals with the threshold value, compare each of values of the second feedbacks with the threshold value, and generate the second pulse numbers based on the second input signals and the second feedbacks if each of values of the second input signals and the second feedbacks is higher than or equal to the threshold value.

11. The device of claim 10, wherein the processor is configured to shift the first sum values by the nonlinear activation function to generate the first shifted sum values, wherein the processor is configured to shift the second sum values by the nonlinear activation function to generate the second shifted sum values.

12. The device of claim 11, wherein the first feedbacks comprise first backpropagation values, and the second feedbacks comprise second backpropagation values, wherein the processor is configured to calculate the second backpropagation values based on the second shifted sum values and the target values, and calculate the first backpropagation values based on the first shifted sum values and the second backpropagation values.

13. The device of claim 12, wherein the nonlinear activation function comprises at least one of a sigmoid function and a rectified linear unit (ReLU) function.

14. A device, comprising: a plurality of first wires configured to receive a plurality of input signals;a plurality of second wires;a plurality of resistors configured to transmit the plurality of input signals to one of the plurality of second wires to generate a sum value; anda processor configured to: shift the sum value by a nonlinear activation function to generate a shifted sum value,calculate a feedback based on the shifted sum value and a target value related to a corresponding input signal of the plurality of input signals,generate a pulse number based on the corresponding input signal and the feedback,apply a voltage pulse to one of the plurality of resistors related to the corresponding input signal based on the pulse number, andmodify a value of the corresponding input signal to be the same as the target value.

15. The device of claim 14, wherein the processor is further configured to respectively shift a plurality of sum values from the plurality of second wires located in columns of an array by the nonlinear activation function to generate plurality of shifted sum values.

16. The device of claim 15, wherein the processor is further configured to calculate a plurality of feedbacks based on the shifted sum values and a plurality of target values, wherein each of the target values corresponds to one of the plurality of input signals.

17. The device of claim 16, wherein the processor is further configured to: compare each of the plurality of input signals with a threshold value,compare each of the plurality of feedbacks with the threshold value, andadopt the plurality of input signals and the plurality of feedbacks when the plurality of input signals and the plurality of feedbacks are higher than the threshold value.

18. The device of claim 17, wherein the plurality of feedbacks comprise backpropagation values.

19. The device of claim 18, wherein the processor is further configured to: generate a plurality of pulse numbers based on the plurality of input signals and the plurality of feedbacks, and correspondingly apply voltage pulses to the plurality of resistors, through one of the plurality of first wires and one of the plurality of second wires, based on the plurality of pulse numbers, so as to change resistances of the plurality of resistors.

20. The device of claim 14, wherein the nonlinear activation function comprises at least one of a sigmoid function and a rectified linear unit (ReLU) function.