Neuron unit

The present invention generally relates to neuron units, neural networks and signal processing methods, and more particularly to a neuron unit which resembles neurons and is applicable to neural computers, a neural network which includes a plurality of such neuron units which are coupled to form a hierarchical network structure and a signal processing method which uses such a neural network.
In a living body, processes such as character recognition, memory by association and control of motion can be carried out quite simply. However, such processes are often extremely difficult to carry out on Neumann computers.
Hence, in order to cope with the problems encountered in the Neumann computers, various models of neuron units and neural networks have been proposed. The neuron unit resembles a neuron of the living body, and the neural network uses such neuron units which form a network so as to carry out parallel information processing and self teaching which are functions peculiar to a nervous system of the living body.
Presently, the neural network is in most cases realized by computer simulation. However, in order to bring out the advantageous features of the neural network, it is necessary to realize the parallel processing by hardware.
Some proposals have been made to realize the neural network by hardware, however, the proposed neural networks cannot realize the self learning function which is another advantageous feature of the neural network. Furthermore, the majority of the proposed neural networks are realized by analog circuits can suffer from the problems which will be described later in conjunction with the drawings.
First, a description will be given of a model of a conventional neural network. FIG. 1 shows one neuron unit 1, and FIG. 2 shows a neural network which is made up of a plurality of such neuron units 1. Each neuron unit 1 of the neural network is coupled to and receives signal from a plurality of neuron units 1, and outputs a signal by processing the received signals. In FIG. 2, the neural network has a hierarchical structure, and each neuron unit 1 receives signals from the neuron units 1 located in a previous layer shown on the left side and outputs a signal to the neuron units 1 located in a next layer shown on the right side.
In FIG. 1, T.sub.ij denotes a weight function which indicates the intensity of coupling (or weighting) between an ith neuron unit and a jth neuron unit. The coupling between first and second neuron units is referred to as an excitatory coupling when a signal output from the second neuron unit increases as a signal received from the first neuron unit increases. On the other hand, the coupling between the first and second neuron units is referred to as an inhibitory coupling when the signal output from the second neuron unit decreases as the signal received from the first neuron unit increases. T.sub.ij >0 indicates the excitatory coupling, and T.sub.ij <0 indicates the inhibitory coupling.
FIG. 1 shows the jth neuron unit 1 which outputs an output signal y.sub.j. When an output signal of the ith neuron unit 1 is denoted by Y.sub.i, the input signal to the jth neuron unit 1 from the ith neuron unit 1 can be described by T.sub.ij Y.sub.i. Since a plurality of neuron units 1 are coupled to the jth neuron unit 1, the input signals to the jth neuron unit 1 can be described by .SIGMA.T.sub.ij y.sub.i. The input signals .SIGMA.T.sub.ij y.sub.i to the jth neuron unit 1 will hereinafter be referred to as an internal potential u.sub.j of the jth neuron unit 1 as defined by the following equation (1).
u.sub.j =.SIGMA.T.sub.ij y.sub.i ( 1)
Next, it will be assumed that a non-linear process is carried out on the input. The non-linear process is described by a non-linear neuron response function using a sigmoid function as shown in FIG. 3 and the following equation (2).
f(x)=1/(1+e.sup.-x) (2)
Hence, in the case of the neural network shown in FIG. 2, the equations (1) and (2) are successively calculated for each weight function T.sub.ij as as to obtain a final output.
FIG. 4 shows an example of a conventional neuron unit proposed in a Japanese Laid-Open Patent Application No. 62-295188. The neuron unit includes a plurality of amplifiers 2 having an S-curve transfer function, and a resistive feedback circuit network 3 which couples outputs of each of the amplifiers 2 to inputs of amplifiers in another layer as indicated by a one-dot chain line. A time constant circuit 4 made up of a grounded capacitor and a grounded resistor is coupled to an input of each of the amplifiers 2. Input currents I.sub.1, I.sub.2, . . . , I.sub.N are respectively applied to the inputs of the amplifiers 1, and output is derived from a collection of output voltages of the amplifiers 2.
An intensity of the coupling (or weighting) between the neuron units is described by a resistance of a resistor 5 (a lattice point within the resistive feedback circuit network 3) which couples the input and output lines of the neuron units. A neuron response function is described by the transfer function of each amplifier 2. In addition, the coupling between the neuron units may be categorized into the excitatory and inhibitory couplings, and such couplings are mathematically described by positive and negative signs on weight functions. However, it is difficult to realize the positive and negative values by the circuit constants. Hence, the output of the amplifier 2 is distributed into two signals, and one of the two signals is inverted so as to generate a positive signal and a negative signal. One of the positive and negative signals derived from each amplifier 2 is appropriately selected. Furthermore, an amplifier is used to realize the sigmoid function shown in FIG. 3.
However, the above described neuron unit suffers from the following problems.
(1) The weight function T.sub.ij is fixed. Hence, a value which is learned beforehand through a simulation or the like must be used for the weight function T.sub.ij, and a self-learning cannot be made.
(2) Because the signal intensity is described by an analog value of potential or current and internal operations are also carried out in the analog form, the output value easily changes due to the temperature characteristic, the drift which occurs immediately after the power source is turned ON and the like.
(3) When the neural network is formed by a large number of neuron units, it is difficult to obtain the large number of neuron units which have the same characteristic.
(4) When the accuracy and stability of one neuron unit are uncertain, new problems may arise when a plurality of such neuron units are used to form the neural network. As a result, the operation of the neural network becomes unpredictable.
On the other hand, as a learning rule used in numerical calculations, there is a method called back propagation which will be described hereunder.
First, the weight functions are initially set at random. When an input is applied to the neural network in this state, the resulting output is not necessarily a desirable output. For example, in the case of character recognition, a resulting output "the character is `L`" is the desirable output when a handwritten character "L" is the input, however, this desirable output is not necessarily obtained when the weight functions are initially set at random. Hence, a correct solution (teaching signal) is input to the neural network and the weight functions are varied so that the correct solution is output when the input is the same. The algorithm for obtaining the varying quantity of the weight functions is called the back propagation.
For example, in the hierarchical neural network shown in FIG. 2, the weight function T.sub.ij is varied using the equation (4) so that E described by the equation (3) becomes a minimum when the output of the jth neuron unit in the output (last) layer is denoted by y.sub.j and the teaching signal with respect to this jth neuron unit is denoted by d.sub.j.
E=.SIGMA.(d.sub.j -y.sub.j).sup.2 ( 3)
.DELTA.T.sub.ij =.differential.E/.differential.T.sub.ij ( 4)
Particularly, when obtaining the weight functions of the output layer and the layer immediately preceding the output layer, an error signal .delta. is obtained using the equation (5), where f' denotes a first order differential function of the sigmoid function f.
.delta..sub.j =(d.sub.j -y.sub.j).times.f'(u.sub.j) (5)
When obtaining the weight functions of the layers preceding the layer which immediately precedes the output layer, the error signal .delta. is obtained using the equation (6).
.delta..sub.j =.delta..sub.j T.sub.ij .times.f'(u.sub.j) (6)
The weight function T.sub.ij is obtained from the equation (7) and varied, where .DELTA.T.sub.ij ' and T.sub.ij ' are values respectively obtained during the previous learning, .eta. denotes a learning constant and .alpha. denotes a stabilization constant.
.DELTA.T.sub.ij =.eta.(.delta..sub.j y.sub.i)+.alpha..DELTA.T.sub.ij '
T.sub.ij =T.sub.ij '+.DELTA.T.sub.ij ( 7)
The constants .eta. and .alpha. are obtained through experience since these constants .eta. and .alpha. cannot be obtained logically. The convergence is slower as the values of these constants .eta. and .alpha. become smaller, and an oscillation tends to occur when the values of these constants .eta. and .alpha. are large. Generally, the constants .eta. and .alpha. are in the order of "1".
The neural network learns in the above described manner, and an input is thereafter applied again to the neural network to calculate an output and learn. By repeating such an operation, the weight function T.sub.ij is determined such that a desirable resulting output is obtained for a given input.
When an attempt is made to realize the above described learning function, it is extremely difficult to realize the learning function by a hardware structure since the learning involves many calculations with the four fundamental rules of arithmetics.
On the other hand, a neural network realized by digital circuits has been proposed in Hirai et al., "Design of Completely Digital Neuro-Chip" Electronic Information and Communication Society, ICD-88-130, Dec. 16, 1988.
FIG. 5 shows a circuit construction of a single neuron. In FIG. 5, each synapse circuit 6 is coupled to a cell circuit 8 via a dendrite circuit 8.
FIG. 6 shows an example of the synapse circuit 6. In FIG. 6, a coefficient multiplier circuit 9 multiplies a coefficient a to an input pulse f, where the coefficient a is "1" or "2" depending on the amplification of a feedback signal. A rate multiplier 10 receives an output of the coefficient multiplier circuit 9. A synapse weighting register 11 which stores a weight function w is connected to the rate multiplier 10.
FIG. 7 shows an example of the cell circuit 8. In FIG. 7, a control circuit 12, an up/down counter 13, a rate multiplier 14 and a gate 15 are successively connected in series. In addition, an up/down memory 16 is connected as shown.
In this proposed neural network, the input and output of the neuron circuit is described by a pulse train, and the signal quantity is described by the pulse density of the pulse train. The weight function is described by a binary number and stored in the memory 16. The input signal is applied to the rate multiplier 14 as the clock and the weight function is applied to the rate multiplier 14 as the rate value, so that the pulse density of the input signal is reduced depending on the rate value. This corresponds to the term T.sub.ij y.sub.i of the back propagation model. The portion which corresponds to .SIGMA. of .SIGMA.T.sub.ij y.sub.i is realized by an OR circuit which is indicated by the dendrite circuit 7.
Because the coupling may be excitatory or inhibitory, the circuit is divided into an excitatory group and an inhibitory group and an OR operation is carried out independently for the excitatory and inhibitory groups. Outputs of the excitatory and inhibitory groups are respectively applied to up-count and down-count terminals of the counter 13 and counted in the counter 13 which produces a binary output. The binary output of the counter 13 is again converted into a corresponding pulse density by use of the rate multiplier 14.
A plurality of the neurons described above are connected to form a neural network. The learning of this neural network is realized in the following manner. That is, the final output of the neural network is input to an external computer, a numerical calculation is carried out within the external computer, and a result of the numerical calculation is written into the memory 16 which stores the weight function. Accordingly, this neural network does not have the self-learning function. In addition, the circuit construction of this neural network is complex because a pulse density of a signal is once converted into a numerical value by use of a counter and the numerical value is again converted back into a pulse density.
Therefore, the conventional neural network or neural network suffer from the problem in that the self-learning function cannot be realized by hardware.
Furthermore, the analog circuits do not provide stable operations, and the learning method using numerical calculation is extremely complex and is unsuited to be realized by hardware. On the other hand, the circuit construction of the digital circuits which provide stable operations is complex.
SUMMARY OF THE INVENTION
Accordingly, it is a general object of the present invention to provide a novel and useful neuron unit, neural network and signal processing method, in which the problems described above are eliminated.
Another and more specific object of the present invention is to provide a neuron unit for processing a plurality of input signals and for outputting an output signal which is indicative of a result of the processing, the neuron unit comprising input line means for receiving the input signals, forward process means coupled to the input line means and including supplying means for supplying weight functions, and operation means for carrying out an operation on each of the input signals using one of the weight functions and for outputting the output signal, and self-learning means coupled to the forward process means and including generating means for generating new weight functions based on errors between the output signal of the forward process means and teaching signals, and varying means for varying the weight functions supplied by the supplying means of the forward process means to the new weight functions generated by the generating means. According to the neuron unit of the present invention, it is possible to realize the functions of a neuron including the self-learning function.
Still another object of the present invention is to provide the neuron unit described above wherein the supplying means and the operation means of the forward process means are made up of analog circuits.
A further object of the present invention is to provide the neuron unit described above wherein the supplying means and the operation means of the forward process means are made up of digital circuits.
Another object of the present invention is to provide the neuron unit described above wherein the input line means includes a plurality of first input lines for receiving first binary input signals which undergo transitions with time and a plurality of second input lines for receiving second binary input signals which undergo transitions with time, the supplying means includes first and second memory means for storing the weight functions, and the operation means includes first gate means for successively obtaining a logical product of one of the first binary input signals received from the first input lines and a corresponding one of the weight functions read out from the first memory means for each of the first binary input signals, second gate means for successively obtaining a logical product of one of the second binary input signals received from the second input lines and a corresponding one of the weight functions read out from the second memory means for each of the second binary input signals, third gate means for obtaining a logical sum of logical products output from the first gate means, fourth gate means for obtaining a logical sum of logical products output from the second gate means, and output means including an inverter for inverting the logical sum output from the fourth gate means and a gate for obtaining one of a logical product and a logical sum of the logical sum output from the third gate means and an inverted logical sum output from the inverter, the gate outputting the output signal of the neuron unit.
Still another object of the present invention is to provide the neuron unit described above wherein the input line means includes a plurality of input lines for receiving binary input signals which undergo transitions with time, the supplying means includes memory means for storing the weight functions and corresponding grouping information, the grouping information indicating one of excitatory and inhibitory groups to which the weight functions belong, and the operation means includes first gate means for successively obtaining a logical product of one of the binary input signals received from the input lines and a corresponding one of the weight functions read out from the memory means for each of the binary input signals, second gate means for obtaining a logical product of one of the grouping information read out from the memory means and a corresponding one of logical products output from the first gate means for each of the logical products output from the first gate means, third gate means for obtaining a logical product of an inversion of one of the grouping information read out from the memory means and a corresponding one of the logical products output from the first gate means for each of the logical products output from the first gate means, fourth gate means for obtaining a logical sum of logical products output from the second gate means, fifth gate means for obtaining a logical sum of logical products output from the third gate means, and output means including an inverter for inverting the logical sum output from the fifth gate means and a gate for obtaining one of a logical product and a logical sum of the logical sum output from the fourth gate means and an inverted logical sum output from the inverter, the gate outputting the output signal of the neuron unit.
A further object of the present invention is to provide the neuron unit described above wherein the input line means includes a plurality of input lines for receiving binary input signals which undergo transitions with time, the supplying means includes first and second memory means for storing the weight functions, and the operation means includes first gate means for successively obtaining a logical product of one of the binary input signals received from the input lines and a corresponding one of the weight functions read out from the first memory means for each of the binary input signals, second gate means for successively obtaining a logical product of one of the binary input signals received from the input lines and a corresponding one of the weight functions read out from the second memory means for each of the binary input signals, third gate means for obtaining a logical sum of logical products output from the first gate means, fourth gate means for obtaining a logical sum of logical products output from the second gate means, and output means including an inverter for inverting the logical sum output from the fourth gate means and a gate for obtaining one of a logical product and a logical sum of the logical sum output from the third gate means and an inverted logical sum output from the inverter, the gate outputting an output signal of the neuron unit.
Another object of the present invention is to provide the neuron unit described above wherein the generating means of the self-learning means generates the new weight functions based on the errors and a learning constant.
Still another object of the present invention is to provide the neuron unit described above which further comprises means for arbitrarily setting the learning constant from outside the neuron unit.
A further object of the present invention is to provide the neuron unit described above which further comprises switching means for switching a mode of the neuron unit between first and second modes, the varying means of the self-learning means being enabled in the first mode to thereby renew the weight functions, the varying means of the self-learning means being disabled in the second mode to thereby fix the weight functions.
Another object of the present invention is to provide the neuron unit described above wherein the supplying means of the forward process means includes memory means for storing the weight functions, and the neuron unit further comprises means for making access to the memory means, so that the weight functions can be written into and read out from the memory means from outside the neuron unit.
Still another object of the present invention is to provide the neuron unit described above which further comprises first memory means for storing the input signals, second memory means for storing the teaching signals, and control means for controlling the first and second memory means to supply the stored input signals to the forward process means and the teaching signals to the self-learning means.
A further object of the present invention is to provide the neuron unit described above wherein the supplying means of the forward process means includes memory means for storing the weight functions, and the neuron unit further comprises setting means for variably setting a data length of the weight functions in the memory means.
Another object of the present invention is to provide the neuron unit described above which further comprises first converter means for converting analog input signals into digital input signals which are supplied to the forward process means as the input signals, and second converter means for converting the output signal of the neuron unit into an analog signal.
Still another object of the present invention is to provide the neuron unit described above which further comprises switching means for switching at least one of the input signals to the forward process means and the output signal of the self-learning means in response to a plurality of external signals.
A further object of the present invention is to provide the neuron unit described above wherein the operation means of the forward process means includes first means for successively carrying out an operation on the input signals in groups of the input signals, second means for successively storing results of operations for the groups, and third means for carrying out an operation on the results of operations simultaneously read out from the second means.
Another object of the present invention is to provide the neuron unit described above wherein the operation means of the forward process means includes first means for successively carrying out an operation on the input signals in groups of the input signals, second means for successively storing results of operations for the groups, third means for carrying out an operation the input signals in a first mode and for carrying out an operation on the results of operations simultaneously read out from the second means in a second mode, and fourth means for setting a mode of the third means to one of the first and second modes.
Still another object of the present invention is to provide the neuron unit described above wherein operations of the forward process means and the self-learning means are carried out on a computer.
A further object of the present invention is to provide the neuron unit described above wherein operations of the forward process means are carried out by hardware, and operations of the self-learning means are carried out on a computer.
Another object of the present invention is to provide a neural network comprising a plurality of neuron units which are coupled to form a hierarchical structure which has a plurality of layers, and a plurality of signal lines coupling outputs of arbitrary neuron units in one layer of the hierarchical structure to inputs of arbitrary neuron units in another layer of the hierarchical structure, each of the neuron units simultaneously processing a plurality of binary input signals and outputting an output signal which is indicative of a result of the processing, the neuron unit comprising input line means for receiving the input signals, forward process means coupled to the input line means and including supplying means for supplying weight functions, and operation means for carrying out an operation on each of the input signals using one of the weight functions and for outputting the output signal, and self-learning means coupled to the forward process means and including generating means for generating new weight functions based on errors between the output signal of the forward process means and teaching signals, and varying means for varying the weight functions supplied by the supplying means of the forward process means to the new weight functions generated by the generating means.
Still another object of the present invention is to provide the neural network described above wherein the operation means of the forward process means includes first means for successively carrying out an operation on the input signals in groups of the input signals, second means for successively storing results of operations for the groups, and third means for carrying out an operation on the results of operations simultaneously read out from the second means.
A further object of the present invention is to provide the neural network described above wherein the operation means of the forward process means includes first means for successively carrying out an operation on the input signals in groups of the input signals, second means for successively storing results of operations for the groups, third means for carrying out an operation the input signals in a first mode and for carrying out an operation on the results of operations simultaneously read out from the second means in a second mode, and fourth means for setting a mode of the third means to one of the first and second modes.
Another object of the present invention is to provide a signal processing system comprising N neural networks which are coupled to carry out a signal processing on a plurality of input signals respectively having N bits, where N is an integer, the N neural networks receiving corresponding bits of the input signals, each of the N neuron units comprising a plurality of neuron units which are coupled to form a hierarchical structure which has a plurality of layers, and a plurality of signal lines coupling outputs of arbitrary neuron units in one layer of the hierarchical structure to inputs of arbitrary neuron units in another layer of the hierarchical structure, each of the neuron units simultaneously processing a plurality of binary input signals and outputting an output signal which is indicative of a result of the processing, the neuron unit comprising input line means for receiving the input signals, forward process means coupled to the input line means and including supplying means for supplying weight functions, and operation means for carrying out an operation on each of the input signals using one of the weight functions and for outputting the output signal, and self-learning means coupled to the forward process means and including generating means for generating new weight functions based on errors between the output signal of the forward process means and teaching signals, and varying means for varying the weight functions supplied by the supplying means of the forward process means to the new weight functions generated by the generating means.
Still another object of the present invention is to provide a signal processing method for processing a plurality of input signals in a neuron unit and for outputting an output signal which is indicative of a result of the processing, the signal processing method comprising the steps of writing weight functions in memory means of the neuron unit from outside the neuron unit, forward process including carrying out an operation on each of the input signals using one of weight functions stored in the memory means and for outputting an operation result as the output signal, and self-learning process including generating new weight functions based on errors between the output signal obtained by the forward process and teaching signals, and varying the weight functions used by the forward process to the new weight functions which are generated by renewing contents of the memory means.
A further object of the present invention is to provide a signal processing method for processing a plurality of input signals in a neuron unit and for outputting an output signal which is indicative of a result of the processing, the signal processing method comprising the steps of writing weight functions in memory means of the neuron unit from outside the neuron unit, forward process including carrying out an operation on each of the input signals using one of weight functions stored in the memory means and for outputting an operation result as the output signal, self-learning process including generating new weight functions based on errors between the output signal obtained by the forward process and teaching signals, and varying the weight functions used by the forward process to the new weight functions which are generated by renewing contents of the memory means, and reading the weight functions stored in the memory means from outside the neuron unit.
Another object of the present invention is to provide a signal processing method for processing a plurality of input signals in a neuron unit and for outputting an output signal which is indicative of a result of the processing, the signal processing method comprising the steps of storing input signal data of the input signals in first memory means and storing teaching signal data of teaching signals in second memory means, forward process including carrying out an operation on each of the input signals using one of weight functions and for outputting an operation result as the output signal, and self-learning process including generating new weight functions based on errors between the output signal obtained by the forward process and the teaching signals, and varying the weight functions used by the forward process to the new weight functions which are generated.
Still another object of the present invention is to provide a computer-implemented method of simulating a neuron unit for processing a plurality of input signals and for outputting an output signal which is indicative of a result of the processing, the computer-implemented method comprising receiving the input signals, carrying out a forward process including supplying weight functions and carrying out an operation on each of the input signals using one of the weight functions and for outputting an operation result as the output signal, and carrying out a self-learning process including generating new weight functions based on errors between the output signal and teaching signals, and varying the weight functions to the new weight functions which are generated.
A further object of the present invention is to provide a computer-implemented method of simulating a self-learning process of a neuron unit which processes a plurality of input signals and outputs an output signal which is indicative of a result of the processing, the neuron unit including input line means for receiving the input signals and forward process means including supplying means for supplying weight functions and operation means for carrying out an operation on each of the input signals using one of the weight functions supplied by the supplying means and for outputting the output signal, the computer-implemented method comprising the step of generating new weight functions based on errors between the output signal output from the forward process means and teaching signals, and varying the weight functions supplied by the supplying means of the forward process means to the new weight functions which are generated.
Another object of the present invention is to provide a neuron unit for processing a plurality of input signals and for outputting an output signal which is indicative of a result of the processing, comprising input line means for receiving the input signals, forward process means, coupled to the input line means, and including supplying means for supplying weight functions and operation means for carrying out an operation on each of the input signals using one of the weight functions and for outputting the output signal, and self-learning means, coupled to the forward process means, and including function generating means for generating new weight functions based on errors between the output signal of the forward process means and teaching signals, and varying means for varying the weight functions supplied by the supplying means of the forward process means to the new weight functions generated by the generating means, where the supplying means includes memory means for storing each weight function in the form of a binary value, and generating means, coupled to the memory means, for generating a random pulse train based on each binary value stored in the memory means, the random pulse train describing each weight function in the form of a pulse signal having a pulse density which is defined by at least one of a number of first values and a number of second values of the pulse signal within a predetermined time, where the first and second values are arranged at random and the first and second values respectively correspond to high and low binary signal levels. According to the neuron unit of the present invention, it is possible to reduce the scale of the required circuitry. In addition, the learning capability of the neuron unit is improved, particularly when the random pulses within the neuron unit are mutually different even when describing the same pulse density.
Other objects and further features of the present invention will be apparent from the following detailed description when read in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows an example of a conventional neuron unit;
FIG. 2 shows a conventional neural network;
FIG. 3 is a graph showing a sigmoid function;
FIG. 4 is a circuit diagram showing a neural network proposed in a Japanese Laid-Open Patent Application No. 62-295188;
FIG. 5 is a system block diagram showing an example of a conventional neuron using digital circuits;
FIG. 6 is a system block diagram showing a synapse circuit shown in FIG. 5;
FIG. 7 is a system block diagram showing a cell circuit shown in FIG. 5;
FIG. 8 is a circuit diagram showing a weight function varying circuit of a first embodiment of a neuron unit according to the present invention;
FIG. 9 is a circuit diagram showing a coefficient multiplier circuit which uses the weight function varying circuit shown in FIG. 8;
FIG. 10 is a circuit diagram showing an adder circuit shown in FIG. 9;
FIG. 11 is a graph showing a characteristic of a first order differential function f';
FIG. 12 is a circuit diagram showing an f' signal generating circuit 30;
FIG. 13 shows an input versus output characteristic of the f' signal generating circuit shown in FIG. 12;
FIG. 14 shows an input versus output characteristic of the f' signal generating circuit when an amplifier having a non-linear input versus output characteristic is provided at an input stage;
FIG. 15 is a system block diagram showing an error signal generating circuit corresponding to the equation (5);
FIG. 16 is a system block diagram showing an error signal generating circuit corresponding to the equation (6);
FIG. 17 is a system block diagram showing a weight function generating circuit corresponding to the equation (7);
FIG. 18 is a system block diagram showing the first embodiment of the neuron unit according to the present invention;
FIG. 19 is a diagram showing a hand-written character which is read on a scanner for explaining an application of the first embodiment to a self-learning type character recognition system;
FIG. 20 is a circuit diagram showing a second embodiment of the neuron unit according to the present invention;
FIG. 21 is a diagram for explaining a pulse train which describes an input signal of the second embodiment;
FIG. 22 is a diagram for explaining a pulse train which describes a weight function of the second embodiment;
FIG. 23 is a diagram for explaining a logical product of the input signal and the weight function;
FIG. 24 is a diagram for explaining an output of the second embodiment;
FIGS. 25 and 26 are diagrams for explaining outputs of excitatory and inhibitory groups;
FIG. 27 is a diagram for explaining an error signal;
FIGS. 28 and 29 respectively are diagrams for explaining positive and negative error signals in the case of an excitatory coupling;
FIGS. 30 and 31 respectively are diagrams for explaining positive and negative error signals in the case of an inhibitory coupling;
FIGS. 32 and 33 respectively are diagrams for explaining examples of thinning out the error signal;
FIGS. 34 and 35 respectively are diagrams for explaining a method of varying the weight function;
FIGS. 36 and 37 respectively are diagrams for explaining a method of obtaining a new weight function for excitatory and inhibitory original weight function;
FIG. 38 is a circuit diagram showing a circuit which corresponds to a connection line between two neuron units in the neural network shown in FIG, 2;
FIG. 39 is a circuit diagram showing a circuit which corresponds to the neuron unit of the second embodiment;
FIG. 40 is a circuit diagram showing a circuit for obtaining an error signal in a final layer of the neural network based on an output of the final layer and a teaching signal;
FIG. 41 is a circuit diagram showing an embodiment of a circuit for grouping excitatory and inhibitory couplings and determining the output in FIGS. 38 and 39;
FIG. 42 is a circuit diagram showing another embodiment of the circuit for grouping excitatory and inhibitory couplings and determining the output;
FIG. 43 is a circuit diagram showing a modification of a gate circuit shown in FIGS. 41 and 42;
FIG. 44 is a circuit diagram showing an essential part of a third embodiment of the neuron unit according to the present invention;
FIGS. 45 through 47 are circuit diagrams for explaining essential parts of a fourth embodiment of the neuron unit according to the present invention;
FIG. 48 is a circuit diagram showing an essential part of a fifth embodiment of the neuron unit according to the present invention;
FIGS. 49 through 51 are system block diagrams respectively showing essential parts of a sixth embodiment of the neuron unit according to the present invention;
FIG. 52 is a system block diagram showing a seventh embodiment of the present invention;
FIG. 53 is a circuit diagram showing an essential part of an eighth embodiment of the present invention;
FIG. 54 is a diagram for explaining pulse sequences for indicating the same weight function;
FIGS. 55A and 55B respectively are circuit diagrams showing an essential part of a ninth embodiment of the present invention for explaining an analog-to-digital conversion;
FIGS. 56A through 56C respectively are circuit diagrams showing another essential part of the ninth embodiment of the present invention for explaining a digital-to-analog conversion;
FIG. 57 is a system block diagram showing a tenth embodiment of the present invention;
FIG. 58 is a system block diagram for explaining the functions of two neuron units;
FIGS. 59 through 62 respectively are system block diagrams for explaining an operation of an eleventh embodiment of the present invention;
FIGS. 63 and 64 respectively are system block diagrams for explaining methods of storing the weight function and its sign;
FIG. 65 is a circuit diagram showing an essential part of a twelfth embodiment of the present invention;
FIGS. 66A, 66B and 66C respectively are system block diagrams showing essential parts of thirteenth, fourteenth and fifteenth embodiments of the present invention;
FIG. 67 is a flow chart for explaining a learning process of a CPU shown in FIG. 66B in the fourteenth embodiment;
FIG. 68 is a flow chart for explaining a forward process of the CPU shown in FIG. 66C in the fifteenth embodiment;
FIGS. 69, 70 and 71 are system block diagrams for explaining further embodiments of the present invention;
FIG. 72 is a system block diagram showing an essential part of a sixteenth embodiment of the neuron unit according to the present invention;
FIG. 73 is a circuit diagram showing the sixteenth embodiment of the neuron unit according to the present invention;
FIG. 74 is a circuit diagram showing a first embodiment of a random number generator shown in FIG. 73;
FIG. 75 is a circuit diagram showing a seventeenth embodiment of the neuron unit according to the present invention;
FIGS. 76 through 90 respectively are time charts for explaining the operation of an eighteenth embodiment of the neuron unit according to the present invention;
FIG. 91 is a circuit diagram showing a circuit which corresponds to a connection line between two neuron units in the neural network applied with the eighteenth embodiment;
FIG. 92 is a circuit diagram showing a circuit which corresponds to the neuron unit of the eighteenth embodiment;
FIG. 93 is a system block diagram showing an embodiment of an error signal output circuit;
FIG. 94 is a system block diagram showing another embodiment of the error signal output circuit;
FIG. 95 is a circuit diagram showing another circuit which corresponds to the connection line between two neuron units in the neural network;
FIGS. 96 through 110 respectively are time charts for explaining the operation of a nineteenth embodiment of the neuron unit according to the present invention;
FIGS. 111 through 114 respectively are circuit diagrams showing a circuit part corresponding to the line coupling the neuron units applied with the nineteenth embodiment;
FIG. 115 is a circuit diagram showing a circuit part corresponding to the neuron unit of the nineteenth embodiment;
FIG. 116 is a circuit diagram showing a circuit part which obtains the error signal in the final layer in the nineteenth embodiment;
FIGS. 117A, 117B and 117C respectively are circuit diagrams showing circuits for realizing the processes shown in FIGS. 79, 98 and 99;
FIG. 118 is a system block diagram showing an embodiment of a pulse train converting circuit;
FIG. 119 is a system block diagram showing an embodiment of a numerical value converting circuit;
FIG. 120 is a system block diagram showing an embodiment of a circuit integrally comprising circuit parts for carrying out a conversion from a numerical value signal into a pulse train signal and vice versa;
FIG. 121 is a system block diagram showing an embodiment of a circuit having two random number generators and two comparators;
FIGS. 122A through 122C are circuit diagrams respectively showing a first embodiment of a random number generator shown in FIG. 74 with slightly different connections;
FIGS. 123A through 123D respectively show bit sequence of the M sequence generated from the random number generators shown in FIGS. 74 and 122A through 122C;
FIGS. 124A through 124D respectively show random number sequences generated from the random number generators shown in FIGS. 74 and 122A through 122C;
FIG. 125 is a circuit diagram showing a second embodiment of the random number generator;
FIG. 126 is a circuit diagram showing an embodiment of a switching circuit shown in FIG. 125;
FIG. 127 is a flow chart for explaining the operation of the second embodiment of the random number generator;
FIG. 128 is a circuit diagram showing a third embodiment of the random number generator;
FIG. 129 is a circuit diagram showing an embodiment of a random pulse generator;
FIG. 130 is a circuit diagram showing another embodiment of the random pulse generator;
FIG. 131 is a circuit diagram showing a fourth embodiment of the random number generator;
FIGS. 132(A)-132(C) (hereinafter collectively referred to as FIG. 132) are a diagram for explaining the relationship of the counter content, feedback position and the characteristic polynomial in the fourth embodiment of the random number generator;
FIG. 133 is a circuit diagram showing a fifth embodiment of the random number generator;
FIG. 134 is a diagram for explaining the operation of the seventh embodiment of the random number generator;
FIG. 135 is a system block diagram showing the first embodiment of the random number generator shown in FIG. 74 together with the essential part of the the neuron unit shown in FIG. 72;
FIG. 136 is a system block diagram showing a sixth embodiment of the random number generator together with the essential part of the neuron unit shown in FIG. 72;
FIG. 137 is a system block diagram showing an essential part of a twentieth embodiment of the neuron unit according to the present invention;
FIG. 138 is a system block diagram showing an essential part of a twenty-first embodiment of the neuron unit according to the present invention;
FIG. 139 is a system block diagram showing an essential part of a twenty-second embodiment of the neuron unit according to the present invention;
FIG. 140 is a circuit diagram showing a bit generating part shown in FIG. 139;
FIG. 141 is a circuit diagram showing an essential part of LFSRs shown in FIG. 139; and
FIG. 142 is a flow chart for explaining the operation of the twenty-second embodiment of the neuron unit according to the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS
A description will be given of a first embodiment of a neuron unit according to the present invention.
In this embodiment, the neuron unit has the self-learning function. Hence, the weight function must be made variable so as to enable the self-learning.
FIG. 8 shows the circuit construction of a weight function varying circuit 20 for varying the weight function. The weight function varying circuit 20 includes a plurality of resistors 21 which describe the weight function, and the weight function is made variable by appropriately switching a switching circuit 22 so as to connect to the resistor 21 which is to be used. The switching circuit 22 may be realized by a generally available switch which is switched and controlled responsive to a binary input from an external controller (not shown). In the case shown in FIG. 8, an analog-to-digital (A/D) converter 23 is used to convert a voltage S from the external controller into a binary value, and this binary value is used to control the switching of the switching circuit 22. In addition, a sign bit of the A/D converter 23 is used to control the switching of a switching circuit 24. The switching circuit 24 determines whether or not an output is to be obtained via an inverting amplifier 25 responsive to the sign bit, so as to switch the output between an excitatory output and an inhibitory output. Accordingly, the weight function is variable depending on the voltage (external signal) S, and it is possible to obtain an output by multiplying the weight function having an arbitrary value to an input signal.
FIG. 9 shows a coefficient multiplier circuit for describing the equations (1) and (2) using the weight function varying circuit 20 shown in FIG. 8. Each weight function varying circuit 20 has the function of multiplying the weight function to an input signal from an immediately preceding layer. Outputs of the weight function varying circuits 20 are added in an adder circuit 26.
The adder circuit 26 may be easily realized by use of a generally available operational amplifier 27 as shown in FIG. 10. In FIG. 10, the operational amplifier 27 is for adding but has the inverting amplifier structure. Hence, an amplifier 28 is used to further invert an output of the operational amplifier 27 to obtain an output of the adder circuit 26.
A non-linear amplifier 29 is connected to an output of the adder circuit 26 as shown in FIG. 9. The input and output of this amplifier 29 have a relationship described by the equation (2). The input to the amplifier 29 corresponds to the internal potential of the equation (1).
Next, a description will be given of a method of forming the external signal S which determines the weight function. This method corresponds to the equations (5) through (7), and circuits are necessary to realize these equations (5) through (7). In the equations (5) and (6), f' is the first order differential function of the sigmoid function shown in FIG. 3 and has a characteristic shown in FIG. 11. An f' signal generating circuit 30 shown in FIG. 12 realizes the first order differential function f'. As shown in FIG. 12, a plurality of amplifiers 31 through 35 are connected in a plurality of stages so that the f' signal generating circuit 30 might have a non-linear characteristic shown in FIG. 11. This f' signal generating circuit 30 has an input versus output characteristic shown in FIG. 13. The f' signal generating circuit 30 does not always accurately realize the characteristic shown in FIG. 11, however, the characteristic can be approximated. In addition, when an amplifier (not shown) having a non-linear input versus output characteristic shown in FIG. 14 is provided at an input stage of the f' signal generating circuit 30, the input versus output characteristic of the f' signal generating circuit 30 becomes as shown in FIG. 14, and the characteristic shown in FIG. 11 can be more closely approximated.
FIG. 15 shows an error signal generating circuit 36 which corresponds to the equation (5). In FIG. 15, the f' signal generating circuit 30 is the same as that shown in FIG. 12, and subjects the internal potential (that is, the input to the amplifier 29 shown in FIG. 9) to the function processing shown in FIG. 13 or 14. On the other hand, a subtracting circuit 37 is provided to obtain an error between an output of the neuron unit in the output layer and a teaching signal. A circuit similar to that shown in FIG. 10 may be used by inverting one input in an amplifier. Outputs of these circuits 30 and 37 are supplied to a multiplier circuit 38 which obtains a product of the two outputs and obtains a result similar to the equation (5).
On the other hand, FIG. 16 shows an error signal generating circuit 39 corresponding to the equation (6). In FIG. 16, the error signal generating circuit 39 includes the weight function varying circuits 20, the adder circuit 26, the f' signal generating circuit 30 and a multiplier circuit 40 which obtains a product of the outputs of the circuits 26 and 30. This circuit construction is equivalent to the equation (6). Accordingly, by inputting the internal potential and the error signal which is generated by the circuit 39 shown in FIG. 16 in another layer or by the circuit 36, it is possible to finally obtain an output which is similar to that obtained by the equation (6).
FIG. 17 shows a weight function generating circuit 41 corresponding to the equation (6). The weight function generating circuit 41 includes a multiplier circuit 42 which may be realized by a generally available multiplier. The multiplier circuit 42 obtains a product of an output of a neuron unit of one layer, the error signal generated in the circuit described above, and a constant .eta.. An output of the multiplier circuit 42 is supplied to an adder circuit 43, and a new T is generated from T and .DELTA.T using a delay circuit 44. Hence, an output of the adder circuit 43 corresponds to the output obtained by the equation (7).
FIG. 18 shows the first embodiment of the neuron unit which is formed from the circuits described above. In a neural network, a neuron unit 45 shown in FIG. 18 corresponds to a part surrounded by a dashed line in FIG. 2, for example.
In FIG. 18, a block B1 corresponds to the circuit shown in FIG. 9, and an output of this block B1 is supplied to each neuron unit of the next layer. A block B2 corresponds to the error signal generating circuit 39 shown in FIG. 16. That is, the block B2-1 of the next layer and the block B2-2 of the neuron unit 45 correspond to the circuit shown in FIG. 16. Similarly, the block B2-1 of the neuron unit 45 and the block B2-2 of the preceding layer correspond to the circuit shown in FIG. 16. Since the neural network as a whole has the multi-layer structure as shown in FIG. 2, the block of the error signal generating circuit 39 can be divided into two at the center to realize an equivalent circuit from two circuit parts.
Blocks B3-1, B3-2, . . . , B3-N shown in FIG. 18 each correspond to the weighing coefficient generating circuit 41 shown in FIG. 17 and the A/D converter 23 shown in FIG. 8. In FIG. 18, however, the illustration of the delay circuit 44 is omitted for the sake of convenience. The weight functions T which are newly obtained in the blocks B3-1, B3-2, . . . , B3-N are used, and each weight function T is varied in the weight function varying circuit 20 shown in FIG. 8. Since the same weight function is used at two locations which are the blocks B1 and B2-1, the two are linked and varied. In other words, the blocks B2-1 and B3 and the weight function varying circuit 20 within the block B1 in FIG. 18 correspond to a self-learning circuit, while the remaining part of the block B1 and the block B2-2 correspond to a neuron circuit which resembles a neuron.
A plurality of the neuron units 45 having the structure shown in FIG. 18 are connected to form a network similar to that shown in FIG. 2. The neural network is realized by adding the error signal generating circuit 36 shown in FIG. 15, for example, at the output part of the final output layer.
A particular case will be described for the circuit construction described above. First, the adder circuit and the like of each block are all made of generally available operational amplifiers, and a plurality of 256-input and 256-output neuron units 45 having the structure shown in FIG. 18 and a plurality of weight function generating circuits 41 shown in FIG. 17 are formed. Next, the input and output lines of the neuron units 45 and the weight function generating circuits 41 are connected to form a neural network having three layers. In the neural network, a first layer includes 256 neuron units 45, a second layer includes four neuron units 45, and a third layer includes five neuron units 45. In addition, an output of the third layer is connected to the error signal generating circuit 36 shown in FIG. 15. When an input is applied to each neuron unit 45 of the first layer in the neural network, the resulting output is not necessarily a desirable value. However, because this neuron unit network has the self-learning circuit, the resulting output eventually becomes the desirable value, that is, the teaching signal.
A description will be given of a case where the above described neural network is applied to a self-learning type character recognition system. First, a hand-written character shown in FIG. 19 is read by a scanner, and the read image is divided into 16.times.16 meshes. The data in each mesh is then applied to each neuron unit 45 of the first layer in the neural network. For the sake of convenience, the data of a mesh which includes a portion of the character is taken as 1V, while the data of a mesh which includes no portion of the character is taken as 0V. The output of the neural network is connected to a voltmeter so that the resulting output is directly displayed on the voltmeter. Out of the five neuron units 45 of the third layer, the neuron unit 45 which outputs the largest output is assumed to output the recognition result. The learning takes place so that when the numbers "1" through "5" are input to the neural network, the five neuron units 45 of the third layer respectively corresponding to the numbers "1" through "5" output the largest output. In other words, when the number "1" is input, the neuron unit 45 of the third layer corresponding to the number "1" outputs the largest output. With respect to a character after sufficient learning, the recognition rate was 100%.
Next, a description will be given of a second embodiment of the neuron unit according to the present invention. In this embodiment, the neuron unit is realized by use of digital circuits according to the following rules [1] through [6].
[1] Input and output signals of the neuron unit, intermediate signals within the neuron unit, the weight function, the teaching signal and the like are all take the form of a pulse train described by binary values "0" and "1".
[2] The signal quantity within the neural network is expressed by the pulse density, that is, the number of "1"s within a predetermined time.
[3] The calculation within the neuron unit is described by a logic operation of pulse trains.
[4] The pulse train expressing the weight function is stored in a memory.
[5] The learning is realized by rewriting the pulse train of the weight function stored in the memory.
[6] When learning, an error is calculated based on a pulse train of the given teaching signal, and the pulse train of the weight function is varied depending on the calculated error. The calculation of the error and the calculation of the deviation of the weight function are carried out by logic operations of pulse trains described by "0"s and "1"s.
FIG. 20 shows a neuron unit 50, and a plurality of such neuron units 50 are connected in a plurality of layers to form a hierarchical neural network shown in FIG. 2, for example. The input and output signals of the neuron unit 50 are all described in binary by "1"s and "0"s and are synchronized. The signal intensity of the input signal y.sub.i is expressed by a pulse density, that is, a number of "1"s existing in a pulse train within a predetermined time. FIG. 21 shows a case where four "1"s and two "0"s of the input signal y.sub.i exist within the predetermined time amounting to six synchronizing pulses. In this case, the input signal y.sub.i has a signal intensity 4/6. It is desirable that the "1"s and "0"s of the input signal y.sub.i are positioned at random within the predetermined time.
On the other hand, the weighting coefficient T.sub.ij is similarly described by a pulse density, and is stored in a memory as a pulse train of "0"s and "1"s. FIG. 22 shows a case where three "1"s and three "0"s of the weight function T.sub.ij exist within the predetermined time amounting to six synchronizing pulses. In this case, the weight function T.sub.ij has a value 3/6. It is desirable that the "1"s and "0"s of the weight function T.sub.ij are positioned at random within the predetermined time.
The pulse train of the weight function T.sub.ij is successively read from the memory responsive to the synchronizing pulses and supplied to each AND gate 51 shown in FIG. 20 which obtains a logical product (y.sub.i .andgate.T.sub.ij) with the pulse train of the input signal y.sub.i. An output of the AND gate 51 is used as an input to the neuron unit 50. Hence, in the case described above, the logical product y.sub.i .andgate. T.sub.ij becomes as shown in FIG. 23 and a pulse train "101000" is obtained. It can be seen from FIG. 23 that the input signal y.sub.i is converted by the weight function T.sub.ij and the pulse density becomes 2/6.
The pulse density of the output signal of the AND gate 51 is approximately the product of the pulse density of the input signal and the pulse density of the weight function, and the AND gate 51 acts similarly as in the case of the analog circuit. The pulse density of the output signal of the AND gate 51 more closely approximates the above product as the pulse train becomes longer and as the locations of the "1"s and "0"s become more at random. When the pulse train of the weight function is short compared to the pulse train of the input signal and no further data can be read out from the memory, the data can be read out from the first data and repeat such an operation until the pulse train of the input signal ends.
One neuron unit 50 receives a plurality of input signals, and a plurality of logical products are obtained between the input signal and the weight function. Hence, an OR circuit 52 obtains a logical sum of the logical products. Since the input signals are synchronized, the logical sum becomes "111000" when the first logical product is "101000" and the second logical product is "010000" for example. FIG. 24 shows the logical products input to the OR circuit 52 and the logical sum .orgate.(y.sub.i .andgate.T.sub.ij) which is output from the OR circuit 52. This corresponds to the calculation of the sum and the non-linear function (sigmoid function) in the case of the analog calculation.
When the pulse densities are low, the logical sum of such pulse densities is approximately the sum of the pulse densities. As the pulse densities become higher, the output of the OR circuit 52 saturates and no longer approximates the sum of the pulse densities, that is, the non-linear characteristic begins to show. In the case of the logical sum, the pulse density will not become greater than "1" and will not become smaller than "0". In addition, the logical sum displays a monotonous increase and is approximately the same as the sigmoid function.
As described above, there are two types of couplings (or weighting), namely, the excitatory coupling and the inhibitory coupling. When making numerical calculations, the excitatory and inhibitory couplings are described by positive and negative signs on the weight function. In the case of the analog neuron unit, when the weight function T.sub.ij indicates the inhibitory coupling and the sign on the weight function T.sub.ij is negative, an inverting amplifier is used to make an inversion and a coupling to another neuron unit is made via a resistance which corresponds to the weight function T.sub.ij.
On the other hand, in this embodiment which uses digital circuits, the couplings are leaded into an excitatory group and an inhibitory group depending on the positive and negative signs on the weight function T.sub.ij. Then, the calculation up to the part where the logical sum of the logical products of the pulse trains of the input signals and the weight functions are carried out for each group. Thereafter, the neuron unit 50 outputs "1" when only the output of the excitatory group is "1" and outputs "0" when only the output of the inhibitory group is "1". When the outputs of the excitatory and inhibitory groups are both "1" or both "0" the neuron unit 50 may output either "1" or "0" or output "1" with a probability of 1/2.
In this embodiment, the neuron unit 50 outputs "1" only when the output of the excitatory group is "1" and the output of the inhibitory group is "0". This may be achieved by obtaining an AND of a NOT of the output of the inhibitory group and the output of the excitatory group as shown in FIG. 25. Hence, the output a of the excitatory group can be described as a=.orgate.(y.sub.i .andgate.T.sub.ij) and the output b of the inhibitory group can be described by b=.orgate.(y.sub.i .andgate.T.sub.ij). In addition, the output y.sub.j of the neuron unit 50 can be described by y.sub.j =a.andgate. b. The neural network can be formed by connecting a plurality of such neuron units 50 in a plurality of layers to form a hierarchical structure similarly as in the case of the neural network shown in FIG. 2. When the entire neural network is synchronized, it is possible to carry out the above described calculation in each layer.
On the other hand, measures may be taken so that the neuron unit 50 outputs "1" excluding the case where the output of the excitatory group is "0" and the output of the inhibitory group is "1". This may be achieved by obtaining an OR of a NOT of the output of the inhibitory group and the output of the excitatory group as shown in FIG. 26. Hence, the output a of the excitatory group can be described as a=.orgate.(y.sub.i .andgate.T.sub.ij) and the output b of the inhibitory group can be described by b=.orgate.(y.sub.i .andgate.T.sub.ij). In addition, the output y.sub.j of the neuron unit 50 can be described by y.sub.j =a.orgate.b.
Next, a description will be given of the learning process.
The Error Signal in the Final Layer
The error signal of each neuron unit 50 in the final layer is calculated, and the weight function of each neuron unit 50 is varied depending on the error signal. A description will now be given of the method of calculating the error signal. In this embodiment, the error signal is defined as follows. That is, the error may take a positive or negative value when the error is described by a numerical value, but in the case of the pulse density, it is impossible to simultaneously describe the positive and negative values. Hence, two kinds of signals, namely, a signal which indicates a positive component and a signal which indicates a negative component are used to describe the error signal. In other words, an error signal .delta..sup.+ j or .delta..sup.- j of the jth neuron unit 50 can be described as follows, where .delta..sup.+ j denotes the positive error signal, .delta..sup.- j denotes the negative error signal, and the output signal y.sub.j and the teaching signal d.sub.j of the jth neuron unit 50 are as shown in FIG. 27.
.delta..sup.+ j.ident.(y.sub.j EXOR d.sub.j) AND d.sub.j
.delta..sup.- j.ident.(y.sub.j EXOR d.sub.j) AND y.sub.j
Therefore, the positive component of the error signal corresponds to the pulses existing on the teaching signal side out of the parts (1, 0) and (0, 1) where the teaching signal pulse and the output signal pulse differ. Similarly, the negative component of the error signal corresponds to the pulses existing on the output signal side out of the parts (1, 0) and (0, 1) where the teaching signal pulse and the output signal pulse differ. In other words, when the positive component of the error signal is added to the output signal and the negative component of the error signal is subtracted, the teaching signal is obtained. As will be described later, the weight function is varied based on such error signal pulses.
The Error Signal in the Intermediate Layer
The error signal is back propagated, so that not only the weight functions of the final layer and the immediately preceding layer but also the weight functions of the layer which precedes the above immediately preceding layer are varied. For this reason, there is a need to calculate the error signal for each neuron unit 50 in the intermediate layer. The error signals from each of the neuron units 50 in the next layer are collected and used as the error signal of a certain neuron unit 50 of the intermediately layer, substantially in the reverse manner as supplying the output signal of the certain neuron unit 50 to each of the neuron units in the next layer. This may be achieved similarly as described above with reference to the equation (7) and FIGS. 21 through 23. That is, the couplings are divided into two groups depending on whether the coupling is an excitatory coupling or an inhibitory coupling, and the multiplication part is described by AND and the .SIGMA. part is described by OR. The only difference in this case is that although y is a single signal .delta. may be positive or negative and thus two error signals must be considered. Therefore, four cases must be considered depending on whether the weight function T is positive or negative and whether the error signal .delta. is positive or negative.
First, a description will be given of the excitatory coupling. In this case, .delta..sup.+ k.andgate.T.sub.ij which is an AND of the positive error signal .delta..sup.+ k of the kth neuron unit in the layer next to a specific layer and the weight function T.sub.jk between the jth neuron unit in the specific layer and the kth neuron unit in the next layer is obtained for each neuron unit in the specific layer. Furthermore, .orgate.(.delta..sup.+ k.andgate.T.sub.jk) which is an OR of .delta..sup.+ k.andgate.T.sub.jk obtained for each neuron unit in the specific layer, and this OR is regarded as the positive error signal .delta..sup.+ j for the specific layer as shown in FIG. 28.
In addition, .delta..sup.- k.andgate.T.sub.jk which is an AND of the negative error signal .delta..sup.+ k of the kth neuron unit in the next layer and the weight function T.sub.jk between the jth neuron unit in the specific layer and the kth neuron unit in the next layer is obtained for each neuron unit in the specific layer. Furthermore, .orgate.(.delta..sup.+ k.andgate.T.sub.jk) which is an OR of .delta..sup.- k.andgate.T.sub.jk obtained for each neuron unit in the specific layer, and this OR is regarded as the negative error signal .delta..sup.- j for the specific layer as shown in FIG. 29.
Next, a description will be given of the inhibitory coupling. In this case, .delta..sup.- k.andgate.T.sub.jk which is an AND of the negative error signal .delta..sup.- k of the kth neuron unit in the layer next to a specific layer and the weight function T.sub.jk between the jth neuron unit in the specific layer and the kth neuron unit in the next layer is obtained for each neuron unit in the specific layer. Furthermore, .orgate.(.delta..sup.- k.andgate.T.sub.jk) which is an OR of .delta..sup.- k.andgate.T.sub.jk obtained for each neuron unit in the specific layer, and this OR is regarded as the positive error signal .delta..sup.+ j for the specific layer as shown in FIG. 30.
In addition, .delta..sup.+ k.andgate.T.sub.jk which is an AND of the positive error signal .delta..sup.+ k of the kth neuron unit in the next layer and the weight function T.sub.jk between the jth neuron unit in the specific layer and the kth neuron unit in the next layer is obtained for each neuron unit in the specific layer. Furthermore, .orgate.(.delta..sup.+ k.andgate.T.sub.jk) which is an OR of .delta..sup.+ k.andgate.T.sub.jk obtained for each neuron unit in the specific layer, and this OR is regarded as the negative error signal .delta..sup.+ j for the specific layer as shown in FIG. 31.
Since one neuron unit may be coupled to another neuron unit by an excitatory or inhibitory coupling an OR of the error signal .delta..sup.+ j shown in FIG. 28 and the error signal .delta..sup.+ j shown in FIG. 30 is regarded as the error signal .delta..sup.+ j of the jth neuron unit. Similarly, an OR of the error signal .delta..sup.- j shown in FIG. 29 and the error signal .delta..sup.- j shown in FIG. 31 is regarded as the error signal .delta..sup.- j of the jth neuron unit.
Therefore, the error signals .delta..sup.+ j and .delta..sup.- j of the jth neuron unit in the specific layer can be described as follows. ##EQU1##
The error signals .delta..sup.+ j and .delta..sup.- j can also be described as follows. ##EQU2##
It is possible to further provide a function corresponding to the learning rate (learning constant). When the rate is "1" or less in numerical calculation, the learning capability is improved. This may be realized by thinning out the pulse train in the case of an operation on pulse trains. Two examples will now be described where the example 1) thins out every other pulses of the original pulse signal in which the pulses are equi-distant from each other and the example 2) thins out every other pulses of the original pulse signal in which the pulses are not equi-distant from each other.
FIG. 32 shows the example 1) for .eta.=0.5 where every other pulses of the original pulse signal are thinned out, .eta.=0.33 where every third pulses of the original pulse signal are thinned out, and .eta.=0.67 where every third pulses of the original pulse signal are thinned out.
FIG. 33 shows the example 2) for .eta.=0.5 where every other pulses of the original pulse signal are thinned out, .eta.=0.33 where every third pulses of the original pulse signal are thinned out, and .eta.=0.67 where every third pulses of the original pulse signal are thinned out.
By thinning out the error signal in the above described manner, it is possible to provide the function corresponding to the learning rate. Such thinning out can easily be realized by use of a generally available counter and/or flip-flop by carrying out a logic operation on a counter output, for example. In a particular case where the counter is used, it is possible to easily set the value of the learning constant .eta. to an arbitrary value, thereby making it possible to control the characteristic of the neural network.
It is not essential to always use the learning constant for the error signal. For example, it is possible to use the learning constant may be used only when carrying out the operation to obtain the weight function. In addition, the learning constant at the time of back-propagating the error signal and the learning constant at the time of carrying out the operation to obtain the weight function may be different. This means that the characteristics of the neuron units in the neural network can be set independently, and it is thus possible to form a system which is easily applicable to general applications. Accordingly, it becomes possible to appropriately adjust the performance of the neural network.
Variation of Each Weighting Coefficient by the Error Signal
The error signal is obtained by the method described above, and each weight function is varied. The method of varying each weight function will now be described. First, an AND is obtained between the error signal and the signal flowing in a line to which the weight function which is to be varied belongs. In other words, .delta..andgate.y is obtained. But since there are two error signals, one positive and one negative, both .delta..sup.+ j.andgate.y.sub.i and .delta..sup.- j.andgate.y.sub.i are obtained as shown respectively in FIGS. 34 and 35. The two signals which are obtained from .delta..sup.+ j .andgate.y.sub.i and .delta..sup.- j.andgate.y.sub.i are respectively denoted by .DELTA.T.sup.+ ij and .DELTA.T.sup.- ij.
Next, a new weight function T.sub.ij is obtained based on .DELTA.T.sub.ij. But since the weight function T.sub.ij in this embodiment is an absolute value component, the new weight function T.sub.ij is obtained differently depending on whether the original weight function T.sub.ij is excitatory or inhibitory. When the original weight function T.sub.ij is excitatory, the component of .DELTA.T.sup.+ ij is increased with respect to the original weight function T.sub.ij and the component of .DELTA.T.sup.- ij is decreased with respect to the original weight function T.sub.ij as shown in FIG. 36. On the other hand, when the original weight function T.sub.ij is inhibitory, the component of .DELTA.T.sup.+ ij is decreased with respect to the original weight function T.sub.ij and the component of .DELTA.T.sup.- ij is increased with respect to the original weight function T.sub.ij as shown in FIG. 37.
The calculations in the neural network are carried out based on the above described learning rules.
Next, a description will be given of actual circuits which form the second embodiment, by referring to FIGS. 38 through 40. FIG. 38 shows a circuit which corresponds to a connection line between two neuron units in the neural network shown in FIG. 2. FIG. 39 shows a circuit which corresponds to the neuron unit 50. FIG. 40 shows a circuit for obtaining the error signal in the final layer based on the output of the final layer and the teaching signal. The circuits shown in FIGS. 38 through 40 are connected as shown in FIG. 2 to form the digital neural network having the self-learning function.
In FIG. 38, an input signal 55 to the neuron unit 50 corresponds to the input signal described with reference to FIG. 21. The value of the weight function described with reference to FIG. 22 is stored in a shift register 56. The shift register 56 has an input 56b and an output 56a and has a function similar to a general shift register. For example, a combination of a random access memory (RAM) and an address controller may be used as the shift register 56.
A logic circuit 58 which includes an AND circuit 57 and corresponds to y.sub.i .andgate.T.sub.ij described with reference to FIG. 23 obtains an AND of the input signal 55 and the weight function within the shift register 56. An output signal of the logic circuit 58 must be grouped depending on whether the coupling is excitatory or inhibitory, but it is preferable from the point of general application to prepare an output 59 for the excitatory group and an output 60 for the inhibitory group and output one of these outputs 59 and 60. For this reason, this embodiment has a memory 61 for storing a bit which indicates whether the coupling is excitatory or inhibitory, and a switching gate circuit 62 is switched depending on the bit which is stored in the memory 61. The switching gate circuit 62 includes two AND gates 62a and 62b and an inverter 62c which inverts the bit which is read out from the memory 61 and is supplied to the AND gate 62a.
In addition, as shown in FIG. 39, gate circuits 63a and 63b which include a plurality of OR gates and correspond to .orgate.(y.sub.i .andgate.T.sub.ij) and described with reference to FIG. 24 are provided to process each input. A gate circuit 64 includes an AND gate 64a and an inverter 64b and outputs an output signal "1" only when the output of the excitatory group is "1" and the output of the inhibitory group is "0" as described in conjunction with FIG. 25.
Next, a description will be given of the error signal. A logic circuit 65 shown in FIG. 40 includes two AND gates and one exclusive-OR gate and generates error signals in the final layer. This logic circuit 65 corresponds to the equations described with reference to FIG. 27. In other words, the logic circuit 65 generates error signals 68 and 69 based on an output signal 66 of the final layer and a teaching signal 67. The calculation of the error signals in the intermediate layer described with reference to FIGS. 28 through 31 is carried out by a gate circuit 72 shown in FIG. 38 which includes two AND gates. The gate circuit 72 outputs output signals 73 and 74 depending on positive and negative signals 75 and 76 of the error signals 68 and 69.
The calculation is carried out for two cases, that is, for the case where the coupling is excitatory and the case where the coupling is inhibitory. A gate circuit 77 which includes four AND gates and two OR gates determines which one of the cases the calculation is to be carried out based on the bit stored in the memory 61 and the positive and negative signals 75 and 76.
A gate circuit 78 which includes OR gates as shown in FIG. 39 carries out the calculations according to the equations described above to obtain the error signals .delta..sup.+ j and .delta..sup.- j. Furthermore, the calculation to obtain the learning rate as described in conjunction with FIGS. 32 and 33 is carried out by a frequency dividing circuit 79 shown in FIG. 39. Finally, a gate circuit 80 which includes three AND gates, an OR gate and an inverter as shown in FIG. 38 calculates the new weight function from the error signal as described in conjunction with FIGS. 34 through 37. The content of the shift register 56, that is, the weight function, is rewritten into the new weight function which is calculated by the gate circuit 80. The gate circuit 80 also carries out the calculation for the case where the coupling is excitatory and the case where the coupling is inhibitory, and one of these two cases is determined by the gate circuit 77.
FIG. 41 shows an embodiment of the grouping system and the output determination system shown in FIGS. 38 and 39. In this case, the grouping is not made at the input stage. A shift register 56.sub.ij which stores the weight function is provided with respect to each input signal 55.sub.ij. An output signal of each AND gate 57.sub.ij is grouped into one of excitatory and inhibitory groups via the switching gate circuit 62 depending on the content of a memory 61.sub.ij. A logical sum is obtained in the OR gate 63a for the excitatory group (excitatory coupling) and a logical sum is obtained in the OR gate 63b for the inhibitory group (inhibitory coupling). Thereafter, the output signal is determined by a logical product processing in the gate circuit 64.
Next, a description will be given of a case where the above described neural network is applied to a self-learning type character recognition system. The first layer of the neural network includes 256 neuron units, the second layer includes 20 neuron units and the third layer includes five neuron units. First, a hand-written character shown in FIG. 19 is read by a scanner, and the read image is divided into 16.times.16 meshes. The data in each mesh is then applied to each neuron unit of the first layer in the neural network. For the sake of convenience, the data of a mesh which includes a portion of the character is taken as "1", while the data of a mesh which includes no portion of the character is taken as "0". The output of the neural network is connected to a light emitting diode (LED) so that the resulting output is directly displayed on the LED. Out of the five neuron units of the third layer, the neuron unit which outputs the largest output is assumed to output the recognition result. The learning takes place so that when the numbers "1" through "5" are input to the neural network, the five neuron units of the third layer respectively corresponding to the numbers "1" through "5" output the largest output. In other words, when the number "1" is input, the neuron unit of the third layer corresponding to the number "1" outputs the largest output. Those input parts of the neuron units not connected to another neuron unit are grounded.
Initially, when each weight function is set at random, the resulting output is not necessarily the desirable value. Hence, the self-learning function is used to newly obtain each weight function, and such renewal of each weight function is repeated a predetermined number of times until the desired value is obtained as the resulting output. In this embodiment, the input signal is "0" or "1", and the input pulse sequence is simply made up of low-level and high-level pulses. The LED is turned ON when the output signal has the high level and is turned OFF when the output signal has the low level. Since the synchronizing pulses (clock) has a frequency of 1000 kHz, the brightness of the LED appears to change to the human eye depending on the pulse density. Hence, the LED which appears to be the brightest corresponds to the answer, that is, the recognition result. With respect to a character after sufficient learning, the recognition rate was 100%.
FIG. 42 shows an embodiment of the grouping system and the output determination system shown in FIGS. 38 and 39. In this case, the grouping is made at the input stage, and the couplings are grouped into an excitatory coupling group a and an inhibitory coupling group b. A shift register 81 has at least two bits and stores the coupling coefficient T.sub.ij with respect to each input signal 55.sub.ij. The output signals of the group a is supplied to the OR gate 63a, while the output signals of the group b are supplied to the OR gate 63b. The output signals of the OR gates 63a and 63b are processed similarly as in the case of the embodiment shown in FIG. 41 in the gate circuit 64.
FIG. 43 shows a modification of the gate circuit 64 shown in FIGS. 41 and 42. This modification of the gate circuit 64 uses an OR gate 64c in place of the AND gate 64a and obtains a logical sum. This process of the gate circuit 64 shown in FIG. 43 corresponds to the process described with reference to FIG. 26.
The circuit for grouping the couplings into the excitatory group and the inhibitory group and determining the output is further disclosed in a U.S. patent application Ser. No. 550,404 filed Jul. 10, 1990, the disclosure of which is hereby incorporated by reference.
Next, a description will be given of a third embodiment of the neuron unit according to the present invention, by referring to FIG. 44. This embodiment further includes learning constant setting means 82 for arbitrarily and variably setting the learning constant which is used in the weight function varying circuit from the outside. In other words, in addition to the fundamental rules [1] through [6] described above, the neuron unit is realized by use of digital circuits according to the following additional rule [7A ].
[7A] The learning constant (learning rate) which is used during the learning process of [6] is made variable, so as to enable general applications of the neural network.
The learning constant setting means 82 is provided in place of the frequency dividing circuit 79 shown in FIG. 39. The learning constant setting means 82 includes a counter 83 which receives the error signal, OR gates 84 through 87 for carrying out a logic operation on outputs of the counter 83 so as to process the learning constant, switches Sa through Sd which are respectively connected to the OR gates 84 through 87, and an AND gate 88 which receives outputs of the OR gates 84 through 87. .eta.=1.0 when the switches Sa through Sd are connected to the high-level side, and .eta.=1/16 when the switches Sa through Sd are connected to the not-high-level side. When the number of switches connected to the high-level side is denoted by N, .eta.=2.sup.N /16. Accordingly, the learning constant can be set arbitrarily by use of the switches Sa through Sd or external signals which replace the switches Sa through Sd.
When the pulse density is used as the clock input to the counter 83, it is possible to provide an AND gate 89 with respect to the error signal input as shown in FIG. 44. The learning constant setting means 82 is of course not limited to that shown in FIG. 44, and in addition, it is possible to provide a plurality of learning constant setting means 82. Furthermore, it is possible to appropriately controlling the learning constant setting means 82 by external signals, it is also possible to make the value of the learning constant which is used for the operation on the weight function different from the value of the learning constant which is used for the back propagation of the error signal.
Next, a description will be given of a fourth embodiment of the neuron unit according to the present invention, by referring to FIGS. 45 through 47. In addition to the fundamental rules [1] through [6] described above, this embodiment realizes the neuron unit by use of digital circuits according to the following additional rule [7B].
[7B ] Two kinds of weight functions, that is, an excitatory weight function and an inhibitory weight function are prepared, and the flexibility of the neural network is improved by determining the result of the operation on the input signal depending on the majority of the kind of weight functions used.
In other words, one neuron unit has the excitatory weight function and the inhibitory weight function, and the resulting output which is obtained by an AND of the input signal and the weight function is processed depending on the ratio of the existing excitatory couplings and inhibitory couplings. This ratio of the existing excitatory couplings and inhibitory couplings means the following. That is, with respect to the plurality of input signals which are subjected to an operation in synchronism, the number of times the resulting output obtained by use of the excitatory weight function is "1" is compared with the number of times the resulting output obtained by use of the inhibitory weight function is "1" and the neuron unit outputs "0" when the latter is greater and otherwise outputs "1". Alternatively, the neuron unit may output "0" when the two as the same.
FIGS. 45 and 46 show circuits for realizing the above. First, a pair of shift registers 90a and 90b are provided with respect to each input signal 55. One of the shift registers 90a and 90b stores the excitatory weight function while the other stores the inhibitory weight function. These shift registers 90a and 90b may have the same construction as the shift register 56. The contents of the shift registers 90a and 90b are successively read out by a reading means (not shown) and supplied to corresponding AND gates 91a and 9lb together with the input signal 55. A known reading means may be used.
Output signals 59 and 60 of the AND gates 91a and 9lb are supplied to a majority determination circuit 92 shown in FIG. 46. The digital signals including the signal 59 which are obtained by using the excitatory weight functions stored in the shift registers 90a are supplied to an amplifier 93a and subjected to an adding process. Similarly, the digital signals including the signal 60 which are obtained by using the inhibitory weight functions stored in the shift registers 90b are supplied to an amplifier 93b and subjected to an adding process. Outputs of the amplifiers 93a and 93b are compared in a comparator 94 which determines the majority. Of course, the majority determination circuit 92 is not limited to that shown in FIG. 46, and any kind of majority determination circuit may be used.
FIG. 47 shows the circuit for grouping the circuit shown in FIG. 45. A pair of shift registers (memories) for storing the excitatory and inhibitory weight functions with respect to each input signal are provided, and the logical product is obtained for each group of shift registers.
In FIG. 47, OR gates 63a and 63b are provided in place of the majority determination circuit 92, similarly as in the circuits shown in FIGS. 41 and 42. The gate circuit 64 may have the construction shown in FIG. 43.
In this embodiment, the pair of shift registers 90a and 90b is provided for each input signal 55. Hence, the rewriting of the weight function using the self-learning function is carried out for each of the shift registers 90a and 90b. For this reason, a self-learning circuit 95 is provided as shown in FIG. 45 to calculate the new weight function as described in conjunction with FIGS. 29 through 31 and the equations for obtaining the error signals .delta..sup.+ j and .delta..sup.- j. This self-learning circuit 95 is connected to the input side of the shift registers 90a and 90b.
According to this embodiment, the coupling of the neuron units is not limited to only the excitatory coupling or the inhibitory coupling. As a result, the neural network has more flexibility and is applicable to general applications.
The frequency dividing circuit 79 show in FIG. 46 may also be replaced by a learning constant setting means such as the learning constant setting means 82 shown in FIG. 44.
In addition, the method of determining the output by the majority determination circuit 92 is not limited to that shown in FIG. 45 in which two memories (shift registers 90a and 90b) are provided with respect to each input signal. For example, this method may be applied similarly to a case where one memory 56 is provided with respect to each input signal. In other words, in addition to the combination of FIGS. 38 and 39, it is also possible to combine FIGS. 38 and 46.
Next, a description will be given of a fifth embodiment of the neuron unit according to the present invention. In this embodiment, a switching circuit is provided to select a first mode in which the weight function is renewed (changed) or a second mode in which the weight function is fixed.
In addition to the fundamental rules [1] through [6] described above, this embodiment realizes the neuron unit by use of digital circuits according to the following additional rule [7C].
[7C] The mode is switched between a first mode in which the weight function is renewed (changed) or a second mode in which the weight function is fixed, where the first mode corresponds to a case where the learning (back propagation) takes place and the second mode corresponds to a case where no learning (only forward process) takes place.
FIG. 48 shows an essential part of the fifth embodiment. In FIG. 48, those parts which are the same as those corresponding parts in FIG. 38 are designated by the same reference numerals, and a description thereof will be omitted.
In the neural network, the forward process and the back propagation are not necessarily carried out constantly at the same time. Depending on the circumstances, the forward process is only required. For example, when making a character recognition using the weight functions which are obtained after the learning process, it is sufficient to carry out the forward process alone. Therefore, the neural network becomes more flexible by switching the mode between the first and second modes. This switching of the mode corresponds to a control which determines whether or not the weight function is to be renewed by the new weight function which is obtained as described above in conjunction with FIG. 36 or 37.
In FIG. 48, a switching circuit 101 switches the mode between the first and second modes responsive to an external switching signal S which is applied to a terminal 101a. The switching circuit 101 includes two AND gates, an OR gate and an inverter which are connected as shown. When selecting the first mode, the switching circuit 101 in response to the switching signal S selectively supplies to the shift register 56 the output value of the gate circuit 80 which is newly obtained, so as to renew the weight function. On the other hand, when selecting the second mode, the switching circuit 101 in response to the switching signal S selectively supplies to the shift register 56 the output value of the shift register 56, so as to maintain the weight function fixed. Therefore, the neural network as a whole can be controlled by the switching of the switching circuit 101 responsive to the switching signal S.
In the second embodiment, for example, the weight function and its sign (polarity) are stored in the memory. However, the memory content will be erased when the power source is turned OFF, and it is necessary to store the memory content in a non-volatile storage medium. On the other hand, when the weight functions and the signs thereof are already prepared through the learning process, it is unnecessary to carry out the learning process and it is simply necessary to write the prepared weight functions and the signs thereof in the memory. For this reason, it would be useful if an access to the memory content can be freely made from the outside.
Next, a description will be given of a sixth embodiment of the neuron unit according to the present invention, in which an access to the memory content can be made from the outside. FIGS. 49 through 51 respectively show essential parts of the sixth embodiment.
In FIG. 49, an output signal 102 which is output from the output 56a of the shift register 56 to the outside. The input 56b of the shift register 56 uses an external signal 103 when reading or writing from the outside, and uses an internal signal 103' when not reading or writing from the outside. Hence, a selector 111 is provided to selectively supply one of the signals 103 and 103' to the shift register 56 depending on an external signal 104. A clock signal 106 is used to shift the content of the shift register 56, and an internal synchronizing signal (clock) 105 is used as it is as the clock signal 106. This internal synchronizing pulse 105 is output to the outside as a clock signal 105'. An external circuit (not shown) may control the read and write with respect to the shift register 56 in synchronism with the clock signal 105'.
Alternatively, it is possible to apply an external clock signal 107 to a selector 111a which also receives the internal synchronizing pulse 105, as shown in FIG. 50. The selector 111a selectively supplies the external clock signal 107 or the internal synchronizing pulse 105 to the shift register 56, and this selector 111a is controlled similarly to the selector 111.
When a combination of a RAM and an address decoder is used in place of the shift register 56, it becomes possible to make a random access to the RAM from the external circuit by connecting an address bus, a data bus and a control signal line for read/write control signals to the external circuit. In the case of a memory having two ports, one port can be used for the read/write from the external circuit and the other port can be used for the internal processing. In the case of a memory having one port, the line from the internal address decoder is switched to the address bus and the like when making the access from the external circuit. This may be achieved by replacing the lines for the signals 105, 106 and 107 by the address bus and the control signal line for the read/write control signals.
Next, a description will be given of a case where the read/write with respect to a plurality of memories is carried out by use of a common bus, by referring to FIG. 51. In FIG. 51, only essential parts of FIGS. 49 and 50 are shown.
First, an address is assigned to each memory 56. A judging circuit 122 is connected to each memory 56 and judges whether or not an address received from the external circuit matches the address assigned to the memory 56. Each memory 56 is selected depending on a judgement result 124 output from the judging circuit 122. When making the read/write with respect to the memory 56, an address signal 120 which designates an arbitrary one of the memories 56 is supplied to the judging circuits 122, so that only the arbitrary memory 56 is read/write enabled.
When the memory 56 has only one port, an external signal 121 from the external circuit is used to indicate whether or not the read/write is to be made. On the other hand, all of the memories 56 are selected when no read/write is to be made. Further, a gate circuit 123 is provided to prevent the data from being supplied to the common bus for external read/write.
When realizing this embodiment by the hardware described above, it is possible to provide the entire hardware within one computer or provide only a portion of the hardware within the computer. In addition, hardware portions having independent functions may be combined to form the entire hardware.
Next, a description will be given of a seventh embodiment of the neuron unit according to the present invention. In addition to the fundamental rules [1] through [6] described above, this embodiment realizes the neuron unit by use of digital circuits according to the following additional rules [7D ] and [8D ].
[7D ] When carrying out the learning process, the input signal and the teaching signal are stored in a memory so as to facilitate the operation of the operator.
[8D ] When carrying out the learning process under the rule [7D], the learning rate is judged from the error in the final resulting output, and the judgement of the learning is made by the neural network itself.
It is assumed for the sake of convenience that the neuron units are connected as shown in FIG. 2 to form the neural network, and that this neural network learns from the teacher.
In order for the neural network to learn, it is necessary to prepare input signals which are input to the input layer and a desired resulting output, that is, a teaching signal which is output from the output layer. In FIG. 2, the input layer is made up of the neuron units on the left, and the output layer is made up of the neuron units on the right.
Immediately after a system such as a character recognition system applied with the neural network is started, that is, before the learning takes place, the weight functions of the neuron units are set at random. For this reason, the resulting output in most cases does not become the desired value with respect to the input signal. Hence, each weight function is varied as described above by applying to the neural network the teaching signal which corresponds to the input signal. Generally, the neural network has many input and output signals. Thus, the input signal and the teaching signal respectively are rarely one, and a group of input signals and a group of teaching signals usually exist. In addition, the learning is not ended by varying the weight functions once, and the learning process is usually carried out several tens of times to several thousands of times or more. As a result, there is a great burden on the operator in this respect.
Accordingly, this embodiment provides a memory for storing the input signal data and the teaching signal data, so that the inputting of the various data during each learning process and the learning process as a whole are simplified, thereby reducing the burden on the operator.
Particularly, groups of appropriate input signal data suited for the learning process and corresponding group of teaching signal data are prepared beforehand. At least one group of input signal data and at least one group of teaching signal data are stored in an external memory which is coupled externally to the neural network. The data stored in the external memory are stored in the form of pulse trains so as to match the form of the input signals to the neural network. Alternatively, it is also possible to store the data in the external memory in the form of numerical values and convert the numerical values to corresponding pulse trains when needed. Each group of input signal data has a corresponding group of teaching signal data, and such corresponding groups are read out from the external memory.
After the above described preparation, the operator instructs the neural network to carry out the learning process. Responsive to the learning instruction, the neural network reads out the corresponding groups of input signal data and teaching signal data from the external memory, and applies the input signal data to the neuron units in the input layer of the neural network. Then, the neural network carries out the forward process described above, the operation result is obtained from the neuron units in the output layer of the neural network. The teaching signal data which are read out from the external memory is then supplied to the neural network so that the neural network can carry out the learning process. Other groups of input signal data and teaching signal data are successively read out from the external memory and the process of supplying the input signal data to the input layer and supplying the teaching signal data to the neural network is repeated if needed. The end of the learning process can be controlled by providing a counter and counting the number of learning processes which are carried out. By appropriately changing the corresponding groups of input signal and teaching signal data which are used for the learning process depending on the counted value in the counter, it is possible to realize an efficient learning process. It is not essential to read out the teaching signal data from the external memory simultaneously as the reading out of the input signal data, and the teaching signal data need only be supplied to the neural network by the time the learning process of the neural network starts.
The learning process can also be controlled as follows in order to realize an efficient learning process. That is, the resulting output from the neural network is compared with the teaching signal data, and it is judged that the neural network has sufficiently learned and the learning process is ended when the error between the resulting output and the teaching signal data becomes less than a predetermined value. This judgement can be realized with ease by comparing and carrying out an operation on the resulting output of the neural network and the teaching signal data when supplying the teaching signal data to the neuron units in the neural network. Furthermore, a counter may be provided similarly as described above, and stop the learning process when the error does not become less than the predetermined value after carrying out a predetermined number of learning processes. In this case, it is possible to reset the data for learning and carry out the learning process again, and this method is extremely effective when E in the equation (4) is trapped at a local minimum, for example.
FIG. 52 shows the seventh embodiment of the present invention applied to the neural network. A system 131 shown in FIG. 52 includes a hierarchical neural network 132, a controller 133, a memory 134 for storing the input signal data, a memory 135 for storing the teaching signal data, and an operation circuit 136. The necessary data are prestored in the memories 134 and 135. The operation circuit 136 receives a resulting output 132R of the neural network 132 and a teaching signal data 135B, and carries out a comparison and a judgement which will be described hereunder.
When the operator instructs the learning process to the controller 133, the controller 133 outputs control signals 132A, 134A and 135A. The memory 134 inputs an input signal data 134B to an input part 132-I of the neural network 132 responsive to the control signal 134A and the memory 135 inputs a teaching signal data 135B to an output part 132-0 of the neural network 132 responsive to the control signal 136A. The neural network 132 starts the learning process responsive to the control signal 132A. The controller 133 includes a circuit for storing the number of learning processes carried out. When the number of learning processes reaches a predetermined number, the controller 133 ends the learning process of the neural network 132 by supplying the control signal 132A. The appropriate data are selectively read out from the memories 134 and 135 and input to the neural network 132 responsive to the control signals 134A and 135A depending on each stage of the learning process.
The learning process may be ended by use of the operation circuit 136. Particularly, the resulting output 132R of the neural network 132 and the teaching signal data 135B are compared, and an error is input to the controller 133 as a judgement result 136A. The controller 133 ends the learning process when the judgement result 136A is less than a predetermined value, but otherwise continues the learning process. Alternatively, the operation circuit 136 may judge whether the learning process is to be ended or continued, and in this case, the operation circuit 136 inputs the judgement result to the controller 133 as the judgement result 136A. Furthermore, in these two cases, the operation circuit 136 may be included within the controller 133. In addition, the learning process of the neural network 132 may be controlled by using a combination of the judgement result 136A and the number of learning processes carried out which is stored within the controller 133.
Of course, the operations of parts of the system 131 may be carried out by a computer.
Next, a description will be given of an eighth embodiment of the present invention. In addition to the fundamental rules [1] through [6] described above, this embodiment realizes the neuron unit by use of digital circuits according to the following additional rule [7E].
[7E] The length of the pulse train, that is, the data length, which describes the weight function is made variable, so as to improve the calculation accuracy of the neural network.
FIG. 53 shows an essential part of the eighth embodiment. In FIG. 53, those parts which are the same as those corresponding parts in FIG. 38 are designated by the same reference numerals, and a description thereof will be omitted.
In FIG. 53, the pulse density of the output signal of the AND circuit 57 is approximately the product of the pulse density of the input signal and the pulse density of the weight function, and the AND circuit 57 functions similarly as when obtaining a logical products of signals in the analog system. The pulse density of the output signal of the AND circuit 57 more closely approximates the product of the pulse densities of the two signals applied thereto when the pulse trains of the two signals are longer and when the "0"s and "1"s are arranged more at random in the pulse trains. The "0"s or "1"s are concentrated when the "0"s or "1"s are not arranged at random.
Accordingly, by making the length of the pulse train (data length) of the weight function variable and setting the length of the pulse train depending on the circumstances, it is possible to arrange the "1"s and "0"s more at random. Especially when the pulse train of the weight function is short compared to the pulse train of the input signal and no more data can be read as the weight function, the read out can return to the beginning of the pulse train of the weight function and repeat such a read out depending on the length of the pulse train of the input signal.
FIG. 54 shows a case where the length of the pulse train of the input signal is 12 and the weight function is 6/12 for three cases, that is, a case where the length of the pulse train of the weight function is 6, a case where the length of the pulse train of the weight function is 8 and a case where the length of the pulse train of the weight function is 12. In the case where the length of the pulse train of the weight function is 6, the length of the pulse train of the weight function is repeated once, that is, the total length of the pulse train of the weight function is shifted once, to match the length of the pulse train of the input signal. In the case where the length of the pulse train of the weight function is 8, the length of the pulse train of the weight function is repeated 0.5 times, that is, 0.5 times the total length of the pulse train of the weight function is shifted once, to match the length of the pulse train of the input signal. It may be readily seen from FIG. 54 that although the weight function is 6/12 for the three cases, the random nature of the pulse train of the weight function improves when the length thereof is varied by repeating the same pulse trains to match the length of the pulse train of the input signal. When the random nature of the pulse train is improved, it is possible to improve the calculation accuracy of the neural network as a whole.
In FIG. 53, a control signal 56c is supplied to the shift register 56 from an external circuit such as a control circuit, so as to control the bit length of the data to be shifted. As a result, it is possible to make the length of the pulse train (data length) of the weight function variable.
Next, a description will be given of a ninth embodiment of the present invention. In addition to the fundamental rules [1] through [6] described above, this embodiment realizes the neuron unit by use of digital circuits according to the following additional rule [7F].
[7F] In the neural network, the input and output signals can be processed in analog signal form.
As described above, the signals processed in the neural network are digital signals, that is, pulse trains. Hence, in order to input analog signals to the neural network, the analog data are converted into pulse trains which contain information in the form of pulse densities. The conversion may be realized by providing a converting unit at the input signal input part of each neuron unit belonging to the input layer of the neural network and providing a converting unit at the teaching signal input part of each neuron unit belonging to the output layer of the neural network.
As a first method of analog-to-digital (A/D) conversion, it is possible to use a circuit 141 shown in FIG. 55A. A comparator 143 of the circuit 141 compares an input signal (analog voltage) 144 with a thermal noise (voltage) 142 of a transistor or the like, and outputs a signal 145 which is supplied to the neuron unit belonging to the input layer of the neural network. The thermal noise 142 is supplied to the comparator 143 as a random number. Hence, the signal 145 output from the circuit 141 is a pulse train which is proportional to the input signal 144 and in which the pulses exist at random intervals.
As a second method of A/D conversion, it is possible to use a circuit 146 shown in FIG. 55B. The circuit 146 includes an amplifier 149, an A/D converter 147 and a memory 148. The memory 148 prestores pulse train data corresponding to various input values. The input signal (analog voltage) 144 is supplied to the A/D converter 147 via the amplifier 149, and a binary output data of the A/D converter 147 is supplied to the memory 148 as an address signal. Based on the address signal, the prestored pulse train data which corresponds to the input signal value is read out from the memory 148 and is supplied to the neuron unit belonging to the input layer of the neural network. Of course, it is possible to omit the amplifier 149.
As a third method of A/D conversion, it is possible to convert an output of the A/D converter into a serial pulse train using a known pseudo random pulse generating circuit (not shown).
Similarly as in the case of the input signals to the neural network, the output signals of the neural network are also pulse trains containing information in the form of pulse densities. Hence, in order to output analog signals from the neural network, the pulse trains (digital data) are converted into analog signals. The conversion may be realized by providing a converting unit at the signal output part of each neuron unit belonging to the input layer of the neural network.
As a first method of digital-to-analog (D/A) conversion, it is possible to use a circuit 150 shown in FIG. 56A. The circuit 150 includes a counter 151, a D/A converter 152 and an amplifier 154. The counter 151 counts the number of pulses received from the neuron unit belonging to the output layer of the neural network and outputs the counted value as a binary data. The binary data is converted into an analog signal 153 by the D/A converter 152 and is amplified by the amplifier 154. The pulses received from the neuron unit belonging to the output layer of the neural network are generated at random time intervals, and thus, when these pulses are supplied to the counter 151 only during reference time intervals, the pulse train which contains the information in the form of the pulse density can be converted into the binary data which indicates the same information. Normally, this converting operation is repeated. Of course, the amplifier 154 may be omitted.
As a second method of D/A conversion, it is possible to use a circuit 155 shown in FIG. 56B. The circuit 155 includes a frequency-to-voltage (F/V) converter 156 and the amplifier 154. The F/V converter 156 generates an output voltage 153 which is dependent on the frequency of the pulses received from the neuron unit belonging to the output layer of the neural network, that is, dependent on the pulse density. The output voltage 153 of the F/V converter 156 is amplified by the amplifier 154. Since the pulse density of the pulse train corresponds to a kind of frequency modulation, the corresponding analog signal (voltage) 153 can easily be obtained by use of the F/V converter 156 which is a generally available element. Again, the amplifier 154 may be omitted.
As a third method of D/A conversion, it is possible to use a circuit 157 shown in FIG. 56C. The circuit 157 includes a driving circuit 158 and an LED 159. The pulses received from the neuron unit belonging to the output layer of the neural network are supplied to the LED 159 via the driving circuit 158. Thus, when the reference clock has a sufficiently high frequency, the brightness of the LED 159 becomes proportional to the pulse density of the pulse train. In this case, it is possible to visually and directly detect the resulting output of the neural network. Hence, this third method facilitates the transmission of the resulting output in the form of an optical signal.
Next, a description will be given of a tenth embodiment of the present invention. In this embodiment, a neural network is formed by a plurality of neuron units such as those of the embodiments described above. A plurality of such neural networks are then connected to form a large neural network as shown in FIG. 57. In this embodiment, six neural networks NW1 through NW6 are connected to form the large neural network, and the six neural networks NW1 through NW6 receive corresponding bits of input signals y1 through y6. In other words, the number of neural networks which are connected is equal to the number of bits of the input signals.
Next, a description will be given of a case where the above described large neural network is applied to a self-learning type character recognition system. The first layer of the neural network includes 256 neuron units, the second layer includes 20 neuron units and the third layer includes five neuron units. First, a hand-written character shown in FIG. 19 is read by a scanner, and the read image is divided into 16.times.16 meshes. The data in each mesh is then applied to each neuron unit of the first layer in the neural network. For the sake of convenience, the data of a mesh which includes a portion of the character is taken as "1" while the data of a mesh which includes no portion of the character is taken as "0". The output of the neural network is connected to an LED so that the resulting output is directly displayed on the LED. Out of the five neuron units of the third layer, the neuron unit which outputs the largest output is assumed to output the recognition result. The learning takes place so that when the numbers "1" through "5" are input to the neural network, the five neuron units of the third layer respectively corresponding to the numbers "1" through "5" output the largest output. In other words, when the number "1" is input, the neuron unit of the third layer corresponding to the number "1" outputs the largest output.
Each input signal data is made up of 128 bits. Hence, 128 identical neural networks were connected to form the large neural network.
Initially, when each weight function is set at random, the resulting output is not necessarily the desirable value. Hence, the self-learning function is used to newly obtain each weight function, and such renewal of each weight function is repeated a predetermined number of times until the desired value is obtained as the resulting output. In this embodiment, the input signal is "0" or "1", and the input pulse train is simply made up of low-level and high-level pulses. The LED is turned ON when the output signal has the high level and is turned OFF when the output signal has the low level. Since the synchronizing pulses (clock) has a frequency of 1000 kHz, the brightness of the LED appears to change to the human eyes depending on the pulse density. Hence, the LED which appears to be the brightest corresponds to the answer, that is, the recognition result. With respect to a character after sufficient learning, the recognition rate was 100%.
In the embodiments described above, the neural network carries out parallel processing. However, as the scale of the neural network becomes large, it becomes difficult to actually produce the required circuits which form the neural network. But if one neuron unit were to have the functions of a plurality of neuron units, it would be possible to reduce the scale of the entire neural network.
Next, a description will be given of an eleventh embodiment of the present invention in which one neuron unit has the functions of two neuron units within the same layer of the neural network.
FIG. 58 shows two neuron units 161 and 162. The neuron units 161 and 162 respectively receive input signals 164 and 165 for the forward process and respectively output output signals 166 and 167 of the forward process. Error signals 170 and 171 are propagated from an immediately next layer of the neural network, and error signals 168 and 169 are back-propagated to an immediately preceding layer of the neural network. Actually, each signal is received via a plurality of signal lines.
FIG. 59 is a diagram for explaining a first method of realizing the functions of the two neuron units 161 and 162 shown in FIG. 58 by a single neuron unit 173. In FIG. 59, those signals which are the same as those corresponding signals in FIG. 58 are designated by the same reference numerals, and a description thereof will be omitted.
Next, a description will be given of the circuit shown in FIG. 59 by referring to FIGS. 60 through 62. First, the input signals 174 and 175 are input and stored in respective memories 182 and 183. The neuron unit 173 carries out an operation on the input signal 174 and stores an output signal in a memory 185. As a result, the forward process with respect to the input signal 174 ends.
Next, as shown in FIG. 61, the input signal 175 which is stored in the memory 183 is read out and supplied to the neuron unit 173. The neuron unit 173 thus carries out an operation on the input signal 175 and outputs the output signal 177. At the same time, the neuron unit 173 outputs the output signal 176 which is stored in the memory 185. Therefore, the output signals 176 and 177 which respectively are the results of the operations carried out on the input signals 174 and 175 are output as the results of the forward process.
An operation is carried out in the immediately next layer of the neural network based on the output signals 176 and 177, and the error signals 180 and 181 are eventually back-propagated. First, the back-propagated error signal 180 is stored in a memory 186. The back-propagated error signal 181 is supplied to the neuron unit 173 which carries out an operation thereon, and an error signal which is output from the neuron unit 173 is stored in a memory 184. Next, the error signal 180 which is stored in the memory 186 and the input signal 174 which is stored in the memory 182 are read out simultaneously and supplied to the neuron unit 173 which carries out an operation thereon. Thus, the neuron unit 173 outputs the error signal 178, and the error signal stored in the memory 184 is output as the error signal 179, thereby ending the operation of the neuron unit 173.
The line to which the signal is to be output and the line from which the signal is to be input may be appropriately selected by providing a switch or the like on each of lines 187 through 190 which are connected to the neuron unit 173.
Next, a description will be given of the weight function and its positive or negative sign which are stored in the neuron unit. It is possible to independently store the weight function and its sign in the respective memories as shown in FIG. 63 by switching switches SW1 and SW2. On the other hand, it is possible to store both the weight function and its sign in the same memory 56 as shown in FIG. 64.
The memory 185 described above is provided to simultaneously output the output signals 176 and 177. Hence, the memory 185 may be omitted when it is unnecessary to simultaneously output the output signals 176 and 177. The memory 184 is provided to simultaneously output the error signals 178 and 179 which are to be back-propagated to the immediately preceding layer of the neural network. Thus, the memory 184 may be omitted similarly when it is unnecessary to simultaneously output the error signals 178 and 179.
The memory 183 is provided to temporarily store the input signals 174 and 175 which are received simultaneously. Hence, when the input signal 175 does not change in the states shown in FIGS. 60 and 62, it is possible to omit this memory 183. The memories 182 and 186 may be omitted for similar reasons.
In the described embodiment, the single neuron unit has the functions of two neuron units. However, it is of course possible for the single neuron unit to have the functions of three or more neuron units. In addition, the single neuron unit may have the functions of two or more neuron units which belong to different layers of the neural network. The circuitry required for the single neuron unit having such functions may be realized with ease using a general integrated circuits (ICs).
Next, a description will be given of a case where a neural network made up of the above described neuron units is applied to a self-learning type character recognition system. The first layer of the neural network includes 256 neuron units, the second layer includes 20 neuron units and the third layer includes five neuron units. First, a hand-written character shown in FIG. 19 is read by a scanner, and the read image is divided into 16.times.16 meshes. The data in each mesh is then applied to each neuron unit of the first layer in the neural network. For the sake of convenience, the data of a mesh which includes a portion of the character is taken as "1" while the data of a mesh which includes no portion of the character is taken as "0". The output of the neural network is connected to an LED so that the resulting output is directly displayed on the LED. Out of the five neuron units of the third layer, the neuron unit which outputs the largest output is assumed to output the recognition result. The learning takes place so that when the numbers "1" through "5" are input to the neural network, the five neuron units of the third layer respectively corresponding to the numbers "1" through "5" output the largest output. In other words, when the number "1" is input, the neuron unit of the third layer corresponding to the number "1" outputs the largest output.
Initially, when each weight function is set at random, the resulting output is not necessarily the desirable value. Hence, the self-learning function is used to newly obtain each weight function, and such renewal of each weight function is repeated a predetermined number of times until the desired value is obtained as the resulting output. In this embodiment, the input signal is "0" or "1", and the input pulse train is simply made up of low-level and high-level pulses. The LED is turned ON when the output signal has the high level and is turned OFF when the output signal has the low level. Since the synchronizing pulses (clock) has a frequency of 1000 kHz, the brightness of the LED appears to change to the human eyes depending on the pulse density. Hence, the LED which appears to be the brightest corresponds to the answer, that is, the recognition result. With respect to a character after sufficient learning, the recognition rate was 100%.
Next, a description will be given of a twelfth embodiment of the present invention. FIG. 65 shows an essential part of this embodiment. In FIG. 65, those parts which are the same as those corresponding parts in FIG. 39 are designated by the same reference numerals, and a description thereof will be omitted.
In FIG. 65, an OR circuit 216 corresponds to the process described above in conjunction with FIG. 24.
When carrying out the parallel processing, the number of input signals becomes extremely large and the interconnections of the neural network becomes extremely complex when making a character recognition or the like. Hence, this embodiment carries out serial processing with respect to a portion of the data, so that the number of interconnections is greatly reduced. In the case of a charge coupled device (CCD), for example, the data is read out serially for each line. Hence, it is more convenient in such a case to carry out the serial processing with respect to a portion of the data.
First, when calculating the outputs a and b described in conjunction with FIG. 25, the calculations can be carried out independently for the output a and the output b. In order to calculate the outputs a and b for 256 inputs, for example, the outputs a and b are first calculated for 64 inputs. The calculation of the outputs a and b for 64 inputs is successively performed four times, and the outputs a and b are finally obtained by taking an OR of the four outputs a and taking an OR of the four outputs b. This principle can be used to carry out the serial processing.
The outputs a and b which are obtained by the first process are stored in memories 215 shown in FIG. 65. The outputs a and b which are obtained by the second through fourth processes are similarly stored in the memories 215. After four outputs a are stored in one memory 215 and four outputs b are stored in the other memory 215, the four outputs a are read out in parallel from the one memory 215 and supplied to one OR circuit 216, and the four outputs b are read out in parallel from the other memory 215 and supplied to the other OR circuit 216. Hence, the calculation ends after four reference clock pulses. The serial processing becomes possible by thereafter carrying out the process y.sub.j =a.andgate.b described in conjunction with FIG. 25 with respect to the outputs of the OR circuits 216. A shift register or the like may be used for the memory 215.
The output signal of the gate circuit 64 is supplied to the next layer of the neural network. However, since the output signal from the gate circuit 64 is supplied to the next layer after a plurality of clock pulses, the operation of the neuron units of the next layer must be delayed accordingly. It is possible to generate clock pulses which are obtained by frequency-dividing the reference clock pulses in a clock generator (not shown) and provide a switch or the like at the neuron unit so that each neuron unit may receive the reference clock pulse or the clock pulse. Alternatively, it is possible to provide a frequency divider within the neuron unit so that the clock pulses can be generated from the reference clock pulses within the neuron unit. In either case, a more general application of the neural network becomes possible when measures are taken so that the neuron unit can selectively receive the reference clock pulse or the clock pulse.
The pulse of the error signal is back-propagated only after the four serial processings end. However, the pulses of the input signal, the weight function and the sign of the weight function which are used for the forward process are required for the learning and back-propagation of the error signal. When the serial processing is carried out, the pulses corresponding to the first three processings are already read out when the operation is carried out on the pulse of the error signal, and it is necessary to again read out the pulses corresponding to the first three processings. Accordingly, after the forward process is carried out by the four serial processings, the first through 64th pulses are again input or read out from the memory and used for the operations necessary for the back-propagation of the error signal and the learning process. Thereafter, the operation is carried out on the next 65th through 128th pulses. The above described operation is repeated a total of four times before ending the operations on the 256 inputs.
As described above, the data used at the time of the forward process are used for the learning process. Thus, the data used at the time of the forward process may be stored in an independent memory and read out at the time of the learning process. Alternatively, a hold circuit may be provided with respect to the input signal which is received from the outside and an address decoder may be provided with respect to the weight function and the like stored in the memory for outputting the same address as the stored weight function, so that the data identical to that at the time of the forward process are input at the time of the learning process.
In addition, a switch 221 may be provided so that the circuit shown in FIG. 65 can be switched between two modes, that is, a mode in which the parallel processing is carried out as described in conjunction with FIG. 39 and a mode in which the serial processing is carried out. In order to input parallel data to the neural network which carries out the serial processing, a parallel-to-serial converter circuit is provided at the input of the neural network, and a known method such as the method using a shift register may be used to realize the parallel-to-serial converter circuit.
Next, a description will be given of a case where the above described neural network is applied to a self-learning type character recognition system. The first layer of the neural network includes 256 neuron units, the second layer includes 20 neuron units and the third layer includes five neuron units. First, a hand-written character shown in FIG. 19 is read by a scanner, and the read image is divided into 16.times.16 meshes. The data in each mesh is then applied to each neuron unit of the first layer in the neural network. For the sake of convenience, the data of a mesh which includes a portion of the character is taken as "1" while the data of a mesh which includes no portion of the character is taken as "0". The output of the neuron unit network is connected to an LED so that the resulting output is directly displayed on the LED. Out of the five neuron units of the third layer, the neuron unit which outputs the largest output is assumed to output the recognition result. The learning takes place so that when the numbers "1" through "5" are input to the neural network, the five neuron units of the third layer respectively corresponding to the numbers "1" through "5" output the largest output. In other words, when the number "1" is input, the neuron unit of the third layer corresponding to the number "1" outputs the largest output.
Initially, when each weight function is set at random, the resulting output is not necessarily the desirable value. Hence, the self-learning function is used to newly obtain each weight function, and such renewal of each weight function is repeated a predetermined number of times until the desired value is obtained as the resulting output. In this embodiment, the input signal is "0" or "1", and the input pulse train is simply made up of low-level and high-level pulses. The LED is turned ON when the output signal has the high level and is turned OFF when the output signal has the low level. Since the synchronizing pulses (clock) has a frequency of 1000 kHz, the brightness of the LED appears to change to the human eyes depending on the pulse density. Hence, the LED which appears to be the brightest corresponds to the answer, that is, the recognition result. With respect to a character after sufficient learning, the recognition rate was 100%.
In the embodiments described heretofore, it is assumed that the forward process and the learning process are both carried out by hardware, that is, circuits. However, these processes may be carried out by software, that is, by executing appropriate programs by a central processing unit (CPU).
Next, descriptions will be given of thirteenth through fifteenth embodiments of the present invention, by referring to FIGS. 66 through 68.
The neuron circuits shown in FIGS. 38 through 47 can be used to form a neuron unit or a neural network, but the entire circuit need not be formed solely of hardware. For example, the signal processing may be carried out by software according to the procedure described with reference to the equations (8) through (29).
In the thirteenth embodiment of the present invention, the functions of the neuron units forming the neural network may be realized by software. In the case of the neural network shown in FIG. 2, the signal processing is carried out by software in an arbitrary neuron unit of the neural network. The signal processing may be carried out by software in one or more neuron units, all of the neuron units, or selected neuron units determined for each layer of the neural network.
FIG. 66A shows the neuron unit which carries out the signal processing by software in the thirteenth embodiment. In FIG. 66A, an input/output apparatus 301 is coupled to a neuron unit which uses a neuron circuit or an apparatus for inputting/outputting signals from/to the neural network. A memory 303 stores data and programs (software) for controlling a central processing unit (CPU) 302, and the signals are processed in the CPU 302. The signal processing procedure is as described above. The software is made according to the procedures shown in FIGS. 67 and 68 and stored in the memory 303.
One neuron unit shown in FIG. 66A may function as a plurality of neuron units depending on the software. In this case, it is necessary to process the signals in time division.
According to this embodiment, the network structure can be modified by simply changing the memory 303 (or changing the contents of the memory 303) without the need to modify the hardware. As a result, the network structure becomes flexible and suited for general applications.
In the fourteenth embodiment, a portion of the functions of one neuron unit is carried out by software. In other words, the forward process is carried out by software. In FIG. 66B, software based on the signal processing procedure shown in FIG. 68 is stored in the memory 303, so as to realize a neuron unit which uses software and can carry out the forward process. In order to realize a neuron unit having the function of carrying out the forward process, the circuit shown in FIG. 38 or 45 is added to the input/output apparatus 301. In either case, the right half of the circuit shown in FIG. 39 and the circuit shown in FIG. 40 are required. The circuit shown in FIG. 44 may be provided depending on the needs. In FIG. 66B, the forward process is realized by the provision of a forward process circuit 304.
According to this embodiment, the network structure can be modified by simply changing the memory 303 (or changing the contents of the memory 303) without the need to modify the hardware. As a result, the network structure becomes flexible and suited for general applications.
General electronic apparatuses usually has a CPU and there is no need to newly provide the CPU 302. In addition, when no learning function is required, it is possible to greatly reduce the necessary hardware.
In the fifteenth embodiment, the learning process is carried out by software. In FIG. 66C, software based on the signal processing procedure shown in FIG. 67 is stored in the memory 303, so as to realize a neuron unit which uses software and can carry out the learning process. In order to realize a neuron unit having the function of carrying out the learning process, the circuits shown in FIGS. 38 and 39, the circuits shown in FIGS. 38 and 46, the circuit shown in FIG. 41, 42 or 47 is added to the input/output apparatus 301. The circuit shown in FIG. 43 may be provided depending on the needs. In FIG. 66C, the learning function is realized by the provision of a learning circuit 305.
According to this embodiment, the network structure can be modified by simply changing the memory 303 (or changing the contents of the memory 303) without the need to modify the hardware. As a result, the network structure becomes flexible and suited for general applications. In addition, the network can cope with a modification to the learning rule. Furthermore, general electronic apparatuses usually has a CPU and there is no need to newly provide the CPU 302.
As described above, the functions of the neuron unit can be realize using software. Moreover, when the signal processing system of the present invention is employed, the signal processing can be made solely by digital logic operations, and a low level language may be used for the required software thereby enabling high-speed processing of the software.
FIG. 67 is a flow chart for explaining the learning process of the CPU 302. In FIG. 67, a step S1 generates the error signals .delta..sup.+ j=(y.sub.j EXOR d.sub.j) AND d.sub.j and .delta..sup.- j=(y.sub.j EXOR d.sub.j) AND y.sub.j between the output signal y.sub.j and the teaching signal d.sub.j as described in conjunction with FIG. 27. A step S2 judges the following processes to be calculated depending on the sign of the weight function which is excitatory or inhibitory. Steps S3e and S4e are then carried out in the case where the coupling between the jth layer and the next kth layer is excitatory. On the other hand, steps S3i and S4i are carried out in the case where the coupling between the jth layer and the next kth layer is inhibitory.
The step S3e obtains an AND between the weight function T.sub.jk and the error signal .delta..sup.+ k, and an AND between the weight function T.sub.jk and the error signal .delta..sup.- k. The step S4e obtains E.sup.+ j=.orgate.(.delta..sup.+ k.andgate.T.sub.jk) which is an OR of all of .delta..sup.+ k .andgate.T.sub.jk obtained in the step S3e. In addition, the step S4e obtains E.sup.- j=.orgate.(.delta..sup.- k.andgate.T.sub.jk) which is an OR of all of .delta..sup.- k.andgate.T.sub.jk obtained in the step S3e.
Similarly, the step S3i obtains an AND between the weight function T.sub.jk and the error signal .delta..sup.+ k, and an AND between the weight function T.sub.jk and the error signal .delta..sup.- k. The step S4i obtains I.sup.+ jk=.orgate.(.delta..sup.+ k.andgate.T.sub.jk) which is an OR of all of .delta..sup.+ k.andgate.T.sub.jk obtained in the step S3i . In addition, the step S4i obtains I.sup.- j=.orgate.(.delta..sup.- k .andgate.T.sub.jk) which is an OR of all of .delta..sup.- k.andgate.T.sub.jk obtained in the step S3i.
A step S5 obtains .delta..sup.+ j=E.sup.+ j.andgate.I.sup.- j which is an OR of E.sup.+ j and I.sup.- j. In addition, the step S5 obtains .delta..sup.- j=E.sup.- j.andgate.I.sup.+ j which is an OR of E.sup.- j and I.sup.+ j. A step S6 thins out the pulse trains of the error signals .delta..sup.+ j and .delta..sup.- j and obtains .eta..delta..sup.+ j and .eta..delta..sup.- j. A step S7 obtains an AND of the input signal Y.sub.i and the thinned out error signals .eta..delta..sup.+ j and .eta..delta..sup.- j. That is, the step S7 obtains .DELTA.T.sup.+ ij=.eta..delta..sup.+ j.andgate.y.sub.i and .DELTA.T.sup.- ij=.eta..delta..sup.- j.andgate.y.sub.i. A step S8 judges the following processes to be calculated depending on the sign of the weight function which is excitatory or inhibitory.
Then, a step S9e is carried out in the case of the excitatory coupling and a step S9i is carried out in the case of the inhibitory coupling. The step S9e renews the weight function T.sub.ij by obtaining T.sub.ij .andgate..DELTA.T.sup.- ij.andgate..DELTA.T.sup.+ ij. On the other hand, the step S9i renews the weight function T.sub.ij by obtaining T.sub.ij .andgate..DELTA.T.sup.+ ij.orgate..DELTA.T.sup.- ij.
A step S10 changes j to i in accordance with the backward process In this case, j is decremented to i. A step S11 thereafter judges whether or not the jth layer of the neural network is the input layer. The process returns to the step S2 when the judgement result in the step S11 is NO. On the other hand, the process ends when the judgement result in the step S11 is YES.
FIG. 68 is a flow chart for explaining the forward process of the CPU 302. In FIG. 68, a step S21 inputs the input signal (pulse train) Y.sub.i. A step S22 supplies a signal Y.sub.i from the ith layer to the jth layer, where j=i+1. A step S23 obtains y.sub.i .andgate.T.sub.ij which is an AND of the input signal y.sub.i and the weight function T.sub.ij. A step S24 judges the following processes to be calculated depending on the sign of the weight function which is excitatory or inhibitory.
A step S25e is then carried out when the coupling is excitatory. On the other hand, a step S25i is carried out when the coupling is inhibitory. The step S25e obtains E=.orgate.(T.sub.ij .andgate.y.sub.i) which is an OR of all y.sub.i .andgate.T.sub.ij obtained in the step S23. The step S25i obtains I=.orgate.(T.sub.ij .andgate.y.sub.i) which is an OR of all y.sub.i .andgate.T.sub.ij obtained in the step S23.
A step S26 obtains y.sub.j =E.andgate.I which is an AND of E and I, or y.sub.j =E.orgate.I which is an OR of E and I. Then, a step S27 increments i by one. In this case, i is incremented to j. A step S28 judges whether or not the ith layer is the output layer of the neural network. When the judgement result in the step S28 is NO, the process returns to the step S22. On the other hand, when the judgement result in the step S28 is YES, a step S29 outputs the signal (pulse train) y.sub.j and the process ends.
Of course, the application of the present invention is not limited to the character recognition system. The present invention may be applied to various other systems such as image recognition systems, motion control systems for robots, and associative information storage systems.
In addition, the structure of the neural network according to the present invention is not limited to the network structure shown in FIG. 2.
FIG. 69 shows a neural network in which a neuron unit 1 included in an aggregate is not coupled to all neuron units 1 included in another aggregate. In the neural network shown in FIG. 2, each neuron unit 1 included in an aggregate is coupled to all neuron units 1 included in another aggregate. But in the present invention, the neuron units 1 included in an aggregate need not be coupled to all neuron units 1 included in another aggregate as may be seen from FIG. 69.
FIG. 70 shows a neural network in which a first aggregate and a last aggregate are coupled via two intermediate aggregates. Of course, the number of intermediate aggregates between the first and last aggregates is not limited to one or two and may be three or more.
FIG. 71 shows a neural network in which a first aggregate and a last aggregate are coupled via a single intermediate aggregate.
In the embodiments described above such as the embodiment shown in FIG. 41, for example, each shift register which stores the weighting coefficient T.sub.ij must have a number of bits corresponding to the required pulse length because the weighting coefficient T.sub.ij is described by a random pulse sequence (pulse train). If the calculations or operations are to be carried out with a high accuracy, it is essential that the pulse length is set long. For example, if the signal accuracy of 7 bits is considered, the random pulse sequence must have a bit length (pulse length) of 128 (=2.sup.7) bits. In this case, in order to store the weighting coefficient T.sub.ij, each corresponding shift register must have 128 bits.
In the neuron unit network which is made up of a plurality of neuron units which are coupled, the number of neuron units which are coupled is several hundred to several thousand, for example. For this reason, an extremely large number of shift registers or memory means having the relatively large bit length is required to store all of the weighting coefficients T.sub.ij. Therefore, the scale of the hardware becomes large and the production cost becomes high. As a result, when the neuron units are formed on a semiconductor integrated circuit device, the number of neuron units which can be formed on the chip area becomes extremely limited. On the other hand, the amount of data to be transferred for the purpose of initially setting each weighting coefficient T.sub.ij into the corresponding shift register or memory means becomes extremely large, thereby requiring a very long time to complete the initial setting operation.
Accordingly, a description will hereinafter be given of embodiments in which the above described problems are eliminated.
FIG. 72 shows an essential part of a sixteenth embodiment of the neuron unit according to the present invention. In FIG. 72, those parts which are the same as those corresponding parts in FIG. 38 are designated by the same reference numerals, and a description thereof will be omitted.
In this embodiment, a random number generator 331, a comparator 332 and a register 333 are provided in place of the shift register 56 shown in FIG. 38. The random number generator 331 generates a random number and supplies the random number t the comparator 332. On the other hand, the register 333 stores a binary data corresponding to the weighting coefficient T.sub.ij. The comparator 332 compares the random number from the random number generator 331 with the data from the register 333, and outputs a pulse indicative of the comparison result to the AND gate 57. As a result, a random pulse sequence having a pulse density corresponding to the binary data stored in the register 333, that is, the weighting coefficient T.sub.ij, is output from the comparator 332.
For example, if the random number generator 331 generates a 7-bit random number from "1" to "127" and the register 333 stores a 7-bit data indicating "30" the comparator 332 can be designed to output a data "1" if the 7-bit random number is less than or equal to "30". In this case, the random number generator 331 generates a number which is less than or equal to "30" 30 times in one random number generation period in which the number from "1" to "127" and excluding "0" is generated once at random. Hence, the comparator 332 outputs a random pulse sequence having a pulse density of "30/127" that is a random pulse sequence in which the data "1" appears 30 times at random within one random number generation period.
Compared to the case where a 128-bit shift register 56 shown in FIG. 38 is used, this embodiment merely requires a 7-bit register 333. Accordingly, it is possible to simplify both the circuit construction and the operation of setting the initial value into the register 333. Although this embodiment uses the random number generator 331 and the comparator 332 in addition to the register 333, the increase in the hardware is small compared to the large hardware reduction that can be achieved by using the 7-bit register 333 in place of the 128-bit shift register 56, particularly when the large number of registers used is taken into consideration.
Of course, the circuit part made up of the random number generator 331, the comparator 332 and the register 333 may be used in place of the shift registers of the other embodiments described above.
Next, a description will be given of this embodiment in more detail, by referring to FIG. 73. In FIG. 73, those parts which are the same as those corresponding parts in FIGS. 38 and 72 are designated by the same reference numerals, and a description thereof will be omitted.
In FIG. 73, the random number generator 331, the comparator 322, the register 333 and a binary counter 334 is provided in place of the N-bit shift register 56 shown in FIG. 38. In this case, the register 333 is a logN-bit binary register.
During the forward process, the weighting coefficient T.sub.ij is stored in the register 333 as a data described by a number of pulses. The comparator 332 compares this data stored in the register 333 with each random number generated from the random number generator 331, and outputs a random pulse sequence having a pulse density dependent on the weighting coefficient T.sub.ij.
FIG. 74 shows a first embodiment of the random number generator 331. In FIG. 74, the random number generator 331 includes an exclusive-OR gate 1305 and 7 flip-flops 1302.sub.1 through 1302.sub.7 which are connected as shown. A clock signal is input to a terminal 1307 and is applied to clock terminals CK of each of the flip-flops 1302.sub.1 through 1302.sub.7. The 7 flip-flops 1302.sub.1 through 1302.sub.7 form a 7-bit linear feedback shift register (LFSR) 1302 together with the exclusive-OR gate 1305. An initial value is set in the LFSR 1302, and the LFSR 1302 thereafter repeats a shift operation. As a result, a number from "1" to "127" and excluding "0" is generated once at random within one random number generation period. This random number which is generated from the LFSR 1302 is defined by bits A.sub.1 through A.sub.7, where A.sub.7 denotes the most significant bit (MSB) and A.sub.1 denotes the least significant bit (LSB), for example. However, the bits A.sub.7 and A.sub.1 may respectively denote the LSB and the MSB.
The output of the flip-flop 1032.sub.1 is input to the exclusive-OR gate 1305 in FIG. 74, but the output of any of the flip-flops 1302.sub.1 through 1302.sub.6 may be input to the exclusive-OR gate 1305.
During the backward process, that is, during the learning process, the register 333 stores the weighting coefficient at the time before the learning process is carried out, and the counter 334 is cleared to zero at the time before the learning process is carried out. The error signal pulse sequences which are collected from the neuron unit of the previous stage and are processed in the gate circuit 78 and the frequency dividing circuit 79 shown in FIG. 39, for example, are input as the signals 75 and 76. Hence, a logic operation is carried out based on the signals 75 and 76, the input signal pulse sequence 55, and the random pulse sequence which is output from the comparator 332 based on the weighting coefficient at the time before the learning process is carried out, and a pulse sequence corresponding to a new weighting coefficient is generated as a result of this logic operation. This pulse sequence corresponding to the new weighting coefficient is input to the counter 334 via the gate circuit 80. The counter 334 counts the number of pulses of this pulse sequence, and the counted value of the counter 334 is transferred to the register 333 when the learning process ends. As a result, the content of the register 333 is updated or corrected.
Accordingly, the weighting coefficient at the time before the learning process is carried out is stored in the register 333 during the learning process. In addition, the register 333 stores the data describing the weighting coefficient by the number of pulses. For this reason, compared to the case where the pulse sequence of the weighting coefficient is stored as it is in the shift register 56 shown in FIG. 38, it is possible to greatly reduce the number of bits of the register 333 and the number of bits of the corresponding counter 334.
For example, if a signal precision is 7 bits, the shift register 56 shown in FIG. 38 would require 128 bits. But according to this embodiment, the 128-bit shift register 56 can be replaced by a 7-bit register 333 and a 7-bit counter 334. Therefore, it is possible to considerably reduce the hardware according to this embodiment. Furthermore, this reduction in the hardware is only for one coupling of the neuron unit, and the effect of reducing the hardware is extremely large in the case of a neural network which includes several hundred to several thousand couplings.
A random pulse generating circuit formed by the random number generator 331 and the comparator 332 requires additional hardware compared to the case where the shift register 56 shown in FIG. 38 is used. However, the random number generator 331 has a simple circuit construction such as that shown in FIG. 74. For this reason, by taking measures such as using the random pulse generating circuit in common among a plurality of couplings, the increase of the hardware due to the random pulse generating circuit does not introduce a serious problem.
Next, a description will be given of a seventeenth embodiment of the neuron unit according to the present invention, by referring to FIG. 75. In FIG. 75, those parts which are the same as those corresponding parts in FIG. 73 are designated by the same reference numerals, and a description thereof will be omitted.
In this embodiment, a circuit part which replaces the shift register 56 shown in FIG. 38 includes the random number generator 331, the comparator 332, the binary register 333, and an up-down counter 335. Inputs of the up-down counter 335 are coupled to the gate circuit 80 as shown in FIG. 75.
The data indicating the weighting coefficient is stored in the binary register 333, and during the forward process, the operation of this embodiment is the same as that of the sixteenth embodiment shown in FIG. 73. On the other hand, during the backward process, that is, during the learning process, the binary register 533 stores the data indicating the weighting coefficient at the time before the learning process is carried out, and the data indicating the same weighting coefficient at the time before the learning process is carried out is stored in the up-down counter 535.
The error signal pulse sequences 75 and 76 which are collected from each neuron unit from the previous stage and processed in the gate circuit 78 shown in FIG. 39, the input signal pulse sequence 55, and random pulse sequence generated from the weighting coefficient at the time before the learning process is carried out are subjected to a logic operation. As a result of this logic operation, a pulse sequence 335S1 for correcting the weighting coefficient towards the positive side and a pulse sequence 335S2 for correcting the weighting coefficient towards the negative side are output from an intermediate part of the gate circuit 80 and input to the up-down counter 335. The data in the up-down counter 335 indicating the weighting coefficient is successively incremented in response to the pulse sequence 335S1 or decremented in response to the pulse sequence 335S2. After the learning process ends, the content of the up-down counter 335 is transferred to the binary register 333.
According to this embodiment, the data stored in the up-down counter 335 and indicating the weighting coefficient is directly increased or decreased using the correcting pulse sequences of the weighting coefficients shown in FIGS. 34 and 35. Therefore, it is possible to further reduce the hardware required to update or correct the weighting coefficient when compared to the sixteenth embodiment.
On the other hand, in order to more efficiently carry out the operations in each neuron unit, it is desirable that the signals transferred within each neuron unit do not have any correlation. Hence, a description will now be given of an embodiment in which the random characteristics of the pulse signal sequences which are transferred during the forward process are improved.
Next, a description will be given of an eighteenth embodiment of the neuron unit according to the present invention. In this embodiment, the neuron unit is realized by use of digital circuits according to the rules [1] through [6] described above with reference to the second embodiment.
A plurality of neuron units 50 respectively having the construction shown in FIG. 20 described above are connected in a plurality of layers to form a hierarchical neural network shown in FIG. 2, for example. The input and output signals of the neuron unit 50 are all described in binary by "1"s and "0"s and are synchronized. The signal intensity of the input signal y.sub.i is expressed by a pulse density, that is, a number of "1"s existing in a pulse train within a predetermined time. FIG. 76 shows a case where four "1"s and two "0"s of the input signal y.sub.i exist within the predetermined time amounting to six synchronizing pulses. Whether the input signal y.sub.i is "0" or "1" is detected at the rising edge or the falling edge of the synchronizing pulse. In this case, the input signal y.sub.i has a signal intensity 4/6. It is desirable that the "1"s and "0"s of the input signal y.sub.i are positioned at random within the predetermined time.
On the other hand, the weighting coefficient T.sub.ij is similarly described by a pulse density, and is stored in a memory as a pulse train of "0"s and "1"s. FIG. 77 shows a case where three "1"s and three "0"s of the weighting coefficient T.sub.ij exist within the predetermined time amounting to six synchronizing pulses. Whether the weighting coefficient T.sub.ij is "0" or "1" is detected at the rising edge or the falling edge of the synchronizing pulse. In this case, the weighting coefficient T.sub.ij has a value 3/6. It is desirable that the "1"s and "0"s of the weighting coefficient T.sub.ij are positioned at random within the predetermined time.
The pulse train of the weighting coefficient T.sub.ij is successively read from the memory responsive to the synchronizing pulses and supplied to each AND gate 51 shown in FIG. 20 which obtains a logical product (y.sub.i .andgate.T.sub.ij) with the pulse train of the input signal y.sub.i. An output of the AND gate 51 is used as an input to the neuron unit 50. Hence, in the case described above, the logical product y.sub.i .andgate.T.sub.ij becomes as shown in FIG. 78 and a pulse train "101000" is obtained. It can be seen from FIG. 78 that the input signal y.sub.i is converted by the weighting coefficient T.sub.ij and the pulse density becomes 2/6.
The pulse density of the output signal of the AND gate 51 is approximately the product of the pulse density of the input signal and the pulse density of the weighting coefficient, and the AND gate 51 acts similarly as in the case of the analog circuit. The pulse density of the output signal of the AND gate 51 more closely approximates the above product as the pulse train becomes longer and as the locations of the "1"s and "0"s become more at random. When the pulse train of the weighting coefficient is short compared to the pulse train of the input signal and no further data can be read out from the memory, the data can be read out from the first data and repeat such an operation until the pulse train of the input signal ends.
One neuron unit 50 receives a plurality of input signals, and a plurality of logical products are obtained between the input signal and the weighting coefficient. Hence, the OR circuit 52 obtains a logical sum of the logical products. Since the input signals are synchronized, the logical sum becomes "111000" when the first logical product is "101000" and the second logical product is "010000", for example. FIG. 79 shows the logical products input to the OR circuit 52 and the logical sum .orgate.(y.sub.i .andgate.T.sub.ij) which is output from the OR circuit 52. This corresponds to the calculation of the sum and the non-linear function (sigmoid function) in the case of the analog calculation.
When the pulse densities are low, the logical sum of such pulse densities is approximately the sum of the pulse densities. As the pulse densities become higher, the output of the OR circuit 52 saturates and no longer approximates the sum of the pulse densities, that is, the non-linear characteristic begins to show. In the case of the logical sum, the pulse density will not become greater than "1" and will not become smaller than "0". In addition, the logical sum displays a monotonous increase and is approximately the same as the sigmoid function.
As described above, there are two types of couplings (or weighting), namely, the excitatory coupling and the inhibitory coupling. When making numerical calculations, the excitatory and inhibitory couplings are described by positive and negative signs on the weighting coefficient. In this embodiment which uses digital circuits, the couplings are divided into an excitatory group and an inhibitory group depending on the positive and negative signs on the weighting coefficient T.sub.ij. Then, the calculation up to the part where the logical sum of the logical products of the pulse trains of the input signals and the weighting coefficients are carried out for each group.
In this embodiment, the neuron unit 50 outputs "1" only when the output of the excitatory group is "1" and the output of the inhibitory group is "0" and otherwise outputs "0". This may be achieved by obtaining an AND of a NOT of the output of the inhibitory group and the output of the excitatory group as shown in FIG. 80. Hence, the output a of the excitatory group, the output b of the inhibitory group, and the output y.sub.i of the neuron unit 50 can respectively be described by the following formulas (1), (2) and (3).
a=.orgate.(y.sub.i .andgate.T.sub.ij(+)) (1)
b=.orgate.(y.sub.i .andgate.T.sub.ij(-)) (2)
y.sub.i =a.andgate.b (3)
The neural network can be formed by connecting a plurality of such neuron units 50 in a plurality of layers to form a hierarchical structure similarly as in the case of the neural network shown in FIG. 2. When the entire neural network is synchronized, it is possible to carry out the above described calculation in each layer.
As will be described later, when coupling a plurality of neuron units 50, the operations within each neuron unit 50 become more effective if the pulse signals exchanged via the couplings have no correlation.
Next, a description will be given of the learning process (back propagation). Basically, the error signal is obtained by either one of the two methods described hereunder, and then, the weighting coefficients are varied depending on the error signal by the method which will also be described hereunder.
First, a description will be given of the method of calculating the error signal of each neuron unit 50 in the final layer based on the output signal and the teaching signal. This teaching signal is a desirable output with respect to a given input of the neuron unit, and is described by a pulse train (sequence) or a number of pulses.
In this embodiment, the error signal is defined as follows. That is, the error may take a positive or negative value when the error is described by a numerical value, but in the case of the pulse density, it is impossible to simultaneously describe the positive and negative values. Hence, two kinds of signals, namely, a signal which indicates a positive component and a signal which indicates a negative component are used to describe the error signal. In other words, an error signal of the jth neuron unit 50 is obtained by counting the number of pulses of the output signal y.sub.j and the teaching signal d.sub.j amounting to one data, and comparing the counted values. If the number of pulses of the teaching signal d.sub.j is greater than that of the output signal y.sub.j, a number of pulses amounting to the difference is output as the positive error signal .delta..sub.j(+). On the other hand, if the number of pulses of the output signal y.sub.j is greater than that of the teaching signal d.sub.j, a number of pulses amounting to the difference is output as the negative error signal .delta..sub.j(-). It is also desirable that the pulses of these error signals .delta..sub.j(+) and .delta..sub.j(-) are positioned at random.
Next, a description will be given of the method of obtaining the error signal of each neuron unit 50 in the intermediate layer. The error signal is back propagated, so that not only the weighting coefficients of the final layer and the immediately preceding layer but also the weighting coefficients of the layer which precedes the above immediately preceding layer are varied. For this reason, there is a need to calculate the error signal for each neuron unit 50 in the intermediate layer. The error signals from each of the neuron units 50 in the next layer are collected and used as the error signal of a certain neuron unit 50 of the intermediate layer, substantially in the reverse manner as supplying the output signal of the certain neuron unit 50 to each of the neuron units in the next layer. This may be achieved similarly as described above for a and b with reference to the formulas (1) and (2) described above and FIGS. 76 through 70. However, although the input signal y is a single signal, there are two error signals, that is, the positive and negative error signals. Hence, it is necessary to consider the two error signals and also consider two cases depending on whether the weighting coefficient T is positive or negative.
First, a description will be given of the excitatory coupling. In this case, with respect to a neuron unit in the intermediate layer, .delta..sub.j(+) .andgate. T.sub.ij which is an AND of the positive error signal .delta..sub.j(+) of the jth neuron unit in the next layer (final layer) and the weighting coefficient T.sub.ij between the jth neuron unit and itself (the neuron unit in the intermediate layer) is obtained for each neuron unit. Furthermore, .orgate.(.delta..sub.j(+) .andgate.T.sub.ij) which is an OR of .delta..sub.j(+) .andgate.T.sub.ij obtained for each neuron unit in the final layer, and this OR is regarded as the positive error signal .delta..sub.j(+) of the neuron unit as shown in FIG. 81.
Similarly, an AND of the negative error signal of the neuron unit in the next layer and the weighting coefficient is obtained, and an OR of the ANDs obtained for the neuron units is regarded as the negative error signal of the neuron unit as shown in FIG. 82.
Next, a description will be given of the inhibitory coupling. In this case, an AND of the negative error signal of the neuron unit in the next layer and the weighting coefficient between this neuron unit and itself is obtained, and an OR the AND obtained for the neuron units is regarded as the positive error signal of the neuron unit as shown in FIG. 83.
Similarly, an AND of the positive error signal of the neuron unit in the next layer and the weighting coefficient is obtained, and an OR of the ANDs obtained for the neuron units is regarded as the negative error signal of the neuron unit as shown in FIG. 84.
Furthermore, an OR of the positive error signal of the excitatory coupling and the positive error signal of the inhibitory coupling, and an OR of the negative error signal of the excitatory coupling and the negative error signal of the inhibitory coupling are obtained for this neuron unit. The error signal .delta..sub.i(+) of this neuron unit is set to "1" only if the result of the former OR is "1" and the result of the latter OR is "0". On the other hand, the error signal .delta..sub.i(-) of this neuron unit is set to "1" only if the result of the former OR is "0" and the result of the latter OR is "1". Therefore, at least one of the results of the ORs is omitted. Otherwise, the results of the ORs may both be omitted. By selecting or omitting the error signal in this manner, it is possible to suppress the correlation of the positive and negative error signals.
Therefore, the following formulas (6) summarize the above with regard to the positive and negative error signals. ##EQU3##
Similarly as in the case of the processing of the error signal in the final layer, the number of pulses of each of the error signals .delta..sub.i(+) and .delta..sub.i(-) is counted and the error signal could be defined as the pulses of the positive error signal .delta..sub.i(+) if the counted value of the positive error signal .delta..sub.i(+) is greater. The error signal could be defined as the pulses of the negative error signal .delta..sub.i(-) if the counted value of the negative error signal .delta..sub.i(-) is greater.
Furthermore, it is possible to further provide a function corresponding to the learning rate (learning constant). When the rate is "1" or less in numerical calculation, the learning capability is improved. This may be realized by thinning out the pulse train in the case of an operation on pulse trains. Two examples will now be described for the learning rate of 0.5 where the example 1) thins out every other pulses of the original pulse signal in which the pulses are equi-distant from each other and the example 2) thins out every other pulses of the original pulse signal in which the pulses are not equi-distant from each other.
FIG. 85 shows the example 1) for .eta.=0.5 where every other pulses of the original pulse signal are thinned out, .eta.=0.33 where every third pulses of the original pulse signal are not thinned out (remained), and .eta.=0.67 where every third pulses of the original pulse signal are thinned out.
FIG. 86 shows the example 2) for .eta.=0.5 where every other pulses of the original pulse signal are thinned out, .eta.=0.33 where every third pulses of the original pulse signal are not thinned out (remained), and .eta.=0.67 where every third pulses of the original pulse signal are thinned out.
By thinning out the error signal in the above described manner, it is possible to provide the function corresponding to the learning rate. Such thinning out can easily be realized by use of a generally available counter and/or flip-flop by carrying out a logic operation on a counter output, for example. In a particular case where the counter is used, it is possible to easily set the value of the learning constant .eta. to an arbitrary value, thereby making it possible to control the characteristic of the neural network.
The error signal is obtained by the method described above, and each weighting coefficient is varied. The method of varying each weighting coefficient will now be described. First, an AND (.delta..sub.j .andgate.y.sub.i) is obtained between the error signal .delta..sub.j(+) or .delta..sub.j(-) of the neuron unit and the output y.sub.i of immediately preceding neuron unit corresponding to a line to which the weighting coefficient which is to be varied belongs, as shown in FIGS. 87 and 88. The two signals which are obtained by this AND operation will respectively be denoted by .DELTA.T.sub.ij(+) and .DELTA.T.sub.ij(-).
Then, a new T.sub.ij is obtained from .DELTA.T.sub.ij. However, since T.sub.ij is an absolute value component, two cases are considered depending on whether the original T.sub.ij is excitatory or inhibitory. If the original T.sub.ij is excitatory, the .DELTA.T.sub.ij(+) component is increased and the .DELTA.T.sub.ij(-) component is decreased with respect to the original T.sub.ij, that is, as shown in FIG. 89. On the other hand, if the original T.sub.ij is inhibitory, the .DELTA.T.sub.ij(+) component is decreased and the .DELTA.T.sub.ij(-) component is increased with respect to the original T.sub.ij, that is, as shown in FIG. 90.
The following formulas (7) summarize the above with respect to the weighting coefficients. ##EQU4##
The calculations are carried out in the neural network based on the above described learning rule.
Next, the circuit construction of this eighteenth embodiment based on the above described algorithm will now be described with reference to FIGS. 91 through 95. In this embodiment, the construction of the neural network as a whole is the same as that shown in FIG. 2, FIG. 91 shows a circuit part corresponding to the line coupling the neuron unit 1, and FIG. 92 shows a circuit part corresponding to the neuron unit 1. A digital neural network having a self-learning function is realized by coupling the circuit parts shown in FIGS. 91 and 92 into a network. In FIGS. 91 and 92, those parts which are the same as those corresponding parts in FIGS. 38 and 39 are designated by the same reference numerals, and a description thereof will be omitted.
In FIG. 91, the input signal 55 to the neuron unit corresponds to the input signal y.sub.i shown in FIG. 76. The value of the weighting coefficient such as the weighting coefficient T.sub.ij shown in FIG. 77 is stored in the shift register 56 which is used as a memory. The logic circuit 58 which includes the AND circuit 57 and corresponds to the operation shown in FIG. 78 obtains an AND of the input signal 55 and the weighting coefficient within the shift register 56. The output signal of this logic circuit 58 must be grouped depending on whether the coupling is excitatory or inhibitory, but it is preferable from the point of general application to prepare the output 59 for the excitatory group and the output 60 for the inhibitory group and to carry out a switching so as to determine which one of the outputs 59 and 60 to output. For this reason, this embodiment has the memory 61 for storing a bit which indicates whether the coupling is excitatory or inhibitory, and the switching gate circuit 62 is switched depending on the bit which is stored in the memory 61.
In addition, as shown in FIG. 92, the gate circuits 63a and 63b which include a plurality of OR gates and corresponds to the operation shown in FIG. 79 are provided to process each input. The gate circuit 64 includes the AND gate 64a and the inverter 64b and outputs an output signal "1" only when the output of the excitatory group is "1" and the output of the inhibitory group is "0" as shown in FIG. 80.
When outputting the operation result of one neuron unit to a next neuron unit, the output signal of the gate circuit 64 must be converted into another pulse train with the same value in order to suppress the correlation of the pulse trains. In this embodiment, pulses of the output signal of the gate circuit 64 amounting to one data is counted in a counter 435, and a counted value is supplied to a random pulse train generator 436. This random pulse train generator 436 outputs a new pulse train dependent on the counted value of the counter 435, and this new pulse train is output as an output signal 437. This random pulse train generator 436 may have a general construction made up of a random number generator and a comparator, such as that shown in FIG. 72.
Next, a description will be given of the learning process. First, error signals 441 and 442 must be generated from a teaching signal 440 and an output signal 439 from an output layer. FIG. 93 shows an embodiment of an error signal output circuit 448 for generating these error signals 441 and 442. The number of pulses of the output signal 439 and the teaching signal 440 are respectively counted in counter 443 and 444, and a subtractor 445 calculates a difference between the counted values. A random pulse train generator 446 generates a random pulse train depending on the difference and outputs the error signal 441 having a pulse density described thereby or, a random pulse train generator 447 generates a random pulse train depending on the difference and outputs the error signal 442 having a pulse density described thereby. In other words, if the teaching signal 440 is greater as a result of the subtraction in the subtractor 445, the error signal 441 becomes a pulse train signal and the error signal 442 becomes "0". On the other hand, if the output signal 439 is greater as a result of the subtraction in the subtractor 445, the error signal 442 becomes a pulse train signal and the error signal 441 becomes "0".
It is also possible to use in place of the error signal output circuit 448 an error signal output circuit 449 which does not use the subtractor 445, as shown in FIG. 94. In FIG. 94, the number of pulses of the output signal 439 and the teaching signal 440 are respectively counted in the counters 443 and 444, and the counted values are converted into pulse trains by the corresponding random pulse train generators 446 and 447. At the same time, a comparator 450 compares the output signal 439 and the teaching signal 440. An output of this comparator 450 and the outputs of the random pulse train generators 446 and 447 are processed in a gate circuit 451 which is made up of AND gates and cinverters. When the comparison is made in the comparator 450, the error signal pulse becomes "1" only when the pulse having the greater value is "1" and the pulse having the smaller value is "0". More particularly, if the teaching signal 440 is greater, the error signal 441 becomes a pulse train signal and the error signal 442 becomes "0". ON the other hand, if the output signal 439 is greater, the error signal 442 becomes a pulse train signal and the error signal 441 becomes "0".
The error signals 441 and 442 are input to the circuit shown in FIG. 92. These error signals 441 and 442 are collected in the respective gate circuits 78 by an operation corresponding to that shown in FIG. 80, and are output as the error signals 75 and 76. The error signals 75 and 76 are supplied to the circuit shown in FIG. 91 wherein the error signals of the intermediate layer are calculated as shown in FIGS. 82 through 84 by a gate circuit 455 which is made up of AND gates and inverters and the gate circuit 72 which includes AND gates. The error signals 73 and 74 which are to be output to the neuron unit of the immediately preceding layer are thus obtained depending on the signs of the error signals.
Accordingly, it is necessary to distinguish the excitatory coupling from the inhibitory coupling. The gate circuit 77 which is made up of AND gates and OR gates determines whether the coupling is excitatory or inhibitory depending on the information which indicates excitatory or inhibitory coupling and is stored in the memory 61.
In addition, the process related to the learning rate shown in FIGS. 85 and 86 is carried out by the frequency dividing circuit 79 shown in FIG. 92. In addition, the gate circuit (weighting coefficient carrying circuit) 80 which is made up of an inverter, AND gates and an OR gate is provided as shown in FIG. 91 to carry out the operations shown in FIGS. 87 through 90, that is, to calculate the new weighting coefficient based on the error signals 75 and 76. The gate circuit 80 is coupled to the input 56b of the shift register 56. In other words, by reading out again the content of the shift register 56 used for the forward process and at the same time transmitting the output of the previous neuron unit as a pulse signal using its counter value, it is possible to update or rewrite the weighting coefficient which is stored in the shift register 56. It is also necessary to distinguish the excitatory coupling from the inhibitory coupling in the gate circuit 80, but this may be carried out by the gate circuit 77.
Similarly as in the case of the forward process, the error signals also need to be converted into random pulse trains. Hence, as shown in FIG. 81, the error signals amounting to one data from the gate circuit 72 are counted in respective counters 462 and 463. Random pulse generators 464 and 465 convert the error signals into random pulse trains based on the corresponding counted values of the counters 462 and 463, and outputs the random pulse trains as the error signals 73 and 74. In this case, the random pulse generators 464 and 465 may also have the general construction made up of a random number generator and a comparator, such as that shown in FIG. 72.
The error signal output circuit 448 shown in FIG. 93 or the error signal output circuit 449 shown in FIG. 94 may be used in place of the error signal output circuit 466 shown in FIG. 91. In either case, the positive and negative error signals 75 and 76 (actually, the error signals obtained via the gate circuit 72) may be input in place of the output signal 439 and the teaching signal 440 shown in FIGS. 93 and 94.
In addition, the circuit shown in FIG. 91 may be replaced by a circuit shown in FIG. 95. In FIG. 95, those parts which are the same as those corresponding parts in FIG. 91 are designated by the same reference numerals, and a description thereof will be omitted. The circuit shown in FIG. 95 uses the error signal output circuit 448 shown in FIG. 93 or the error signal output circuit 449 shown in FIG. 94 in place of the gate circuit 455. In this case, only one of the positive error signal and the negative error signal constantly has a value other than "0".
In this embodiment, the weighting coefficient may be stored in the shift register 56 in the form of a binary number and converted into a pulse train when carrying out the operation. After carrying out the operation, the pulses of the pulse train are counted and the counted binary value is stored in the shift register 56.
It is desirable that the pulse trains used for the forward process and the pulse trains used for the learning process have no correlation, so as to optimize the learning capability of the neuron unit. Next, a description will be given of a nineteenth embodiment of the neuron unit according to the present invention, in which this demand is satisfied.
If the weighting coefficient is only excitatory or inhibitory with respect to one input, a result F.sub.ij of the excitatory group of couplings and a result I.sub.ij of the inhibitory group of couplings may be obtained from the following formulas (8) and (9). ##EQU5##
If the weighting coefficients can be both excitatory and inhibitory, F.sub.j and I.sub.j may be obtained from the following formulas (10) and (11) or the formulas (12) and (13). ##EQU6##
In the formulas (12) and (13), if the weighting coefficient is only one of excitatory and inhibitory with respect to one input, Y.sub.Fij and Y.sub.Iij is described by the following formulas (14) and (15). On the other hand, if the weighting coefficients may be both excitatory and inhibitory with respect to one input, Y.sub.Fij and Y.sub.Iij is described by the following formulas (16) and (17). ##EQU7##
If the result F.sub.j of the excitatory group and the result I.sub.j of the inhibitory group which are obtained do not match, the result F.sub.j of the excitatory group is output of the neuron unit. In other words, if the result F.sub.j of the excitatory group is "0" and the result I.sub.j of the inhibitory group is "1" the output is "0".
In order to realize the above function, an AND of the output of the excitatory group and a NOT of the output of the inhibitory group is calculated. FIG. 96 shows this case, and may be described by the following formula (18).
y.sub.j =F.sub.j .andgate.I.sub.j (18)
On the other hand, the output is "1" if the result F.sub.j of the excitatory group is "1" and the result I.sub.j of the inhibitory group is "0".
An OR of the output of the excitatory group and a NOR of the output of the inhibitory group is calculated. FIG. 97 shows this case, and may be described by the following formula (19).
y.sub.j =F.sub.j .orgate.I.sub.j (19)
If the result F.sub.j of the excitatory group and the result I.sub.j of the inhibitory group match, the output may be "0" or "1" and it is also possible to output a separately prepared second input signal E.sub.j. If it is also possible to provide a memory which stores the pulse density or the number of pulses in the form of a binary number with respect to the second input signal E.sub.j, the binary number is to be read convert it into the pulse density, and output should be a result of an operation which is obtained from a logical product of the pulse density and the second input signal E.sub.j. Similarly as in the case of the weighting coefficient with respect to the input signal, the contents of this memory may be read again from the beginning if all of the content is read out.
FIG. 98 shows the case where the second input signal is output, and may be described by the following formula (20).
y.sub.j =[(F.sub.j XOR I.sub.j) AND F.sub.j ] OR
[(F.sub.j XOR I.sub.j) AND E.sub.j ] (20)
Furthermore, FIG. 99 shows the last case where T'.sub.j denotes the content (coefficient) of the memory provided with respect to the second input signal E.sub.j, and may be described by the following formula (21).
y.sub.j =[(F.sub.j XOR I.sub.j) AND F.sub.j ] OR
[(F.sub.j XOR I.sub.j) AND (E.sub.j AND T'.sub.j)] (21)
The network of the neural network 50 has the hierarchical structure such as that shown in FIG. 2, similarly to the back propagation. Hence, by synchronizing the entire network, the calculation in each layer can be realized by the above described functions.
Next, a description will be given of the learning (back propagation)process. Basically, the error signal is obtained by one of the following methods, and the weighting coefficient is then varied by the method described in the following. However, in the following description, the pulse train describing the weighting coefficient and the pulse trains described by the formulas up to the formula (17) may have the same pulse density or the same number of pulses, but it is assumed that the pulse positions differ.
The error signal of each neuron unit in the final layer is calculated, and the weighting coefficient of each neuron unit is varied depending on the error signal. A description will now be given of the method of calculating the error signal. In this embodiment, the error signal is defined as follows. That is, the error may take a positive or negative value when the error is described by a numerical value, but in the case of the pulse density, it is impossible to simultaneously describe the positive and negative values by one pulse train. Hence, two kinds of signals, namely, a signal which indicates a positive component and a signal which indicates a negative component are used to describe the error signal. In other words, the error signals of the jth neuron unit can be described as shown in FIG. 100. The positive component (+) of the error signal corresponds to the pulses existing on the teaching signal where the teaching signal pulse and the output signal pulse differ. Similarly, the negative component of the error signal corresponds to the pulses existing on the output signal side out of the parts (1, 0) and (0, 1) where the teaching signal pulse and the output signal pulse differ. In other words, when the positive component of the error signal is added to the output signal and the negative component of the error signal is subtracted, the teaching signal is obtained. As will be described later, the weighting coefficient is varied based on such error signal pulses.
Therefore, a positive error signal .delta..sub.j(+) and a negative error signal .delta..sub.j(-) of the jth neuron unit can respectively be described by the following formulas (22) and (23), where y.sub.j and d.sub.j respectively denote the output signal and the teaching signal of the jth neuron unit.
.delta..sub.j(+) =y.sub.j .andgate.d.sub.j (22)
.delta..sub.j(-) =y.sub.j .andgate.d.sub.j (23)
The error signal is back propagated, so that not only the weighting coefficients of the final layer and the immediately preceding layer but also the weighting coefficients of the layer which precedes the above immediately preceding layer are varied. For this reason, there is a need to calculate the error signal for each neuron unit in the intermediate layer. The error signals from each of the neuron units in the next layer are collected and used as the error signal of a certain neuron unit of the intermediate layer, substantially in the reverse manner as supplying the output signal of the certain neuron unit to each of the neuron units in the next layer. This may be achieved similarly as described above with reference to the formulas (8) through (10) and FIGS. 76 through 79, 96 and 97. That is, the couplings are divided into two groups depending on whether the coupling is an excitatory coupling or an inhibitory coupling, and the multiplication part is described by AND and the .SIGMA.0 part is described by OR. The only difference in this case is that although y.sub.j is always a positive signal .delta..sub.j may be positive or negative and thus two error signals must be considered. Therefore, four cases must be considered depending on whether the weighting coefficient T.sub.ij is positive or negative and whether the error signal .delta..sub.j is positive or negative.
First, a description will be given of the excitatory coupling. In this case, .delta..sub.k(+) .andgate.T.sub.jk which is an AND of the positive error signal .delta..sub.k(+) of the kth neuron unit in the layer next to a specific layer and the weighting coefficient T.sub.jk between the jth neuron unit in the specific layer and the kth neuron unit in the next layer is obtained for each neuron unit in the specific layer. Furthermore, .orgate.(.delta..sub.k(+) .andgate.T.sub.jk) which is an OR of .delta..sub.k(+) .andgate.T.sub.jk is obtained for each neuron unit in the specific layer, and this OR is regarded as the positive error signal .delta..sub.j(+) for the specific layer as shown in FIG. 101 if there are n neuron units in the next layer. The following formulas (24) through (26) can thus be made. ##EQU8##
Similarly, .delta..sub.k(-) .andgate.T.sub.jk which is an AND of the negative error signal .delta..sub.k(-) of the kth neuron unit in the layer next to a specific layer and the weighting coefficient T.sub.jk between the jth neuron unit in the specific layer and the kth neuron unit in the next layer is obtained for each neuron unit in the specific layer. Furthermore, .orgate.(.delta..sub.k(-) .andgate.T.sub.jk) which is an OR of .delta..sub.k(-) .andgate.T.sub.jk is obtained for each neuron unit in the specific layer, and this OR is regarded as the negative error signal .delta..sub.j(-) for the specific layer as shown in FIG. 102 if there are n neuron units in the next layer. The following formulas (27) through (29) can thus be made. ##EQU9##
Next, a description will be given of the inhibitory coupling. In this case, .delta..sub.k(-) .andgate.T.sub.jk which is an AND of the negative error signal .delta..sub.k(-) of the kth neuron unit in the layer next to a specific layer and the weighting coefficient T.sub.jk between the jth neuron unit in the specific layer and the kth neuron unit in the next layer is obtained for each neuron unit in the specific layer. Furthermore, .orgate.(.delta..sub.k(-) .andgate.T.sub.jk) which is an OR of .delta..sub.k(-) .andgate.T.sub.jk is obtained for each neuron unit in the specific layer, and this OR is regarded as the positive error signal .delta..sub.j(+) for the specific layer as shown in FIG. 103 if there are n neuron units in the next layer. The following formulas (30) through (32) can thus be made. ##EQU10##
Similarly, .delta..sub.k(+) .andgate.T.sub.jk which is an AND of the positive error signal .delta..sub.k(+) of the kth neuron unit in the layer next to a specific layer and the weighting coefficient T.sub.jk between the jth neuron unit in the specific layer and the kth neuron unit in the next layer is obtained for each neuron unit in the specific layer. Furthermore, .orgate.(.delta..sub.k(+) .andgate.T.sub.jk) which is an OR of .delta..sub.k(+) .andgate.T.sub.jk is obtained for each neuron unit in the specific layer, and this OR is regarded as the negative error signal .delta..sub.j(-) for the specific layer as shown in FIG. 104 if there are n neuron units in the next layer. The following formulas (33) through (35) can thus be made. ##EQU11##
Since one neuron unit may be coupled to another neuron unit by an excitatory or inhibitory coupling, an OR of the error signal .delta..sub.j(+) obtained as shown in FIG. 101 and the error signal .delta..sub.j(+) obtained as shown in FIG. 103 is regarded as the error signal .delta..sub.j(+) of the jth neuron unit. Similarly, an OR of the error signal .delta..sub.j(-) obtained as shown in FIG. 102 and the error signal .delta..sub.j(-) obtained as shown in FIG. 104 is regarded as the error signal .delta..sub.j(-) of the jth neuron unit.
Therefore, the error signals .delta..sub.j(+) and .delta..sub.j(-) of the jth neuron unit in the specific layer can be described by the formulas (36) or (37). ##EQU12##
In addition, when both excitatory and inhibitory weighting coefficients are provided with respect to one input, the error signals may be described by the following formulas (38) or (39). ##EQU13##
It is possible to further provide a function corresponding to the learning rate (learning constant). When the rate is "1" or less in numerical calculation, the learning capability is improved. This may be realized by thinning out the pulse train in the case of an operation on pulse trains. Two examples will now be described where the example 1) thins out every other pulses of the original pulse signal in which the pulses are equi-distant from each other and the example 2) thins out every other pulses of the original pulse signal in which the pulses are not equi-distant from each other.
FIG. 105 shows the example 1) for .eta.=0.5 where every other pulses of the original pulse signal are thinned out, .eta.=0.33 where every third pulses of the original pulse signal are thinned out, and .eta. =0.67 where every third pulses of the original pulse signal are thinned out.
FIG. 106 shows the example 2) for .eta.=0.5 where every other pulses of the original pulse signal are thinned out, .eta.=0.33 where every third pulses of the original pulse signal are thinned out, and .eta.=0.67 where every third pulses of the original pulse signal are thinned out.
The error signal is obtained by the method described above, and each weighting coefficient is varied. An AND of the signal flowing through the line to which the weighting coefficient to be varied belongs and the error signal, that is .delta..sub.j .andgate.y.sub.i, is obtained. Since there are the positive and negative error signals, the two error signals are respectively obtained as shown in FIGS. 107 and 108. As a result, .delta..sub.j(+) .andgate.y.sub.i and .delta..sub.j(-) .andgate.y.sub.i are respectively obtained as the positive and negative weighting coefficients.
The two weighting coefficients obtained above will be denoted by .DELTA.T.sub.ij(+) and .DELTA.T.sub.ij(-). Next, a new T.sub.ij is obtained based on .DELTA.T.sub.ij, but T.sub.ij is an absolute value component. For this reason, two cases must be considered depending on whether the original T.sub.ij is excitatory or inhibitory. In the excitatory case, the component of .DELTA.T.sub.ij(+) is increased with respect to the original T.sub.ij, and the component of .DELTA.T.sub.ij(-) is decreased with respect to the original T.sub.ij, as shown in FIG. 109. On the other hand, in the inhibitory case, the component of .DELTA.T.sub.ij(+) is increased and the component of .DELTA.T.sub.ij(-) is decreased with respect to the original T.sub.ij, as shown in FIG. 110. FIGS. 109 and 110 can respectively be described by the following formulas (40) and (41). ##EQU14##
The calculations are carried out in the network based on the learning rule described above.
Next, the circuit construction of this nineteenth embodiment based on the above described algorithm will now be described with reference to FIGS. 111 through 117. In this embodiment, the construction of the neural network as a whole is the same as that shown in FIG. 2, FIGS. 111 through 114 respectively show a circuit part corresponding to the line coupling the neuron units 1, and FIG. 115 shows a circuit part corresponding to the neuron unit 1. FIG. 116 shows a circuit part which obtains the error signal in the final layer based on the teaching signal and the output of the final layer. A digital neural network having a self-learning function is realized by coupling the circuit parts shown in FIGS. 111 through 115 into a network. In FIGS. 111 through 116, those parts which are the same as those corresponding parts in FIGS. 38 through 40 are designated by the same reference numerals, and a description thereof will be omitted.
In FIG. 111, the input signal 55 to the neuron unit corresponds to the input signal y.sub.i shown in FIG. 76. The value of the weighting coefficient such as the weighting coefficient T.sub.ij shown in FIG. 77 is stored in the memory 56. Pulse train converting circuits 527a and 527b are connected to the output side of the shift register 56 to convert the stored numerical value into a corresponding pulse train signal. The pulse train converting circuit 527a is provided for the forward process, and the pulse train converting circuit 527b is provided for the learning process and the error signal propagation. In addition, a converting circuit 527c is provided on the input side of the memory 56 to convert the pulse train signal into a corresponding numerical value. The memory 56 and the converting circuits 527a, 527b and 527c are coupled to n lines which are required to describe a numerical value. Although not shown in FIG. 111, signals for controlling the read and write operations of the memory 56 such as an output enable signal and a write enable signal are input to the memory 56.
The input signal 55 and the weighting coefficient which is stored in the memory 56 and converted by the pulse train converting circuit 527a are supplied to the AND gate 57 of the logic circuit 58. The logic circuit 58 carries out the operation corresponding to to the operation shown in FIG. 78. The output signal of this logic circuit 58 must be grouped depending on whether the coupling is excitatory or inhibitory, but it is preferable from the point of general application to prepare the output 59 for the excitatory group and the output 60 for the inhibitory group and to carry out a switching so as to determine which one of the outputs 59 and 60 to output. For this reason, this embodiment has the memory 61 for storing a bit which indicates whether the coupling is excitatory or inhibitory, and the switching gate circuit 62 is switched depending on the bit which is stored in the memory 61.
The output signal of the logic circuit 58 may be fixed to one of the excitatory or inhibitory groups if there is no need to make the switching. For example, FIG. 112 shows the excitatory case, and FIG. 113 shows the inhibitory case. The circuits shown in FIGS. 112 and 113 are respectively equivalent to the case where the bit stored in the memory 61 is fixed to "0" and "1". In addition, it is possible to provide the memory for each of the excitatory and inhibitory weighting coefficients, as shown in FIG. 114. In FIG. 114, a logic circuit 58A includes the memory 56 for storing the excitatory weighting coefficient, and a logic circuit 58B includes the memory 56 for storing the inhibitory weighting coefficient.
In addition, as shown in FIG. 115, the gate circuits 63a and 63b which include a plurality of OR gates and corresponds to the operation shown in FIG. 79 are provided to process each input. The gate circuit 64 includes the AND gate 64a and the inverter 64b and outputs an output signal "1" only when the output of the excitatory group is "1" and the output of the inhibitory group is "0", as shown in FIG. 96. The processes shown in FIGS. 79 and 96 through 99 may easily be realized by logic circuits in a similar manner.
The gate circuit 64 may have a circuit construction shown in FIG. 117A when realizing the process shown in FIG. 79, a circuit construction shown in FIG. 117B when realizing the process shown in FIG. 98, and a circuit construction shown in FIG. 117C when realizing the process shown in FIG. 99. In FIG. 117A, the gate circuit 64 is made up of an inverter 64b and an OR gate 64c. In FIG. 117B, the gate circuit 64 is made up of an exclusive-OR gate 64d, an inverter 64e, two AND gates 64f and 64g, and an OR gate 64h, and the second input signal E.sub.j is input to the AND gate 64g. In FIG. 117C, the gate circuit 64 includes a memory 64i, an AND gate 64j, and a pulse train converting circuit 527d in addition to the elements of the gate circuit 64 shown in FIG. 117B. In FIG. 117C, the memory 64i stores the coefficient T' with respect to the second input signal E.sub.j in a numerical form, and the pulse train converting circuit 527d is the same as the pulse train converting circuit 527a.
Next, a description will be given of the error signal. The logic circuit 65 shown in FIG. 116 which is made up of two AND gates and an exclusive-OR gate generates the error signals in the final layer. The operation of this logic circuit 65 corresponds to the formulas (6) and (7), and the error signals 68 and 69 are generated based on the teaching signal 67 and the output signal 66 from the final layer. In the formula (37) which calculates the error signals in the intermediate layer, the process of obtaining E.sub.j(+) and E.sub.j(-) is carried out by the gate circuit 72 which is made up of AND gates as shown in FIG. 111, and thus, the outputs 73 and 74 corresponding to the positive and negative components are obtained. In this case, the weighting coefficient which is obtained after the learning process is used. However, it is also possible to use the weighting coefficient at the time before the learning process is carried out, and the circuit for this case can easily be derived.
In addition, the excitatory coupling and the inhibitory coupling must be treated separately, and the kind of coupling can be determined by the gate circuit 77 based on the excitatory or inhibitory information stored in the memory 61 and the positive and negative error signals 75 and 76. As shown in FIG. 111, the gate circuit 77 is made up of AND gates and OR gates.
In the circuits shown in FIGS. 112 and 113 where the kind of coupling is fixed to the excitatory and inhibitory couplings, respectively, the circuits are respectively equivalent to the cases where the content of the memory 61 is fixed to "0" and "1" in FIG. 111.
In the circuit shown in FIG. 114 where the memory 56 of the logic circuit 58A for indicating the excitatory coupling and the memory 56 of the logic circuit 58B for indicating the inhibitory coupling are provided independently with respect to one input, the circuit corresponding to the formulas (39) is shown as the gate circuit 72.
The operation of the formula (8) for collecting the error signals, that is, the remaining part of the formulas (37), is carried out by the gate circuit 78 shown in FIG. 115 which is made up of OR gates. The processes shown in FIGS. 105 and 108 corresponding to the learning rate are carried out by the frequency dividing circuit 79 shown in FIG. 113. The frequency dividing circuit 79 can easily be realized by the use of flip-flops. It is not essential to provide the frequency dividing circuit 79, and the locations of the frequency dividing circuit 79 is not limited to those shown in FIG. 112. For example, the frequency dividing circuit 79 may be provided at appropriate position such as those shown in FIGS. 111 through 114.
Finally, the part for calculating the new weighting coefficient from the error signal, that is, the part corresponding to the processes shown in FIGS. 109 and 110, is the gate circuit 80 shown in FIG. 111 which is made up of AND gates, an inverter and an OR gate. The gate circuit 80 rewrites the content of the memory 56, that is, the value of the weighting coefficient T.sub.ij. This gate circuit 80 also needs to be operate depending on whether the coupling is the excitatory or inhibitory coupling. The gate circuit 77 determines the kind of coupling. In the cases of FIGS. 112 and 113, the coupling is fixed to the excitatory coupling and the inhibitory coupling, respectively, and no circuit corresponding to the gate circuit 77 is necessary. On the other hand, in the case of FIG. 114, the gate circuits 80A and 80B respectively correspond to the excitatory coupling and the inhibitory coupling, because in this case there are two kinds of couplings.
In this embodiment, the memory 56 stores the weighting coefficient in the form of a binary number. When using the weighting coefficient, this numerical value is converted into the pulse train signal or, the conversion is made from the pulse train signal into the numerical value. The pulse train converting circuits 527a, 527b and 527c are provided for the purpose of making such conversions.
Basically, the pulse train converting circuits 527a and 527b have a construction shown in FIG. 118. In FIG. 118, a comparator 552 compares the numerical value read from the memory 56 and a random number generated from a random number generator 551, and outputs a pulse train signal having a value "0" or "1" depending on the result of the comparison. A number of reference clocks must be determined with respect to the random number generator 551, and this number may be preset or arbitrarily set from the outside. The random number generator 551 generates the random numbers from "0" to "preset number of reference clocks minus one" in synchronism with the reference clocks.
For example, the random number generator 551 may be made of a LFSR. The comparator 552 compares the numerical data read from the memory 56 and the random number data output from the random number generator, and outputs "1" if the numerical data is greater than the random number data and outputs "0" if the numerical data is smaller than or equal to the random number data. As a result, a pulse train signal corresponding to the numerical value stored in the memory 56 is output from the comparator 552.
On the other hand, the numerical value converting circuit 527c has a construction shown in FIG. 119. In FIG. 119, the converting circuit 527c is made up of a counter 553. Although not shown in FIG. 119, a clear signal, a signal for making a signal line on which the counted value is output to a high impedance state, and reference clocks are input to the counter 533. This counter 533 counts the number of "1"s of the input signal at the rising edge or the falling edge of the reference clock, and the counted value in the counter 553 is output with respect to the memory 56 as the numerical value data.
In other words, the output signal line for the counted value of the counter 553 is initially set to the high impedance state, and the counter 553 is cleared. Then, the output enable signal of the memory 56 is controlled to read out the data from the shift register 56 amounting to the number of reference clocks. As a result, a pulse train signal describing (data of the memory 56)/(number of reference clocks) is output from the comparator 552. The read out from the memory 56 is stopped when the number of reference clocks reaches the preset number, and the counted value of the counter 553 is output via the output signal line. In addition, the write enable signal to the memory 56 is controlled so as to write this counted value in the memory 56. The control signals with respect to the memory 56 can easily be generated from a counter, a sequencer or the like.
The circuit part for carrying out the conversion from the numerical value signal into the pulse train signal and the circuit part for carrying out the conversion from the pulse train signal into the numerical value signal may be integrally formed as shown in FIG. 120. In this case, an up-down counter 554 having a similar function as the memory 56 is provided. A control signal generating circuit 555 for generating the counter control signals for controlling the operation of the up-down counter 554 is connected to the up-down counter 554. The control signal generating circuit 555 generates the counter control signals based on the pulse train signal which describes the weighting coefficient at the time after the learning process is carried out and is obtained from the circuit 80, and the pulse train signal which describes the weighting coefficient from the comparator 552.
More particularly, the control signal generating circuit 555 generates a control signal for not counting by the up-down counter 554 if the pulses of the weighting coefficient are "0" and the pulses of the weighting coefficient at the time after the learning process is carried out are "0". The control signal generating circuit 555 generates a count-up signal if the pulses of the weighting coefficient are a combination of "0" and "1". On the other hand, the control signal generating circuit 555 generates a count-down signal if the pulses of the weighting coefficient are a combination of "1" and "0". Further, the control signal generating circuit 555 also generates a control signal for not counting in the up-down counter 554 if the pulses of the weighting coefficient are "1".
Therefore, the new weighting coefficient (numerical representation) is obtained from the present weighting coefficient (pulse train representation) and the weighting coefficient (pulse train representation) at the time after the learning process is carried out. Hence, it is possible to realize functions similar to "memory", "numerical value to pulse train signal conversion" and "pulse train signal to numerical value signal conversion".
When providing a plurality of pulse train converting circuits 527a and 527b with respect to one memory, two random number generators 551a and 551b and two comparators 552a and 552b are provided as shown in FIG. 121. In FIG. 121, those parts which are the same as those corresponding parts in FIGS. 118 and 120 are designated by the same reference numerals, and a description thereof will be omitted. In this case, it is preferable that the random number generators 551a and 551b generate mutually different random number sequences.
The technique for representing the signals by the pulse densities is not limited to the use of circuits, that is, hardware. The same technique can be simulated on a computer by software. In the computer, the operations are carried out serially, but the computing speed is greatly improved compared to the case where calculations are made using analog values, because the technique described above only requires logic operations of binary values of "0" and "1". In general, the operations in accordance with the four rules of arithmetic require a large number of machine cycles for one calculation, but the machine cycles required for the logic operation is small. In addition, when only logic operations are required, it is easier to use low-level languages suited for high-speed processing.
Not all of the parts described above need to be realized by hardware, and a part may be realized by software. In addition, the circuitry is not limited to those described above, and other equivalent circuits may be employed. Furthermore, it is also possible to employ negative logic.
Next, a description will be given of a particular example applied with the above described nineteenth embodiment. For example, it is assumed for the sake of convenience that the neural network has three layers as shown in FIG. 2, and that the first, second and third layers respectively are made up of 256, 20 and 5 neuron units. In addition, it is assumed that all of the neuron units of the first and second layers are coupled to each other, and that all of the neuron units of the second and third layers are coupled to each other. This neural network was applied to a self-learning type character recognition system.
First, a hand-written character shown in FIG. 19 was read by a scanner, and the read image was divided into 16.times.16 meshes. The mesh where the character part exists was regarded as "1" and the mesh where no character part exists was regarded as "0". The 256 data of the meshes were input to the first layer of the neural network. The 5 neuron units in the output (third) layer were made to correspond to the number from "1" to "5" and the learning process was carried out so that the output of the corresponding neuron unit becomes "1" when the number is input and the output of the other neuron units become "0".
In this case, a 7-bit LFSR was used as the random number generator 551 and the number of reference clocks was set to 127. Four kinds of such 7-bit LFSRs exist, and the four kinds were arranged at random for each of the random number generators 551 (551a, 551b). In addition, the LFSR is accessible from the outside, and a random numerical value is set as an initial value. If the weighting coefficients are set at random, the output result does not necessarily become a desired value. For this reason, each weighting coefficient is newly obtained using the self-learning function of this embodiment, and such an operation was repeated until the desired output was obtained. The input is "1" or "0" and the input pulse train is a monotonous signal made up of the low level or the high level.
The final outputs were connected to LEDs via transistors, so that each LED is turned OFF in response to a low level signal and is turned ON in response to a high level signal. The synchronizing clock was set to 1000 kHz, so that the brightness of the LEDs appears different depending on the pulse density of the signal applied thereto. The brightest LED corresponds to the answer, and it was confirmed that the recognition rate was 100% with respect to a character which was sufficiently learned by the learning process.
Next, a description will be given of the reason why the random pulse sequence generated from the random number generators 331 shown in FIGS. 74 and 122A through 122C actually becomes regular to a certain extent. The random number generators 331 shown in FIGS. 122A through 122C have constructions which are basically the same as that of the random number generator 331 shown in FIG. 74, except for one of the signals input to the exclusive-OR gate 1305.
If the bit stored in the flip-flop 1302.sub.1 at a time t is denoted by A.sub.t, a bit A.sub.(t-i+j) which is stored in the flip-flop 1302.sub.1 a time i-j before is stored in the flip-flop 1302.sub.1+i at a time (t+j). Hence, a bit A.sub.(t-i-1) is stored in the flip-flop 1302.sub.1+1 at a time (t-1).
For example, in the case of the random number generator 331 shown in FIG. 122B, the bit A.sub.t stored in the flip-flop 1302.sub.1 of the first stage is the value which is output from the exclusive-OR gate 1305 based on the bits stored in the flip-flops 1302.sub.4 and 1302.sub.7 of the fourth and seventh stages at the time (t-1). Because the exclusive-OR gate 1305 carries out a modulo-2 addition of the bits, the bit A.sub.t stored in the flip-flop 1302.sub.1 of the first stage can be described by the following formula (42), where "mod2" denotes a modulo-2 addition.
A.sub.t =A.sub.(t-4) +A.sub.(t-7) (mod2) (42)
When setting an initial value into the LFSR 1302 having 7 stages, each of the 7 bits can either take the value "0" or "1" and there are 2.sup.7 =128 possible initial values. If all of the 7 bits of the initial value are "0" the value output from the LFSR 1302 will not change, and the value "0000000" is excluded from the initial value. If at least one of the 7 bits of the initial value has the value "1", the value output from the LFSR 1302 will successively change to one of the 127 possible values with a predetermined period.
If the bit A.sub.t stored in the flip-flop 1302.sub.1 of the first stage can be described by the formula (42), it is known that the bit sequence of the bit A.sub.t is a pseudo random sequence having a period (2.sup.7 -1). In other words, each of the random number generators 331 shown in FIGS. 74 and 122A through 122C can generate (2.sup.7 -1) bit sequences in one period, and it is thus possible to output integers from "1" to "127" by reading the 7 bits of the bit sequences as a binary number.
When reading the bit sequences output from the random number generator 331 as the binary number, it is possible to consider the bit A.sub.1 output from the first stage as the LSB and the bit A.sub.7 output from the last (seventh) stage as the MSB or vice versa. But for the sake of convenience, it will be assumed hereunder that the bit A.sub.1 output from the first stage is the LSB and the bit A.sub.7 output from the last stage is the MSB of the binary value.
If it is assumed that c.sub.i (=1, 2, . . . , p) is an integer "0" or "1" where c.sub.p =1, a recurrence formula which describes A.sub.t by c.sub.i becomes as follows.
A.sub.t =c.sub.1 A.sub.(t-1) +c.sub.2 A.sub.(t-2) +. . . +c.sub.p A.sub.(t-p) (mod2) (43)
A characteristic polynomial of this recurrence formula (43) can be described as follows.
f(x)=1+c.sub.1 x+c.sub.2 x.sup.2 +. . . +c.sub.p x.sup.p (44)
A random number sequence which is made up of random pulses generated by the recurrence formula (43) has a period which is 2.sup.p -1 or less in length, and the characteristic polynomial which has a maximum period within this range is called a primitive polynomial. The bit sequence of A.sub.t generated by such a primitive polynomial and having the period of 2.sup.p -1 is called a p order maximum length linearly recurring sequence (hereinafter simply referred to as a M sequence).
For example, the bit sequence of A.sub.t generated by the formula (42) can be described by a seventh order M sequence corresponding to the primitive polynomial f(x)=1+x.sup.4 +x.sup.7. In the case of the random number generators 331 using the LFSR 1302 having 7 stages and generating the bit sequence of the seventh order M sequence, the number of kinds of random number generators 331 is limited to 4 as shown in FIGS. 74 and 122A through 122C The primitive polynomial and the recurrence formula of each of the random number generators 331 shown in FIGS. 74 and 122A through 122C are shown below.
[A] Random number generator 331 shown in FIG. 74:
Primitive Polynomial: f(x) 1+x+x.sup.7
Recurrence Formula: A.sub.t =A.sub.(t-1) +A.sub.(t-7) (mod2)
Bit Sequence of M sequence: FIG. 123A
Generated Random Number Sequence: FIG. 124A
[B] Random number generator 331 shown in FIG. 122A:
Primitive Polynomial: f(x) 1+x.sup.3 +x.sup.7
Recurrence Formula: A.sub.t =A.sub.(t-3) +A.sub.(t-7) (mod2)
Bit Sequence of M sequence: FIG. 123B
Generated Random Number Sequence: FIG. 124B
[C] Random number generator 331 shown in FIG. 122B:
Primitive Polynomial: f(x) 1+x.sup.4 +x.sup.7
Recurrence Formula: A.sub.t =A.sub.(t-4) +A.sub.(t-7) (mod2)
Bit Sequence of M sequence: FIG. 123C
Generated Random Number Sequence: FIG. 124C
[D] Random number generator 331 shown in FIG. 122C:
Primitive Polynomial: f(x) 1+x.sup.6 +x.sup.7
Recurrence Formula: A.sub.t =A.sub.(t-6) +A.sub.(t-7) (mod2)
Bit Sequence of M sequence: FIG. 123D
Generated Random Number Sequence: FIG. 124D
Therefore, the random number generators 331 shown in FIGS. 74 and 122A through 122C respectively generate bit sequences of the M sequence such that the period of the random pulses becomes a maximum, and the random nature of the generated pulses in each period is extremely satisfactory. Since it is known beforehand that the period of the bit sequence output from such random number generators 331 is a maximum, it is possible to generate signals having random pulse positions by modulating the generated pulses by the pulse density or the number of pulses.
For example, in the case where a signal having a pulse density of 10/127 is demanded of the random number generator 331 which generates 127 random numbers, it is possible to generate a signal having random pulse positions and the pulse density of 10/127 by outputting the random pulses if the number generated is "1" to "10" and blocking the random pulses if the number generated is "11" to "127". This signal passing and blocking functions can be realized by the comparator 332 shown in FIG. 72, for example.
As described above, the random number generator 331 can make the period of the pseudo random pulses or the random numbers a maximum because the bit sequence of the M sequence is generated. However, the pulses generated from the random number generator 331 are pseudo random pulses and are reproducible. In other words, when the random number generators 331 are driven continuously, the same pulses are output after each period, and the random nature of the generated pulses over a plurality of periods is poor.
Next, a description will be given of other embodiments of the random number generator 331 in which the random nature of the pulses generated over a plurality of periods is improved.
FIG. 125 shows a second embodiment of the random number generator 331. In FIG. 125, those parts which are the same as those corresponding parts in FIGS. 74 and 122A through 122C are designated by the same reference numerals, and a description thereof will be omitted. In FIG. 125, the illustration of the signal lines for inputting the clock signal will be omitted so as to simplify the drawing.
In FIG. 125, a switching circuit 1309 receives the bit b.sub.6 output from an output part 1304 of the flip-flop 1302.sub.7 of the last stage and the output of the exclusive-OR gate 1305, and selectively outputs one of these signals in response to a control signal which is received from a terminal 1314. The output of the switching circuit 1309 is supplied to an input part 1306 of the flip-flop 1302.sub.1 of the first stage. In this case, the exclusive-OR gate 1305 receives the bits b.sub.0 and b.sub.6 respectively output from the output parts 1303 and 1304 of the flip-flops 1302.sub.1 and 1302.sub.7. Hence, the LFSR 1302 is formed by the flip-flops 1302.sub.1 through 1302.sub.7, the exclusive-OR gate 1305 and the switching circuit 1309.
FIG. 126 shows an embodiment of the switching circuit 1309. This switching circuit 1309 includes an OR gate 1310, AND gates 1311 and 1312, and an inverter 1313 which are connected as shown. The control signal is input to the terminal 1314, and the bits b.sub.0 and b.sub.6 output from the output parts 1303 and 1304 of the flip-flops 1302.sub.1 and 1302.sub.7 are respectively input to terminals 1317 and 1316. An output of the OR gate 1310 is output from a terminal 1318 and is supplied to the input part 1306 of the flip-flop 1302.sub.1 as the output of the switching circuit 1309.
Normally, the switching circuit 1309 selectively outputs the output of the exclusive-OR gate 1305 in response to the control signal. In this case, the connection of the random number generator 331 shown in FIG. 125 is the same as that shown in FIG. 74. But since the random pulses will be repeated periodically if this connection is fixed, this embodiment switches the connection of the switching circuit 1309 in response to the control signal after a predetermined number of bits are shifted in the LFSR 1302.
For example, this predetermined number of bits corresponds to the number of bits which are shifted in the LFSR 1302 during one period of the random pulses. When the connection of the switching circuit 1309 is switched to selectively output the bit b.sub.6 from the flip-flop 1302.sub.7, the initial value set in the LFSR 1302 after one period of the random pulses is changed from the original initial value by shifting an arbitrary number of bits in the LFSR 1302. Thereafter, the connection of the switching circuit 1309 is returned to selectively output the output of the exclusive-OR gate 1305. Therefore, it is possible to guarantee the random nature of the random pulses over a plurality of periods of the random pulses.
Next, a description will be given of the operation of the random number generator 331 shown in FIG. 125, by referring to FIG. 127.
In FIG. 127, a step S1 sets a predetermined binary value in the LFSR 1302 from an external unit (not shown) via the input part 1306 of the flip-flop 1302.sub.1 of the first stage. A step S2 supplies the control signal to the switching circuit 1309 via the terminal 1314 so that the output of the exclusive-OR gate 1305 is coupled to the input part 1306 of the flip-flop 1302.sub.1 via the switching circuit 1309, and applies the clock signal to the clock terminals of the flip-flops 1302.sub.1 through 1302.sub.7 so as to successively shift the bits in the LFSR 1302 in response to the clock signal. A step $3 obtains the exclusive-OR of the bit output from the flip-flop 1302.sub.1 of the first stage and the bit output from the flip-flop 1302.sub.7 of the last stage, thereby feeding back the output of the exclusive-OR gate 1305 to the input part 1306 of the flip-flop 1302.sub.1 of the first stage. A step S4 successively shifts the content of the LFSR 1302 while inputting the output of the exclusive-OR gate 1305. As a result, a step S5 generates random numbers or pulses described by each content of the LFSR 1302. The generated random numbers are described by binary numbers, but may be converted into decimal numbers ranging from "1" to "127".
A step S6 decides whether or not one period of the random pulses have been generated. The process returns to the step S3 if the decision result in the step S6 is NO. On the other hand, if the decision result in the step S6 is YES, a step S7 supplies the control signal to the switching circuit 1309 via the terminal 1314 so as to switch the connection of the switching circuit 1309 for a certain time. As a result, the switching circuit 1309 selectively outputs the output of the flip-flop 1302.sub.7 of the last stage to the flip-flop 1302.sub.1 of the first stage. In this particular case, the certain time is set so that the content of the LFSR 1302 is shifted by 1 bit. Hence, a step S8 applies the clock signal to the flip-flops 1302.sub.1 through 1302.sub.7 and shifts the content of the LFSR 1302 by 1 bit so as to set a new initial value which is different from the original initial value, and the process returns to the step S2. Of course, the step S8 may shift the content of the LFSR 1302 by an arbitrary number of bits by appropriately setting the certain time in the step S7.
Therefore, after one period of the random pulses, the original initial value set in the LFSR 1302 is changed to a new initial value which is different from the original initial value. For this reason, the random nature of the random pulses generated from the random number generator 331 shown in FIG. 125 is improved compared to that of the random number generators 331 shown in FIGS. 74 and 122A through 122C.
The exclusive-OR gate 1305 shown in FIG. 125 receives the outputs of the flip-flops 1302.sub.1 and 1302.sub.7. However, the exclusive-OR gate 1305 may receive the outputs of the flip-flop 1302.sub.7 and a flip-flop other than the flip-flop 1302.sub.1, as may be readily understood from the random number generators 331 shown in FIGS. 74 and 122A through 122C described above. Furthermore, the connection of the switching circuit 1309 may be switched before one period of the random pulses elapses.
Next, a description will be given of a third embodiment of the random number generator 331, by referring to FIG. 128. In FIG. 128, those parts which are the same as those corresponding parts in FIG. 125 are designated by the same reference numerals, and a description thereof will be omitted.
In FIG. 128, the switching circuit 1309 is coupled between one of the inputs of the exclusive-OR gate 1305 and the output parts 1303.sub.1 and 1303.sub.2 of the flip-flops 1302.sub.1 and 1302.sub.2. The switching circuit 1309 selectively outputs the output from one of the flip-flops 1302.sub.1 and 1302.sub.2 in response to the control signal which is received via the terminal 1314, and the connection of the switching circuit 1309 is switched for every period of the random pulses. The exclusive-OR gate 1305 receives the output of the switching circuit 1309 and the output of the flip-flop 1302.sub.7, and supplies the output thereof to the flip-flop 1302.sub.1 of the first stage.
According to the random number generator 331 shown in FIG. 128, it is possible to change the inputs of the exclusive-OR gate 1305 for every one period of the random pulses. As a result, the period of the random pulses effectively becomes approximately twice those of the random number generators 331 shown in FIGS. 74 and 122A through 122C, and the random character of the random pulses is improved.
Of course, the output parts which are connected to the switching circuit 1309 is not limited to the output parts 1303.sub.1 and 1303.sub.2 of the flip-flops 1302.sub.1 and 1302.sub.2, and the output parts of other flip-flops may be used.
Next, a description will be given of an embodiment of a random pulse generator, by referring to FIG. 129.
The random pulse generator 4331 shown in FIG. 129 includes a LFSR 1410, an output control circuit 1411, and an AND gate 1412 which are connected as shown. Reference clock pulses are applied to a terminal 1407. M clock pulses are required to generate random pulses amounting to a maximum period of the random pulses. The clock pulses supplied to the LFSR 1410 and the output control circuit 1411, and the AND gate 1412 receives the outputs of the LFSR 1410 and the output control circuit 1411.
The output control circuit 1411 is made up of AND gates 1413 and 1414, frequency dividing counters 1415 and 1416, and an inverter 1417 which are connected as shown. The frequency dividing counter 1415 has a frequency dividing ratio of 1/N and the frequency dividing counter 1416 has a frequency dividing ratio of 1/M, where N<M.
If predetermined bits are initially set in the LFSR 1410 and the clock pulses are input to the terminal 1407, the random pulses generated by the LFSR 1410 in response to the clock pulses are output via the AND gate 1412.
As described above, the pulses generated by the LFSR 1410 are pseudo random pulses and the same pulses are repeated after one period. Hence, in this embodiment, after the LFSR 1410 generates random pulses amounting to one period thereof in response to M clock pulses, N clock pulses are thereafter input to the LFSR 1410 to resume the generation of the random pulses and improve the random nature of the generated random pulses.
In the initial state, the clock pulses are only input to the 1/M frequency dividing counter 1416 of the output control circuit 1411 via the AND gates 1413 and 1414. Hence, the initial state of the output control circuit 1411 is set so that the 1/M frequency dividing counter 1416 outputs a value "0". This output value "0" of the 1/M frequency dividing counter 1416 is inverted by the inverter 1417, and the output value "1" of the inverter 1417 is input to one input of the AND gate 1412. As a result, the AND gate 1412 passes the random pulses generated by the LFSR 1410.
When the number of clock pulses received from the terminal 1407 reaches M from the initial state of the output control circuit 1411, the output value of the 1/M frequency dividing counter 1416 becomes "1" and the inverter 417 outputs the value "0" which is input to one input of the AND gate 1412. Thus, the AND gate 1412 blocks the random pulses which are generated by the LFSR 1410. In this state, the clock pulses are input to only the 1/N frequency dividing counter 1415 of the output control circuit 1411 via the AND gates 1413 and 1414. The 1/N frequency dividing counter 1415 continuously outputs the value "0" until the number of clock pulses received reaches N. When the number of clock pulses input to the terminal 1407 reaches N in the state where the random pulses generated from the LFSR 1410 are blocked by the AND gate 1412, the output value of the 1/N frequency dividing counter 1415 becomes "1", and the output control circuit 1411 returns to its initial state. After the output control circuit 1411 returns to its initial state, the AND gate 1412 resumes output of the random pulses generated from the LFSR 1410.
Therefore, when the random pulses amounting to one period are output from the LFSR 1410, the output of the random pulses is resumed after N clock pulses are input to the LFSR 1410. As a result, the same pulses are prevented from being output after one period, and the random nature of the random pulses generated from the random number generator 331 is improved.
Next, a description will be given of another embodiment of the random pulse generator, by referring to FIG. 130. In FIG. 130, those parts which are the same as those corresponding parts in FIG. 129 are designated by the same numerals, and a description thereof will be omitted.
The random pulse generator 4331 shown in FIG. 130 includes the LFSR 1410, the output control circuit 1411, AND gates 1419 and 1420, a LFSR 1421, and an inverter 1422 which are connected as shown. In other words, the output of the AND gate 1419 is connected directly to the input of the LFSR 1410, while the output of the AND gate 1420 is coupled to the input of the LFSR 1410 via the LFSR 1421. Furthermore, the output of the output control circuit 411 is fed back directly to the AND gate 1419 on one hand, and is fed back to the AND gate 1420 via the inverter 1422 on the other. The generation of the random pulses in response to the clock pulses is made by the n order LFSR 14 10. The m order LFSR 1421 has a number of stages smaller that that of the n order LFSR 1410 (n>m) , and functions as a set value changing means for changing the number N of clock pulses input to the LFSR 1410 every time after the output control circuit 1411 operates.
Similarly as in the case of the random pulse generator 4331 shown in FIG. 129, after the LFSR 1410 outputs the random pulses amounting to one period in response to M clock pulses, the output of the random pulses is stopped and is resumed after N' clock pulses are input to the LFSR 1410. In this state, when the output value of the output control circuit 1411 becomes "0" in order to block the output random pulses of the LFSR 1410, the clock pulses from the terminal 1407 are input directly to the output control circuit 1411 on one hand and is input to the LFSR 1410 via the LFSR 1421 on the other. Hence, the number of clock pulses input to the LFSR 1410 is irregularly modulated until N clock pulses are input to the output control circuit 1411 and the output of the random pulses is resumed, and the number N' of clock pulses input to the LFSR 1410 is changed every time the output control circuit 1411 operates. In other words, when the random pulses amounting to one period are output from the LFSR 1410, the output of the random pulses is stopped and is resumed after a certain number of clock pulses are input to the LFSR 1410, where this certain number changes for each period of the random pulses. Consequently, the random nature of the generated random pulses is extremely satisfactory in this embodiment.
Of course, the set value changing means is not limited to the LFSR 1421 which changes the number N of clock pulses by modulation of the clock pulses, and the number N of clock pulses may be changed by other set value changing means.
Next, a description will be given of a fourth embodiment of the random number generator 331, by referring to FIG. 131. In FIG. 131, those parts which are the same as those corresponding parts in FIG. 128 are designated by the same reference numerals, and a description thereof will be omitted.
The random number generator 331 shown in FIG. 131 includes a 7-bit shift register 1501, a selector 1502, a 2-bit binary counter 1503, and an exclusive-OR gate 1305 which are connected as shown. The shift register 1501, the exclusive-OR gate 1305 and the selector 1502 form the LFSR 1302. Count clock pulses are input to a terminal 1507, and shift clock pulses are input to a terminal 1506. An output 1505 of the shift register 1501 is made up of bits b.sub.0 through b.sub.6, where b.sub.0 is regarded as the LSB of the binary random value and the bit b.sub.6 is regarded as the MSB of the binary random value.
The binary counter 1503 sequentially outputs values "00", "01", "10" and "11" when counting the count clock pulses from the terminal 1507, and the output of the binary counter 1503 is supplied to the selector 1502. As shown in FIG. 132 (A) and (B), the selector 1502 outputs a register data bit r.sub.1 of the shift register 1501 in response to the output "00" of the binary counter 1503, a register data bit r.sub.2 in response to the output "01", a register data bit r.sub.3 in response to the output "10", and a register data bit r.sub.4 in response to the output "11". The output of the selector 1502 is fed back to the shift register 1501 via the exclusive-OR gate 1305.
For example, if the initial value of the binary counter 1503 is "00" and an initial value "0000001" is set in the shift register 1501, the selector 1502 first selectively feeds back the register data bit r.sub.1 and the random numbers 64, 1, 3, 7, 15, 31, . . . described by the characteristic polynomial 1+x+x.sup.7 shown in FIG. 132 (C) are sequentially output from the shift register 1501 by inputting 127 shift clock pulses via the terminal 1506. Next, at least one count clock is input to the terminal 1507. For example, if one count clock is input to the terminal 1507, the count of the binary counter 1503 advances by one, that is, changes from the initial value "00" to the value "01" and the selector 1502 thereafter switches the feedback from the register data bit r.sub.1 to the register data bit r.sub.6.
When 127 shift clock pulses are input from the terminal 1506 in this state, the random numbers 64, 1, 2, 4, 8, 16, 32, . . . described by the characteristic polynomial 1+x.sup.6 +x.sup.7 shown in FIG. 132 (C) are sequentially output from the shift register 1501. These random numbers are generated in a sequence different from that of before.
The count clock pulses input to the terminal 1507 need only be controlled so that at least one count clock pulse is input for every predetermined period. Further, the binary counter 1503 is not limited to the 2-bit counter and may be an n bit counter. In this case, m bits out of the n bits are decoded and the decoded output is used to control the selector 1502.
As in the case of the previously described embodiments of the random number generators, the number of bits of the shift register 1501 forming the LFSR 1302 is not limited to 7. If the number of bits of the shift register 1501 is other than 7, the feedback position (that is, the register data bits) used may be other than the 4 positions shown in FIG. 131. In other words, the number of bits of the shift register 1501 and the number and positions of the feedback are not limited to those of the described embodiment.
According to this embodiment of the random number generator 331 shown in FIG. 131, the feedback position is cyclically updated as shown in FIG. 132 every time the random number sequence is repeatedly generated, so that a different kind of random number sequence is generated each time.
Next, a description will be given of a fifth embodiment of the random number generator 331, by referring to FIG. 133. In FIG. 133, those parts which are the same as those corresponding parts in FIG. 131 are designated by the same reference numerals, and a description thereof will be omitted.
The random pulse generator 331 shown in FIG. 133 includes a first LFSR 1302A, a 2-bit shift register 1513, and a second LFSR 1302 which is identical to the LFSR 1302 shown in FIG. 131.
The selector 1502 of the LFSR 1302 receives the output of the 2-bit shift register 1513 and switches the feedback position of the shift register 1501 based thereon. The selector 1502 selects the register data bit r.sub.11 of the shift register 1501 if the output of the shift register 1513 is "00", selects the register data bit r.sub.16 if the output of the shift register 1513 is "01", selects the register data bit r.sub.14 if the output of the shift register 1513 is "10", and selects the register data bit r.sub.13 if the output of the shift register 1513 is "11".
The shift out terminal of the first LFSR 1302A is connected to the shift input terminal of the shift register 1513, so that the shift out data of the first LFSR 1302A can be input to the shift register 1513. The feedback position of the first LFSR 1302A is fixed to the register position r.sub.31, so as to generate the random number sequence described by the characteristic formula 1+x+x.sup.7.
For example, an initial value "11" is set in the shift register 1513, an initial value "0000001" is set in the shift register 1501 of the second LFSR 1302, and an initial value "1011100" is set in the shift register 1501A of the first LFSR 1302A. In this case, the content of the shift register 1503 is "11" and the register data bit r.sub.13 of the shift register 1501 is selectively fed back to the shift register 1501 via the selector 1502 and the exclusive-OR gate 1305. By inputting 127 shift clock pulses to the terminal 1506 in this state, the random number sequence made up of the random numbers 64, 1, 2, 4, 9, 18, 36, . . . described in binary are sequentially output as the output 1505 of the second LFSR 1302.
Next, at least one shift clock pulse is input to the terminal 1512. For example, if only one shift clock pulse is input to the terminal 1512, the content of the shift register 1513 changes from "11" to "01", and the feedback position of the second LFSR 1302 changes from r.sub.13 to r.sub.16. When 127 shift clock pulses are input again to the terminal 1506 in this state, the random number sequence made up of the random numbers 64, 1, 2, 4, 8, 16, 32, . . . are obtained as the output 1505 of the second LFSR 1302. The sequence of the random numbers generated in this case is different from that of the random numbers generated previously. Thereafter, the content of the shift register 1513 changes to "01", "00", "10", . . . as shown in FIG. 134 every time the random pulses amounting to one period are output from the second LFSR 1032. Thus, it is possible to generate the random numbers having mutually different sequences for each period of the random pulses.
In addition, the binary number sequence generated by the first LFSR 1302A is a M sequence signal. For this reason, the content of the shift register 1501A changes at random at every repetition.
Of course, the number of bits of the registers 1501 and 1501A is not limited to 7, and the feedback position of the first LFSR 1302A is not limited to that shown in FIG. 133.
FIG. 135 shows the first embodiment of the random number generator 331 shown in FIG. 74 together with the essential part of the neuron unit shown in FIG. 72.
In FIG. 135, the register 333 is set from a keyboard (not shown) or the like. A decimal number is set from the keyboard and this decimal number is converted into a binary signal and output to the comparator 332. The binary output of the register 333 is fixed to a value from 1 to 127 if the register 333 has 7 bits, for example. If the decimal value "10" is set in the register 333, for example, the comparator 332 compares the 7-bit output of the register 333 describing the decimal number "10" with the 7-bit output from the LFSR 1302 in response to each clock pulse applied to the terminal 1307. The comparator 332 outputs the value "1" via a terminal 1390 if the 7-bit output of the LFSR 1302 is less than or equal to the 7-bit output of the register 333, and otherwise outputs the value "0". Hence, the output value of the comparator 332 has the value "1" at a rate of 10/127.
But since the random pulses output from the LFSR 1302 has a periodic nature dependent on the order of the LFSR 1302 as described above, it is preferable to improve the random nature of the output value of the comparator 332.
FIG. 136 shows a sixth embodiment of the random number generator 331 together with the essential part of the neuron unit shown in FIG. 72. In FIG. 136, those parts which are the same as those corresponding parts in FIGS. 125, 126 and 135 are designated by the same reference numerals, and a description thereof will be omitted.
According to this embodiment, the random nature of the output value of the comparator 332 is improved compared to that shown in FIG. 135 because the 7-bit output of the LFSR 1302 does not have a cyclic pattern due to the provision of the switching circuit 1309.
In FIG. 136, the initial value of the LFSR 1302 may be set from a terminal 1393 by switching the connection of a switch 1392 to a terminal 1394 from the terminal 1318.
Next, a description will be given of a twentieth embodiment of the neuron unit according to the present invention, by referring to FIG. 137. In FIG. 137, those parts which are the same as those corresponding parts in FIGS. 135 and 136 are designated by the same reference numerals, and a description thereof will be omitted.
In this embodiment, a random pulse generating part 3100 is made up of the random number generator 331, the comparator 332 and the register 333 shown in FIG. 72, where the random number generator 331 has a construction identical to that shown in FIG. 135. If a decimal number set in a register 33300 is "10" the random pulse sequence which is output from the terminal 1390 is the same after 10 random pulses are generated. In other words, after 10 random pulses are generated, the output bits of the LFSR 1302 of the random number generator 331 are all "0" until one period of the random pulses ends, and there is no need to generate any random pulses from the LFSR 1302 after 10 random pulses are generated. Hence, this embodiment utilizes this feature of the LFSR 1302 and obtains a predetermined number of random pulses by inputting no clock pulse to the LFSR 1302 after the number of random pulses are generated, so as to eliminate the periodic nature of the LFSR 1302.
In this embodiment, a random pulse counter 1616 is connected to the output of the random pulse generating part 33100, and the 7-bit output of the counter 1616 is supplied to a comparator 33200, as shown in FIG. 137. In addition, the clock pulses input to the terminal 1307 are supplied to the random pulse generating part 33100 via an AND gate 1623 which also receives the output of the comparator 33200.
The comparator 33200 outputs the value "1" if the 7-bit output of the counter 1616 is smaller than the 7-bit output of the register 33300, and otherwise outputs the value "0". The 7-bit output of the register 33300 has a fixed value selected from "1" to "127", for example.
If the number of random pulses counted in the counter 1616, that is, the value output from the counter 1616, is denoted by R, and the fixed value output from the register 33300 is denoted by Q, the comparator 33200 outputs the value "1" if R<Q and otherwise outputs the value "0". Because the output of the comparator 33200 is fed back to the AND gate 1623, the supply of the clock pulses to the random pulse generating part 33100 is stopped when R=Q. As a result, the generation of the random pulses by the random pulse generating part 33100 is interrupted, and the periodic nature of the random pulses output from the random pulse generating part 33100 is eliminated.
The generation of the random pulses from the random pulse generating part 33100 for next period of the random pulses, without the need to reset the initial value of the LFSR 1302. Accordingly, the random characteristic of the random pulses output via the terminal 1390 is improved.
The fifth embodiment is particularly effective if the value Q is small. However, if the value Q is large, it becomes frequently necessary to input a number of clock pulses amounting to more than one period of the random pulses generated by the LFSR 1302 in order to count the value Q. Hence, in this case, it is necessary to change the initial value of the LFSR 1302 as done in the sixth embodiment which will be described hereunder.
FIG. 138 shows an essential part of a twenty-first embodiment of the neuron unit according to the present invention. In FIG. 138, those parts which are the same as those corresponding parts in FIG. 137 are designated by the same reference numerals, and a description thereof will be omitted.
In FIG. 138, a frequency dividing counter 1626 is provided between the AND gate 1623 and the random pulse generating part 33100. This frequency dividing counter 1626 has the function of counting the clock pulses which are supplied from the terminal 1307 to the random pulse generating part 33100 via the AND gate 1623, and the function of supplying the output thereof to the random pulse generating part 33100 only when the number of clock pulses counted corresponds to one period of the generated random pulses. In other words, the frequency dividing counter 1626 also functions to change the initial value set in the LFSR 1302 of the random pulse generating part 33100.
Because the initial value set in the LFSR 1302 of the random pulse generating part 33100 is changed by the output of the frequency dividing counter 1626 after the frequency dividing counter 1626 counts a number of clock pulses required to generate the random pulses amounting to one period, the random pulse sequence output from the random pulse generating part 33100 is different for each period of the random pulses. As a result, the periodic nature of the pulses output from the random pulse generating part 33100 is eliminated, and the random characteristic of the random pulse sequence output via the terminal 1390 is improved. Of course, the frequency dividing counter 1626 is reset automatically after the frequency dividing counter 1626 counts the number of clock pulses required to generate the random pulses amounting to one period.
The method of changing the initial value set in the LFSR 1302 of the random pulse generating part 33100 is of course not limited to that using the frequency dividing counter 1626 shown in FIG. 138, and various other variations are possible.
In the neuron unit according to the present invention, it is necessary to use a plurality of weighting coefficients. Hence, a plurality of circuit systems having a circuit construction such as those shown in FIGS. 72 and 135 through 138 must be provided. In other words, it is necessary to provide a plurality of random number generators. When providing a plurality of random number generators, it is preferable to provide random number generators having the same construction in order to minimize the cost and simplify the production process.
However, if the random number generators are identical in construction and are set with the same initial value, the random number sequences generated therefrom naturally become the same. For this reason, it is desirable to take measures so that mutually different initial values are set in the random number generators. Furthermore, if all of the bits of the initial value are "0", the random numbers which are generated will be fixed to "0", and it is therefore necessary to prevent such an initial value from being set in the random number generators.
FIG. 139 shows an essential part of a twenty-second embodiment of the neuron unit according to the present invention. FIG. 139 only shows a plurality of random number generators which are linked so that mutually different initial values are set therein at the time of initialization when the power source is turned ON, for example. In FIG. 139, those parts which are the same as those corresponding parts in FIG. 125 are designated by the same reference numerals, and a description thereof will be omitted.
In FIG. 139, a plurality of LFSRs 13021 through 13025 are linked via respective switching circuits 1309. In addition, a bit generating part 1701 is connected to the LFSR 13021 via a switching circuit 1309.
For example, the bit generating part 1701 is made up of a LFSR having 5 bits as shown in FIG. 140. In addition, the LFSRs 13021 through 13027 respectively have constructions which are basically the same as that of the LFSR 302 shown in FIG. 125 except for the feedback position, and respectively have 7 bits, for example. FIG. 141 shows the constructions of only the LFSRs 13022 and 13023 for the sake of convenience. In FIG. 141, the illustration of the clock signal line is omitted for the sake of convenience. In FIGS. 140 and 141, those parts which are the same as those corresponding parts in FIG. 125 are designated by the same reference numerals, and a description thereof will be omitted. Normally, each switching circuit 1309 is connected to input the output of the exclusive-OR gate 1305 to the flip-flop 1302.sub.1 of the first stage of the respective LFSR 1302J, where J=1 to 5.
FIG. 142 is a flow chart for explaining the operation of the twenty-second embodiment of the neuron unit according to the present invention.
In FIG. 142, a step S11 sets a predetermined initial value into the bit generating part 1701. A step S12 controls each switching circuit 1309 to disconnect the connection of the output of the exclusive-OR gate 1305 to the input of the corresponding LFSR 1302J and to connect the output of the LFSR 1302J to the input of the LFSR 1302(J+1). In this embodiment, the output of the LFSR 13025 is not connected to a switching circuit, and the switching circuit 1309 connects the output of the bit generating part 1701 to the LFSR 13021. When the LFSRs 13021 through 13025 are linked by the switching circuits 1309, a shift register having 35 (=5.times.7) stages is formed.
In this state, a step S13 inputs 29 (=7.times.(5-1)+1) or more clock pulses to the bit generating part 1701 and each of the LFSRs 13021 through 13025. As a result, at least one bit having the value "1" is set in each of the LFSRs 13021 through 13025 by the bit sequence output from the bit generating part 1701 in response to the clock pulses. After the input of such a bit sequence, a step S14 controls each switching circuit 1309 to return the connection to the normal connection. In other words, 5 7-bit LFSRs are formed from 1 35-bit shift register by the switching of the switching circuits 1309.
In this embodiment, a low-level control signal (that is, a value "0") supplied to each switching circuit 1309 via the terminal 1314 connects the output of the exclusive-OR gate 1305 to the input of the corresponding LFSR 1302J, and a high-level control signal (that is, a value "1") switches the connection to the output of the LFSR of the preceding stage. Such a control signal may be generated from a control circuit (not shown) by known means.
Because the LFSR forming the bit generating part 1701 is 5 bits and the initial value thereof is set so that the number of continuous bits having the value "0" is less than 5, at least one bit having the value "1" is set in each of the LFSRs 13021 through 13025. Therefore, it is possible to prevent all of the bits of the initial value from becoming "0" in each of the LFSRs 13021 through 13025. Furthermore, because the setting of the initial value is made via the linearly connected LFSRs 13021 through 13025, mutually different initial values are automatically set in each of the LFSRs 13021 through 13025 approximately at the same time.
It is desirable that the bit sequence output from the bit generating part 1701 is also at random. If the period of the bit sequence output from the bit generating part 1701 is an integral multiple of the number of stages of each LFSR 1302J, the period of the initial value of each LFSR 1302J becomes short. For this reason, it is desirable that the period of the bit sequence output from the bit generating part 1701 is not an integral multiple of the number of stages of each LFSR 1302J.
For example, if the period of the bit sequence output from the bit generating part 1701 is "35", the same bit is input for every 5 (=35/7) 7-bit LFSRs, and thus, the random characteristic of the generated random pulses will deteriorate if more than 5 7-bit LFSRs are linked. But if the period of the sequence output from the bit generating part 1701 is "36", it is possible to simultaneously set mutually different initial values to a maximum of 36 7-bit LFSRs without deteriorating the random characteristic of the generated random pulses. Therefore, the concept of linking a plurality of LFSRs to simultaneously set mutually different initial values is particularly effective when used in the neuron unit according to the present invention.
Further, the present invention is not limited to these embodiments, but various variations and modifications may be made without departing from the scope of the present invention.

Number	Date	Country
1-343891	Dec 1989	JPX
2-55523	Mar 1990	JPX
2-58515	Mar 1990	JPX
2-58548	Mar 1990	JPX
2-60738	Mar 1990	JPX
2-67938	Mar 1990	JPX
2-67939	Mar 1990	JPX
2-67940	Mar 1990	JPX
2-67941	Mar 1990	JPX
2-67942	Mar 1990	JPX
2-67943	Mar 1990	JPX
2-67944	Mar 1990	JPX
2-272827	Oct 1990	JPX
3-351410	Dec 1991	JPX
4-11217	Jan 1992	JPX
4-21284	Feb 1992	JPX
4-53204	Mar 1992	JPX
4-65637	Mar 1992	JPX
4-120360	May 1992	JPX
4-224702	Jul 1992	JPX

Number	Name	Date	Kind
4893255	Tomlinson, Jr.	Jan 1990
4951243	Tomlinson, Jr.	Aug 1990

Neuron unit

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

US Classifications

Field of Search

US

International Classifications

Abstract

Description

Claims

Priority Claims (20)

BACKGROUND OF THE INVENTION

US Referenced Citations (2)

Foreign Referenced Citations (1)

Non-Patent Literature Citations (8)

Divisions (1)

Continuation in Parts (1)

Entry
Ito, T., et al., "A Neural Network Model Extracting Features From Speech Signals", Electronic Information Communication Society Papers, 87/2, vol. J70-D, No. 2.
Murray et al., "Asynchronous Arithmetic for VLSI Neural Systems", Electronic Letters, Jun. 4, 1987, vol. 23, No. 12.
Murray et al., "A Novel Computation and Signalling Method for VLSI Neural Networks", Digest of Technical Papers, ESSCIRC '87, Thirteenth European Solid-State Circuits Conference, Sep. 23-25, 1987.
Kamada et al., "Digital Neuron Model", Electronic Information Communication Society Spring National Conference, 1988.
Yoshida et al., "Neural Coupling Model of Visual Domain", Electronic Communication Society National Conference, 1974.
K. Fukushima, "Cognitron", Electronic Communication Society National Conference, 1974.
K. Fukushima, "Improvement in Pattern-Selectivity of a Cognitron", Electronic Communication Society Papers '79/10, vol. J62-A, No. 10.
W. Banzhaf, "On a Simple Stochastic Neuron-like Unit", Biological Cybernetics, vol. 60, No. 2, 1988, pp. 153-160.