Method and apparatus for parallel implementation of neural networks

Abstract

Long and short term memory equations for neural networks are implemented by means of exchange of signals which carry information in the form of both binary and continuously modulated energy emissions. In one embodiment, array of parallel processors exhibits behavior of cooperative-competitive neural networks. Parallel bus interconnections and digital and analog processing of analog information contained in the exchanged energy emissions are employed with generally local synchronization of the processors.

Description

TECHNICAL FIELD

This invention relates to methods of and apparatuses for hardware implementations of neural networks. The invention has particular application in hardware implementations of neural networks describable by sets of first order ordinary differential equations or by sets of finite difference equations with changeable connectivity pattern of the processors and changeable degree of information exchange among the processors.

BACKGROUND

A crucial requirement for a neural network model to be of practical use is its stability. Neural network models whose global stability can be proved are usually described by sets of first order ordinary differential equations of the type ##EQU1## where F.sub.P (t) and D.sub.PQ (t) are the short and the long term memory traces, respectively. Function G(y;z) is generally linear in the second argument; the summation is over the processors Q which are the nearest neighbors of processor P; T.sub.PQ typically are fixed coefficients taking both positive and negative values, and function s'(F.sub.P) is generally of sigmoid type, i.e., it has a threshold, it is monotonic, and it is bounded. Function H is typically some implementation of Hebbian learning rule.

Such equations were employed for a number of methods for pattern recognition, for construction of content addressable memory, and for determining good solutions of complex optimization problems.

Usually the hardware implementations of the neural network theories consist of digital, or software simulations of the neural network differential equations. Such techniques are time-intensive when used on serial digital computers and, therefore, are slow and expensive. For high speed information exchange among the processors it is advantageous to use parallel interconnections and analog processes to carry the information.

Various attempts have been made to produce analog neural network hardware. One method involves dynamical holograms for redirection of optical signal emitted by coherent sources of the processors. The obvious disadvantage of this method is the difficulty in manufacturing the required integrated optoelectronics and in creation of high quality dynamical hologram while maintaining the information about the processors' interconnections stored in it.

Another means to achieve hardware implementation, known in the art, involves manufacture of networks of resistors on a chip. Disadvantages of this method are that the connectivity patterns are constant and once set are impossible to change. In addition, interconnections among the processors take much space on the chip. Finally, because of the general fanout limitations in digital electronics, the fanout for fixed resistor networks can not be expected to be much larger than 100. For comparison the human brain fanout reaches 100,000.

Therefore, there exists a need for simple, cost effective, and energy efficient means for analog, parallel implementation of neural networks with changeable interconnection pattern and large fanout.

DISCLOSURE OF THE INVENTION

The implementation of any neural network model describable by first order differential equations or their finite difference equivalents is performed by arrangement of linear arrays of processors into two or higher dimensional arrays. Each processor of a linear array emits and absorbs energy emissions from one or more common pipeline buses which serve as common data channels. In one embodiment of the present invention the energy emissions consist of particles of half integer spin, such as electrons, and the corresponding pipelines are strips of conductive material. In some embodiments of the present invention, the conductive material has finite resistivity, in others the resistivity is zero and the conductive material is superconductive. In some embodiments of the present invention, the energy emissions are in the form of radiant energy and consist of particles of integer spin, such as photons, and corresponding pipelines are common radiant energy waveguides.

In accordance with the present invention, an adaptive communications network comprises a plurality of processors coupled together by an energy communication channel. Each of a plurality of processors transmits energy into a channel and detect energy from a channel. The energy emitted and detected is modulated and demodulated according to codes. The codes are representative of present states of the processors and are changed according to a rule which depends on the demodulated inputs to energy detectors. The detectors detect the aggregate energy in the energy channel from plural processors.

An initial implementation of the present invention is presented in application Ser. No. 07/026,479. An alternative implementation is presented here and in application Ser. No. 07/113,636.

The linear arrays which are adjacent within the two or higher dimensional array may share one or more of the pipelines. In one embodiment of the present invention each linear array is connected to two or less pipeline buses, in others there are more than two pipeline buses per linear array. In one embodiment of the present invention with two pipeline buses, each pipeline bus is shared by two or less linear arrays, where one of said linear arrays can only emit energy emissions into said pipeline bus and the other of the two said linear arrays can only absorb energy emissions from the pipeline bus to form a feedforward neural network. In others, both linear arrays can emit and absorb the energy emissions from the same pipeline bus to form a feedback neural network.

In the preferred embodiment each processor of the linear array comprises one or more of the following: input gate, output gate, random code generator, delay/memory circuit, integrator, adder, analog processor.

Each processor is generally locally synchronized with a global base frequency f.

The input and the output gates are controlled by the random code generator which opens or closes them to the respective bus pipelines according to a random binary code at integer multiples of the inverse of the base frequency. When the input gate is open it admits energy emissions from the input bus pipeline. When the output gate is open energy emissions of predetermined amplitude enter the output pipeline bus. When the gates are closed both emission and absorption of the energy emissions do not occur.

The delay/memory circuit controls the timing and the length of the periodic activation of the random code generator within certain period of time T.sub.p /f, where T.sub.p is the integer number of pulses of time duration 1/f, called the pulse length. In one embodiment of the present invention the timing t.sub.d, called delay time, and the length of activation, called the code length, are stored within the delay/memory circuit as two integer numbers of pulses each of time duration 1/f signifying the timing when the random code generator is turned on, and the duration of its activation. In other embodiments the timing and the length of activation are supplied externally.

During its activation, the random code generator generates a random binary code which controls opening and closing of the gates. The total number of nonzero values of the code is controlled by the processor's external input from pipeline bus and by the average value of the generated code function.

In this particular embodiment of the present invention, the integrators convert piece-wise continuous functions corresponding to the binary codes into continuous functions by averaging the code functions over the time equal to pulse length T.sub.p /f. In others, the averaging is over the time interval equal to the code length T.sub.c /f.

Analog processors apply non-linear processing to their inputs which are continuous functions supplied by the integrators.

Adders perform linear and non-linear analog summations of their inputs.

Smallness of said processors and low power consumption provide for capability for advantageously large number of processors to be linked in the dynamically determined interactive neural network, creating a massively parallel data processing device. The use of common bus pipelines provides for an advantageously high fanout per processor.

In certain embodiments of the present invention, the apparatus is incorporated into an automated pattern recognition device, called Data Recognition Device, where it is used for extraction of features of patterns from data flows to which said Data Recognition Device is subjected. In other embodiments it is incorporated into said Data Recognition Device as a content addressable memory for storage and retrieval of the detected pattern features.

Various other features of the method and the apparatus of the present invention will become obvious to those skilled in the art upon reading the disclosure set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of a preferred embodiment of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead is placed upon illustrating the principles of the invention.

FIG. 1 is an illustration of the bus interconnection structure of a two-dimensional array of processors.

FIG. 2 is a block-diagram of a single processor with arrowed lines depicting flow of control signals.

FIG. 3 illustrates two sample random codes and their second correlation.

FIG. 4 illustrates averaging of a code by an integrator.

FIG. 5 illustrates non-linear processing by an analog processor.

FIG. 6 illustrates a typical energy emission by a processor.

PREFERRED MODE OF THE INVENTION

Referring now to FIG. 1 one example of the embodiment of the present invention is shown as a linear array 1 of the processors 2 on a single chip. In this embodiment of the present invention, each processor 2 is connected to one bus pipeline 3 with connector 4 for absorption and to the second pipeline bus 3 with connector 5 for emission of the energy emissions from said pipeline bus to form a feedforward neural network. In other embodiments connectors 4 and 5 are connecting said processor to both of said pipeline buses to form a feedback neural network. Connections 7 provide for external control access, for sampling pipeline bus signals, and for connection to additional processor assembly chips. Connections 9 are for the input signals, and connections 10 are for output signals.

Referring now to FIG. 2 a block diagram of a single processor 2 is shown. Base clock frequency f is supplied to the delay/memory circuit 6 through connections 7 as shown on FIG. 3. The first few pulses, up to a hundred in number, are reserved for a binary address of the processor among the processors belonging to the same linear array and data for changing parameters of the addressed processor. This binary address and data are not detected by circuits 20 and 32 discussed below. Said delay/memory circuit, with a period equal to pulse length, periodically activates the random code generator 8 for a time duration T.sub.c, called code length. In the present embodiment of this invention the code lengths of different processors 2 of the array 1 are the same, in others they vary from processor to processor. Said activation occurs with a delay t.sub.d which is an integer multiple of 1/f. The integer numbers corresponding to the duration of said activation T.sub.c and said delay t.sub.d are stored internally in the delay/memory circuit. For a code length T.sub.c =10,000 and delay range of variation equal to 10,000, the amount of the required memory is approximately 14 bits, which is advantageously low. In other embodiments of the present invention the timing of the delay and the duration of the activation are controlled externally through access connections 7 by assigning a different address to each processor in a given linear array. For 10,000 processors per linear array the address requires the advantageously low number of 14 bits.

During the period of its activation T.sub.c /f random code generator 8 generates a binary random code by advantageously utilizing the noise present in the system. For each processor P this binary code may be described as a piece-wise constant function of time D.sub.P (t), whose discontinuities occur only at integer multiples of 1/f. When D.sub.P (t)=0, the corresponding gate is closed; when D.sub.P (t)=1, the corresponding gate is open. The average value of the code P determines its short term memory trace by the formulas ##EQU2## For any two processors P and Q the corresponding long term memory trace is defined as the ratio of the second correlation of the two codes to the product of the averages of the respective codes

D.sub.PQ (t)=<D.sub.P (t) D.sub.Q (t)>/F.sub.P (t) F.sub.Q (t) Random code generator 8 comprises a memory cell, which holds the integer number equal to the number of positive pulses within the code length T.sub.c of its code function D.sub.P (t). The code function generated by the random number generator 8 is updated in real time pulse by pulse according to the formula ##EQU3## where G.sub.P (t) is a function of the external input and F.sub.P (t), sign G.sub.P (t) is the sign of the function G.sub.P (t). Function S.sub.P (t) is determined by ##EQU4##

In this particular embodiment of the present invention said updating corresponds to the pulse by pulse updating generated by the following algebra

0+(-1)=0 0+(+1)=1 1+(-1)=0 1+(+1)=1 where (0,1) represent code function D.sub.P (t) values and (-1,+1) represent the sign of the update function G.sub.P (t). In other embodiments other choices of the sixteen possible local algebras are used. In yet others, non-local updating algebras are used.

Randomness of the codes ensures that for large samples, when t.sub.d =0, T.sub.c =T.sub.p, for any two processors P and Q

<D.sub.P (t) D.sub.Q (t)>=<D.sub.P (t)><D.sub.Q (t)> For large codes said updating results in the following short and long term memory equations d/dt F.sub.P (t)=<G.sub.P (t) S.sub.P (t)> d/dt D.sub.PQ (t)=0 D.sub.PQ (t)=T.sub.p (t.sub.d.sup.P -t.sub.d.sup.R +T.sub.c)/T.sub.c.sup.2 where without loss of generality the delay time t.sub.d.sup.P of processor P is larger or equal to the delay time t.sub.d.sup.Q of processor Q.

The random binary code D.sub.PQ (t) generated by random code generator 8 controls opening and closing of input gate 20 and output gate 28.

The controlling signal from random code generator 8 to input gate 20 may be subjected to operation of inversion by inverter 10, which replaces all zeros of a code with ones and all one with zeros.

The controlling signal from random code generator 8 to output gate 28 is separated into two signals, one of which undergoes integration by integrator 18, as shown in FIG. 4, and subsequently undergoes analog processing by analog processor 14, as shown on FIG. 5. The output signal E.sub.18 of integrator 18 has the form

E.sub.18 (t)=F.sub.P (t) Output gate 28 comprises a source of energy emissions and a modulator. In this embodiment the energy emissions of the source are composed of electrons, in others they are composed of photons. When the code function D.sub.P (t) is zero the source does not emit; during the time when said code function is equal to one, source emits energy with amplitude B.sub.P (t) determined by the output signal of analog processor 14 according to the formula B.sub.P (t)=s(F.sub.P (t)) where the general form of function s(F.sub.P) is shown on FIG. 5. FIG. 6 shows a typical time dependence of the emitted signal.

The binary signal generated by random code generator 8 and processed by integrator 18 is sent to analog processor 16. Integrator 18 and analog processor 16, in their purpose are identical to integrators 22, 32 and analog processors 14, 24, 30. The output signal of analog processor 16 has the form

E.sub.16 (t)=h(F.sub.P (t)) where h(F.sub.P) is some advantageously chosen function.

The input signal from pipeline bus 3 is separated into two signals, one of which goes to integrator 32 and analog processor 30. Second controlling signal arrives to analog processor 30 from integrator 18. The input signals to the gate 20 and integrator 32 have the form ##EQU5## The output signal of analog processor 30 has the form ##EQU6## where g(F.sub.P) is some advantageously chosen function.

The second of said two signals passes through gate 20. Referring now to FIG. 3, for a code function D.sub.P (t) given by graph 3a) and for the pipeline bus input signal given by graph 3b), the output signal of the gate 20 is given by the graph 3c) or by the formula ##EQU7## Thus, the decoded input is a function of the aggregate energy on the bus from plural processors Q. The output signal from the gate 20 is averaged by integrator 22 and further processed by analog processor 24. Analog processor 24 receives the second controlling signal from integrator 18. The output signal of the processor 24 is given by the formula

E.sub.24 (t)=f(F.sub.P (t))<E.sub.20 (t)> where f(F.sub.P (t)) is some advantageously chosen function.

The signals from analog processors 24, 30 are linearly combined in adder 26, and are non-linearly modulated by the output from the integrator 18. The output signal of adder 26 has the following form ##EQU8## The output signal of adder 26 controls updating of the code generated by the random code generator 8 by controlling the total number of positive pulses generated for the new code. The average of the binary code output of the random code generator 8 is given according to the formula ##EQU9## When the code length T.sub.c and the pulse length T.sub.p are large we can use the formula ##EQU10## and for any two processors the output of the random code generator 8 is updated by the formula ##EQU11## where we chose g(F.sub.P)=s'(F.sub.P), f(F.sub.P)=s'(F.sub.P)F.sub.p and substituted s'(F.sub.Q) instead of F.sub.Q s(F.sub.Q) according to our choice of function s(F.sub.Q).

The formula above describes a general cooperative competitive neural network with constant long term memory traces, which include Hopfield neural networks, where external input denotes the input signals through access interconnections 7.

By varying the timing of the delay in the delay/memory circuit 6 and the duration of activation of random code generator 8 one can advantageously change the long term memory traces D.sub.PQ. In one embodiment of the present invention changing of the long term memory traces is controlled externally, in others it occurs internally as the result of internal controlling of said delays and code lengths T.sub.c.

By making delay time t.sub.d equal to the code pulse T.sub.p one can turn off some of the processors in the assembly and, therefore, change the connectivity pattern among the plurality of the processors of the neural network.

Claims

1. A circuit assembly forming an adaptive communications network with changeable connectivity pattern, comprising:

an assembly comprising a plurality of processors each comprising an energy detector, an energy source, and an analog logical circuit;

a common energy data channel into which energy sources emit and from which energy detectors detect energy signals;

means for feeding external data into said energy data channel;

means for modulation of the energy signals emitted by said energy sources as a function of output codes assigned to each said processor;

means for demodulation of the energy detected by said energy detectors, as a function of input codes assigned to each said processor to provide demodulated inputs which are functions of the input codes and of the aggregate energy in the energy channel from plural processors and received by the energy detectors; and

means for changing said input codes and output codes according to a rule which depends on the demodulated input of each of said energy detectors.

2. A circuit assembly as defined in claim 1, wherein the assembly of processors is a two dimensional array.

3. A circuit assembly as defined in claim 1, wherein said input and said output codes are binary codes.

4. A circuit assembly as defined in claim 1, wherein said input and said output codes are integer-valued codes.

5. A circuit assembly as defined in claim 1, wherein said rule for changing said input codes depends on said demodulated input averaged over a certain time.

6. A circuit assembly as defined in claim 1 wherein the input codes and output codes are functions of a random binary code.

7. A method of changing the magnitude of information exchange among a plurality of processors comprising:

representing said information in the form of modulated energy;

arranging said processors to emit said energy into a common energy channel;

arranging each processor to detect energy from said common energy channel by a detector;

encoding said emitted energy as a function of output codes assigned to each said processor;

decoding said detected energy as a function of input codes assigned to each said processor to provide a decoded input which is a function of the input codes and of the aggregate energy in the energy channel from plural processors and received by the detector; and

making changes in said input codes and output codes as a function of the decoded input of each said processor.

8. A method of changing the magnitude of information exchange among a plurality of processors as defined in claim 7, wherein said input and said output codes are binary.

9. A method of changing the magnitude of information exchange among a plurality of processors as defined in claim 7, wherein said input codes and said output codes are integer valued.

10. A method as defined in claim 7 wherein the input codes and output codes are functions of a random binary code.

11. An adaptive communications network comprising;

a plurality of processors; and

an energy communication channel coupling the processors;

each of a plurality of processors transmitting energy into the communication channel as a function of its present state, and each of a plurality of processors receiving a portion of the aggregate energy, transmitted by plural processors and received by a detector, as a function of its present state, the state of each processor being adapted to the received portion of the aggregate energy from plural transmitting processors over time.

12. A network as claimed in claim 11 wherein the received portion of the aggregate energy over time is a function of the match in an input switching code of the receiving processor and output switching codes of plural transmitting processors.

13. A network as claimed in claim 11 further comprising means for selectively enabling groups of processors to provide a changeable pattern of processors connected through the channel.

14. A network as claimed in claim 13 wherein the processors are selectively enabled by addressing the processors through the communication channel at times when aggregate energy is not detected by the detector.

15. A network of changeable connectivity pattern comprising:

a communication channel;

a plurality of processors which communicate by simultaneously broadcasting energy in the communication channel; and

means for selectively enabling individual processors to detect aggregate energy simultaneously broadcast from a plurality of processors through the communication channel.

16. A network of changeable connectivity pattern comprising:

a communication channel;

a plurality of processors which communicate by broadcasting energy in the communication channel; and

means for selectively enabling processors to detect energy from the communication channel; wherein the processors are selectively enabled by addressing the processors through the communication channel at times when aggregate energy is not detected by the detector.