Patent Application
20190026627
One or more aspects of embodiments according to the present invention relate to artificial neural networks, and more particularly to a variable precision neuromorphic architecture.
Artificial neural networks (or, as used herein, simply “neural networks”) may perform machine learning and decision-making using data processing that may be computationally costly, e.g., including significant numbers of multiply accumulate (MAC) operations. This computational cost may result in slow processing, or in high power consumption and equipment cost if speed is to be improved.
Thus, there is a need for an improved artificial neural network.
Aspects of embodiments of the present disclosure are directed toward a neuromorphic architecture for providing variable precision in a neural network, through programming. Logical pre-synaptic neurons are formed as configurable sets of physical pre-synaptic artificial neurons, logical post-synaptic neurons are formed as configurable sets of physical post-synaptic artificial neurons, and the logical pre-synaptic neurons are connected to the logical post-synaptic neurons by logical synapses each including a set of physical artificial synapses. The precision of the weights of the logical synapses may be varied by varying the number of physical pre-synaptic artificial neurons in each of the logical pre-synaptic neurons, and/or by varying the number of physical post-synaptic artificial neurons in each of the logical post-synaptic neurons.
According to an embodiment of the present invention there is provided a neural network, including: a plurality of pre-synaptic artificial neurons; a plurality of post-synaptic artificial neurons; and a plurality of artificial synapses, each of the artificial synapses being connected between a respective pre-synaptic artificial neuron of the pre-synaptic artificial neurons and a respective post-synaptic artificial neuron of the post-synaptic artificial neurons, each of the artificial synapses having a respective weight, each of the pre-synaptic artificial neurons including a respective multiplying circuit programmable to amplify its output signal by a gain factor selected from a set of N gain values being, respectively, A, 2A, 4A, . . . 2N-1A, wherein N is an integer greater than 1 and A is a constant, each of the pre-synaptic artificial neurons being programmed to amplify its output signal by a gain factor that is different from that of the other pre-synaptic artificial neurons, each of the post-synaptic artificial neurons including a respective multiplying circuit programmable to amplify its input signal, and each of the post-synaptic artificial neurons being programmed to amplify its output signal by a gain factor that is different from that of the other post-synaptic artificial neurons.
In one embodiment, each of the pre-synaptic artificial neurons is configured to produce, as an output signal, a voltage.
In one embodiment, each of the weights is a conductance of a resistive element.
In one embodiment, each resistive element is configured to operate in one of: a first state, in which the resistive element has a first conductance; and a second state, in which the resistive element has a second conductance different from the first conductance.
In one embodiment, each resistive element is a programmable resistive element within a spin-transfer torque random access memory cell.
In one embodiment, all of the weights have the same first conductance and all of the weights have the same second conductance.
In one embodiment, each of the post-synaptic artificial neurons is configured to receive, as an input signal, a current.
In one embodiment, each of the post-synaptic artificial neurons has a respective multiplying circuit programmable to amplify its output signal by a gain factor selected from a set of M gain values being, respectively, B, 2NB, 42NB, . . . 2(M-1)B, wherein M is an integer greater than 1 and A is a constant.
According to an embodiment of the present invention there is provided a neural network including: a plurality of logical pre-synaptic neurons; a plurality of logical post-synaptic neurons; and a plurality of logical synapses, a first logical pre-synaptic neuron of the logical pre-synaptic neurons having an input and including N pre-synaptic artificial neurons, N being an integer greater than 1, each of the N pre-synaptic artificial neurons having a respective input, all of the inputs of the pre-synaptic artificial neurons being connected to the input of the first logical pre-synaptic neuron, a first logical post-synaptic neuron of the logical post-synaptic neurons having an output and including: M post-synaptic artificial neurons, M being an integer greater than 1; and a summing circuit having: an output connected to the output of the first logical post-synaptic neuron, and a plurality of inputs, each of the M post-synaptic artificial neurons having a respective output, the output of each of the post-synaptic artificial neurons being connected to a respective input of the plurality of inputs of the summing circuit.
In one embodiment, each of the N pre-synaptic artificial neurons includes a respective multiplying circuit programmable to amplify its output signal by a gain factor selected from a set of N gain values being, respectively, A, 2A, 4A, . . . 2NA, wherein A is a constant.
In one embodiment, each of the M post-synaptic artificial neurons includes a respective multiplying circuit programmable to amplify its output signal by a gain factor selected from a set of M gain values being, respectively, B, 2NB, 42NB, . . . 2(M-1)NB, wherein A is a constant.
In one embodiment, all of the pre-synaptic artificial neurons differ only with respect to their respective programmed gain factors.
In one embodiment, all of the post-synaptic artificial neurons differ only with respect to their respective programmed gain factors.
In one embodiment, an input of each pre-synaptic artificial neuron is a digital input; the multiplying circuit of each pre-synaptic artificial neuron is a digital multiplying circuit connected to the input of the pre-synaptic artificial neuron; and each pre-synaptic artificial neuron further includes a digital to analog converter having an input connected to an output of the digital multiplying circuit and an output connected to an output of the pre-synaptic artificial neuron.
In one embodiment, an output of each post-synaptic artificial neuron is a digital output; the multiplying circuit of each post-synaptic artificial neuron is a digital multiplying circuit connected to the output of the post-synaptic artificial neuron; and each post-synaptic artificial neuron further includes an analog to digital converter having an input connected to an input of the post-synaptic artificial neuron and an output connected to an input of the digital multiplying circuit.
In one embodiment, the first logical post-synaptic neuron further includes a digital summing circuit having M inputs each connected to a respective one of the outputs of the M post-synaptic artificial neurons and an output connected to the output of the first logical post-synaptic neuron.
In one embodiment, each of the pre-synaptic artificial neurons is configured to produce, as an output signal, a voltage; each of the logical synapses includes a plurality of artificial synapses, each of the artificial synapses having a respective weight, each weights being a conductance of a resistive element; and each of the post-synaptic artificial neurons is configured to receive, as an input signal, a current.
In one embodiment, each resistive element is configured to operate in one of: a first state, in which the resistive element has a first conductance; and a second state, in which the resistive element has a second conductance different from the first conductance.
In one embodiment, each resistive element is a programmable resistive element within spin-transfer torque random access memory cell.
According to an embodiment of the present invention there is provided a neural network, including: a plurality of pre-synaptic artificial neurons; a plurality of post-synaptic artificial neurons; and means for forming a plurality of connections, each connection being between a respective pre-synaptic artificial neuron of the pre-synaptic artificial neurons and a respective post-synaptic artificial neuron of the post-synaptic artificial neurons, each of the pre-synaptic artificial neurons including a respective multiplying circuit programmable to amplify its output signal by a gain factor selected from a set of N gain values being, respectively, A, 2A, 4A, . . . 2N-1A, wherein N is an integer greater than 1 and A is a constant, each of the pre-synaptic artificial neurons being programmed to amplify its output signal by a gain factor that is different from that of the other pre-synaptic artificial neurons, each of the post-synaptic artificial neurons including a respective multiplying circuit programmable to amplify its input signal, and each of the post-synaptic artificial neurons being programmed to amplify its output signal by a gain factor that is different from that of the other post-synaptic artificial neurons.
These and other features and advantages of the present invention will be appreciated and understood with reference to the specification, claims, and appended drawings wherein:
The detailed description set forth below in connection with the appended drawings is intended as a description of exemplary embodiments of a variable precision neuromorphic architecture provided in accordance with the present invention and is not intended to represent the only forms in which the present invention may be constructed or utilized. The description sets forth the features of the present invention in connection with the illustrated embodiments. It is to be understood, however, that the same or equivalent functions and structures may be accomplished by different embodiments that are also intended to be encompassed within the spirit and scope of the invention. As denoted elsewhere herein, like element numbers are intended to indicate like elements or features.
Referring to
Each of the post-synaptic artificial neurons 110 may have at its input a circuit such as the transimpedance amplifier of
A modified circuit, such as that of
In some embodiments each weight is controllable or programmable to operate at any time in one of two states, e.g., a high-resistance state and a low-resistance state. Each such weight may be implemented or constructed, for example, as the programmable resistive element within a spin-transfer torque random access memory (STT-RAM) cell (e.g., an STT-RAM cell based on a magnetic tunneling junction (MTJ) device). Accordingly, in an embodiment such as that of
Artificial synapses 115 with relatively low precision, e.g., with two or three states, such as those illustrated in
Referring to
Referring to
In some embodiments, in a given layer all of the pre-synaptic artificial neurons 105 may be identical (except for the respective gain factors of their respective multipliers (i.e., they may differ only with respect to these gain factors)), all of the post-synaptic artificial neurons 110 may be identical (except for the respective gain factors of their respective multipliers (i.e., they may differ only with respect to these gain factors)), and all of the synapses may be identical (except for the respective programmed weights). As such, a neural network may be fabricated in which each layer has weights with a bit precision that may be selected, after fabrication, by suitable programming. Such a neural network may be said to have a neuromorphic architecture.
Referring to
Referring to
Referring to
In light of the foregoing, some embodiments provide a neuromorphic architecture for providing variable precision in a neural network, through programming. Logical pre-synaptic neurons are formed as configurable sets of physical pre-synaptic artificial neurons, logical post-synaptic neurons are formed as configurable sets of physical post-synaptic artificial neurons, and the logical pre-synaptic neurons are connected to the logical post-synaptic neurons by logical synapses each including a set of physical artificial synapses. The precision of the weights of the logical synapses may be varied by varying the number of physical pre-synaptic artificial neurons in each of the logical pre-synaptic neurons, and/or by varying the number of physical post-synaptic artificial neurons in each of the logical post-synaptic neurons.
Each of the digital circuits mentioned herein may be, or may be a portion of, a processing circuit. The term “processing circuit” is used herein to mean any combination of hardware, firmware, and software, employed to process data or digital signals. Processing circuit hardware may include, for example, application specific integrated circuits (ASICs), general purpose or special purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field programmable gate arrays (FPGAs). In a processing circuit, as used herein, each function is performed either by hardware configured, i.e., hard-wired, to perform that function, or by more general purpose hardware, such as a CPU, configured to execute instructions stored in a non-transitory storage medium. A processing circuit may be fabricated on a single printed circuit board (PCB) or distributed over several interconnected PCBs. A processing circuit may contain other processing circuits; for example a processing circuit may include two processing circuits, an FPGA and a CPU, interconnected on a PCB.
It will be understood that, although the terms “first”, “second”, “third”, etc., may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed herein could be termed a second element, component, region, layer or section, without departing from the spirit and scope of the inventive concept.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the inventive concept. As used herein, the terms “substantially,” “about,” and similar terms are used as terms of approximation and not as terms of degree, and are intended to account for the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art.
As used herein, the singular forms “a” and “an” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising”, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of,” when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list. Further, the use of “may” when describing embodiments of the inventive concept refers to “one or more embodiments of the present invention”. Also, the term “exemplary” is intended to refer to an example or illustration. As used herein, the terms “use,” “using,” and “used” may be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively.
It will be understood that when an element or layer is referred to as being “on”, “connected to”, “coupled to”, or “adjacent to” another element or layer, it may be directly on, connected to, coupled to, or adjacent to the other element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly on”, “directly connected to”, “directly coupled to”, or “immediately adjacent to” another element or layer, there are no intervening elements or layers present.
Although exemplary embodiments of a variable precision neuromorphic architecture have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Accordingly, it is to be understood that a variable precision neuromorphic architecture constructed according to principles of this invention may be embodied other than as specifically described herein. The invention is also defined in the following claims, and equivalents thereof.
The present application claims priority to and the benefit of U.S. Provisional Application No. 62/535,187, filed Jul. 20, 2017, entitled “VARIABLE PRECISION NEUROMORPHIC ARCHITECTURE”, the entire content of which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62535187 | Jul 2017 | US |
