Patent Application
20230297816
The present invention relates to the field of digital computer systems, and more specifically, to a resonator network system.
Various embodiments provide a method and resonator network system and computer program product as described by the subject matter of the independent claims. Advantageous embodiments are described in the dependent claims. Embodiments of the present invention can be freely combined with each other if they are not mutually exclusive.
In accordance with an embodiment, a system is presented. The system includes a neuromorphic memory device with a crossbar array structure that includes row lines and column lines. The neuromorphic memory device further includes resistive memory elements coupled between the row lines and the column lines at junctions formed by the row and column lines. The resistive memory elements of each set of row lines store a value of respective code hypervectors of a codebook. Each codebook includes candidate code hypervectors. The system further includes an input buffer that stores an input hypervector. The system further includes a plurality of estimate buffers each associated with a different subset of row lines and a different codebook and initially store estimated hypervectors that are derived from respective code hypervectors that are stored in the associated subset of row lines. The system further includes a control circuit connected to the neuromorphic memory device. The control circuit computes an unbound hypervector using the input hypervector and all the estimated hypervectors, computes an attention vector that indicates a similarity of the unbound hypervector with one estimated hypervector, computes a linear combination of the one estimated hypervector, weighted by the attention vector, and stores the computed linear combination in the estimate buffer that is associated with the one estimated hypervector.
In accordance with another embodiment, a method for factorizing an input hypervector is presented. The method includes storing respective code hypervectors of an associated codebook within row lines of a neuromorphic memory device with a crossbar array structure that includes the row lines and column lines and resistive memory elements coupled between the row lines and the column lines at junctions formed by the row and column lines. Each associated codebook includes candidate code hypervectors. The method further includes storing an input hypervector in an input buffer. The method further includes storing estimated hypervectors within a plurality of estimate buffers each associated with a different subset of row lines and a different codebook. The estimated hypervectors are derived from respective code hypervectors that are stored in the associated subset of row lines. The method further includes computing an unbound hypervector using the input hypervector and all the estimated hypervectors. The method further includes computing an attention vector that indicates a similarity of the unbound hypervector with one estimated hypervector. The method further includes computing a linear combination of the one estimated hypervector, weighted by the attention vector. The method further includes storing the computed linear combination in the estimate buffer that is associated with the one estimated hypervector.
In accordance with yet another embodiment, a computer program product is presented. The computer program product includes a computer-readable storage medium having computer-readable program code embodied therewith. The computer-readable program code, when evoked by a processor, causes the processor to store respective code hypervectors of an associated codebook within row lines of a neuromorphic memory device with a crossbar array structure that includes the row lines and column lines and resistive memory elements coupled between the row lines and the column lines at junctions formed by the row and column lines. Each associated codebook includes candidate code hypervectors. The computer-readable program code, when evoked by a processor, further causes the processor to store an input hypervector in an input buffer and store estimated hypervectors within a plurality of estimate buffers each associated with a different subset of row lines and a different codebook. The estimated hypervectors are derived from respective code hypervectors that are stored in the associated subset of row lines. The computer-readable program code, when evoked by a processor, further causes the processor to compute an unbound hypervector using the input hypervector and all the estimated hypervectors and further causes the processor to compute an attention vector that indicates a similarity of the unbound hypervector with one estimated hypervector. The computer-readable program code, when evoked by a processor, further causes the processor to compute a linear combination of the one estimated hypervector, weighted by the attention vector and further causes the processor to store the computed linear combination in the estimate buffer that is associated with the one estimated hypervector.
In the following embodiments of the invention are explained in greater detail, by way of example only, making reference to the drawings in which:
The descriptions of the various embodiments of the present invention will be presented for purposes of illustration but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Resonator networks as known by one of ordinary skill in the art are discussed in the following references: E. Paxon Frady et al. “Resonator networks for factoring distributed representations of data structures”, Neural Computation 2020 and Spencer J. Kent et al. “Resonator Networks outperform optimization methods at solving high-dimensional vector factorization,” Neural Computation 2020. Given a hypervector formed from an element-wise product of two or more atomic hypervectors (each from a fixed codebook), the resonator network may find its factors. The resonator network may iteratively search over the alternatives for each factor individually rather than all possible combinations until a set of factors is found that agrees with the input hypervector.
Data structures may represent cognitive concepts, such as colors, shapes, positions, etc. Each cognitive concept may comprise items e.g., items of the color concept may comprise red, green, blue etc. The data structure may contain a combination (e.g., product) of multiple components that each represent a cognitive concept. For example, the data structure may be an image of a red disk in the bottom right and a green rectangle in the top left, wherein the cognitive concepts may be the color, shape, size, and position. In another example, a data structure may form a distributed representation of a tree, wherein each leaf in the tree may represent a concept, and each type of traversal operation in the tree may represent concepts.
The data structure may be encoded by an encoder into a hypervector that uniquely represents the data structure. A hypervector may be a vector of bits, integers, real or complex numbers. The hypervector is a vector having a dimension higher than a minimum dimension, e.g., 100. The hypervector may be holographic with independent and identically distributed (i.i.d) components. The hypervector being holographic means that each bit position in the hypervector may have an equal weight, in contrast to a conventional model with most significant bits and least significant bits. The encoder may combine hypervectors that represent individual concepts with operations in order to represent a data structure. For example, the above mentioned image may be described as a combination of multiplication (or binding) and addition (or superposition) operations as follows: (bottom right*red*disk)+(top left*green*rectangle). The encoder may represent the image using hypervectors that represent the individual concepts and associated operations to obtain the representation of the image as a single hypervector that distinctively represents the knowledge that the disk is red and placed at the bottom right and the rectangle is green and placed at the top left. The encoder may be defined by a vector space of a set of hypervectors which encode a set of cognitive concepts and algebraic operations on this set. The algebraic operations may, for example, comprise a superposition operation and a binding operation. In addition, the algebraic operations may comprise a permutation operation. The vector space may, for example, be a D-dimensional space, where D>100. The hypervector may be a D-dimensional vector comprising D numbers that define the coordinates of a point in the vector space. The D-dimensional hypervectors may be in {−1, +1}D and thus may be referred to as “bipolar.” For example, a hypervector may be understood as a line drawn from the origin to the coordinates specified by the hypervector. The length of the line may be the hypervector's magnitude. The direction of the hypervector may encode the meaning of the representation. The similarity in meaning may be measured by the size of the angles between hypervectors. This may typically be quantified as a dot product between hypervectors. The encoder may be a decomposable (i.e., factored) model to represent the data structures. This may be advantageous as the access to the hypervectors may be decomposed into the primitive or atomic hypervectors that represent the individual items of the concepts in the data structure. For example, the encoder may use a Vector Symbolic Architecture (VSA) technique in order to represent the data structure by a hypervector. The encoder may enable to perform an elementwise multiply operation. The encoder may, for example, comprise a trained feed-forward neural network.
Hence, the encoding of data structures may be based on a predefined set of F concepts, where F>1 and candidate items that belong to each of the F concepts. Each candidate item may be represented by a respective hypervector. Each concept may be represented by a matrix of the hypervectors representing candidate items of the concept, e.g., each column of the matrix may be a distinct hypervector. The matrix may be referred to as codebook and the hypervector representing one item of the concept may be referred to as code hypervector. The components of the code hypervector may, for example, be randomly chosen. For example, a codebook that represents the concept of colors may include seven possible colors as candidate items, a codebook that represents the concept of shapes may include 26 possible shapes as candidate items, etc. The codebooks that represent the set of concepts may be referred to as X1, X2 . . . XF respectively. Each codebook Xi may comprise Mx
Querying such data structures through their hypervector representations may require decoding the hypervectors. Decoding such hypervectors may be performed by testing every combination of code hypervectors. However, this may be resource intensive. The present subject matter may solve this issue by using a resonator network system. The resonator network system may utilize an iterative approach. In particular, the resonator network system can efficiently decode a given hypervector without needing to directly test every combination of factors, making use of the fact that the superposition operation is used for the encoding of multiple concept items in the given hypervector and the fact that randomized code hypervectors may be highly likely to be close to orthogonal in the vector space, meaning that they can be superposed without much interference. For that, the resonator network system may search for possible factorizations of the given hypervector by combining a strategy of superposition and clean-up memory. The clean-up memory may reduce some crosstalk noise between the superposed concept items. The resonator network system combines the strategy of superposition and clean-up memory to efficiently search over the combinatorially large space of possible factorizations.
Assuming for a simplified description of the iterative process of the resonator network system that the set of concepts comprises three concepts i.e., F=3, but it is not limited thereto. The codebooks/matrices representing the set of concepts may be referred to as X, Y and Z respectively (i.e., X=X1, Y=X2 and Z=X3.). The codebook X may comprise Mx code hypervectors x1 . . . xM
The iterative process may be implemented according to the present subject matter using the resonator network system. The resonator network system may include a neuromorphic memory device. The neuromorphic memory device is provided with a crossbar array structure comprising row lines and column lines and resistive memory elements coupled between the row lines and the column lines at junctions formed by the row and column lines. The row lines may be split into sets of rows lines associated with the set of F concepts respectively. For example, for three concepts, the first set of Mx consecutive row lines may be assigned to the first concept (X), the subsequent second set of My consecutive row lines may be assigned to the second concept (Y) and the subsequent third set of Mz consecutive row lines may be assigned to the third concept (Z). The resistive memory elements that belong to the first set of row lines may represent values of the code hypervectors of the codebook X (that is, the matrix X may be stored in the resistive memory elements of the first set of row lines). The resistive memory elements that belong to the second set of row lines may represent values of the code hypervectors of the codebook Y (that is, the matrix Y may be stored in the resistive memory elements of the second set of row lines). The resistive memory elements that belong to the third set of row lines may represent values of the code hypervectors of the codebook Z (that is, the matrix Z may be stored in the resistive memory elements of the first set of row lines). The neuromorphic memory device may, thus, enable to store the values of the codebooks and to perform in-memory operations on those values. This may provide a clean-up memory that is built from the codebooks X, Y and Z, which contain all the code hypervectors that are possible factors of the inputs.
Given the hypervector s that represents the data structure and given the set of predefined concepts, an initialization step may be performed by initializing an estimate of the hypervector that represents each concept of the set of concepts. The initial estimates {circumflex over (x)}(0), ŷ(0) and {circumflex over (z)}(0) may, for example, be defined as a superposition of all candidate code hypervectors of the respective concept e.g., {circumflex over (x)}(0)=sign(Σi=1, . . . , M
And, for each current iteration t of the iterative process, the following may be performed. Unbound hypervectors {tilde over (x)}(t), {tilde over (y)}(t) and {tilde over (z)}(t) may be computed. Each of the unbound hypervectors may be an estimate of the hypervector that represents the respective concept of the set of concepts. Each of the unbound hypervectors may be inferred from the hypervector s based on the estimates of hypervectors for the other remaining F−1 concepts of the set of concepts which are currently stored in the set of estimate buffers. The unbound hypervectors may be computed as follows: {tilde over (x)}(t)=s⊙ŷ(t)└{circumflex over (z)}(t)), {tilde over (y)}(t)=s⊙{circumflex over (x)}(t)⊙{circumflex over (z)}(t) and {tilde over (z)}(t)=s⊙{circumflex over (x)}(t)⊙ŷ(t), where ⊙ refers to elementwise multiplication (i.e., the Hadamard product). This first definition of the unbound hypervectors may particularly be advantageous if the estimate hypervectors {circumflex over (x)}(t), ŷ(t) and {circumflex over (z)}(t) may be computed in parallel and the unbound hypervectors may be computed in parallel.
However, in case the estimate hypervectors {circumflex over (x)}(t), ŷ(t) and {circumflex over (z)}(t) are computed sequentially (referred to as sequential case) e.g., {circumflex over (x)}(t) followed by ŷ(t) and then {circumflex over (z)}(t), the unbound hypervectors may be computed as follows: {tilde over (x)}(t)=s⊙ŷ(t)⊙{circumflex over (z)}(t), {tilde over (y)}(t)=s⊙{circumflex over (x)}(t+1)⊙{circumflex over (z)}(t) and {tilde over (z)}(t)=s⊙{circumflex over (x)}(t+1)⊙ŷ(t+1). This second definition of the unbound hypervectors may simplify the resonator system structure but still provide accurate results. For example, considering large problem sizes where it takes larger number of time steps to converge, there may be little effect whether {circumflex over (x)}(t+1) or {circumflex over (x)}(t) is used for the computation of ŷ(t). For that, the resonator network system may comprise a circuit that is configured to read the input buffer and the set of estimate buffers and to apply the elementwise multiplications. This may be referred to as an inference step. The inference step may, however, be noisy if many estimates (e.g., F−1 is high) are tested simultaneously.
The unbound hypervectors {tilde over (x)}(t), {tilde over (y)}(t) and {tilde over (z)}(t) may be noisy. This noise may result from crosstalk of many quasi-orthogonal code hypervectors and may be reduced through clean-up memory. After providing the unbound version of a hypervector of a given concept, clean-up memory may be used to find the similarity of each code hypervector of said concept to the unbound version of the hypervector. This may be referred to as a similarity step. The similarity may be computed as a dot product of the codebook that represents said concept by the unbound version of the hypervector, resulting in an attention vector ax(t), ay(t) and az(t) respectively.
The attention vector may be referred to herein as similarity vector. The similarity vectors ax(t), ay(t) and az(t) have sizes Mx, My and Mz respectively and may be obtained as follows: ax(t)=XT {tilde over (x)}(t)∈M
M
M
A weighted superposition of the similarity vectors ax(t), ay(t) and az(t) may be performed. This may be referred to as the superposition step. The superposition step may be performed on the similarity vectors ax(t), ay(t) and az(t) as follows: {circumflex over (x)}(t+1)=ax(t)X, ŷ(t+1)=ay(t)Y and {circumflex over (z)}(t+1)=az(t)Z respectively, in order to obtain the current estimates {circumflex over (x)}(t+1), ŷ(t+1) and {circumflex over (z)}(t+1) respectively of the hypervectors that represent the set of concepts. In other words, the superposition step generates each of the estimates {circumflex over (x)}(t+1), ŷ(t+1) and {circumflex over (z)}(t+1) representing the respective concept by a linear combination of the candidate code hypervectors (provided in respective matrices X, Y and Z), with weights given by the respective similarity vectors ax(t), ay(t) and az(t). For example, for computing the estimate hypervector {circumflex over (x)}(t+1), the similarity vector ax(t) may be input (back) to the first set of row lines so that the linear combination may be performed in-memory using the stored values of the codebook X. The elements of the estimate hypervector {circumflex over (x)}(t+1) may be obtained from the column lines respectively. For computing the estimate hypervector ŷ(t+1), the similarity vector ay(t) may be input (back) to the second set of row lines so that the linear combination may be performed in-memory using the stored values of the codebook Y. The elements of the estimate hypervector ŷ(t+1) may be obtained from the column lines respectively. For computing the estimate hypervector {circumflex over (z)}(t+1), the similarity vector az(t) may be input (back) to the third set of row lines so that the linear combination may be performed in-memory using the stored values of the codebook Z. The elements of the estimate hypervector {circumflex over (z)}(t+1) may be obtained from the column lines respectively.
The iterative process may stop if a convergence criterion is fulfilled. The convergence criterion may, for example, require that the value of at least one element of each similarity vector ax(t), ay(t) and az(t) exceeds a predefined threshold. In another example, the convergence criterion may require a predefined number of iterations to be reached.
The present subject matter may provide a micro architecture to implement a high dimensional vector factorizing resonator network system using crossbar arrays where dot product operations are in-memory. The present subject matter may efficiently factorize the hypervector representing a data structure into the primitives from which it is composed. For example, given a hypervector formed from an element-wise product of two or more hypervectors, its factors (i.e., the two or more hypervectors) may be efficiently found. This way, a nearest-neighbor lookup may need only search over the alternatives for each factor individually rather than all possible combinations. This may reduce the number of operations involved in every iteration of the resonator network system and hence reduce the complexity of execution. This may also solve larger size problems (at fixed dimensions) and improve the robustness against noisy input hypervectors.
The resonator network system 100 comprises a neuromorphic memory device 101 with a crossbar array structure of memristors that provide local data storage. The crossbar array structure includes r row lines 131.1-131.r (collectively referred to as 131) and D column lines 132.1-132.D (collectively referred to as 132), where the row lines 131 and column lines 132 are interconnected at junctions via electronic devices 133. The number of columns corresponds to the size of the hypervectors, where the number of row lines r may be equal to the number of all items represented by the codebooks e.g., r=Σi=1FMx
The row lines 131 may comprise F sets of consecutive row lines 135.1-135.F which are associated respectively with the F concepts. Each set of row lines 135.1-F may comprise a number of row lines that correspond respectively to the items of the respective concept. The column lines 132 may be associated with the D elements of the hypervector respectively. The electronic devices 133 that are at the junction of each set of row lines and the column lines may represent values of the codebook associated with said set of row lines. For example, electronic devices 133 that are at the junction of the set of row lines 135.i and the column lines 132 may represent values of the codebook Xi so that each column of the codebook Xi may be stored in the electronic devices 133 along a respective row line, where i varies between 1 and F, e.g., xi1 may be stored on the electronic devices 133 along the first row line of the set of row lines 135.i . . . and
may be stored in the Mx
The neuromorphic memory device 101 may comprise peripheral circuits 137 and 138. Each of the peripheral circuits 137 and 138 comprises elements that enable to input data and obtain output results. For example, each of the peripheral circuits 137-138 may comprise transimpedance amplifiers (TIAs) to convert collected currents to voltages and analog-to-digital converters (ADCs) to transform the voltages to the binary numbers. Each of the peripheral circuits 137-138 may, for example, comprise sense amplifiers and digital-to-analog converters (DACs) to convert binary numbers to input voltages.
The resonator network system 100 further comprises a control circuit 103. The control circuit 103 may comprise buffers and a computation logic 121. The buffers comprise an input buffer 123 for storing an input hypervector s which may represent a data structure such as an image. The buffers further comprise a set of F estimate buffers 128.1-128.F for storing the estimates of hypervectors (e.g., {circumflex over (x)}1 . . . {circumflex over (x)}F) of the set of F concepts respectively. The computation logic 121 is configured to compute an unbound vector using the input hypervector s and the corresponding estimate hypervectors e.g., the computation logic 121 may compute an unbound hypervector {tilde over (x)}i as follows: {tilde over (x)}i=s⊙{circumflex over (x)}1 . . . ⊙{circumflex over (x)}i−1⊙{circumflex over (x)}i+1 . . . ⊙{circumflex over (x)}F using the input hypervector s and the estimates of hypervectors (having index different from i, where i varies between 1 and F, the number of concepts). The F−1 estimates {circumflex over (x)}1 . . . {circumflex over (x)}i−1, {circumflex over (x)}i+1 . . . {circumflex over (x)}F, may be obtained from the corresponding estimate buffers 128.1-F using a selector 127. The computed unbound hypervector {tilde over (x)}i may be stored in temporal input buffer 124. The estimate hypervectors (e.g., {circumflex over (x)}1 . . . {circumflex over (x)}F) may, for example, be received as outputs of the neuromorphic memory device 101. The output of the neuromorphic memory device 101 may, for example, be stored in temporal output buffer 125 and may be stored on the corresponding estimate buffer 128.1-F using a selector 126.
Reading from the neuromorphic memory device 101 may be achieved via a vector-matrix multiplication which can be performed in constant time, meaning that data may possibly be retrieved during any operation cycle. In practice, however, read operations may only be performed at specific times (e.g., according to the general algorithm run at the control circuit 103) and not necessarily at each operation cycle. Generally, data can be retrieved via a multiply-accumulate operation which may be parameterized by values corresponding to the states of the electronic devices 133.
The resonator network system 100 may be configured to perform the iterative process of the resonator network in order to factorize the input hypervector s stored in the input buffer 123 into individual individual hypervectors representing the set of F concepts respectively.
The resonator network system 200 comprises a neuromorphic memory device 201 with a crossbar array structure of memristors that provide local data storage. The resonator network system 200 may support bipolar vectors and bipolar dot product as opposed to binary operations in resonator network system 100. The crossbar array structure includes row lines 231.1-231.r (collectively referred to as 231) and column lines 232.1-232.D (collectively referred to as 232). The row lines 231 may comprise F sets of consecutive row lines 235.1-235.F which are associated respectively with the F concepts. The row lines 231 and column lines 232 are interconnected at junctions via electronic devices 233 (e.g., memristive devices). For ease of illustration, only one electronic device 233 is labeled with a reference number in &Xl+
&
The neuromorphic memory device 201 may, for example, form a crossbar of PCM devices. Current through a single memristor device is a product of the voltage applied across it and its conductance value. Matrix-vector multiplication may be computed by applying voltages along the row lines/column lines of the neuromorphic memory device 201 and summing up currents (using Kirchhoff's current law) along the column lines/row lines. Net output current is usually measured using analog to digital converter (ADC) for further computations.
The neuromorphic memory device 201 may comprise peripheral circuits 237, 238 and 239. Each of the peripheral circuits 237, 238 and 239 comprises elements that enable to input data and/or obtain output results. For example, the peripheral circuit 239 may comprise transimpedance amplifiers (TIAs) to convert collected currents to voltages and ADCs to transform the voltages to the binary numbers. The peripheral circuit 237 or 238 may, for example, comprise DACs to convert binary numbers to input voltages.
The resonator network system 200 further comprises a control circuit 203. The control circuit 203 may comprise buffers and a computation logic 221. The buffers comprise an input buffer 223 for storing an input hypervector s which may represent a data structure such as an image. The buffers further comprise a set of F estimate buffers 228.1-228.F for storing the estimate hypervectors (e.g., {circumflex over (x)}1 . . . {circumflex over (x)}F) of the set of F concepts respectively. The computation logic 221 is configured to compute an unbound vector using the input hypervector s and the corresponding estimate hypervectors e.g., the computation logic 221 may compute an unbound hypervector {tilde over (x)}i as follows {tilde over (x)}i=s⊙{circumflex over (x)}1 . . . ⊙{circumflex over (x)}i−1 ⊙{circumflex over (x)}i+1 . . . ⊙{circumflex over (x)}F using the input hypervector s and the estimate hypervectors (having index different from i, where i varies between 1 and F the number of concepts). The F−1 estimate hypervectors {circumflex over (x)}1 . . . {circumflex over (x)}i−1, {circumflex over (x)}i+1 . . . {circumflex over (x)}F may be obtained from the corresponding estimate buffers 228.1-F using a selector 227. The computed unbound hypervector {tilde over (x)}i may be stored in temporal buffer 224. The estimate hypervectors (e.g., {circumflex over (x)}1 . . . {circumflex over (x)}F) may, for example, be received as outputs of the neuromorphic memory device 101. The output of the neuromorphic memory device 101 may, for example, be received from the peripheral circuit 239 and may be stored on the corresponding estimate buffer 228.1-F using a selector 226.
The resonator network system 200 may be configured to perform the iterative process of the resonator network in order to factorize the input hypervector s stored in the input buffer 223 into individual hypervectors representing the set of F concepts respectively.
In the initial state S301 of the resonator network system 100 as shown in
In the following state S302 as shown in
In state S303 as shown in
In state S304 as shown in
In state S305 as shown in
In state S306 as shown in
In state S307 as shown in
In state S308 as shown in
The application of the equal attentions to obtain each initial estimate hypervector is an exemplary method for obtaining that initial estimate hypervector but it is not limited thereto. For example, the initial estimate hypervectors may be stored (directly) e.g., by a user, in respective estimate buffers 128.1-3 without using the computation capability of the neuromorphic memory device 101.
In state S309 as shown in
In state S310 as shown in
In state S311 as shown in
In state S312 as shown in
In state S313 as shown in
In state S314 as shown in
In state S315 as shown in
In state S316 as shown in
In state S317 as shown in
Steps S309 to S317 may be repeated until a predefined convergence criterion is fulfilled. For example, steps S309 to S317 may be repeated until one of the attention values in each codebook exceeds a threshold near to the value 1. Given the range of attention values (close to equal attentions on all vectors in the codebook) that are applied on the bitlines, the values of the estimate hypervectors {circumflex over (z)} and {circumflex over (x)} may not change in two iterations as shown in
A data structure may be represented in block 401 by a hypervector s using an encoder such as a VSA based encoder. The data structure may, for example, be a query image representing a visual scene. The encoder may be a feed-forward neural network that is trained to produce the hypervector s as a compound hypervector describing the input visual image. The image may comprise colored MNIST digits. The components of the image may be the color, shape, vertical and horizontal locations of the letters in the image. The encoder may, for example, be configured to compute a hypervector for each letter in the image by multiplying the related quasi-orthogonal hypervectors drawn from four fixed codebooks of four concepts: color codebook (with 7 possible colors), shape codebook (with 26 possible shapes), vertical codebook (with 50 locations), and horizontal codebook (with 50 locations). The product vectors for every letter may be added (component-wise) to produce the hypervector s describing the whole image. The hypervector s may, for example, be stored in the input buffer 123.
The set of concepts comprises F concepts. The codebooks/matrices representing the set of concepts may be referred to as X1, X2 . . . XF respectively. Each codebook Xi may comprise Mx
The hypervector s may be estimated to be as follows s=x1α
For each concept i of the set of F concepts, blocks 403 to 413 may be performed.
An unbound version {tilde over (x)}i(t) of the hypervector {circumflex over (x)}i(t) representing the concept i may be computed in blocks 403. The index i refers to one concept that is being processed and the index t refers to the current iteration of the method. The unbound version of the hypervector {circumflex over (x)}i(t) may be referred to as the unbound hypervector {tilde over (x)}i(t). The unbound hypervector {tilde over (x)}i(t) may be computed by using the input hypervector s stored in the input buffer 123 and the first estimates of hypervectors of the concepts j other than the concept i and which are stored in the set of estimate buffers 128.j (i.e., j varies between 1 and F and j is different from i). The computation logic 221 may, for example, compute the unbound hypervector {tilde over (x)}i(t) as follows {tilde over (x)}i(t)=s⊙{circumflex over (x)}1(t) . . . └{circumflex over (x)}i−1(t)⊙{circumflex over (x)}i+1(t) . . . ⊙{circumflex over (x)}F(t).
The D elements of the unbound hypervector {tilde over (x)}i(t) may be input in block 405 to the column lines 132.1-D respectively. This may trigger the in-memory computation of the product {tilde over (x)}i(t)Xi using the memresistive electronic devices 133. This product {tilde over (x)}i(t)Xi defines the attention vector ax
The Mx
The attention vector ax
The D elements of the estimate hypervector {circumflex over (x)}i(t+1) may be received in block 411 through the column lines 132.1-D respectively.
The estimate hypervector {circumflex over (x)}i(t+1) may be stored in block 413 in the respective estimate buffer 128.i.
It may be determined (block 415) whether the attention vectors ax
In case the attention vectors ax
In case the attention vectors ax
The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be accomplished as one step, executed concurrently, substantially concurrently, in a partially or wholly temporally overlapping manner, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
