Patent Grant
11176043
The present invention relates to testing of a computer technology and, more specifically, to a distributed memory architecture.
Artificial neural networks (ANNs), such as deep neural networks have had a revolutionary impact on the field of computing technology, particularly, artificial intelligence. ANNs have facilitated achieving improved performance as well as results on tasks such as image recognition, speech recognition, translation, image captioning, etc. This, in turn, has resulted in efforts directed towards the design of accelerators for both an inference phase and a training phase of an ANN. ANNs, typically, handle large volumes of data during training as well as during inference. Hence, the ANNs can require memory devices that facilitate handling storage and retrieval of such large volumes of data in an efficient manner.
According to one or more embodiments of the present invention, a method for using a distributed memory device in a memory augmented neural network system includes receiving, by a controller, an input query to access data stored in the distributed memory device, the distributed memory device comprising a plurality of memory banks. The method further includes determining, by the controller, a memory bank selector that identifies a memory bank from the distributed memory device for memory access, wherein the memory bank selector is determined based on a type of workload associated with the input query. The method further includes computing, by the controller and by using content based access, a memory address in the identified memory bank. The method further includes generating, by the controller, an output in response to the input query by accessing the memory address.
According to one or more embodiments of the present invention, a neural network system, includes a distributed memory device with several memory banks, and a controller coupled with the distributed memory device. The controller performs a method to access data stored in the distributed memory device. The method includes receiving an input query to access data stored in the distributed memory device. The method further includes determining a memory bank selector that identifies a memory bank from the distributed memory device for memory access, wherein the memory bank selector is determined based on a type of workload associated with the input query. The method further includes computing, by using content based access, a memory address in the identified memory bank. The method further includes generating an output in response to the input query by accessing the memory address.
According to one or more embodiments of the present invention, a memory address determination apparatus includes a selection unit, and a weighting unit. The memory address determination apparatus performs a method to access data stored in a distributed memory device. The method includes receiving, by the selection unit, a memory bank selector that identifies a memory bank from the distributed memory device for a memory access. The memory bank selector is determined based on a type of workload associated with an input query received by a controller of a neural network. The method further includes, computing, by the weighting unit, a memory address, using content based access, in the memory bank identified. The memory address is used to access data from the distributed memory to generate an output in response to the input query.
According to one or more embodiments of the present invention, a computer-implemented method includes receiving, by a controller neural network, an input. The method further includes generating, by the controller neural network, parameters including (i) a key vector and (ii) a selector vector to interface with several memory banks that are in communication with the controller neural network via a memory subsystem. The method further includes selecting, by the memory subsystem, at least one of the memory banks for the controller neural network to access, using the selector vector and the key vector. The method further includes accessing, by the controller neural network, a memory address from the selected memory bank using the key vector.
Additional technical features and benefits are realized through the techniques of the present invention. Embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.
The specifics of the exclusive rights described herein are particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of the embodiments of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
The diagrams depicted herein are illustrative. There can be many variations to the diagrams or the operations described therein without departing from the spirit of the invention. For instance, the actions can be performed in a differing order or actions can be added, deleted or modified. Also, the term “coupled” and variations thereof describe having a communications path between two elements and do not imply a direct connection between the elements with no intervening elements/connections between them. All of these variations are considered a part of the specification.
Embodiments of the present invention facilitate improvements to computing technology, and particularly neural networks, by providing technical features to implement a distributed memory architecture when using memory-augmented neural networks (MANN), such as a neural turing machine (NTM). MANNs are a class of recurrent neural network (RNN) that separate computation from memory. The key distinction between MANNs and other RNNs such as long short-term memory cells (LSTM) is the existence of an external memory unit. A controller network in the MANN receives input, interacts with the external memory unit via read and write heads and produces output.
The controller 110 generates and emits parameters (kt) to a read head 130 and/or to the write head 140. The read head 130 determines the address from which values are read from the memory device 120. The write head 140 determines the address to which the values are stored into the memory device 120. In one or more embodiments of the present invention, the read head 130 generates a distribution of weights to access the memory device 120 and read from a location. In one or more embodiments of the present invention, the write head 140 can erase and add new vectors to the memory device 120. In one or more embodiments of the present invention, the read head 130 and the write head 140 are memory pointers containing addresses of the memory device 120 from which the data values are to be read, or written into.
In the present solutions, the memory device 120 is a single contiguous memory that includes multiple memory locations that are accessed using a 2-dimensional matrix representation. The memory location that the input query 105 accesses is determined by a weight vector associated with the controller 110. The weight vector is normalized so that its value ranges from zero to one and the sum of the weight vector equals one. Accordingly, consider a normalized weight vector wt where t denotes time and wt (i) denotes an element in the weight vector at an index i and time t. Further, consider that the memory locations in the memory device 120 are represented as a matrix with N rows and M columns and
t represents the memory matrix at the time t. In other words,
t is a snapshot of the memory contents at time t.
When reading from the memory device 120, the memory matrix t is combined with the weight vector wt to determine a read vector rt at time t in a linear way. This can be expressed as:
The read vector rt represents the memory addresses from which the data values are read as part of the output 150.
Writing to the memory device 120 can include erasing and adding content to the memory device 120. Erase operation can be applied to remove information that is not required in the memory device 120. To erase the value of particular cell(s) in the memory matrix, an erase vector et is used. The erase vector has the same length as the weight vector wt. Typically, the erase vector contains values from 0 and 1. To erase values and to get an updated memory matrix t* at time t, the memory matrix
t−1 from the most recent time t−1 is combined with a result of multiplication of the weight vector and the erase vector as follows:
t*(i)=
t−1(i)·(1−wt(i)et)
Storing content into the memory cells uses the weight vector and an add vector at which lists the values that are to be added into the memory locations. The memory matrix Mt* is first determined using the erase operation, and then the new data values are added into the memory device 120 by determining an updated memory matrix t at time t using:
t*(i)=
t(i)+wt(i)at
As can be seen, the addressing for both the reading and writing operations depends on the weight vector that the controller 110 determines. The controller 110 can use attention network calculation and different addressing schemes to compute the weight vector. For the content-based addressing, values selected from the memory device 120 are based on similarity to the data values in the input query 105. Consider that the controller 110 determines a key vector kt that is used to compare with each memory vector t(i) to determine a similarity. The similarity can be measured using various techniques, for example, a cosine similarity can be computed. It is understood that other similarity functions K can be used in one or more embodiments of the present invention.
Accordingly, the weight vector is computed in present solutions as:
Here, β represents a key strength that determines a concentration of the weight vector, and it is a predetermined vector that a user can configure prior to or during runtime. Based on the value of β, the controller 110 tunes attention to a particular location in the memory device 120. For example, when the value of β is low, the controller 110 can focus on all the locations equally, and when the β is high, the controller 110 can focus on a particular location(s). The β can be accordingly varied if the data values are to be retrieved/stored by the controller 110, in particular parts of the memory device 120.
A technical challenge with such existing solutions is that the memory device 120 is restricted to a particular size. This is because in case the memory device 120 is too large, the matrix multiplications and other matrix calculations noted above can take time, and also additional resources. This can increase the time required to access the memory locations, and in turn the data values in the memory device 120. For faster access to memory, smaller sized memory devices 120 can be used, which then limits the memory size, and in turn, the operations that can be performed by the MANN 100. Technical solutions provided by one or more embodiments of the present invention address such technical challenges by facilitating the controller 110 to operate a memory bank that includes several memory matrices t.
The memory bank selector 250 is used to select one of several memory banks 225 from the memory device 220 that is a distributed memory device. In the distributed memory device 220, the memory banks 225 are implemented as individual matrices. Accordingly, a memory location l in each memory bank 225 represents a separate memory location. The memory bank selector 250 is a trainable parameter, which the read head 230 and the write head 240 use to select a particular memory bank 225 to read and write, respectively. The memory bank selector 250 can be, or can produce at least one of the following: preferred biases, circuit timings, stability and/or write assist parameters for the memory banks 225. The addressing operation is limited to the selected memory bank 225, rather than the whole memory device 220, accordingly making the address computations more manageable and efficient. Accordingly, embodiments of the present invention provide an improvement to computing technology by facilitating the MANN 200 to use large sized memory devices 220 that include several memory banks 225, each representing independent memory matrices t. Alternatively, or in addition, by facilitating such independent memory matrices in the memory device 220, the address computations can be performed more efficiently than existing solutions. Accordingly, embodiments of the present invention provide an improvement to existing computing technology, particularly MANNs. Such technical improvements provide a practical application that can be used for several technical fields such as image recognition, audio recognition, handwriting recognition, economic analysis, and several other fields where MANNs can be used.
Each memory bank 225 can be a separate memory hardware in one or more embodiments of the present invention. For example, the separate memory banks 225 can be different physical types of memory with attributes such as high-density, fast write, fast read, latency, low standby power, endurance, non-volatility and other such attributes. The memory banks 225 can be a combination of static random access memory (SRAM), register files, dynamic random access memory (DRAM), flash memory, resistive random access memory (RRAM), magnetoresistive RAM (MRAM), ferroelectric RAM (FeRAM) and other types of memory.
Alternatively, or in addition, each memory bank 225 can be a partition in a single memory hardware. Alternatively still, the memory banks 225 can be formed by a combination of multiple memory hardware, each including multiple memory banks 225 as separate partitions.
The memory bank selector 250 is trained to select a memory bank 225 from the distributed memory device 220 trainable selector to steer content based on type of computational work to be performed such as maximized model size with dense memory, fast operation, power conservation, etc. Based on the type of computation to be performed, the controller 210 determines the memory bank(s) 225 to be used for storing/retrieving data.
Alternatively, the distributed memory device 220 includes homogenous set of memory banks 225. In such cases, the controller 210 is trained to dynamically select the memory bank(s) 225 to be used based on biases, preferred circuit timings, or desired attributes such as access disturb stability, low-voltage write margin, and low standby power, and the like or a combination thereof. In one or more embodiments of the present invention, the trainable selector 250 facilitates the controller 210 to configure selected bias, implementation of read stability assist or write assist, or preferred circuit timing based on type of computational work to be performed.
Accordingly, in one or more embodiments of the present invention, the distributed memory device 220 includes a heterogeneous combination of memory banks 225, the memory banks 225 having different types of attributes that are advantageous for different types of computational work. In one or more embodiments of the present invention, the distributed memory device 220 includes homogenous combination of memory banks 225, where the memory banks 225 have a common set of attributes. The memory bank selector 250 produces allocation of memory type based on training and/or type of workload associated with an incoming input query 105. The memory bank selector 250 produces preferred bias or circuit timings, or sets stability assist or write assist parameters for memory banks 225 based on training and/or the type of workload.
The method 300 further includes generating address computation parameters including the key vector and the memory bank selector 250 by the controller 210, at block 320. The controller 210 is trained to determine the key vector and the selector 250 using feed forward training, or any other type of training techniques for a neural network. Accordingly, given the input query 105, the controller 210 outputs the key vector and the memory bank selector 250 using inference. In one or more embodiments of the present invention, the memory bank selector 250 is a vector of length equal to the number of memory banks 225 in the distributed memory device 220.
Further, the method 300 includes using the parameters determined by the controller 210 to access the memory locations in the distributed memory device 220, at block 330. Accessing the memory locations includes determining the memory bank(s) 225 using the selector 250, at block 332. As described herein, the selector 250 selects one or more memory banks 225 based on training and a type of workload in the input query 105. Accordingly, the controller 210 determines the type of workload caused by the input query 105. A type of workload can be selected from a list of predetermined types of workloads to which the controller 110 refers. This can be a classification of the input query 105 performed by the controller 210 based on training. Each type of workload is assigned to a particular set of memory banks 225. For example, in cases where computation time required for the query is short, a set of memory banks 225 with lower latency may be selected so that the computations are not stalled for memory access. Alternatively, or in addition, if the workload is computationally expensive, a particular set of memory banks 225 with higher latency may be used because the memory access can take slightly longer time as the computations are being performed. Alternatively, or in addition, other criteria can be used for selecting the memory banks 225, such as power consumption, data distribution, and the like.
Further, once the memory bank(s) 225 are selected, content-based addressing is used to read from the selected memory bank(s) 225, at block 334.
t. The read head 230 and/or the write head 240 selects a memory bank 225 by performing a selector operation (represented as “[1]”), which is a dot product of the memory bank selector 250 (st) and the memory banks 225. Further, the input query 110 and the key vector (kt) are used to compute the similarity function (which can include one or more dot product). The similarity function computation is depicted as “[2].” After these operations are completed for all memory banks 225 in parallel, the output is the read/write address that can be used to access the memory locations for the input query 105.
The calculations in equation form can be depicted as:
Softmax(K[kt⊙st,⊙st]·β)
Here, softmax function, is a function that takes as input a vector of n real numbers, and normalizes it into a probability distribution consisting of n probabilities proportional to the exponentials of the input numbers. That is, prior to applying softmax, some vector components could be negative, or greater than one; and might not sum to 1; but after applying softmax, each component will be in the interval (0,1), and the components will add up to 1, so that they can be interpreted as probabilities. Here, the output of the softmax function represents the memory address that is to be accessed for the input query 105. In one or more embodiments of the present invention, the memory addresses generated in this manner are “soft addresses”, i.e., the memory address is not a single index/address, but a distribution over the memory addresses in the memory device 120.
Referring back to the flowchart from
Accordingly, the method 300 facilitates the MANN 200 to use the distributed memory device 220 to read/write data into independent memory banks 225 in an efficient manner.
The memory address determination apparatus 500 includes a selection unit 510 and a weighting unit 520. The selection unit 510 can be a multiplexer or any other component that facilitates selection of one of several inputs. In this case, the selection unit 510 is coupled with the several memory banks 225 from the distributed memory device 220. The memory bank selector 250 provides a controlling signal to the selection unit 510 to select particular memory bank(s) 225. The selected memory banks 225 are interfaced to the weighting unit 520 via the selection unit 510. The weighting unit 520 computes the weight vector using the key vector and key strength, along with the selected memory banks 225.
Further, the cosine similarity function computation unit 620 includes a dot product unit 730 that computes a dot product of the key vector and the vector of selected memory banks. The dot product unit 730 has a similar behavior, where, at each clock cycle, two inputs (a value of the key vector kt and a value of the vector Mt) are provided. Internally, during the same clock cycle, these inputs are multiplied and accumulated by the dot product unit 730. After the last two inputs are provided, the accumulated value becomes a valid output and is fed into the divider, as a numerator.
When the inputs of the divider 740 are valid, the divider 740 divides the numerator with the denominator, the result of which is the cosine similarity K[kt, Mt].
Referring again to
Embodiments of the present invention facilitate a MANN system to distribute content across several memory banks, and accordingly allowing the MANN system to have a larger storage capacity than in existing systems. Embodiments of the present invention determine a memory bank selector, which is a trainable parameter that a controller of the MANN system learns using neural network training technique(s). The memory bank selector can select one or more memory banks. Further, as described herein, in embodiments of the present invention, the query key and memory matrix are compared after the memory bank selection. This limits the similarity operation to be performed with a much smaller size memory (e.g., 1 bank), than the entire memory, and accordingly increases computational efficiency.
Turning now to
As shown in
The computer system 900 comprises an input/output (I/O) adapter 906 and a communications adapter 907 coupled to the system bus 902. The I/O adapter 906 may be a small computer system interface (SCSI) adapter that communicates with a hard disk 908 and/or any other similar component. The I/O adapter 906 and the hard disk 908 are collectively referred to herein as a mass storage 910.
Software 911 for execution on the computer system 900 may be stored in the mass storage 910. The mass storage 910 is an example of a tangible storage medium readable by the processors 901, where the software 911 is stored as instructions for execution by the processors 901 to cause the computer system 900 to operate, such as is described herein below with respect to the various Figures. Examples of computer program product and the execution of such instruction is discussed herein in more detail. The communications adapter 907 interconnects the system bus 902 with a network 912, which may be an outside network, enabling the computer system 900 to communicate with other such systems. In one embodiment, a portion of the system memory 903 and the mass storage 910 collectively store an operating system, which may be any appropriate operating system, such as the z/OS or AIX operating system from IBM Corporation, to coordinate the functions of the various components shown in
Additional input/output devices are shown as connected to the system bus 902 via a display adapter 915 and an interface adapter 916 and. In one embodiment, the adapters 906, 907, 915, and 916 may be connected to one or more I/O buses that are connected to the system bus 902 via an intermediate bus bridge (not shown). A display 919 (e.g., a screen or a display monitor) is connected to the system bus 902 by a display adapter 915, which may include a graphics controller to improve the performance of graphics intensive applications and a video controller. A keyboard 921, a mouse 922, a speaker 923, etc. can be interconnected to the system bus 902 via the interface adapter 916, which may include, for example, a Super I/O chip integrating multiple device adapters into a single integrated circuit. Suitable I/O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols, such as the Peripheral Component Interconnect (PCI). Thus, as configured in
In some embodiments, the communications adapter 907 can transmit data using any suitable interface or protocol, such as the internet small computer system interface, among others. The network 912 may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. An external computing device may connect to the computer system 900 through the network 912. In some examples, an external computing device may be an external webserver or a cloud computing node.
It is to be understood that the block diagram of
The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer-readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer-readable program instructions described herein can be downloaded to respective computing/processing devices from a computer-readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing/processing device.
Computer-readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source-code or object code written in any combination of one or more programming languages, including an object-oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer-readable program instruction by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer-implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments described herein.
Various embodiments of the invention are described herein with reference to the related drawings. Alternative embodiments of the invention can be devised without departing from the scope of this invention. Various connections and positional relationships (e.g., over, below, adjacent, etc.) are set forth between elements in the following description and in the drawings. These connections and/or positional relationships, unless specified otherwise, can be direct or indirect, and the present invention is not intended to be limiting in this respect. Accordingly, a coupling of entities can refer to either a direct or an indirect coupling, and a positional relationship between entities can be a direct or indirect positional relationship. Moreover, the various tasks and process steps described herein can be incorporated into a more comprehensive procedure or process having additional steps or functionality not described in detail herein.
The following definitions and abbreviations are to be used for the interpretation of the claims and the specification. As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “contains” or “containing,” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, a mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but can include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.
Additionally, the term “exemplary” is used herein to mean “serving as an example, instance or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms “at least one” and “one or more” may be understood to include any integer number greater than or equal to one, i.e. one, two, three, four, etc. The terms “a plurality” may be understood to include any integer number greater than or equal to two, i.e. two, three, four, five, etc. The term “connection” may include both an indirect “connection” and a direct “connection.”
The terms “about,” “substantially,” “approximately,” and variations thereof, are intended to include the degree of error associated with measurement of the particular quantity based upon the equipment available at the time of filing the application. For example, “about” can include a range of ±8% or 5%, or 2% of a given value.
For the sake of brevity, conventional techniques related to making and using aspects of the invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs to implement the various technical features described herein are well known. Accordingly, in the interest of brevity, many conventional implementation details are only mentioned briefly herein or are omitted entirely without providing the well-known system and/or process details.
| Number | Name | Date | Kind |
|---|---|---|---|
| 10042804 | Watson, Jr. | Aug 2018 | B2 |
| 10410119 | Grefenstette et al. | Sep 2019 | B2 |
| 10496286 | Zawodny | Dec 2019 | B2 |
| 10942923 | Zhang | Mar 2021 | B1 |
| 20060190425 | Chang | Aug 2006 | A1 |
| 20070038798 | Bouchard | Feb 2007 | A1 |
| 20140269718 | Goyal | Sep 2014 | A1 |
| 20190324759 | Yang | Oct 2019 | A1 |
| 20200004685 | Guim Bernat | Jan 2020 | A1 |
| 20200008299 | Tran | Jan 2020 | A1 |
| 20210073219 | Barbas | Mar 2021 | A1 |
| Entry |
|---|
| Graves, A. et al., “Hybrid computing using a neural network with dynamic external memory,” 2016, vol. 000, doi:10.1038/Nature 20101, 21 pages. |
| Graves, A. et al., “Neural Turing Machines,” arXiv:1410.5401v2 [cs.NE], Dec. 10, 2014, 26 pages. |
| Khan, A. et al., “Memory Augmented Control Networks,” arXiv preprint arXiv:1709.05706, 2017, 19 pages. |
| Munkhdalai, T. et al., “Metalearned Neural Memory,” Advances in Neural Information Processing Systems, pp. 13310-13321, 2019. |
| Number | Date | Country | |
|---|---|---|---|
| 20210311874 A1 | Oct 2021 | US |
