Patent Grant
11481624
Embodiments of the present disclosure are directed to an AI module that leans an optimal program method for a specific NAND memory device in terms of maximal capacity, minimal program noise, and maximal program speed.
To provide a combination of price and performance for a wide range of high-density storage applications, modern NAND flash memory chips enables storing several bits of data in each flash cell. This scheme is called a multi-level cell. A multi-level cell partitions the threshold voltage range of a flash cell into several voltage states and the written value is extracted from the cell voltage level. However, storing more bits per cell decreases the dynamic voltage range of each voltage state making it more susceptible to noise. As a result thereof, the reliability of flash memory decreases significantly and the performance of current programming algorithms is not sufficient.
During the process of programming a target voltage to a memory cell, the signal is subject to various distortions, noise conditions and other impairments. For example:
Due to the above, programming a NAND flash is a noisy process: actions do not always go as planned and the writing outcome is not deterministic. As a result, programming algorithm performance is severely degraded. The role of a programming algorithm is to write with maximal level of reliability, overcoming those impairments. Since cell voltage levels at each cell cannot be precisely predicted after applying the programming pulse, there cannot be a plan for the entire sequence of pulses. A policy is needed that maps optimal actions to each of the cells voltage states.
Exemplary embodiments of the present disclosure are directed to a programming method based on reinforcement learning in which a software agent that operates or controls the voltage writing in the memory cells learns to act and optimize its performance by interacting with the NAND flash environment. The interaction with the environment is done by applying programming actions to the NAND flash and receiving feedback in the form of observations related to the cell voltage states in the NAND flash and rewards related to the programming quality. To increase the learning speed, an optimal programming method is learned using advanced AI techniques such as hierarchical reinforcement learning (HRL). Using hierarchical learning exploits the NAND flash hierarchical structure, i.e. block, word line and level ranks. By breaking the task to those building blocks or ranks, a group of smaller subtasks can be integrated and HRL solution methods are used.
Methods according to embodiments of the disclosure apply a reinforcement learning framework to learn an optimal programming policy for each NAND flash, thus giving the optimal action in each state, rather than making a programming plan ahead. Methods according to embodiments can utilize the hierarchical structure of the NAND flash to reduce the problem complexity, and can utilize function approximations, such as a deep neural network (DNN), to generalize across actions and states. A programming method according to an embodiment yields high performance in an environment with inter cell programming disturbances and other NAND flash channel noise, and can out-perform the established methods in both programming quality, low bit-error-rate (BER), and programming speed.
According to an embodiment of the disclosure, there is provided a method of programming a NAND memory device, where the NAND memory device comprises a plurality of blocks, each block comprises a plurality of wordlines and an associated agent, and each wordline comprises a plurality of cells and a plurality of voltage levels and an associated agent, and each voltage level comprises an agent. The method includes the steps of receiving, by a block agent, data from a user and a block state from the memory device, and determining a block action from the user data and block state, receiving, by a wordline agent, the block action from the block agent and a wordline state from the memory device, and determining a wordline action from the block action and the wordline state, receiving, by a level agent, the wordline action from the wordline agent and a level state from the memory chip, and programming, by the level agent, cells in the wordline determined from the wordline action and the level state to target voltage levels determined from the wordline action and the level state.
According to a further embodiment of the disclosure, the method includes updating the level agent based on a reward output by the level agent, where the reward measures a difference between the target voltage levels of the cells in the wordline and the actual voltage levels programmed to the cells, updating the wordline agent based on a reward output by the wordline agent, where the reward measures a difference between the target voltage levels of the wordline and the actual voltage levels programmed to the wordline, and updating the block agent based on a reward output by the block agent, where the reward measures a difference between the target voltage levels of all wordlines in the block and the actual voltage levels programmed to all wordlines in the block. Updating an agent comprises updating weights of the agent's neural network using a gradient descent algorithm using ∇ΘJ(Θ)=Eπθ[∇Θ log πθ(s, a)Aw(s, a)], where Θ represents the neural network weights, J is an accumulated reward defined as the expectation of a gradient with respect to Θ of the log of distributions defined by means μ and standard deviations σ output by the agent, times an advantage function, which is a function of the agent's reward.
According to another embodiment of the disclosure, there is provided a method of programming a NAND memory device, where the NAND memory device comprises a plurality of blocks, each block comprises a plurality of wordlines and an associated agent, and each wordline comprises a plurality of cells and a plurality of voltage levels and an associated agent, and each voltage level comprises an agent, the plurality of blocks being a highest rank in a plurality of ranks, the plurality of wordlines being a middle rank in the plurality of ranks, and the plurality of levels being a lowest rank in the plurality of ranks. The method includes the steps of receiving, by an agent at a given rank in the plurality of ranks, parameters from a higher rank agent in the hierarchy of ranks and a state from the memory device, where an agent at the highest rank in the hierarchy of ranks receives its parameters from a user, determining, by the agent, an action from the parameters and the state, passing the action as parameters to a lower rank agent in the hierarchy of ranks, where an agent at the lowest rank in the hierarchy of ranks programs cells in the memory device to target voltage levels determined from the parameters and the state, and updating the agent based on a reward output by the agent. The reward measures a difference between the target voltage levels of the cells and the actual voltage levels programmed to the cells.
According to a further embodiment of the disclosure, a block agent receives data from a user and a block state from the memory device, determines a block action from the user data and block state, and passes the block action as parameters to a wordline agent. A block reward measures a difference between target voltage levels of all wordlines in the block and actual voltage levels programmed to all wordlines in the block.
According to a further embodiment of the disclosure, the block state is a vector (Li, Hi) for each wordline i in the block, where Li is a lowest programmed level in the wordline i, and Hi is a highest level not yet programmed in the wordline i, and the block action is set of triples (WLk, μi, μj) where WLk, is a wordline to be programmed and μi, μj is a range of permitted levels to program.
According to a further embodiment of the disclosure, the block agent outputs a set of means μ and standard deviations a for minimum and maximum voltage levels to be programmed for each wordline WLk, forms Gaussian distributions for the minimum and maximum voltage levels from the means μ and standard deviations σ for the minimum and maximum voltage levels, and randomly samples Gaussian distributions for the minimum and maximum voltage levels to determine the block agent action.
According to a further embodiment of the disclosure, a reward of the block agent is R=ΣwordlinesΣL∈levelsΣC∈L|vL−vC|/|wordlines|, where the sums are over all wordlines in the block, all voltage levels in each wordline, and over all cells of each voltage level L, vL is the target voltage per cell, vC is an actual voltage programmed to the cell, and |wordlines| is the number of wordlines. Updating the block agent comprises updating weights of the block agent neural network using a gradient descent algorithm using ∇ΘJ(Θ)=Eπθ[∇Θ log πθ(s, a)Aw(s, a)], where Θ represents the NN weights, J is an accumulated reward defined as the expectation of a gradient with respect to Θ of the log of the distributions defined by the means p and standard deviations σ output by the block agent, times an advantage function, which is a function of the reward.
According to a further embodiment of the disclosure, a wordline agent receives as its parameters a block action from a block agent and a wordline state from the memory device, determines a wordline action from the block action and the wordline state, and passes the wordline action as parameters to a level agent, and a wordline reward measures a difference between the target voltage levels of the wordline and the actual voltage levels programmed to the wordline.
According to a further embodiment of the disclosure, the parameters of the wordline agent a set of triples (WLk, μi, μj), where WLk, is the wordline to be programmed and μi, μj is a range of permitted levels to program. The wordline state is a vector (μk, σk) for each level in the wordline WL where μk is the mean for level k and σk is the variance of level k on the current wordline, and the wordline action is a set of triples (Li, M, C) where Li is an identifier that specifies the level to program, M is the maximal number of level agent actions to perform, and C is the target voltage for the cells.
According to a further embodiment of the disclosure, the wordline agent calculates a mean p and standard deviation a of the voltage levels and a mean p and standard deviation a of the number of pulses, and a level decision bit vector whose length is equal to a number of available voltage levels and which has a bit that corresponds to the target voltage level set to a different value from other bits in the vector, forms a Gaussian distribution from the mean μ and standard deviation σ for the number of pulse voltages, and a Gaussian distribution from the mean μ and standard deviation σ for the voltage levels, randomly samples each Gaussian distribution to determine the target voltage level to program to the cells in the wordline, and a maximum number of voltage pulses to use in programming the target voltage levels to the cells, and outputs an index of the bit in the level decision bit vector that corresponds to the target voltage level, a chosen voltage level to program and a number of pulses to program that chosen voltage level.
According to a further embodiment of the disclosure, the target voltage level sampled from the voltage level Gaussian distribution is drawn from a range of voltages about the target voltage level According to a further embodiment of the disclosure, a reward of the wordline agent is R=ΣL∈levelsΣC∈L|vL−vC|/|wordline|, where the sums are over all voltage levels and over the cells of voltage level L, vL is the target voltage per cell, vC is an actual voltage programmed to the cell, and |wordline| is the number of levels. Updating the wordline agent comprises updating weights of the wordline agent neural network using a gradient descent algorithm using ∇ΘJ(Θ)=Eπθ[∇Θ log πθ(s, a)Aw(s, a)], where Θ represents the NN weights, J is an accumulated reward defined as the expectation of a gradient with respect to Θ of the log of the distributions defined by the means μ and standard deviations σ output by the wordline agent, times an advantage function, which is a function of the reward.
According to a further embodiment of the disclosure, a level agent receives as its parameters a wordline action from a wordline agent and a level state from the memory chip, programs cells in the wordline determined from the wordline action and the level state to target voltage levels determined from the wordline action and the level state. A level reward measures a difference between the target voltage levels of the cells in the wordline and the actual voltage levels programmed to the cells.
According to a further embodiment of the disclosure, the parameters of the level agent is are set of triples (Li, M, C) where Li is an identifier that specifies the level to program, M is the maximal number of level agent actions to perform, and C is the target voltage for the cells. The level state is a voltage vector for cells in the level from the memory chip, an output of the level agent is a pulse voltage level to be used to program the cells, a threshold voltage level, and a bit vector of those cells in the wordline to be programmed, and the level action applies a number of pulses at the pulse voltage level to those cells specified by the bit vector to be programmed, where the number of pulses is less than than M.
According to a further embodiment of the disclosure, the level agent calculates a mean μ and standard deviation σ for possible pulse voltage levels, forms a Gaussian distribution from the mean μ and standard deviation σ for possible pulse voltage levels, randomly samples the Gaussian distribution to determine an actual pulse voltage level used to program the cells in the wordline, determines a threshold voltage level where cells with a voltage level below the threshold voltage level are programmed and cells with a voltage level equal to or greater than the threshold voltage level are not programmed, and outputs the bit vector that specifies the cells to be programmed.
According to a further embodiment of the disclosure, a reward of the level agent is R=ΣC∈L|vL−vC|/|cells|, where the sum is over the cells of voltage level L, vL is the target voltage per cell, vC is an actual voltage programmed to the cell, and |cells| is the number of cells. Updating the level agent comprises updating weights of the level agent neural network using a gradient descent algorithm using ∇ΘJ(Θ)=Eπθ[∇Θ log πθ(s, a)Aw(s, a)], where θ represents the NN weights, J is an accumulated reward defined as the expectation of a gradient with respect to Θ of the log of the distributions defined by the means μ and standard deviations σ output by the level agent, times an advantage function, which is a function of the reward.
According to another embodiment of the disclosure, there is provided a system for programming a NAND memory device, where the NAND memory device comprises at least one block, the at least one block comprises a plurality of wordlines, and each wordline comprises a plurality of cells and a plurality of voltage levels. The NAND memory device programming system includes at least one block agent, a plurality of wordline agents, one for each wordline in the at least one block, and a plurality of level agents, one for each voltage level in each wordline in the at least one block. The at least one block agent programs the entire block by reading the state of its associated block from the memory device and successively calling each wordline agent to perform writing to specific wordlines in the block, each of the plurality of wordline agents programs a wordline by reading the state of its associated wordline from the memory device and successively calling each level agent to program individual voltage levels on each wordline by using programming pulses on the memory device, and each level agent programs a voltage level to a group of cells by reading the states of its associated cells from the memory device and successively programming each voltage level to selected cells in the wordline. Each level agent returns control back to the calling wordline agent with acknowledgment after finishing programming the levels, each wordline agent returns control back to the calling block agent with acknowledgment after finishing programming the wordline, and the NAND memory device programming system terminates after all wordlines in the block have been written.
According to a further embodiment of the disclosure, the block agent, the plurality of wordline agents, and the plurality of level agents each read the memory device to obtain a reward for their previous action.
According to a further embodiment of the disclosure, each agent is an actor-critic neural network that is trained by a gradient descent algorithm using the agent's reward returned by the memory device. Training an agent comprises updating weights of the agent's neural network using a gradient descent algorithm using ∇ΘJ(Θ)=Eπθ[∇Θ log πθ(s, a)Aw(s, a)], where θ represents the neural network weights, J is an accumulated reward defined as the expectation of a gradient with respect to Θ of the log of distributions defined by means p and standard deviations σ output by each agent, times an advantage function, which is a function of the reward for each agent.
Exemplary embodiments of the disclosure as described herein generally provide systems and methods for training an agent to program different voltage levels to different cells. While embodiments are susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure.
NAND flash memories have a three-level hierarchy of block, wordline, and cell. The number of blocks varies between chip types but is usually in the order of several thousand per chip. A flash memory device can then consist of one or more such chips, so overall the amount of blocks per flash memory device varies quite a bit. The number of wordlines per block varies as well. Different chip types can have 64, 128 or 256 wordlines, and this may change in the future. A cell is a physical part of the wordline, i.e. a wordline is basically a long string of cells.
A level is a conceptual entity, in that each cell is programmed to have a certain voltage level and then cells are grouped according to their voltage levels into N groups, in which case there are N level in the wordline. The cell voltage determines which level it belongs to, and accordingly what information it encodes. Every cell at the same level has the same information stored in it. The number of levels per wordline varies, according to the writing scheme. The number of levels is 2 to the power of the number of bits per cell that are written. For example, for 3 bits per cell, there would be 8 levels per wordline, but this may also vary, even in the same block, according to how many bits per cell are written on in a specific wordline.
Hierarchical reinforcement learning (HRL) is a framework for combining learning on different scales. According to embodiments of the disclosure, there are three different agents acting on three different scales, block, wordline and cell scales, all combined under the HRL framework. A single action of a higher level agent is an entire episode of a lower level agent, i.e. the action of the higher level agent defines the parameters under which the lower level agent executes a series of lower level actions, which together comprise its episode. However, each agent in the hierarchy has its own decision model to allow the agent to choose it's actions. These models are policy networks. An exemplary policy network is an actor-critic model.
The inhibit vector marks all the cells that need to be programmed with zeros and those that should not be programmed with ones. The inhibit vector can be very large (around 147K cells) so it is impossible for a policy network to output it as a decision, i.e. individually decide for each cell in the vector whether it should be one or zero. So instead, embodiments of the disclosure use a different solution. Methods according to embodiments output a voltage threshold, and place ones in the inhibit vector for all cells whose voltage exceeds the chosen threshold. The remaining cells are left as zeros. This way, a network according to an embodiment only has to output one number, instead of 147K numbers, aside from its power output, which is also a single number, and which is separate from the inhibit vector.
Embodiments of the disclosure seek to train an agent to program different voltage levels to different cells. However, since the state-action space is too large for brute-force reinforcement learning, embodiments of the disclosure exploit the hierarchical structure of a NAND flash and decompose the task into several subtasks on different levels. This way a group of smaller subtasks can be integrated and learning becomes feasible. The hierarchical structure stems from prior knowledge of the NAND flash structure and is depicted in
A method according to an embodiment uses agents to perform actions on different scales. Structurally, the agents are quite similar. Agents according to embodiments are actor-critic type reinforcement learning algorithms, with the actor and critic being neural networks. The agents all have input from the higher level agent, except the block level agent which is the highest level, which defines their working parameters, as well as input from the chip, called here a state, which the agents use to decide their next action and reward for previous actions taken, received through the lower level agent's return values. The agents all output an action to a lower level agent, except the level agent which acts directly on the chip, calculate a reward and return to the calling higher level agent after they are done.
Referring to the
As depicted in
1. Level rank agent: Efficiently write each voltage level of the word-line, minimizing the distance to the target level. After the voltage level has been written, control passes to the wordline agent.
2. Word-line rank agent: Determines which voltage levels to program for a given wordline, and directs the lower rank agent to program those levels, while minimizing interference between different levels on the same word-line. The number of possible levels is 2n, where n is the number of bits per cell. After the whole wordline has been written control passes to the block agent.
3. Block rank agent: Determines which wordline in a block to program, while minimizing interference between word-lines on the same block. After all wordlines have been written, the NAND programming terminates.
According to embodiments, although an exact mathematical model of the VNAND flash programming is not assumed, VNAND flash programming can be generally formulated as a Markov Decision Process (MDP). In particular, each rank in the VNAND hierarchy can be formulated as a MDP, i.e., for each rank in the hierarchy an MDP can be defined as described below.
Communication Between Hierarchy Ranks
As mentioned above, the hierarchical structure stems from prior knowledge on the NAND flash structure. From this hierarchical structure, the communication between the different building blocks can be derived.
NAND Programming
According to an embodiment, programming a NAND device can be described by the following process.
1. Each cell in a wordline WL has a voltage vcell, and
1.1. After an erase operation,
1.2. Each cell has a target voltage vtarget, and
1.3. The vector
2. There is a program agent, which can apply a series of pulses to WL that change the state
3. The agent's goal is to change the
4. After each pulse,
5. Δ
6. A pulse with selected parameters is referred to as an action.
The new WL state depends only on the old WL state and that last action taken. This type of process is known as a Markov Decision Process (MDP). An MDP models decision making in situations where outcomes are partly random and partly under the control of a decision maker.
Markov Decision Processes
An MDP is defined by:
1. A set of states s∈S, where a state is the information used to determine what happens next.
2. A set of actions a∈A.
3. A transition function T(s,a,s′)=P(s′|s,a), which is the probability that action a acting on state s leads to state s′.
4. A reward function R(s,a,s′), which is a scalar feedback signal that indicates how well the agent is doing at state s. The agent's job is to maximize the cumulative reward.
5. A start state.
6. A terminal state (optional).
7. The Markov condition: given the present state, the future and past are independent:
P(St+1=SSt=st,At=at,St−1=st−1,At−1. . . ,S0=s0)=P(St+1=S
St=st,At−=at).
The task of an MDP is to find a policy for the decision maker, which is a function that specifies the action π(s) that the decision maker will choose when in state. Once a Markov decision process is combined with a policy in this way, this fixes the action for each state and the resulting combination behaves like a Markov chain, since the action chosen in state is completely determined by π(s), and Pr(st+1=sst=s, at=a) reduces to st+1=s
st=s, a Markov transition matrix. The goal is to choose a policy that will maximize a cumulative sum of the reward function: Σt=0∞R(st, at, st+1).
NAND Programming as an MDP
NAND programming is a very complex MDP. The cell voltage increment Δ
Reinforcement learning (RL) is an area of machine teaming concerned with how agents ought to take actions in an environment (MDP) so as to maximize some cumulative reward. RL does not assume knowledge of an exact mathematical model of the MDP and target large MDPs where exact methods become infeasible.
An RL algorithm according to an embodiment of the disclosure includes the following aspects.
Policy:
A policy π gives an action for each state: π*:S→A.
An optimal policy is one that maximizes an expected utility, if followed, e.g., a minimum mean square error (MSE), etc.
Value Function:
A value function is the prediction of a future reward, and is used to evaluate the goodness/badness of a state and policy:
Vz(s)=Ez[Rt+1+γRt+2+γRt+3+ . . . |St=s],γ∈[0,1].
Utility of Q(s,a):
Q*(s,a)=the expected utility when starting out having taken action a from state s and acting optimally thereafter
Bellman Equation:
The Bellman equations characterize optimal values:
V*(s)=maxαΣs′T(s,a,s′)[R(s,a,s′)+γV*(s′)].
Values can be computed by iteration:
Vk+1(s)←maxαΣs′T(s,a,s′)[R(s,a,s′)+γVk(s′)].
Evaluation:
For a fixed current policy π, find the value function by policy evaluation. Iterate until values converge:
Vk+1π
Policy Extraction:
For fixed values, a better policy can be obtained using policy extraction using a one-step look-ahead:
πi+1(s)=arg maxαΣs′T(s,a,s′)[R(s,a,s′)+γVπ
On-Policy Learning:
An on-policy learner learns the value of the policy being carried out by the agent, including the exploration steps. An exemplary on-policy learning algorithm is the SARSA (State-action-reward-state-action) algorithm.
Off-Policy Learning:
An off-policy learner learns the value of the optimal policy independently of the agent's actions. An exemplary off-policy learning algorithm is the Q-learning algorithm.
Hierarchical RL—Introduction:
The state-action space for RL of wordlines is extremely large. For example, a typical wordline has 16 KBs of cells, each cell has a voltage range from −3000:10:5000 mV, equivalent to 800 values, and there are 128 WLs per block. This yields a total state space of 800×16×8×1000×128≈1.3×1010 states. Brute force RL learning is too slow to handle this many states. Deep RL can address this situation by approximating the value function with a neural network (NN).
Hierarchical reinforcement learning (h-RL) is a sub-area of RL that splits the agent into sub-agent hierarchies. h-RL is a technology that solves problems that could not be solved by traditional RL algorithms.
For h-RL theory please refer to: “Recent advances in Hierarchical Reinforcement learning”, Barto, 2003, https://people.cs.umass.edu/˜mahadeva/papers/hrl.pdf, the contents of which are herein incorporated by reference in their entirety.
According to embodiments, hierarchical reinforcement learning enables the task of approximating the value function to decoupled into a three level hierarchy, with single level programming as the low level, wordline programming as the intermediate level, and block programming at the highest level. Each level relatively independent of the others, and each can be modeled as an MDP and learned in parallel. The task can be simplified or made more complex, by using more or less complex loss functions.
According to embodiments, a reinforcement learning model can be based on existing algorithms or on human expertise. A reinforcement learning model according to an embodiment can learn from the existing algorithm or expert by imitation. The reinforcement learning model can then be improved on the wordline level, and after finding a basic stable policy, the reinforcement learning model can then be adapted to program a block.
Project Description
Exemplary embodiments of the disclosure provide an AI module referred to herein as reinforcement learning pulse programming (RLPP) that learns an optimal programming method for a specific NAND memory device in terms of its maximal capacity, minimal program noise, and maximal program speed. Exemplary embodiments of the disclosure can also provide an RLPP that can enable a 6-bit technology of future NAND memory devices.
The agents at each level are neural networks (NNs) and a can be trained separately. Training is performed once, typically by the manufacturer of the NAND device after the device is manufactured, and the trained programs are subsequently invoked each time a user writes data to the NAND device. The agents can be trained simultaneously, but this training is slow. According to an embodiment, the level agents are trained first, followed by the wordline agents, followed by the block agents. However, embodiments are not limited thereto, and the training order can be reversed, using surrogates for the lower level agents.
Actor-Critic Algorithms:
Actor-Critic is the name of a single algorithm that receives a state from the chip and instructions, such as working parameters, from a higher rank in the hierarchy, translates the state and parameters into something it can work with, which can differ for each rank in the hierarchy, passes it through its internal neural network, which again can differ for each rank in the hierarchy, takes the net's output and translates it into an action for the next lower rank in the hierarchy to execute, or an action on the chip at the lowest rank in the hierarchy. The name Actor-Critic comes from the way the neural network outputs two types of output simultaneously, which are considered the “actor” and “critic” but are jointly used to train the network. After training is complete, only the actor part is still relevant to determine the next action. For example, the block agent can receive an instruction to write a block 20. The block agent then reads the state of that block, translates it into input for its neural network, passes it through the net to get the output, takes the output and translates it into an action for the wordline agent, such as programming levels 30-40 on wordline 15, and passes control to the wordline agent. After the wordline agent is done, if network is being trained, the block agent receives a reward from the wordline agent and updates the model. Regardless, it then reassesses the state of the block and decides on its next action, etc., until it decides it is done.
An Actor-Critic agent is a single network with two outputs: an action and a valuation of the current state. An MDP describes the world the agent works in, i.e. the interaction with the chip. MDP basically means that the result of an action depends only on the action taken and the state of the chip at the time of the action. Thus, only the current state of the chip need be considered to decide on an action, i.e. the only input the network needs is the state of the chip.
Level Rank
According to embodiments, a level rank's agent forms a tight distribution around the target voltage. For each action (pulse+inhibit), the result is non deterministic and depends on the voltage levels of the target cells and neighboring cells. Hence, the following sets of states, actions and rewards can be derived naturally.
Set of States: Different actions are required depending on (1) the current voltage level in the programmed cells, and (2) the current voltage level in the neighboring cells. Both of the preceding can be used to represent a state. Voltage levels can be quantized into bins to decrease computational demands, as shown in the table of
Set of Actions: Programming cells is accomplished by using the following atomic actions:
(1) selecting a cell, which is controlled by the inhibit vector, which is a binary vector that selects the indexes of the cells to which the pulse will be given;
(2) applying the voltage pulse on the selected cells; and
(3) stopping the current action and transferring control back to the calling function, i.e., the wordline agent.
Hence, for a level agent, the actions are the preceding actions.
Level Agent
A level agent according to an embodiment controls the strength of the voltage pulses to the chip and the verify level, which is a threshold that determines which cells are to be programmed. Cells whose voltage is above the verify level are not programmed, while cells below the verify level are programmed. The parameters used by the level agent are received from the wordline agent, and is a triple of an identifier of the voltage level to be programmed, the target voltage level for the cells to be programmed, and the number of pulses to be applied to program the cells.
According to an embodiment, a level agent is an Actor-Critic variant of a Policy Neural Network model as described above. The input to the NN can be extracted from data read from the chip, which is a voltage vector for all cells in the wordline that are to be programmed. Given the cells to be programmed, the voltage range is divided into bins, as shown in
The output of a level agent NN according to an embodiment is a pulse voltage level to be used to program to the cells, a list of those cells to be programmed, and the verify level, as described above. Note that the pulse voltage level differs from the target voltage level for the cells, and is typically much greater than the target voltage level. For example, to program a cell to 5 volts, pulse voltages of 20 volts may be used. These three outputs are used to form the inhibit vector, which has a bit for each cell in the wordline, and where the bit is set to one for those cells to be programmed. In particular, the NN simultaneously outputs a mean μ and standard deviation σ for each pulse voltage, and a mean μ and standard deviation σ for possible verify levels. These means and standard deviations are the possible actions of the level agent, and are used to define Gaussian distributions. These distributions are then randomly sampled to determine the actual actions of the level agent.
An inhibit vector according to an embodiment is a bit vector whose length corresponds to the number of cells in the wordline, is set by the level agent, and tells the chip which cells to program and which cells not to program. The inhibit vector is determined based on the sampled verify level and the list of target cells to be programmed. The level agent sets the inhibit vector first by the level it needs to program, i.e. other levels in the wordline are immediately added to the inhibit vector, which it gets from the wordline agent. Then it decides which cells to add to the inhibit vector according to how close the cells' voltages are to the intended target voltage, so that they do not overshoot the voltage. For example, if a cell's voltage is already the same as the level it is intended to be, i.e. it is at the target, there is no need to apply more voltage to it, which would likely overshoot the target voltage, so its corresponding bit in the inhibit vector is set to 0.
According to an embodiment, the level agent programs the specified number of pulses to the cells selected by the inhibit vector, and outputs a reward, which is a measure of the programming quality that can be defined as a function of the distance from the desired target voltage. According to an embodiment, a reward is calculated from R1=ΣC∈L|vL−vC|/|cells|, R2=ΣC∈L|vL−vC|2/|cells|, R=(R1+R2)/2, and the level-reward=Rcurrent−Rprevious, where current reward is what was calculated on this measurement and previous is the saved measurement from the previous action, where the sum is over the cells of voltage level L, vL is the target voltage per cell, vC is the actual voltage programmed to the cell, and |cells| is the number of cells.
During training, according to an embodiment, the output of the level agent is used to tune the parameters of the NN, where the NN parameters are the weights of the network. This tuning can be done using a gradient descent optimization, a standard algorithm for tuning NNs. According to an embodiment, a gradient descent method can be implemented by an actor critic method. The gradient can be expressed as
∇ΘJ(Θ)=Eπθ[∇Θ log πθ(s,a)Aw(s,a)],
where Θ represents the NN parameters, and J is an accumulated reward defined as the expectation of the gradient with respect to Θ of the log of the current NN policy, which is a function of the state and actions, times an advantage function, which is also a function of the state and actions. According to an embodiment, the state is the histogram that was the input to the NN, and the actions are the cells and voltage levels actually programmed to the chip. The current NN policy is the distribution defined by the mean μ and standard deviation σ output by the NN, and the advantage function is typically a function of the reward, such as the current reward—the previous reward.
After the level agent has performed its action, it returns a parameter indicating that control is being passed back to the wordline agent. According to an embodiment, the reward is not passed up to the higher rank agent, in this case the wordline agent.
Word Line Rank
According to an embodiment, a wordline agent is a NN that implements an Actor-Critic algorithm as described above, and makes the levels as evenly distributed as possible.
Set of States: Embodiments of the disclosure represent a wordline using the mean and variance for each level, represented as two input vectors:
Set of Actions: According to embodiments, the set of actions includes four separate decisions. These decisions then define the parameters for a level agent, which the wordline agent uses to execute its lower level tasks, i.e., the wordline agent creates a level agent with the parameters it chooses, lets it run its course and return, and then reevaluates its state and decides on the next action to take:
Reward: According to an embodiment, the wordline reward is simply Rwordline=(ΣL∈levelsRL)/|levels|, wherein RL is the reward for level L calculated as above for the level agent case, and |levels| is the number of levels.
The Wordline Agent:
A wordline agent according to an embodiment takes all cells in a wordline, divides the cells into groups based on voltage levels, and reads from the chip a vector of μ and σ for all levels in the wordline. Note that in some embodiments, each voltage level is associated with an index, such as a positive integer. The input to the wordline agent is the vector of s and a for all levels in the wordline, an initial target voltage for each group of cells, and a budget, which is a maximum number of pulses to use to program each level. The parameters for the wordline agent, which are received from the block agent, are the wordline to be programmed, and indices of the minimum and maximum voltages μα and μβ to be programmed to that wordline.
Specifically, a wordline agent NN according to an embodiment outputs a mean μ and standard deviation σ of the voltage levels and a mean μ and standard deviation σ of the maximum number of pulses, which are used to form Gaussian distributions for the number of pulse voltages and for the voltage levels, and a level decision bit vector whose length is equal to the number of available voltage levels and which has a bit that corresponds to a target voltage level set to a different value from the other bits in the vector. For example, in some embodiments, the bit corresponding to the target voltage level is set to 1 and all other bits are set to 0. In other embodiments, the bit corresponding to the target voltage level is set to 0 and all other bits are set to 1. The Gaussian distributions for the voltage levels and the maximum number or pulses are randomly sampled in the ranges about the respective initial target voltage and budget to determine a target voltage level to program to the cells in the wordline, and a maximum number of voltage pulses to use in programming that target voltage level.
The output is not a range of voltage levels (or targets). It is a single target voltage. Note that the wordline agent receives an initial target voltage for each level as part of its input (mentioned in the previous paragraph) and then samples the aforementioned Gaussian distribution to choose a target voltage from a range around the initial given target
The action of the wordline agent are the parameters passed to the level agent. The parameters are a triple that includes the maximum number of pulses to program a cell to the target voltage, as sampled from its corresponding Gaussian distribution, the target voltage or target voltage interval, as sampled from its corresponding Gaussian distribution, and the index of the selected voltage level as read from the level decision vector.
A reward of the wordline agent according to an embodiment is Rwordline=(ΣL∈levelsRL)/|levels|, wherein RL is the reward for level L calculated as above for the level agent case, and |levels| is the number of levels. The wordline agent NN is updated during training by a gradient descent optimization, similar to the level agent, and a repeated description thereof will be omitted.
The training of the wordline agent can introduce high variance that should be mitigated. According to an embodiment, an actor-critic model is used to help stabilize the wordline agent. In an actor-critic algorithm, the critic evaluates the executed policy and updates it frequently, for a high learning rate, while the actor executes policy πθ(s, a)AW(s, a) influenced by the critic and updates slowly.
Block Rank
According to an embodiment, the block rank's agent exploits the close relation of word-lines lying on the same string-select line (SSL). For every word-line in the block, the set of states is the lowest level programmed and the highest level not yet programmed. The set of actions includes selection of the word-line that will be programmed and the range of permitted levels to program. The block reward may be average of the wordline rewards summed over all programmed word-lines.
The Block Agent:
According to an embodiment, a block agent model is an actor-critic NN that can exploit the relation of the wordlines lying on the same SSL and on the same position in their SSLs.
A block agent according to an embodiment takes as its parameters an index of the block to be programmed and the data to program the block with, where data values are >=0 and <=the maximum number of voltage levels. According to one embodiment, the input is a vector of two numbers (Li, Hi) for each wordline in the block, where Li is the lowest programmed level, and Hi is the highest level not yet programmed. For example, if the block has 10 wordlines, then the input will have 10×2=20 numbers.
The output of a block agent according to an embodiment, i.e., the block agent's action, is the wordline WLk, to be programmed and a set of means μ and standard deviations σ for the minimum and maximum voltage levels to be programmed for each wordline WLk, which are used to form Gaussian distributions which are then randomly sampled to determine the actions, the range (μi, μj) of permitted levels to program. The actions are passed as parameters to the wordline agent. An optional output is the maximum number of pulses allowed.
According to an embodiment, the reward for the block agent is Rblock=(ΣWL∈wordlinesRWL)/|wordlines|, where RWL is the reward for each wordline, and |wordlines| is the number of wordlines.
Hierarchy Overview
During a training phase, the reward at each rank is used to change the rank agent's behavior, i.e., how it learns, as expressed by the update formula in the update column:
∇ΘJ(Θ)=Eπθ[∇Θ log πθ(s,a)Aw(s,a)].
where Θ represents the NN parameters, and J is an accumulated reward defined as the expectation of the gradient with respect to Θ of the log of the current NN policy, which is a function of the state and actions, times an advantage function Aw(s, a), which is also a function of the state and actions, which, measures how good the action was. The formula ∇ΘJ(Θ)=Eπθ[∇Θ log πθ(s, a)Aw(s, a)] is used to update s when training the networks, and is an average over the policy being used, i.e. the average of all rewards received for all actions, of the gradient of the policy, i.e. the log of the probabilities of the actions given the state, multiplied by the advantage function which measures the difference between the value predicted for the state and action, and the value actually calculated after the fact. This formula is used update the network after actions are taken and rewards are received for the actions.
Results
An incremental step program pulse (ISPP) algorithm is a standard algorithm for programming a level, i.e. without any kind of machine learning.
System Implementation
It is to be understood that embodiments of the present disclosure can be implemented in various forms of hardware, software, firmware, special purpose processes, or a combination thereof. In some embodiments, the present disclosure can be implemented in the memory controller of a flash memory, but it could migrate to the flash memory chip itself. The state can read from the chip by the controller, which passes the state to an algorithm according to an embodiment. In other embodiments, the present disclosure can be implemented in software as an application program tangible embodied on a computer readable program storage device. The application program can be uploaded to, and executed by, a machine comprising any suitable architecture.
The computer system 71 also includes an operating system and micro instruction code. The various processes and functions described herein can either be part of the micro instruction code or part of the application program (or combination thereof) which is executed via the operating system. In addition, various other peripheral devices can be connected to the computer platform such as an additional data storage device and a printing device.
It is to be further understood that, because some of the constituent system components and method steps depicted in the accompanying figures can be implemented in software, the actual connections between the systems components (or the process steps) may differ depending upon the manner in which the present invention is programmed. Given the teachings of the present invention provided herein, one of ordinary skill in the related art will be able to contemplate these and similar implementations or configurations of the present invention.
While the present invention has been described in detail with reference to exemplary embodiments, those skilled in the art will appreciate that various modifications and substitutions can be made thereto without departing from the spirit and scope of the invention as set forth in the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9530461 | Akers | Dec 2016 | B2 |
| 10564864 | Ha | Feb 2020 | B1 |
| 11074004 | Casmira | Jul 2021 | B2 |
| Entry |
|---|
| Andrew G. Barto, et al. “Recent Advances in Hierarchical Reinforcement Learning,” Discrete Event Dynamic Systems: Theory and Applications, 13, 41-77, 2003. |
| Number | Date | Country | |
|---|---|---|---|
| 20210103806 A1 | Apr 2021 | US |
