Device and method for managing die groups

Abstract

The embodiments described herein methods and devices that enhance the endurance of a non-volatile memory (e.g., flash memory). The method includes obtaining, for each of the plurality of die, an endurance metric. The method also includes sorting the plurality of die into a plurality of die groups based on their corresponding endurance metrics, where each die group includes one or more die and each die group is associated with a range of endurance metrics. In response to a write command specifying a set of write data, the method further includes writing the write data to the non-volatile memory by writing in parallel subsets of the write data to the one or more die assigned to a single die group of the plurality of die groups.

Description

TECHNICAL FIELD

The disclosed embodiments relate generally to memory systems, and in particular, to improving the endurance of a storage medium, such as flash memory.

BACKGROUND

Semiconductor memory devices, including flash memory, typically utilize memory cells to store data as an electrical value, such as an electrical charge or voltage. A flash memory cell, for example, includes a single transistor with a floating gate that is used to store a charge representative of a data value. Increases in storage density have been facilitated in various ways, including increasing the density of memory cells on a chip enabled by manufacturing developments, and transitioning from single-level flash memory cells to multi-level flash memory cells, so that two or more bits can be stored by each flash memory cell.

Generally, non-volatile memory devices, such as flash memory devices, include a plurality die. The amount of program-erase (PE) cycles that the plurality of die within a same non-volatile memory device can sustain before being considered operationally defunct varies significantly (e.g., by a factor of three). As such, a non-volatile memory device's endurance is, typically, only as robust as its weakest die. Thus, a method for managing the varying endurance capabilities of die within a non-volatile memory device is desired.

SUMMARY

The disclosed device and method improves the endurance of non-volatile memory. An endurance metric for each die of a non-volatile memory is obtained. The die are logically grouped in die groups based on their corresponding endurance metric. Thereafter, memory operations are performed on a single die group, where the die group includes a plurality of die with similar endurance metrics.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood in greater detail, a more particular description may be had by reference to the features of various embodiments, some of which are illustrated in the appended drawings. The appended drawings, however, merely illustrate the more pertinent features of the present disclosure and are therefore not to be considered limiting, for the description may admit to other effective features.

FIG. 1 is a block diagram illustrating an implementation of a data storage system in accordance with some embodiments.

FIG. 2A is a block diagram illustrating an implementation of a management module in accordance with some embodiments.

FIG. 2B is a diagram illustrating a characterization vector table included in FIG. 2A in accordance with some embodiments.

FIG. 2C is a diagram illustrating a characterization vector included in FIG. 2B in accordance with some embodiments.

FIG. 3 is a schematic diagram of non-volatile memory in accordance with some embodiments.

FIG. 4 is a block diagram of a plurality of die groups in accordance with some embodiments.

FIG. 5 is a block diagram of a die group mapping in accordance with some embodiments.

FIGS. 6A-6B illustrate a flowchart representation of a method of managing a storage system in accordance with some embodiments.

FIG. 7 illustrates a flowchart representation of a method of managing a storage system in accordance with some embodiments.

In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.

DETAILED DESCRIPTION

The various embodiments described herein include devices and/or methods that may improve the endurance of a storage medium, such as a flash memory. Some embodiments include devices and/or methods for managing a storage system by grouping die with similar endurance metrics into die groups.

Some embodiments include a method of managing a storage system that comprises a storage controller and non-volatile memory (e.g., flash memory), the non-volatile memory comprising a plurality of die and each die comprising a plurality of blocks (sometimes also herein called subunits). In some embodiments, the method is performed at the storage controller. The method includes obtaining, for each of the plurality of die, an endurance metric. The method also includes sorting the plurality of die into a plurality of die groups based on their corresponding endurance metrics, where each die group includes one or more die and each die group is associated with a range of endurance metrics. In response to a write command specifying a set of write data, the method further includes writing the write data to the non-volatile memory by writing in parallel subsets of the write data to the one or more die assigned to a single die group of the plurality of die groups.

Some embodiments include a method of managing a storage system that comprises a storage controller and non-volatile memory (e.g., flash memory), the non-volatile memory comprising a plurality of die and each die comprising a plurality of blocks (sometimes also herein called subunits). In some embodiments, the method is performed at the storage controller with a plurality of channels communicatively coupled to the non-volatile memory, where each channel is communicatively coupled with a respective subset of the plurality of die. The method includes: obtaining, for each of the plurality of die, an endurance metric; and, for each channel, ranking the die in the channel in accordance with the endurance metrics of the respective die in the channel. The method also includes: based on the ranking of the die in each of the channels, assigning to each die group of a plurality of die groups, one or more die from each channel, where the die assigned to a respective die group have a rank, or range of ranks, corresponding to the respective die group, and each die group has a distinct corresponding rank or range of ranks. In response to a write command specifying a set of write data, the method further includes writing the write data to the non-volatile memory by writing in parallel subsets of the write data to the plurality of die assigned to a single die group of the plurality of die groups.

Some embodiments include a storage controller comprising: one or more processors; a host interface configured to couple the storage controller to a host; a storage medium interface configured to couple the storage controller to non-volatile memory comprising a plurality of die, each die comprising a plurality of blocks; and a storage controller storing instructions, which when executed by the one or more processors, cause the storage controller to perform the operations of any of the methods described herein.

Some embodiments include a storage controller comprising: a host interface configured to couple the storage controller to a host; a storage medium interface configured to couple the storage controller to non-volatile memory comprising a plurality of die and each die comprising a plurality of blocks; and means for performing the operations of any of the methods described herein.

Some embodiments include a storage system, comprising: a host interface configured to couple the storage system to a host; non-volatile memory comprising a plurality of die, each die comprising a plurality of blocks; and a storage controller with one or more processors and memory storing one or more programs, which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods described herein.

Some embodiments include a non-transitory computer readable storage medium, storing one or more programs for execution by one or more processors of a storage controller communicatively coupled with a non-volatile memory comprising a plurality of die and each die comprising a plurality of blocks, the one or more programs including instructions for performing any of the methods described herein.

Numerous details are described herein in order to provide a thorough understanding of the example embodiments illustrated in the accompanying drawings. However, some embodiments may be practiced without many of the specific details, and the scope of the claims is only limited by those features and aspects specifically recited in the claims. Furthermore, well-known methods, components, and circuits have not been described in exhaustive detail so as not to unnecessarily obscure more pertinent aspects of the embodiments described herein.

FIG. 1 is a diagram of an implementation of a data storage system 100, in accordance with some embodiments. While some example features are illustrated, various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example embodiments disclosed herein. To that end, as a non-limiting example, data storage system 100 includes a storage controller 120, and a storage medium 130, and is used in conjunction with a computer system 110. In some embodiments, storage medium 130 is a single flash memory device while in other embodiments storage medium 130 includes a plurality of flash memory devices. In some embodiments, storage medium 130 is NAND-type flash memory or NOR-type flash memory. Further, in some embodiments, storage controller 120 is a solid-state drive (SSD) controller. However, other types of storage media may be included in accordance with aspects of a wide variety of embodiments.

Computer system 110 is coupled to storage controller 120 through data connections 101. However, in some embodiments computer system 110 includes storage controller 120 as a component and/or a sub-system. Computer system 110 may be any suitable computer device, such as a computer, a laptop computer, a tablet device, a netbook, an internet kiosk, a personal digital assistant, a mobile phone, a smart phone, a gaming device, a computer server, or any other computing device. Computer system 110 is sometimes called a host or host system. In some embodiments, computer system 110 includes one or more processors, one or more types of memory, a display and/or other user interface components such as a keyboard, a touch screen display, a mouse, a track-pad, a digital camera and/or any number of supplemental devices to add functionality.

Storage medium 130 is coupled to storage controller 120 through connections 103. Connections 103 are sometimes called data connections, but typically convey commands in addition to data, and optionally convey metadata, error correction information and/or other information in addition to data values to be stored in storage medium 130 and data values read from storage medium 130. In some embodiments, however, storage controller 120 and storage medium 130 are included in the same device as components thereof. Furthermore, in some implementations memory controller 120 and storage medium 130 are embedded in a host device, such as a mobile device, tablet, other computer or computer controlled device, and the methods described herein are performed by the embedded memory controller. Storage medium 130 may include any number (i.e., one or more) of memory devices including, without limitation, non-volatile semiconductor memory devices, such as flash memory. For example, flash memory devices can be configured for enterprise storage suitable for applications such as cloud computing, or for caching data stored (or to be stored) in secondary storage, such as hard disk drives. Additionally and/or alternatively, flash memory can also be configured for relatively smaller-scale applications such as personal flash drives or hard-disk replacements for personal, laptop and tablet computers.

Storage medium 130 is divided into a number of addressable and individually selectable blocks, such as selectable portion 131. In some embodiments, the individually selectable blocks are the minimum size erasable units in a flash memory device. In other words, each block contains the minimum number of memory cells that can be erased simultaneously. Each block is usually further divided into a plurality of pages and/or word lines, where each page or word line is typically an instance of the smallest individually accessible (readable) portion in a block. In some embodiments (e.g., using some types of flash memory), the smallest individually accessible unit of a data set, however, is a sector, which is a subunit of a page. That is, a block includes a plurality of pages, each page contains a plurality of sectors, and each sector is the minimum unit of data for reading data from the flash memory device.

For example, one block comprises any number of pages, for example, 64 pages, 128 pages, 256 pages or another suitable number of pages. Blocks are typically grouped into a plurality of zones. Each block zone can be independently managed to some extent, which increases the degree of parallelism for parallel operations and simplifies management of storage medium 130.

As noted above, while data storage densities of non-volatile semiconductor memory devices are generally increasing, a drawback of increasing storage density is that the stored data is more prone to being stored and/or read erroneously. As described in greater detail below, error control coding can be utilized to limit the number of uncorrectable errors that are introduced by electrical fluctuations, defects in the storage medium, operating conditions, device history, write-read circuitry, etc., or a combination of these and various other factors.

In some embodiments, storage controller 120 includes a management module 121, an input buffer 123, an output buffer 124, an error control module 125 and a storage medium interface (I/O) 128. Storage controller 120 may include various additional features that have not been illustrated for the sake of brevity and so as not to obscure more pertinent features of the example embodiments disclosed herein, and that a different arrangement of features may be possible. Input and output buffers 123, 124 provide an interface to computer system 110 through data connections 101. Similarly, storage medium I/O 128 provides an interface to storage medium 130 though connections 103. In some embodiments, storage medium I/O 128 includes read and write circuitry, including circuitry capable of providing reading signals to storage medium 130 (e.g., reading threshold voltages for NAND-type flash memory).

In some embodiments, management module 121 includes one or more processing units (CPUs, also sometimes called processors) 122 configured to execute instructions in one or more programs (e.g., in management module 121). In some embodiments, one or more CPUs 122 are shared by one or more components within, and in some cases, beyond the function of storage controller 120. Management module 121 is coupled to input buffer 123, output buffer 124 (connection not shown), error control module 125 and storage medium I/O 128 in order to coordinate the operation of these components.

Error control module 125 is coupled to storage medium I/O 128, input buffer 123 and output buffer 124. Error control module 125 is provided to limit the number of uncorrectable errors inadvertently introduced into data. In some embodiments, error control module 125 is executed in software by the one or more CPUs 122 of management module 121, and, in other embodiments, error control module 125 is implemented in whole or in part using special purpose circuitry to perform encoding and decoding functions. To that end, error control module 125 includes an encoder 126 and a decoder 127. Encoder 126 encodes data by applying an error control code to produce a codeword, which is subsequently stored in storage medium 130.

When the encoded data (e.g., one or more codewords) is read from storage medium 130, decoder 127 applies a decoding process to the encoded data to recover the data, and to correct errors in the recovered data within the error correcting capability of the error control code. Those skilled in the art will appreciate that various error control codes have different error detection and correction capacities, and that particular codes are selected for various applications for reasons beyond the scope of this disclosure. As such, an exhaustive review of the various types of error control codes is not provided herein. Moreover, those skilled in the art will appreciate that each type or family of error control codes may have encoding and decoding algorithms that are particular to the type or family of error control codes. On the other hand, some algorithms may be utilized at least to some extent in the decoding of a number of different types or families of error control codes. As such, for the sake of brevity, an exhaustive description of the various types of encoding and decoding algorithms generally available and known to those skilled in the art is not provided herein.

During a write operation, input buffer 123 receives data to be stored in storage medium 130 from computer system 110. The data held in input buffer 123 is made available to encoder 126, which encodes the data to produce one or more codewords. The one or more codewords are made available to storage medium I/O 128, which transfers the one or more codewords to storage medium 130 in a manner dependent on the type of storage medium being utilized.

A read operation is initiated when computer system (host) 110 sends one or more host read commands on control line 111 to storage controller 120 requesting data from storage medium 130. Storage controller 120 sends one or more read access commands to storage medium 130, via storage medium I/O 128, to obtain raw read data in accordance with memory locations (addresses) specified by the one or more host read commands. Storage medium I/O 128 provides the raw read data (e.g., comprising one or more codewords) to decoder 127. If the decoding is successful, the decoded data is provided to output buffer 124, where the decoded data is made available to computer system 110. In some embodiments, if the decoding is not successful, storage controller 120 may resort to a number of remedial actions or provide an indication of an irresolvable error condition.

Flash memory devices utilize memory cells to store data as electrical values, such as electrical charges or voltages. Each flash memory cell typically includes a single transistor with a floating gate that is used to store a charge, which modifies the threshold voltage of the transistor (i.e., the voltage needed to turn the transistor on). The magnitude of the charge, and the corresponding threshold voltage the charge creates, is used to represent one or more data values. In some embodiments, during a read operation, a reading threshold voltage is applied to the control gate of the transistor and the resulting sensed current or voltage is mapped to a data value.

The terms “cell voltage” and “memory cell voltage,” in the context of flash memory cells, means the threshold voltage of the memory cell, which is the minimum voltage that needs to be applied to the gate of the memory cell's transistor in order for the transistor to conduct current. Similarly, reading threshold voltages (sometimes also called reading signals and reading voltages) applied to a flash memory cells are gate voltages applied to the gates of the flash memory cells to determine whether the memory cells conduct current at that gate voltage. In some embodiments, when a flash memory cell's transistor conducts current at a given reading threshold voltage, indicating that the cell voltage is less than the reading threshold voltage, the raw data value for that read operation is a “1” and otherwise the raw data value is a “0.”

FIG. 2A is a block diagram illustrating an exemplary management module 121 in accordance with some embodiments. Management module 121 typically includes: one or more processing units (CPUs) 122 for executing modules, programs and/or instructions stored in memory 202 and thereby performing processing operations; memory 202; and one or more communication buses 204 for interconnecting these components. One or more communication buses 204, optionally, include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. Management module 121 is coupled to buffer 123, buffer 124, error control module 125, and storage medium I/O 128 by one or more communication buses 204. Memory 202 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 202, optionally, includes one or more storage devices remotely located from the CPU(s) 122. Memory 202, or alternatively the non-volatile memory device(s) within memory 202, comprises a non-transitory computer readable storage medium. In some embodiments, memory 202, or the non-transitory computer readable storage medium of memory 202, stores the following programs, modules, and data structures, or a subset or superset thereof:

• data read module 206 for reading data from storage medium 130 comprising flash memory (e.g., one or more flash memory devices each comprising a plurality of die);
• data write module 208 for writing data to storage medium 130;
• data erase module 210 for erasing data from storage medium 130;
• die group module 212 for dividing storage medium 130 into a plurality of die groups and managing the plurality of die groups, including but not limited to:
  • endurance metric module 214 for obtaining an endurance metric for each of the plurality of die;
  • ranking module 216 for ranking the plurality of die based on their corresponding endurance metrics;
  • grouping module 218 for assigning the plurality of die to die groups based on their corresponding rankings, including die group mapping 220 associating logical addresses of die to a die group; and
  • a selection module 222 for selecting a die group for a memory operation;
• characterization vector table 234 that includes a collection of characterization vectors 236 each storing characterization data for a respective portion of storage medium 130 (e.g., a flash memory device, die, block zone, block, word line, word line zone or page portion of storage medium 130); and
• logical to physical mapping 238 associating logical addresses with physical addresses for respective portions of storage medium 130 (e.g., a distinct flash memory device, die, block zone, block, word line, word line zone or page portion of storage medium 130).

Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules may be combined or otherwise re-arranged in various embodiments. In some embodiments, memory 202 may store a subset of the modules and data structures identified above. Furthermore, memory 202 may store additional modules and data structures not described above. In some embodiments, the programs, modules, and data structures stored in memory 202, or the non-transitory computer readable storage medium of memory 202, provide instructions for implementing any of the methods described below with reference to FIGS. 6A-6B.

Although FIG. 2A shows a management module 121, FIG. 2A is intended more as functional description of the various features which may be present in a management module than as a structural schematic of the embodiments described herein. In practice, and as recognized by those of ordinary skill in the art, the programs, modules, and data structures shown separately could be combined and some programs, modules, and data structures could be separated.

FIG. 2B is a block diagram illustrating an implementation of a characterization vector table 234 in accordance with some embodiments. Characterization vector table 234 includes a collection of characterization vectors 236, that each store characterization data associated with a respective portion of a storage medium 130 (e.g., a distinct device, die, block zone, block, word line, word line zone or page portion of storage medium 130). In some embodiments, each vector (e.g., vector 236-1, vector 236-2, . . . , vector 236-n) in the collection of characterization vectors 236 stores characterization data derived during each of at least two time periods (e.g., time T-1 and time T-2, or Xprogram-erase (PE) cycles and 2X PE cycles). In some embodiments, the characterization data stored in characterization vectors 236 is statistically derived. For example, without limitation, in some embodiments in which a storage medium 130 (FIG. 1) (e.g., one or more flash memory devices, each comprising a plurality of die) includes a plurality of die, characterization vector table 234 includes at least one characterization vector for each die. In another example, in some embodiments, characterization vector table 234 includes a set of distinct characterization vectors 236 for each die in storage medium 130, and the set of distinct characterization vectors 236 for each die includes at least one distinct characterization vector for each block in the die. More detailed example embodiments of characterization vectors 236 are described below with reference to FIG. 2C.

FIG. 2C is a schematic diagram of an implementation of a characterization vector 240 (e.g., corresponding to one of characterization vectors 236 shown in FIG. 2B) for a respective die of the plurality of die in accordance with some embodiments. In some embodiments, characterization data stored in characterization vector 240 for the respective die includes storage medium characterization parameter values such endurance metric field 242 indicating the endurance capabilities of the respective die (e.g., a number of program-erase (PE) cycles sustained by a representative block within the respective die during a calibration routine), a PE cycles field 244 indicating a current count of the number of PE cycles performed on the respective die, and an average bit error rate (BER) 246 indicating an average BER of one or more blocks within the respective die. In some embodiments, the one or more storage medium characterization parameter values provide an indication of at least one of: a physical characteristic associated with the respective portion the storage medium (e.g., a distinct flash memory device, die, block zone, block, word line, word line zone or page portion of storage medium 130, FIG. 1), a mode of operation associated with the respective portion of the storage medium, a history of use associated with the respective portion of the storage medium, a condition characteristic associated with the respective portion of the storage medium, a read type associated with the respective portion of the storage medium, and location(s) associated with the respective portion of the storage medium.

FIG. 3 is a schematic diagram of non-volatile memory (e.g., flash memory) in accordance with some embodiments. In some embodiments, storage controller 120 is communicatively coupled to storage medium 130 (e.g., flash memory comprising one or more flash memory devices each comprising a plurality of die) via a set of channels (e.g., connections 103, FIG. 1). In FIG. 3, 16 channels communicatively couple flash memory 300 (e.g., a flash memory device with 128 die) to storage controller 120. In FIG. 3, a subset of the plurality of die (e.g., 8 die) of flash memory 300 are connected to each channel. In this way, storage controller 120 is configured to perform memory operations on 16 die in parallel by utilizing the 16 channels. In some embodiments, the die are physically soldered to a channel and each die is connected to only one channel.

FIG. 4 is a block diagram of a plurality of die groups in accordance with some embodiments. As was the case in FIG. 3, in FIG. 4, non-volatile memory (e.g., flash memory) comprises a plurality of die (e.g., 128 die) and the plurality of die are divided into subsets (e.g., 8 die per subset), where each subset of die is connected to a channel (e.g., one of 16 channels).

For example, prior to ordinary operational use, a calibration routine is run on a representative block from each die of flash memory, which simulates end of life conditions. For example, the BER for data read from a representative block of the respective die measured after a predefined number of PE cycles are performed on the representative block is the endurance metric. In another example, the total number of PE cycles performed on the representative block of the respective die when the BER for data read from the representative block meets or exceeds a predefined threshold is the endurance metric. In some embodiments, the endurance metric (e.g., a BER or number of PE cycles) for a respective die is stored in endurance metric field 242 of characterization vector 236 associated with the respective die.

In some embodiments, storage controller 120 or a component thereof (e.g., endurance metric module 214, FIG. 2A) is configured to obtain an endurance metric for each die from characterization vector table 234, and storage controller 120 or a component thereof (e.g., ranking module 216, FIG. 2A) is configured to rank the die in each channel based on their corresponding endurance metrics. In FIG. 4, the 8 die in channel 0 are ranked from highest to lowest. With respect to channel 0, die 3 is the highest ranked die (e.g., rank 1) because the endurance metric associated with die 3 is higher than the other die in channel 0. With respect to channel 0, die 6 is the lowest ranked die (e.g., rank 8) because the endurance metric associated with die 6 is lower than the other die in channel 0.

In some embodiments, storage controller 120 or a component thereof (e.g., grouping module 218, FIG. 2A) is configured to group the plurality of die in the flash memory device into die groups based on their corresponding rankings. In some embodiments, one or more die in each channel is assigned to a die group, where the die assigned to a respective die group have a rank, or range of ranks, corresponding to the respective die group, and each die group has a distinct corresponding rank or range of ranks. In FIG. 4, the die in each channel with ranks 1-2 (e.g., die 3 and 7 in channel 0, and die 6 and 0 in channel 15) are assigned to die group 0, the die in each channel with ranks 3-4 (e.g., die 0 and 1 in channel 0, and die 3 and 7 in channel 15) are assigned to die group 1, the die in each channel with ranks 5-6 (e.g., die 5 and 4 in channel 0, and die 5 and 2 in channel 15) are assigned to die group 2, and the die in each channel with ranks 7-8 (e.g., die 2 and 6 in channel 0, and die 4 and 1 in channel 1) are assigned to die group 3.

In some embodiments, a plurality of channels (e.g., 16 channels) communicatively couples the storage controller to the flash memory and each of the plurality of channels is communicatively coupled with a respective subset of the plurality of die (e.g., 8 or 16 die per channel). In some implementations, the die assigned to any particular die group include unequal numbers of die from the various channels. For example, if one channel (e.g., a first channel) has a greater number of die in the endurance metric range of corresponding to a particular die group than another channel (e.g., a second channel), the particular die group will include a greater number of die from the first channel than from the second channel.

Furthermore, the number of die from a respective channel that are assigned to a respective die group varies based on the endurance metric of the die in the respective channel. For example, zero die in a respective channel are assigned to a die group because the endurance metrics of the die in the respective channel do not meet the range of endurance metrics associated with the respective die group. In another example, all die in a respective channel are assigned to a single die group based on their respective endurance metrics because the endurance metrics of the die in the respective channel all meet the range of endurance metrics associated with the respective die group.

FIG. 5 is a diagram of a die group mapping corresponding to the plurality of die groups in FIG. 4 in accordance with some embodiments. In some embodiments, die group mapping 220 is stored in storage controller memory and includes the logical address (or range of logical addresses, or a representation thereof) for die assigned to each die group. In FIG. 5, die group mapping 220 associates the logical address (or range of logical addresses) for all die in channels 0-15 with their assigned die groups. In FIG. 5, the logical addresses for die 3 and 7 in channel 0 (e.g., logical addresses 504 and 506, respectively) are included in entry 502 associated with die group 0.

FIGS. 6A-6B illustrate a flowchart representation of a method 600 of managing a storage system that comprises a storage controller and non-volatile memory (e.g., flash memory), the non-volatile memory comprising one or more non-volatile memory devices each comprising a plurality of die and each die comprising a plurality of blocks (sometimes also called subunits). In some embodiments, method 600 is performed by storage controller 120 in FIG. 1, where a plurality of channels (e.g., 16 channels) communicatively couple storage controller 120 to the flash memory. In some embodiments, each channel is communicatively coupled with a respective subset of the plurality of die (e.g., 8 or 16 die per channel).

In some embodiments, method 600 is performed by storage controller 120 or one or more components of storage controller 120 (e.g., management module 121, FIGS. 1 and 2A). In some embodiments, method 600 is governed by instructions that are stored in a non-transitory computer readable storage medium and the instructions are executed by one or more processors of storage controller 120, such as one or more processing units (CPUs) 122 of management module 121.

The storage controller obtains (602) an endurance metric (e.g., a BER or number of PE cycles) for each of the plurality of die. In some embodiments, the endurance metric is any information indicating an estimated longevity or lifespan for a respective die. In some embodiments, the endurance metrics are based on the results of a calibration routine performed on a representative die from each die prior to ordinary operations (e.g., during initialization or at the factory). In some embodiments, the endurance metrics are based on real data (e.g., an average BER of blocks within each die) after a period of ordinary operations. In some embodiments, the endurance metric for a respective die is based on an amount of time required to erase data from the respective die (e.g., the erase time increases as the respective die ages). In some embodiments, the endurance metric for a respective die is based on a length of time or number of processing cycles required to successfully decode a codeword stored in the respective die (e.g., a number of iterations or cycles required to decode a low-density parity check code (LDPC) encoded codeword). Storage controller 120 or a component thereof (e.g., endurance metric module 214, FIG. 2A) is configured to obtain the endurance metric for a respective die from a characteristic vector 236 associated with the respective die that is stored in characterization vector table 234.

In some embodiments, each die is associated with (604) a single channel. For example, each of the plurality of die comprising the flash memory is physically connected with or soldered to a single channel.

In some embodiments, obtaining the endurance metric for each of the plurality of die includes performing (606) a calibration routine on a representative block in each of the plurality of die to generate the endurance metric (e.g., a BER or number of PE cycles) for each of the plurality of die. For example, prior to ordinary operational use (e.g., during initialization or at the factory), a calibration routine is performed on a representative block from each die of the flash memory to simulate end of life conditions. In some embodiments, the BER for data read from a representative block of the respective die measured after a predefined number of PE cycles are performed on the representative block is the endurance metric. In some embodiments, the total number of PE cycles performed on the representative block of the respective die when the BER for data read from the representative block meets or exceeds a predefined threshold is the endurance metric. In some embodiments, the endurance metric for the respective die is stored in endurance metric field 242 of characterization vector 236 associated with the respective die.

For each channel, the storage controller ranks (608) the die in the channel in accordance with the endurance metrics of the respective die in the channel. In some embodiments, storage controller 120 or a component thereof (e.g., ranking module 216, FIG. 2A) is configured to rank the die in each channel based on the endurance metrics of the die in the channel. In FIG. 4, for example, the 8 die in channel 0 are ranked from highest to lowest. With respect to channel 0, die 3 is the highest ranked die (e.g., rank 1) because the endurance metric associated with die 3 is higher than the other die in channel 0. With respect to channel 0, die 6 is the lowest ranked die (e.g., rank 8) because the endurance metric associated with die 6 is lower than the other die in channel 0.

The storage controller assigns (610) to each die group (sometimes also called a “super device”) of a plurality of die groups, one or more die from each channel (e.g., one or two die per channel are typically assigned to a die group) based on the ranking of the die in each of the channels, where the die assigned to a respective die group have a rank, or range of ranks, corresponding to the respective die group, and each die group has a distinct corresponding rank or range of ranks. In some embodiments, storage controller 120 or a component thereof (e.g., grouping module 218, FIG. 2A) is configured to group the plurality of die in the flash memory into die groups based on their corresponding rankings. In some embodiments, one or more die in each channel is assigned to a die group. In some embodiments, the number of die assigned to a first die group of the plurality of die groups is unequal to the number of die assigned to a second die group of the plurality of die groups. In FIG. 4, for example, the die in each channel with ranks 1-2 (e.g., die 3 and 7 in channel 0, and die 6 and 0 in channel 15) are assigned to die group 0, the die in each channel with ranks 3-4 (e.g., die 0 and 1 in channel 0, and die 3 and 7 in channel 15) are assigned to die group 1, the die in each channel with ranks 5-6 (e.g., die 5 and 4 in channel 0, and die 5 and 2 in channel 15) are assigned to die group 2, and the die in each channel with ranks 7-8 (e.g., die 2 and 6 in channel 0, and die 4 and 1 in channel 15) are assigned to die group 3. In some embodiments, storage controller 120 or a component thereof (e.g., grouping module 218, FIG. 2A) is also configured to store the logical address (or range of logical addresses, or a representation thereof) for die assigned to each die group in die group mapping 220.

In response to a write command specifying a set of write data, the storage controller writes (612) the write data to the flash memory by writing in parallel subsets of the write data to the plurality of die assigned to a single die group of the plurality of die groups. For example, with reference to FIG. 1, storage controller 120 receives a host write command via control line 111 and a set of write data via data connections 101 from computer system 110 (e.g., a host). In response to the host write command, storage controller 120 or a component thereof (e.g., data write module 208, FIG. 2A) is configured to write data to the flash memory by writing, in parallel, subsets of the write data to the plurality of die assigned to a single die group of the plurality of die groups. For example, with respect to FIG. 4, the write data is striped across the die comprising the single die group (e.g., die group 0) by concurrently writing subsets of the write data to each of the 16 rank 1 die in die group 0, in parallel, using the 16 channels.

In some embodiments, writing the write data includes mapping (614) logical addresses of the write data to physical addresses in the flash memory in accordance with the die assigned to each die group of the plurality of die groups. In some embodiments, storage controller or a component thereof maps the logical address (or range of logical addresses) of the write data to physical addresses in the flash memory in logical to physical mapping 238. In some embodiments, logical to physical mapping 238 is stored at storage controller 120 or remote from storage controller 120.

In some embodiments, the storage controller selects (616) a die group based on an operation to be performed and the endurance metrics of die assigned to the die group. In some embodiments, storage controller 120 or a component thereof (e.g., selection module 222, FIG. 2A) is configured to select a die group of the plurality of die groups based on the operation to be performed (e.g., host write or garbage collection operation) and the endurance metrics of die assigned to the die group. For example, selection module 222 selects a die group comprising highly ranked die for write operations. In another example, selection module 222 selects a die group comprising lower ranked die for garbage collection/recycling operations.

In some embodiments, after a first predefined condition occurs, the storage controller stores (618) an average bit error rate for one or more blocks (e.g., a group of blocks or “superblock”) of each die. In some embodiments, in response to detecting the first predefined condition, storage controller 120 or a component thereof is configured to sample the current BER for one or more blocks of each die, average the BERs of the one or more blocks of each die, and store current the average BER for each die. In some embodiments, the current the average BER of a respective die is stored in average BER field 246 of characterization vector 236 associated with the respective die. In some embodiments, the first predefined condition is a periodic trigger (e.g., every X hours or every N PE cycles performed on the respective die) or an event (e.g., shutdown, power loss, etc.).

In some embodiments, after a second predefined condition occurs (620), the storage controller obtains (622), for each of the plurality of die, an updated endurance metric. In some embodiments, storage controller 120 or a component thereof (e.g., endurance metric module 214, FIG. 2A) is configured to store an updated endurance metric for each of the plurality of die. In some embodiments, the updated endurance metric is based on the average bit error rate for the one or more blocks of each die, which was stored in response to detecting the first predefined condition. In some embodiments, the updated endurance metric is based on an erase time associated with the die or the amount of time required to successfully decode a codeword stored in the die. In some embodiments, the updated endurance metric for a respective die is stored in endurance metric field 242 of characterization vector 236 associated with the respective die and replaces the previous value stored in endurance metric field 242. In some embodiments, the second predefined condition is a periodic trigger (e.g., every Y days, every M PE cycles performed on the respective die, etc.).

In some embodiments, after the second predefined condition occurs (620), the storage controller re-ranks (624), for each channel, the die in the channel in accordance with the updated endurance metrics of the respective die in the channel. In some embodiments, storage controller 120 or a component thereof (e.g., ranking module 216, FIG. 2A) is configured to re-rank the die in each channel based on updated endurance metric associated with each die in the channel.

In some embodiments, after the second predefined condition occurs (620), the storage controller assigns (626) to each die group one or more die from each channel based on the re-ranking of the die in each of the channels. In some embodiments, storage controller 120 or a component thereof (e.g., grouping module 218, FIG. 2A) is configured to group the plurality of die in the flash memory into die groups based on their corresponding re-rankings.

In some embodiments, after assigning one or more die from each channel to each die group based on the re-ranking of the die in each of the channels, the storage controller updates (628) a die group mapping, the die group mapping includes a table of logical addresses of die comprising each die group. In some embodiments, after assigning one or more die from each channel to each die group based on the re-ranking of the die in each of the channels, storage controller 120 or a component thereof (e.g., grouping module 218, FIG. 2A) is configured to update die group mapping 220 to reflect the logical address (or range of logical addresses, or a representation thereof) for die assigned to each die group based on the re-ranking.

FIG. 7 illustrates a flowchart representation of a method 700 of managing a storage system that comprises a storage controller and non-volatile memory (e.g., flash memory), the non-volatile memory comprising one or more non-volatile memory devices each comprising a plurality of die and each die comprising a plurality of blocks (sometimes also called subunits). In some embodiments, method 700 is performed by storage controller 120 or one or more components of storage controller 120 (e.g., management module 121, FIGS. 1 and 2A). In some embodiments, method 700 is governed by instructions that are stored in a non-transitory computer readable storage medium and the instructions are executed by one or more processors of storage controller 120, such as one or more processing units (CPUs) 122 of management module 121.

The storage controller obtains (702) an endurance metric (e.g., a BER or number of PE cycles) for each of the plurality of die. In some embodiments, the endurance metric is any information indicating an estimated longevity or lifespan for a respective die. In some embodiments, the BER for data read from a representative block of the respective die measured after a predefined number of PE cycles are performed on the representative block is the endurance metric. In some embodiments, the total number of PE cycles performed on the representative block of the respective die when the BER for data read from the representative block meets or exceeds a predefined threshold is the endurance metric. In some embodiments, the endurance metric for a respective die is based on an amount of time required to erase data from the respective die (e.g., the erase time increases as the respective die ages). In some embodiments, the endurance metric for a respective die is based on a length of time or number of processing cycles required to successfully decode a codeword stored in the respective die (e.g., a number of iterations or cycles required to decode a low-density parity check code (LDPC) encoded codeword). Storage controller 120 or a component thereof (e.g., endurance metric module 214, FIG. 2A) is configured to obtain the endurance metric for a respective die from a characteristic vector 236 associated with the respective die that is stored in characterization vector table 234.

The storage controller sorts (704) the plurality of die into a plurality of die groups based on their corresponding endurance metrics, where each die group includes one or more die and each die group is associated with a range of endurance metrics. For example, die with substantially similar endurance metrics are assigned to a same die group. As such, when memory operations are performed on the die of a particular die group, the die in the particular die group will decay at a substantially similar rate. In some embodiments, the number of die assigned to a first die group of the plurality of die groups is unequal to the number of die assigned to a second die group of the plurality of die groups.

In some embodiments, a plurality of channels (e.g., 16 channels) communicatively couples the storage controller to the flash memory and each of the plurality of channels is communicatively coupled with a respective subset of the plurality of die (e.g., 8 or 16 die per channel). In some implementations, the die assigned to any particular die group include unequal numbers of die from the various channels. For example, if one channel (e.g., a first channel) has a greater number of die in the endurance metric range of corresponding to a particular die group than another channel (e.g., a second channel), the particular die group will include a greater number of die from the first channel than from the second channel.

Furthermore, the number of die from a respective channel that are assigned to a respective die group varies based on the endurance metric of the die in the respective channel. For example, zero die in a respective channel are assigned to a die group because the endurance metrics of the die in the respective channel do not meet the range of endurance metrics associated with the respective die group. In another example, all die in a respective channel are assigned to a single die group based on their respective endurance metrics because the endurance metrics of the die in the respective channel all meet the range of endurance metrics associated with the respective die group.

In response to a write command specifying a set of write data, the storage controller writes (706) the write data to the flash memory by writing in parallel subsets of the write data to the plurality of die assigned to a single die group of the plurality of die groups. For example, storage controller 120 receives a host write command via control line 111 and a set of write data via data connections 101 from computer system 110 (e.g., a host). In response to the host write command, storage controller 120 or a component thereof (e.g., data write module 208, FIG. 2A) is configured to write data to the flash memory by writing (e.g., in parallel when possible) subsets of the write data to the plurality of die assigned to a single die group of the plurality of die groups.

In some embodiments, after a predefined condition occurs, the storage controller obtains, for each of the plurality of die, an updated endurance metric. In some embodiments, storage controller 120 or a component thereof (e.g., endurance metric module 214, FIG. 2A) is configured to store an updated endurance metric for each of the plurality of die. In some embodiments, the updated endurance metric for a respective die is stored in endurance metric field 242 of characterization vector 236 associated with the respective die and replaces the previous value stored in endurance metric field 242. In some embodiments, the second predefined condition is a periodic trigger (e.g., every Y days, every M PE cycles performed on the respective die, etc.).

In some embodiments, after the predefined condition occurs, the storage controller re-sorts the plurality of die into a plurality of updated die groups based on their corresponding updated endurance metrics, where each updated die group includes one or more die and each updated die group is associated with a range of endurance metrics.

In some embodiments, after re-sorting the plurality of die into a plurality of updated die groups based on their corresponding updated endurance metrics, the storage controller updates a die group mapping, where the die group mapping includes a table of logical addresses of die comprising each die group.

It will be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first die group could be termed a second die group, and, similarly, a second die group could be termed a first die group, which changing the meaning of the description, so long as all occurrences of the “first die group” are renamed consistently and all occurrences of the “second die group” are renamed consistently. The first die group and the second die group are both die groups, but they are not the same die group.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain principles of operation and practical applications, to thereby enable others skilled in the art.

Claims

1. A method of managing a storage system that comprises a storage controller and non-volatile memory, the non-volatile memory comprising a plurality of die and each die comprising a plurality of subunits, the method comprising: at the storage controller coupled with the non-volatile memory: for each of the plurality of die, obtaining an endurance metric;sorting the plurality of die into a plurality of die groups based on their corresponding endurance metrics, wherein each die group includes one or more die and each die group is associated with a corresponding range of endurance metrics, and wherein a plurality of the die groups each includes two or more die;storing a die group mapping in the storage controller that indicates which die are assigned to each die group of the plurality of die groups, wherein the storage controller includes a logical address to physical address mapping distinct from the die group mapping; andin response to a write command specifying a set of write data, writing the set of write data to the non-volatile memory by writing in parallel distinct respective subsets of the set of write data to distinct respective die assigned to a single die group of the plurality of die groups.

2. The method of claim 1, wherein obtaining the endurance metric for each of the plurality of die includes performing a calibration routine on a representative subunit in each of the plurality of die to generate the endurance metric for each of the plurality of die.

3. The method of claim 1, wherein the plurality of die groups includes at least four die groups, and writing the write data includes mapping, in the logical address to physical address mapping, logical addresses of the write data to physical addresses in the non-volatile memory in accordance with the die group mapping in the storage controller that indicates which die are assigned to each die group of the plurality of die groups.

4. The method of claim 1, further comprising: selecting a die group based on an operation to be performed and the endurance metrics of die assigned to the die group.

5. The method of claim 1, further comprising: after a predefined condition occurs: obtaining, for each of the plurality of die, an updated endurance metric; andre-sorting the plurality of die into a plurality of updated die groups based on their corresponding updated endurance metrics, wherein each updated die group includes one or more die and each updated die group is associated with a corresponding range of endurance metrics.

6. The method of claim 5, further comprising: after re-sorting the plurality of die into a plurality of updated die groups, updating the die group mapping, the die group mapping including a table of logical addresses of die comprising each die group.

7. The method of claim 1, wherein a plurality of channels communicatively couple the storage controller to the non-volatile memory, wherein each channel is communicatively coupled with a respective subset of the plurality of die, the method further comprising: for each channel, ranking the die in the channel in accordance with the endurance metrics of the respective die in the channel; andwherein sorting the plurality of die into a plurality of die groups based on their corresponding endurance metrics includes assigning, based on the ranking of the die in each of the channels, to each die group of a plurality of die groups, one or more die from each channel, wherein the die assigned to a respective die group have a rank, or range of ranks, corresponding to the respective die group, and each die group has a distinct corresponding rank or range of ranks.

8. The method of claim 7, further comprising: after a predefined condition occurs: obtaining, for each of the plurality of die, an updated endurance metric;for each channel, re-ranking the die in the channel in accordance with the updated endurance metrics of the respective die in the channel; andbased on the re-ranking of the die in each of the channels, assigning to each die group one or more die from each channel.

9. The method of claim 8, further comprising: after assigning one or more die from each channel to each die group based on the re-ranking of the die in each of the channels, updating the die group mapping, the die group mapping including a table of logical addresses of die comprising each die group.

10. A storage controller, comprising: one or more processors;a host interface configured to couple the storage controller to a host;a storage medium interface configured to couple the storage controller to non-volatile memory comprising a plurality of die, each die comprising a plurality of subunits; anda storage controller memory storing instructions, which, when executed by the one or more processors, cause the storage controller to perform operations comprising: for each of the plurality of die, obtaining an endurance metric;sorting the plurality of die into a plurality of die groups based on their corresponding endurance metrics, wherein each die group includes one or more die and each die group is associated with a corresponding range of endurance metrics, and wherein a plurality of the die groups each includes two or more die;storing a die group mapping in the storage controller that indicates which die are assigned to each die group of the plurality of die groups, wherein the storage controller includes a logical address to physical address mapping distinct from the die group mapping; andin response to a write command specifying a set of write data, writing the set of write data to the non-volatile memory by writing in parallel distinct respective subsets of the set of write data to distinct respective die assigned to a single die group of the plurality of die groups.

11. The storage controller of claim 10, wherein obtaining the endurance metric for each of the plurality of die includes performing a calibration routine on a representative subunit in each of the plurality of die to generate the endurance metric for each of the plurality of die.

12. The storage controller of claim 10, wherein the plurality of die groups includes at least four die groups, and writing the write data includes mapping, in the logical address to physical address mapping, logical addresses of the write data to physical addresses in the non-volatile memory in accordance with the die group mapping in the storage controller that indicates which die are assigned to each die group of the plurality of die groups.

13. The storage controller of claim 10, wherein the instructions cause the storage controller to perform operations further comprising: selecting a die group based on an operation to be performed and the endurance metrics of die assigned to the die group.

14. The storage controller of claim 10, wherein the instructions cause the storage controller to perform operations further comprising: after a predefined condition occurs: obtaining, for each of the plurality of die, an updated endurance metric; andre-sorting the plurality of die into a plurality of updated die groups based on their corresponding updated endurance metrics, wherein each updated die group includes one or more die and each updated die group is associated with a corresponding range of endurance metrics.

15. The storage controller of claim 14, wherein the instructions cause the storage controller to perform operations further comprising: after re-sorting the plurality of die into a plurality of updated die groups, updating the die group mapping, the die group mapping including a table of logical addresses of die comprising each die group.

16. The storage controller of claim 10, wherein: a plurality of channels communicatively couple the storage controller to the non-volatile memory, wherein each channel is communicatively coupled with a respective subset of the plurality of die; andthe instructions cause the storage controller to perform operations further comprising: for each channel, ranking the die in the channel in accordance with the endurance metrics of the respective die in the channel; andwherein sorting the plurality of die into a plurality of die groups based on their corresponding endurance metrics includes assigning, based on the ranking of the die in each of the channels, to each die group of a plurality of die groups, one or more die from each channel, wherein the die assigned to a respective die group have a rank, or range of ranks, corresponding to the respective die group, and each die group has a distinct corresponding rank or range of ranks.

17. A non-transitory computer readable storage medium, storing one or more programs for execution by one or more processors of a storage controller communicatively coupled with non-volatile memory comprising a plurality of die and each die comprising a plurality of subunits, the one or more programs including instructions that when executed by the one or more processors cause the storage controller to: for each of the plurality of die, obtain an endurance metric;sort the plurality of die into a plurality of die groups based on their corresponding endurance metrics, wherein each die group includes one or more die and each die group is associated with a corresponding range of endurance metrics, and wherein a plurality of the die groups each includes two or more die;storing a die group mapping in the storage controller that indicates which die are assigned to each die group of the plurality of die groups, wherein the storage controller includes a logical address to physical address mapping distinct from the die group mapping; andin response to a write command specifying a set of write data, write the set of write data to the non-volatile memory by writing in parallel distinct respective subsets of the set of write data to distinct respective die assigned to a single die group of the plurality of die groups.

18. The non-transitory computer readable storage medium of claim 17, wherein obtaining the endurance metric for each of the plurality of die includes performing a calibration routine on a representative subunit in each of the plurality of die to generate the endurance metric for each of the plurality of die.

19. The non-transitory computer readable storage medium of claim 17, wherein the plurality of die groups includes at least four die groups, and writing the write data includes mapping, in the logical address to physical address mapping, logical addresses of the write data to physical addresses in the non-volatile memory in accordance with the die group mapping in the storage controller that indicates which die are assigned to each die group of the plurality of die groups.

20. The non-transitory computer readable storage medium of claim 17, wherein the one or more programs further include instructions that when executed by the one or more processors cause the storage controller to: select a die group based on an operation to be performed and the endurance metrics of die assigned to the die group.

21. The non-transitory computer readable storage medium of claim 17, wherein the one or more programs further include instructions that when executed by the one or more processors cause the storage controller to: after a predefined condition occurs: obtain, for each of the plurality of die, an updated endurance metric; andre-sort the plurality of die into a plurality of updated die groups based on their corresponding updated endurance metrics, wherein each updated die group includes one or more die and each updated die group is associated with a corresponding range of endurance metrics.

22. The non-transitory computer readable storage medium of claim 21, wherein the one or more programs further include instructions that when executed by the one or more processors cause the storage controller to: after re-sorting the plurality of die into a plurality of updated die groups, update the die group mapping, the die group mapping including a table of logical addresses of die comprising each die group.

23. The non-transitory computer readable storage medium of claim 17, wherein: a plurality of channels communicatively couple the storage controller to the non-volatile memory, wherein each channel is communicatively coupled with a respective subset of the plurality of die; andthe one or more programs further include instructions that when executed by the one or more processors cause the storage controller to: for each channel, rank the die in the channel in accordance with the endurance metrics of the respective die in the channel; andwherein sorting the plurality of die into a plurality of die groups based on their corresponding endurance metrics includes assigning, based on the ranking of the die in each of the channels, to each die group of a plurality of die groups, one or more die from each channel, wherein the die assigned to a respective die group have a rank, or range of ranks, corresponding to the respective die group, and each die group has a distinct corresponding rank or range of ranks.