Patent Application
20170116117
The present disclosure is generally related to data storage devices and more particularly to storage descriptors corresponding to portion(s) of memory.
Storage devices enable users to store and retrieve data. For example, some storage devices include non-volatile memory to store data and a controller that coordinates access to the non-volatile memory and performs error detection/correction. According to some industrial standards/protocols, a host device may provide a read or write command to a data storage device. In non-volatile memory express (NVMe) systems, read or write commands may be associated with physical locations in a host memory at the host device. In some cases, the physical locations are indicated to the data storage device via scatter gather list(s) (SGL(s)).
A SGL may include multiple descriptors. One type of SGL descriptor, known as a data block descriptor, describes a host buffer. During a write operation, a controller of a storage device may retrieve data from host buffers described by the SGL and provide the data to the non-volatile memory for storage. During a read operation, the controller may send data read from the non-volatile memory for storage in the host buffers described by the SGL. Due to SGL size or other conditions, segments of a SGL (where each segment includes one or more descriptors) may be stored at a host device in non-contiguous fashion. To illustrate, each segment of the SGL (other than the last segment) may include pointer to a “next” segment, to enable the controller to retrieve and process each portion of the entire SGL.
When the data being transferred to the host buffers (e.g., during a read operation of the non-volatile memory) becomes available out-of-order, multiple descriptors of the SGL may be traversed, in order, to identify the appropriate host buffer for the data. For example, consider a SGL that describes X host buffers, such as host buffer 1, host buffer 2, . . . host buffer X. If data to be stored in the Nth host buffer (N<X) becomes available first, then descriptors 1, 2, . . . (N−1) would be traversed before reaching the Nth descriptor and determining the physical location (e.g., address) of the Nth host buffer at the host device. However, traversing large portions (or the entirety) of the SGL for out-of-order data operations may result in performance loss (e.g., reduced throughput) at the data storage device. In addition, including sufficient memory in the controller to store an entire pre-fetched SGL for traversal may increase cost, complexity, or both associated with the controller.
The present disclosure describes systems and methods of using metrics to improve performance of out-of-order operations, such as when SGLs are used during read or write operations in accordance with NVMe. In accordance with the described techniques, a controller of a data storage device may collect, on-the-fly, statistical information about host buffers described by SGL descriptors. For example, the statistical information can include an average host buffer size and a size deviation. In other examples, more metrics, fewer metrics, and/or different metrics may be used. When data is available out-of-order, the controller may perform a non-linear search based on the metric(s) to identify the host buffer corresponding to the data. To illustrate, the controller may use the metric(s) to determine a subset (e.g., range) of SGL descriptors for the data, where one of the SGL descriptors in the subset corresponds to the host buffer for the data. By traversing through the SGL descriptors in the subset rather in the entire SGL to determine the physical memory location of the host buffer for the data, the controller may improve overall performance of the data storage device.
In some examples, the metrics are calculated and updated while fetching the SGL, before initiating data transfer with a non-volatile memory of the data storage device. Alternatively, the metrics can be updated in parallel with data transfer operations. For example, the metrics can be updated based on a later-fetched portion of the SGL while data transfer is performed based on an earlier-fetched portion of the SGL.
Particular aspects of the disclosure are described below with reference to the drawings. In the description, common or similar features or components may be designated by common reference numbers. As used herein, “exemplary” may indicate an example, an implementation, and/or an aspect, and should not be construed as indicating a preference or a preferred implementation.
Referring to
The data storage device 110 may include the memory device 112, such as a non-volatile memory device. The memory device 112 may include a flash memory (e.g., a NAND flash memory) or a resistive memory, such as a resistive random access memory (ReRAM), as illustrative examples. In some examples, the memory device 112 may have a three-dimensional (3D) memory configuration. As used herein, a 3D memory device may include multiple physical levels of storage elements (instead of having a single physical level of storage elements, as in a planar memory device). As an example, the memory device 112 may have a 3D vertical bit line (VBL) configuration. In a particular implementation, the memory device 112 is a non-volatile memory having a 3D memory array configuration that is monolithically formed in one or more physical levels of arrays of memory cells having an active area disposed above a silicon substrate. Alternatively, the memory device 112 may have another configuration, such as a two-dimensional (2D) memory configuration or a non-monolithic 3D memory configuration (e.g., a stacked die 3D memory configuration).
In some examples, the memory device 112 includes multiple memory dies. In such examples, when data is stored in the memory device 112, the data may be “striped” across one or more of the memory dies. Similarly, reading such data may include accessing one or more of the memory dies. In the illustrated example, the memory device 112 is illustrated as storing four distinct and differently-sized data items—Data 1, Data 2, Data 3, and Data 4, each of which may correspond to different files, parts of the same file, or other user data.
The data storage device 110 may include a controller 130 coupled to the memory device 112. In some implementations, the controller 130 corresponds to a semiconductor die (distinct from semiconductor die(s) of the memory device 112) that includes components of the controller 130. The controller 130 may include a read/write buffer 116, a scatter gather list (SGL) buffer 118, SGL fetch and statistical metric logic 120, a first memory 122 (e.g., random access memory (RAM)), or any combination thereof. The logic 120 may be implemented by software (e.g., instructions) executable by a processor (not shown) of the controller 130 to perform operations described herein. Alternatively, the logic 120 may include hardware (e.g., one or more circuits) configured to perform operations described herein. It should be noted that although the read/write buffer 116 and the SGL buffer 118 are shown as being part of the controller 130, in alternative embodiments one or both of the buffers 116, 118 may be external to the controller 130.
In an illustrative embodiment, the controller 130 may include additional components, such as an error correction coding (ECC) engine to detect and correct errors in data stored in the memory device 112. Examples of ECC schemes that may be used include, but are not limited to, Reed Solomon, Bose-Chaudhuri-Hocquenghem (BCH), low-density parity check (LDPC), and Turbo Code.
The access device 150 is configured to provide data to be stored at the memory device 112 (e.g., as part of a write command) and to request data to be read from the memory device 112 (e.g., as part of a read command) In an illustrative embodiment, the access device 150 may include a mobile telephone, a music player, a video player, a gaming console, an electronic book reader, a personal digital assistant (PDA), a computer (e.g., a laptop computer, a desktop computer, a tablet computer, etc.), another electronic device, or any combination thereof.
The access device 150 may include a processor (not shown) and memory accessible to the processor. For example, the memory of the access device 150 may include a second memory 156. At least a portion of the second memory 156 may be designated or allocated by the access device 150 for use by the controller 130 of the data storage device 110. For example, at least a portion of the second memory 156 may be allocated for use as host memory buffer(s) (HMB(s)). The second memory 156 may include volatile memory or non-volatile memory.
In an illustrative embodiment, the data storage device 110 and the access device 150 communicate via a non-volatile memory express (NVMe) interface. The access device 150 may include one or more submission queues 152 to store NVMe commands, such as read commands and write commands, that are to be performed by the data storage device 110. For example, a NVMe read command may indicate an amount of data to be read from the memory device 112 (which may be specified in numbers of logical blocks, bytes, etc., and may include “bit buckets” of data that are read from the memory device 112 but not transferred to the access device 150), a starting address (e.g., virtual address, physical address, logical block address, etc.) for the read operation, and a data pointer (DPTR) specifying where the read data is to be stored on the access device 150. Similarly, a NVMe write command may indicate an amount of data to be written, a destination address of the memory device 112 for the write operation, and a DPTR specifying a location on the access device 150 that the data is to be transferred from. When a SGL is used for a read command or a write command, the DPTR may enable the controller 130 to access at least an initial descriptor of a SGL. In one example, the DPTR may include the first segment or descriptor of the SGL. Alternatively, the DPTR may include a pointer to the first segment or descriptor of the SGL. In either case, and additional segments or descriptors of the SGL may be retrieved using a “next” pointer of the first segment or descriptor.
The access device 150 may also include one or more completion queues 154 into which the data storage device 110 may insert results of executing commands retrieved or received from the submission queue(s) 152. In an illustrative embodiment, the access device 150 includes a submission queue and a completion queue for each processor core in each processor of the access device 150. The access device 150 may also include additional submission queues and completion queues, such as for controller administration/management. In a particular embodiment, the data storage device 110 and the access device 150 use interrupt-driven communication. For example, to request that a read command 131 be executed, the access device 150 may insert the read command 131 into the submission queue(s) 152 and transmit a first interrupt to the data storage device 110. The controller 130 may respond to the first interrupt by retrieving the read command 131 from the submission queue(s) 152 and executing the read command 131. The data storage device 110 may store results (e.g., data, a completion code, an error code, etc.) of executing the read command 131 in the completion queue(s) 154 and may send a second interrupt to the access device 150 to indicate that execution of the read command 131 has been completed.
During operation, the access device 150 may allocate one or more portions of the second memory 156 for use by the controller 130. In the example of
The access device 150 may generate a SGL 160 to describe the allocated buffers, such as the allocated buffers 172-178. Generally, a SGL may be divided into segments, where each segment includes one or more descriptors. Examples of SGL descriptors may include data block descriptors, bit bucket descriptors, segment descriptors, and “last” segment descriptors. A SGL data block descriptor may include an address and a size, where the address specifies a starting physical memory address of a corresponding buffer and the size specifies the length (e.g., in bytes) of the buffer. To illustrate, in the example of
To request Data 1, Data 2, Data 3, and Data 4 be read from the memory device 112, the access device 150 may generate the read command 131. The controller 130 may receive the read command 131 and may send a SGL request 132 based on data (e.g., the DPTR) included in the read command 131. The controller 130 may receive SGL descriptors 133 in response to the SGL request 132. The SGL descriptors 133 may include one or more of the descriptors 161-168 of the SGL 160, or information determined therefrom. All or a subset of retrieved SGL descriptors 133 may be buffered in the SGL buffer 118, as shown.
The logic 120 may determine metrics based on SGL descriptors received by the controller 130 and may store the metrics in the first memory 122. In the example of
The controller 130 may also issue a data request 142 for Data 1, Data 2, Data 3, and Data 4 in response to the read command 131. In an illustrative example, the data request 142 is based on a size and an offset. In a particular aspect, the offset is measured from a “start” of the read command 131, such as a starting address specified by the read command 131. The data request 142 may be for particular memory sector addresses on particular memory dies or chips of the memory device 112. In a particular aspect, the memory device 112 includes read/write circuitry configured to sense bit values and store the bit values in latches, and the bit values may subsequently “toggled out” from the latches to the read/write buffer 116 of the controller 130 (illustrated in
Because SGL descriptors can indicate host buffers that have variable size, to determine the address A4 for the fourth buffer 178, current systems may traverse each descriptor of the SGL 160, starting from the initial descriptor 161, until the descriptor 164 for the fourth buffer 178 is found. In accordance with the described techniques, however, instead of traversing each and every descriptor of the SGL 160 until the descriptor for the fourth buffer 178 is found, the logic 120 may identify a subset 170 of descriptors based on the metrics stored at the first memory 122. In
The logic 120 may perform an in-order traversal of the descriptors 163-167 in the subset 170 until the descriptor 164 corresponding to Data 4 is identified, and may then determine the physical address A4 of the fourth buffer 178 based on the descriptor 164. If descriptor(s) of the subset 170 are unavailable in the SGL buffer 118 (e.g., the SGL buffer 118 may too small to store the entire SGL 160), the controller 130 may retrieve the descriptor(s) from the access device 150 (e.g., via another SGL request 132). After the address A4 is identified, the controller 130 may send Data 4 to the access device 150 for storage in the fourth buffer 178 based on the address A4, as shown at 134. As additional data (e.g., Data 1-3) becomes available, the controller 130 may determine the physical addresses for the additional corresponding host buffers (e.g., the buffers 172-176) and may send the additional data to the access device 150, as shown at 135. In some examples, an ECC engine of the controller 130 corrects one or more bit error(s) in the data before the data is provided to the access device 150.
After all requested data is provided to the access device 150, the controller may issue a completion command 136 indicating that the read command 131 has been executed. Alternatively, if the read command 131 fails, the completion command 136 may indicate why the read command 131 failed (e.g., invalid memory address, invalid host buffer, etc.).
The system 100 of
Referring to
The access device 150 may send the read command 131 to the controller 130, as shown. In a particular example, the read command 131 includes a SGL pointer. The controller 130 may send the SGL request 132 based on the SGL pointer and may receive the SGL descriptor(s) 133 in response to the SGL request 132. In a particular example, sending the SGL request 132 and receiving the SGL descriptor(s) 133 corresponds to a pre-fetch operation at the data storage device 110.
As the SGL descriptor(s) 133 are received, the controller 120 may determine statistical metrics, as shown at 201. For example, statistical metrics may include, but are not limited to, the average size 124 or the size deviation 126 described with reference to
As described with reference to
Referring to
In the illustrated example, the SGL includes sixty-four data block descriptors. The first data block descriptor indicates that a first host buffer length has a length (i.e., size) of 552 bytes, the second data block descriptor indicates that a second host buffer has a length of 2656 bytes, etc. As data block descriptors are processed, a sum of buffer lengths 301, average buffer length 302, and distance from average buffer length 303 may be determined (e.g., by the logic 120 of
The various statistical metrics illustrated in
where floor( ) is the rounding down function. In the described example, the average buffer length after all sixty-four data block descriptors have been encountered is 2777, and the deviation is 14. Thus, in the described example, the starting SGL descriptor of the subset is determined as:
The subset of descriptors to be searched may thus start at SGL descriptor 32 and may include deviation=14 descriptors (i.e., the subset of descriptors may be from SGL descriptor 32 to SGL descriptor 45, as shown in
Referring to
The method 400 may also include determining a metric based on the sizes included in the plurality of descriptors, at 404. For example, referring to
The method 400 may further include storing data indicating the first metric in the first memory, at 406. The data may enable the controller to locate a portion of the multiple portions by determining, based on the metric, a subset of the plurality of descriptors, the subset including a descriptor corresponding to the portion. For example, referring to
Referring to
The method 500 may also include determining a subset of the plurality of descriptors based on the metric, at 504. For example, referring to
The method 500 may further include determining a physical address corresponding to the portion of the second memory based on a descriptor of the subset of descriptors, at 506. For example, referring to
The methods 400, 500 of
In some implementations, a computer-readable medium stores instructions executable by a processing module to perform operations described herein. For example, the computer-readable medium, the processing module, or both may be included in the data storage device 110, the memory device 112, the controller 120, the first memory 122, the logic 120, the access device 150, the second memory 156, or any combination thereof
Although various components depicted herein are illustrated as block components and described in general terms, such components may include one or more microprocessors, state machines, or other circuits configured to enable such components to perform one or more operations described herein. For example, the logic 120 may represent physical components, such as hardware controllers, state machines, logic circuits, or other structures, to enable the controller 130 to perform operations described herein.
Alternatively or in addition, one or more components described herein may be implemented using a microprocessor or microcontroller programmed to perform operations, such as one or more operations of the methods 400, 500 of
In some examples, the data storage device 110 may be coupled to, attached to, or embedded within one or more accessing devices, such as within a housing of the access device 150. For example, the data storage device 110 may be embedded within the access device 150 in accordance with a Joint Electron Devices Engineering Council (JEDEC) Solid State Technology Association Universal Flash Storage (UFS) configuration. To further illustrate, the data storage device 110 may be integrated within an electronic device, such as a mobile telephone, a computer (e.g., a laptop, a tablet, or a notebook computer), a music player, a video player, a gaming device or console, a component of a vehicle (e.g., a vehicle console), an electronic book reader, a personal digital assistant (PDA), a portable navigation device, or other device that uses internal non-volatile memory.
In one or more other implementations, the data storage device 110 may be implemented in a portable device configured to be selectively coupled to one or more external devices, such as a host device. For example, the data storage device 110 may be removable from the access device 150 (i.e., “removably” coupled to the device). As an example, the data storage device 110 may be removably coupled to the access device 150 in accordance with a removable universal serial bus (USB) configuration.
In some implementations, the system 100, the data storage device 110, or a component thereof may be integrated within a network-accessible data storage system, such as an enterprise data system, an NAS system, or a cloud data storage system, as illustrative examples. In some implementations, the data storage device 110 may include a solid state drive (SSD). The data storage device 110 may function as an embedded storage drive (e.g., an embedded SSD drive of a mobile device), an enterprise storage drive (ESD), a cloud storage device, a network-attached storage (NAS) device, or a client storage device, as illustrative, non-limiting examples. In some implementations, the data storage device 110 may be coupled to the access device 150 via a network. For example, the network may include a data center storage system network, an enterprise storage system network, a storage area network, a cloud storage network, a local area network (LAN), a wide area network (WAN), the Internet, and/or another network.
To further illustrate, the data storage device 110 may be configured to be coupled to the access device 150 as embedded memory, such as in connection with an embedded MultiMedia Card (eMMC®) (trademark of JEDEC Solid State Technology Association, Arlington, Va.) configuration, as an illustrative example. The data storage device 110 may correspond to an eMMC device. As another example, the data storage device 110 may correspond to a memory card, such as a Secure Digital (SD®) card, a microSD® card, a miniSD™ card (trademarks of SD-3C LLC, Wilmington, Del.), a MultiMediaCard™ (MMC™) card (trademark of JEDEC Solid State Technology Association, Arlington, Va.), or a CompactFlash® (CF) card (trademark of SanDisk Corporation, Milpitas, Calif.). The data storage device 110 may operate in compliance with a JEDEC industry specification. For example, the data storage device 110 may operate in compliance with a JEDEC eMMC specification, a JEDEC Universal Flash Storage (UFS) specification, one or more other specifications, or a combination thereof.
The first memory 122 and/or the memory device 112 may include a resistive random access memory (ReRAM), a flash memory (e.g., a NAND memory, a NOR memory, a single-level cell (SLC) flash memory, a multi-level cell (MLC) flash memory, a divided bit-line NOR (DINOR) memory, an AND memory, a high capacitive coupling ratio (HiCR) device, an asymmetrical contactless transistor (ACT) device, or another flash memory), an erasable programmable read-only memory (EPROM), an electrically-erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a one-time programmable memory (OTP), another type of memory, or a combination thereof In a particular embodiment, the data storage device 110 is indirectly coupled to the access device 150 via a network. For example, the data storage device 110 may be a network-attached storage (NAS) device or a component (e.g., a solid-state drive (SSD) component) of a data center storage system, an enterprise storage system, or a storage area network. The first memory 122 and/or the memory device 112 may include a semiconductor memory device.
Semiconductor memory devices include volatile memory devices, such as dynamic random access memory (“DRAM”) or static random access memory (“SRAM”) devices, non-volatile memory devices, such as resistive random access memory (“ReRAM”), magnetoresistive random access memory (“MRAM”), electrically erasable programmable read only memory (“EEPROM”), flash memory (which can also be considered a subset of EEPROM), ferroelectric random access memory (“FRAM”), and other semiconductor elements capable of storing information. Each type of memory device may have different configurations. For example, flash memory devices may be configured in a NAND or a NOR configuration.
The memory devices can be formed from passive and/or active elements, in any combinations. By way of non-limiting example, passive semiconductor memory elements include ReRAM device elements, which in some embodiments include a resistivity switching storage element, such as an anti-fuse, phase change material, etc., and optionally a steering element, such as a diode, etc. Further by way of non-limiting example, active semiconductor memory elements include EEPROM and flash memory device elements, which in some embodiments include elements containing a charge region, such as a floating gate, conductive nanoparticles, or a charge storage dielectric material.
Multiple memory elements may be configured so that they are connected in series or so that each element is individually accessible. By way of non-limiting example, flash memory devices in a NAND configuration (NAND memory) typically contain memory elements connected in series. A NAND memory array may be configured so that the array is composed of multiple strings of memory in which a string is composed of multiple memory elements sharing a single bit line and accessed as a group. Alternatively, memory elements may be configured so that each element is individually accessible, e.g., a NOR memory array. NAND and NOR memory configurations are exemplary, and memory elements may be otherwise configured.
The semiconductor memory elements located within and/or over a substrate may be arranged in two or three dimensions, such as a two dimensional memory structure or a three dimensional memory structure. In a two dimensional memory structure, the semiconductor memory elements are arranged in a single plane or a single memory device level. Typically, in a two dimensional memory structure, memory elements are arranged in a plane (e.g., in an x-z direction plane) which extends substantially parallel to a major surface of a substrate that supports the memory elements. The substrate may be a wafer over or in which the layer of the memory elements are formed or it may be a carrier substrate which is attached to the memory elements after they are formed. As a non-limiting example, the substrate may include a semiconductor such as silicon.
The memory elements may be arranged in the single memory device level in an ordered array, such as in a plurality of rows and/or columns. However, the memory elements may be arrayed in non-regular or non-orthogonal configurations. The memory elements may each have two or more electrodes or contact lines, such as bit lines and word lines.
A three dimensional memory array is arranged so that memory elements occupy multiple planes or multiple memory device levels, thereby forming a structure in three dimensions (i.e., in the x, y and z directions, where the y direction is substantially perpendicular and the x and z directions are substantially parallel to the major surface of the substrate). As a non-limiting example, a three dimensional memory structure may be vertically arranged as a stack of multiple two dimensional memory device levels. As another non-limiting example, a three dimensional memory array may be arranged as multiple vertical columns (e.g., columns extending substantially perpendicular to the major surface of the substrate, i.e., in the y direction) with each column having multiple memory elements in each column. The columns may be arranged in a two dimensional configuration, e.g., in an x-z plane, resulting in a three dimensional arrangement of memory elements with elements on multiple vertically stacked memory planes. Other configurations of memory elements in three dimensions can also constitute a three dimensional memory array.
By way of non-limiting example, in a three dimensional NAND memory array, the memory elements may be coupled together to form a NAND string within a single horizontal (e.g., x-z) memory device levels. Alternatively, the memory elements may be coupled together to form a vertical NAND string that traverses across multiple horizontal memory device levels. Other three dimensional configurations can be envisioned wherein some NAND strings contain memory elements in a single memory level while other strings contain memory elements which span through multiple memory levels. Three dimensional memory arrays may also be designed in a NOR configuration and in a ReRAM configuration.
Typically, in a monolithic three dimensional memory array, one or more memory device levels are formed above a single substrate. Optionally, the monolithic three dimensional memory array may also have one or more memory layers at least partially within the single substrate. As a non-limiting example, the substrate may include a semiconductor such as silicon. In a monolithic three dimensional array, the layers constituting each memory device level of the array are typically formed on the layers of the underlying memory device levels of the array. However, layers of adjacent memory device levels of a monolithic three dimensional memory array may be shared or have intervening layers between memory device levels.
Alternatively, two dimensional arrays may be formed separately and then packaged together to form a non-monolithic memory device having multiple layers of memory. For example, non-monolithic stacked memories can be constructed by forming memory levels on separate substrates and then stacking the memory levels atop each other. The substrates may be thinned or removed from the memory device levels before stacking, but as the memory device levels are initially formed over separate substrates, the resulting memory arrays are not monolithic three dimensional memory arrays. Further, multiple two dimensional memory arrays or three dimensional memory arrays (monolithic or non-monolithic) may be formed on separate chips and then packaged together to form a stacked-chip memory device.
Associated circuitry is typically used for operation of the memory elements and for communication with the memory elements. As non-limiting examples, memory devices may have circuitry used for controlling and driving memory elements to accomplish functions such as programming and reading. This associated circuitry may be on the same substrate as the memory elements and/or on a separate substrate. For example, a controller for memory read-write operations may be located on a separate controller chip and/or on the same substrate as the memory elements.
One of skill in the art will recognize that this disclosure is not limited to the two dimensional and three dimensional exemplary structures described but cover all relevant memory structures within the spirit and scope of the disclosure as described herein and as understood by one of skill in the art. The illustrations of the embodiments described herein are intended to provide a general understanding of the various embodiments. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Those of skill in the art will recognize that such modifications are within the scope of the present disclosure.
The above-disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments, that fall within the scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description.
