1. Technical Field
The present invention relates to storage subsystems, and more specifically, to systems and methods for optimizing execution of storage access commands.
2. Description of the Related Art
Solid-state storage subsystems are widely used to store various types of data. They are often used as an alternative to disk-based storage, particularly in portable electronic devices that often require a combination of large memory capacity and portability. One limitation of solid-state storage subsystems is the need to perform block erase operations when storing data within (programming) the non-volatile memory (“NVM”) array. Another limitation is the finite number of write-erase cycles associated with non-volatile memory storage arrays, whose non-volatile storage components can lose the ability to retain data stored thereon after as little as hundreds of thousands or millions of write/erase cycles. This may necessitate performing periodic wear leveling operations to distribute write-erase cycles evenly across memory blocks and/or periodic bad block management operations to identify and mark inoperative memory blocks. Consequently, a significant problem with solid-state storage subsystems is that their programming performance depends on whether block erase, wear leveling, and bad block management operations need to be performed in advance of programming the memory blocks. The resulting programming performance variance can be especially problematic when a storage subsystem is used for time-critical tasks such as backing up data during a power failure or brownout, caching data, programming data sets having a high priority, or executing time-critical non-standard storage access commands.
Systems and methods for an architecture for optimizing execution of storage access commands are disclosed. The architecture enables a storage subsystem to execute storage access commands while satisfying one or more optimization criteria. The architecture thereby provides predictable execution times of storage access commands performed on a storage subsystem. In order to optimize execution of storage access commands, in one embodiment the host system sends a calibration request specifying a storage access command and an optimization criterion. In response to the calibration request, the storage subsystem determines the execution speeds of the storage access command within the non-volatile memory storage array and selects at least one region within the non-volatile memory storage array having the execution speed that satisfies the optimization criterion. Subsequently, when the host system desires that a storage access command be executed in satisfaction of the optimization criterion, the storage subsystem executes the command within the selected region.
Preferred embodiments of the invention will now be described with reference to the drawings summarized below. These drawings and the associated description are provided to illustrate preferred embodiments of the invention, and not to limit the scope of the invention.
Systems and methods for optimizing execution of storage access commands will now be described with reference to the drawings. Where possible, the same reference numbers are used throughout the drawings to refer to the same or like components. The present description is intended to illustrate certain preferred embodiments, but other embodiments that are apparent to those of ordinary skill in the art, including embodiments which do not provide all of the benefits and features set forth herein, are also within the scope of this invention. As one example, some embodiments may omit some or all of the data backup features described herein. Thus, nothing in this detailed description is intended to suggest that any particular feature or component is essential. The invention is defined by the claims.
1. Existing Approaches of Using a Storage Subsystem
This section describes a typical environment in which the various inventive features may be employed, and describes some of the significant problems with existing approaches of programming non-volatile memory arrays.
The storage subsystem 120 comprises a controller 125 and a storage subsystem non-volatile memory storage array 121. The non-volatile memory storage array 121 is arranged in memory address locations 122. As is conventional, the controller 125 is configured to write data and read data from the non-volatile memory storage array 121 in response to receiving storage access commands from the host system 110. The controller 125 preferably implements a wear leveling algorithm, as is known in the art, to distribute write operations across memory blocks of the non-volatile memory storage array and a bad block management algorithm to recognize and mark inoperative memory blocks of the non-volatile memory storage array. The storage subsystem may receive its power from the host system.
If a sudden power loss condition arises, critical data stored in the volatile memory 116 of the host system can be backed up to the storage subsystem 120 for later retrieval. To provide power for the backup operation, the host system 110 may implement hardware that comprises a capacitor 130 that charges up when the host system operates normally, as in
2. Performance of Existing Storage Subsystems
According to the datasheets of major non-volatile memory storage array manufacturers, a conventional single-level cell non-volatile memory storage array has a typical programming operation time of 200-220 μs (per 2 KB page), a typical block erase operation time of 1.5 ms (per 128 KB block comprising 64 program pages), and a typical wear leveling operation time of 14.3 ms (estimated to comprise one block move and one block erase operation). However, the maximum programming operation time is 500-750 μs, the maximum block erase operation time is 2 ms, and the maximum wear leveling operation time is 50 ms.
Under ideal conditions, no block erase or wear leveling operations need to be performed prior to a programming operation. Hence, the ideal data rate corresponding to 200 μs programming operation time is 10 MB/sec. However, when the programming operation time is greater than the typical and block erase or wear leveling operations are needed, conditions become non-ideal. As illustrated in
For example, under ideal conditions 128 MB of data can be stored within a storage subsystem in 12.8 seconds (128 MB÷10 MB/sec). But, with only a change in the programming operation time from 200 μs to 750 μs, storing 128 MB of data would take at least 48 seconds. In addition, if wear leveling operations need to be performed on just 10% of the blocks (and assuming 128 KB block size), storing 128 MB of data would take at least 53 seconds.
Such a long delay can be become prohibitively costly when time-critical tasks need to be performed. For example, when power is lost and 128 MB of data needs to be backed up from the volatile memory 116 to the storage subsystem 120, the capacitor 130 needs to be large and expensive (as capacitance is linear to the cost and the size of a capacitor) in order to provide such large “hold” times for backing up data.
3. Architecture for Optimizing Execution of Storage Access Commands
An architecture will now be described that addresses at least some of the above problems by selecting areas of the non-volatile memory storage array where the execution speeds of storage access commands satisfy one or more optimization criteria and, optionally, minimizing overhead operations (for example, block erase, wear leveling, and bad block management operations) in those areas. A host system can use such areas when it desires optimized execution of storage access commands by a storage subsystem. The architecture may be used with any of a variety of different standard storage interfaces and protocols, including but not limited to ATA, SATA, SCSI, USB, RS232/423, PCMCIA, FIREWIRE, FibreChannel, PCI EXPRESS bus, SD, MMC, or MMC Plus.
The storage subsystem 120 comprises a controller 125 and a storage subsystem non-volatile memory storage array 121. The non-volatile memory storage array 121 may consist of solid-state non-volatile memory devices, such as flash memory devices. Other types of memory elements, such as solid-state volatile RAM devices and magnetic disk drives, may additionally or alternatively be used. The non-volatile memory storage array 121 comprises memory address locations, which may be arranged into bytes, words, pages, blocks, sectors, clusters, files, directories, partitions, or the like according to a particular storage file system, including but not limited to FAT, FAT32, NTFS, HFS, HFS+, ext, ext2, ext3, ext4, exFAT, JFFS, JFFS2, LogFS, or YAFFS. The controller 125 is configured execute storage access commands communicated from the host system 110 within the non-volatile memory storage array 121.
The storage subsystem 120 can have one or more areas, or regions, within the non-volatile memory storage array 121 that provide an optimized execution of storage access commands. Relatively high performance regions typically exist naturally in a non-volatile memory storage array. They can be found and further calibrated to execute storage access commands with ideal or nearly ideal speed. In one embodiment, a high performance (optimized) region is selected by the storage subsystem 120 in response to a calibration request received from the host system 110. The calibration request specifies a storage access command and an optimization criterion, such that the execution of the storage access command is optimized according to the optimization criterion. In response to the calibration request, the controller 125 performs calibration (finds a high performance region) of the non-volatile memory storage array 121 by executing the specified storage access commands within areas of the non-volatile memory storage array 121 and recording the execution speeds. An area having the execution speed satisfying the optimization criterion is selected as the optimized region.
In a preferred embodiment, the storage access command whose execution is optimized is programming and the optimization criterion is the fastest programming time. As a result, in response to a calibration request from the host system 110, the controller 125 performs calibration of the non-volatile memory storage array 121 by executing programming operations within areas of the non-volatile memory storage array 121 and recording the programming times. An area with the fastest programming time (which may be calculated as an average over multiple operations and/or sub-regions) is selected as the optimized region.
The selected region can be further optimized by performing block erase operations in advance of the host system's desire to use the region, such that block erase operations will not need to be performed when, for example, a programming operation is performed within the region. To further optimize programming performance within the selected region, wear leveling and bad block management operations can be performed in advance or turned off for the region. In addition, any other overhead operations can be similarly performed in advance or disabled in the selected region. For example, during calibration the region can be selected as a contiguous non-volatile memory storage array area to eliminate address computation delays during subsequent programming operations. Consequently, by disabling or performing in advance the overhead operations, the selected region can provide optimized performance of storage access commands.
In one embodiment, calibration can be performed by the host system 110. For example, the host system may communicate a series of storage access commands to areas within the non-volatile memory storage array 121, record the speed of execution of the commands, and select as the location of the optimized region an area within the non-volatile memory storage array 121 having the speed of execution that satisfies the desired optimization criterion. For example, the communicated storage access commands can be programming operations, and the desired optimization criterion can be the fastest programming time. In this embodiment, the calibration request 118 becomes a series of storage access commands, and the optimization criterion is not a part of the calibration request (as the host system selects the optimized region's location according to the optimization criterion).
In another embodiment, execution speeds or averaged, median, or the like execution speeds of the storage access commands whose execution has been optimized within the selected region can be returned to the host system 110. For example, this information can be embedded in the response of the calibration command communicated to the storage subsystem 120. When calibration is performed by the host system 110, the host system can record the execution speeds. The returned execution speeds or averaged, median, or the like execution speeds can be used by the host system as additional information to consider when planning how to perform time-critical tasks such as backing up data during a power failure or brownout, caching data, programming data sets having a high priority, or executing time-critical non-standard storage access commands.
When the host system 110 desires to use the selected region for optimized execution of standard or non-standard storage access commands, the host embeds an identification of the selected region within the storage access command communicated to the storage subsystem 120. In one embodiment in response to a calibration request 118, the host system receives an identification of the selected region, for example, embedded in the response of the calibration request. Then, the host system 110 can embed the received identification of the region within the storage access command, thus specifying to the storage subsystem that the command is to be executed within the selected region. For example, as further explained below, the identification can be a location (a memory address) within the selected region, a special pattern within the structure of the storage access command, an explicit or implicit parameter of a non-standard storage access command, or the like.
In one embodiment, the identification of the selected region can be a special pattern embedded (for example in the data portion) within the storage access command communicated by the host system 110 to the storage subsystem 120. Such a pattern can be known a priori by the host system. Alternatively, such a pattern can be communicated to the host system by the storage subsystem in response to the calibration request, in response to a standard read command, in response to a non-standard storage access command, or the like. The pattern can be embedded into the command portion or data portion of the storage access command communicated to the storage subsystem 120. The controller 125 would then parse the command, discover the embedded pattern, and execute the command within the selected region. Otherwise, if the embedded pattern is not discovered, the controller 125 would execute the command within non-optimized areas of the non-volatile memory storage array 121.
In another embodiment, multiple segments (zones) are defined within the non-volatile memory storage array 121. A zoning mechanism for a storage subsystem is fully described in co-pending U.S. patent applications No. 11/480,303 filed Jun. 30, 2006 and No. 11/480,276 filed Jun. 30, 2006, and is hereby fully incorporated by reference. As a brief summary, zones are created according to zone definitions, which comprise starting and ending memory address of the non-volatile memory storage array 121. At least one zone would correspond to a selected region and at least one other zone would correspond to areas within the non-volatile memory storage array where execution of storage access commands has not been optimized.
In one preferred embodiment, a region where execution of storage commands has been optimized can be mapped as a logical drive on the host system 110.
In another preferred embodiment, a region where execution of storage access commands has been optimized can be mapped as a file on the host system 110. When a host system application 111 desires optimized execution of a storage access command, it writes the command to the mapped file, thereby embedding an identification that the command be executed within the optimized region. The operating system 112 translates this “virtual” location into memory addresses within the optimized region within the non-volatile memory storage array 121, and the controller 115 communicates the storage access command to the storage subsystem 120. Subsequently, the controller 125 executes the command within the optimized region. One benefit of this method is that no special user privileges, such as an “administrator privilege” for Windows or “raw disk” access for Linux, are needed for applications 111 to use the optimized region. Another benefit of this method is that the mapped file may be marked as “read-only,” “system,” “hidden,” or the like to protect the optimized region from unintended or malicious uses.
In another embodiment, a host system application 111 can desire that a non-standard storage access command (not a part of the standard storage interface command set supported by the host system and the storage subsystem) be executed within the selected region. A non-standard storage access command may be embedded within a standard write storage access command and be communicated to the storage subsystem in such manner. A non-standard storage access command may implicitly or explicitly embed an identification of the selected region. For example, when a storage subsystem 120 receives a particular non-standard storage access command (such as Cache Write, Dual Channel Programming, CopyBack, or the like) embedded into a standard write command communicated by the host system 110, the storage subsystem controller 125 may implicitly, based on a priori configuration, execute the command within the selected region. Alternatively, for example, a non-standard storage access command may embed an identification of the selected region in any of the ways explained above, and the storage subsystem controller 125 will execute the command within the selected region.
In another embodiment, execution speeds of storage access commands executed within a selected region can be returned to the host system 110. For example, this information can be embedded in the response of a standard or non-standard storage access command executed within the region. Optionally, execution speeds or averaged, median, or the like execution speeds can be recorded and returned by the storage subsystem 120 periodically (for example, after a certain number of storage access commands have been executed within the selected region) to the host system 110. The returned execution speeds or averaged, median, or the like execution speeds can be used by the host system as additional information to consider when planning how to perform time-critical tasks such as backing up data during a power failure or brownout, caching data, programming data sets having a high priority, or executing time-critical non-standard storage access commands.
A calibration request 118 communicated by the host system 110 to the storage subsystem 120 can comprise single or multiple standard storage access commands in accordance with the specific storage interface and protocol supported by the host system and the storage subsystem. For example, if a region is being calibrated to optimize the execution speed of programming commands, the host system may communicate a series of programming operations to areas within the non-volatile memory storage array 121, record their execution times, and select as the location of the optimized region an area within the non-volatile memory storage array having the speed of execution that satisfies a desired optimization criterion (for example, the fastest programming time).
Alternatively, the calibration request can comprise single or multiple non-standard storage access commands. A non-standard storage access command may specify a storage access command and one or more optimization criteria, such that execution of the storage access command is optimized according to the one or more optimization criteria. Alternatively, a non-standard storage access command may implicitly (for example, by a special command code) communicate to the storage subsystem that a particular storage access command is to be optimized according to an optimization criteria configured a priori. For example, a non-standard storage access command may implicitly communicate programming as the storage access command to be optimized, and the storage subsystem will know that the optimization criterion is the fastest programming time.
The host system 110 may use a standard operating system device driver to communicate calibration requests to the storage subsystem 120 and to communicate storage access commands for execution within the selected region. Alternatively, the host system may use a special (non-standard) operating system device driver, especially where non-standard storage commands are being communicated to the storage subsystem. For reasons of compatibility and portability, it is more advantageous to use a standard operating system device driver.
Consequently, the present architecture enables a storage subsystem to execute storage access commands with predictable and ideal or nearly ideal execution speeds. This can be advantageous for executing time-critical tasks, including but not limited to backing up data during a power failure or brownout, caching data, programming data sets having a high priority, or executing time-critical non-standard storage access commands.
4. An Example of Using Architecture for Optimizing Execution of Storage Access Commands
When the host system 110 has retrieved backup data stored in the optimized region 128 (or has determined that none is stored), the host system prepares the optimized region for a future time-critical operation (for example, programming of backup data). As shown in
Although described primarily in the context of a storage subsystem that includes a non-volatile memory storage array arranged into segments or zones, the invention is not so limited. For example, in some embodiments, the storage subsystem's memory array may comprise volatile memory elements, and/or may be arranged in units other than zones. In addition, as mentioned above, the solid-state non-volatile memory elements can be replaced or supplemented with magnetic disk drives or another type of storage device.
While certain embodiments of the invention have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the invention. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention. The invention is defined by the accompanying claims.
Number | Name | Date | Kind |
---|---|---|---|
4644494 | Muller | Feb 1987 | A |
4937736 | Chang et al. | Jun 1990 | A |
5018096 | Aoyama | May 1991 | A |
5640529 | Hasbun | Jun 1997 | A |
5781783 | Gunther et al. | Jul 1998 | A |
5860137 | Raz et al. | Jan 1999 | A |
5929590 | Tang | Jul 1999 | A |
6052799 | Li et al. | Apr 2000 | A |
6134631 | Jennings, III | Oct 2000 | A |
6173360 | Beardsley et al. | Jan 2001 | B1 |
6286087 | Ito et al. | Sep 2001 | B1 |
6324627 | Kricheff et al. | Nov 2001 | B1 |
6484229 | Ichikawa et al. | Nov 2002 | B1 |
6633963 | Ellison et al. | Oct 2003 | B1 |
6640268 | Kumar | Oct 2003 | B1 |
6654850 | Fox et al. | Nov 2003 | B2 |
6694381 | Lo et al. | Feb 2004 | B1 |
6792519 | Constable et al. | Sep 2004 | B2 |
6968434 | Kamano et al. | Nov 2005 | B2 |
7003644 | Heath et al. | Feb 2006 | B2 |
7024410 | Ito et al. | Apr 2006 | B2 |
7114051 | Guu et al. | Sep 2006 | B2 |
7139871 | Mizuno | Nov 2006 | B2 |
7139890 | Moran et al. | Nov 2006 | B2 |
7149046 | Coker et al. | Dec 2006 | B1 |
7170788 | Wan et al. | Jan 2007 | B1 |
7213117 | Wakabayashi et al. | May 2007 | B2 |
7224604 | Lasser | May 2007 | B2 |
7287118 | Chang et al. | Oct 2007 | B2 |
7307881 | Chen et al. | Dec 2007 | B2 |
7315917 | Bennett et al. | Jan 2008 | B2 |
7330954 | Nangle | Feb 2008 | B2 |
7408804 | Hemink et al. | Aug 2008 | B2 |
7441067 | Gorobets et al. | Oct 2008 | B2 |
7447807 | Merry et al. | Nov 2008 | B1 |
7450436 | Salessi et al. | Nov 2008 | B2 |
7467253 | Yero | Dec 2008 | B2 |
7509441 | Merry et al. | Mar 2009 | B1 |
7515471 | Oh et al. | Apr 2009 | B2 |
7609565 | Lee | Oct 2009 | B2 |
7654466 | Maeda et al. | Feb 2010 | B2 |
7870128 | Jensen et al. | Jan 2011 | B2 |
20020073272 | Ko et al. | Jun 2002 | A1 |
20030162549 | Carlsson | Aug 2003 | A1 |
20030163633 | Aasheim et al. | Aug 2003 | A1 |
20030182496 | Yoo | Sep 2003 | A1 |
20030188092 | Heath et al. | Oct 2003 | A1 |
20030200400 | Nangle | Oct 2003 | A1 |
20040015653 | Trantham | Jan 2004 | A1 |
20050160195 | Bruner et al. | Jul 2005 | A1 |
20050196165 | Dybsetter et al. | Sep 2005 | A1 |
20060095699 | Kobayashi et al. | May 2006 | A1 |
20060143426 | Wu | Jun 2006 | A1 |
20060184736 | Benhase et al. | Aug 2006 | A1 |
20060190696 | Ito et al. | Aug 2006 | A1 |
20060236392 | Thomas et al. | Oct 2006 | A1 |
20060294338 | Fisher et al. | Dec 2006 | A1 |
20070033362 | Sinclair | Feb 2007 | A1 |
20070050536 | Kolokowsky | Mar 2007 | A1 |
20070079065 | Bonella et al. | Apr 2007 | A1 |
20070079097 | Karnowski et al. | Apr 2007 | A1 |
20070136553 | Sinclair | Jun 2007 | A1 |
20070192538 | Dawkins | Aug 2007 | A1 |
20070208604 | Purohit et al. | Sep 2007 | A1 |
20070233939 | Kim | Oct 2007 | A1 |
20070245065 | Kagan et al. | Oct 2007 | A1 |
20070247933 | Kagan | Oct 2007 | A1 |
20080019189 | Lin | Jan 2008 | A1 |
20080019196 | Lin | Jan 2008 | A1 |
20080082726 | Elhamias | Apr 2008 | A1 |
20080091872 | Bennett et al. | Apr 2008 | A1 |
20080098164 | Lee et al. | Apr 2008 | A1 |
20080126449 | Haitsma | May 2008 | A1 |
20080162798 | Lofgren et al. | Jul 2008 | A1 |
20080270678 | Cornwell et al. | Oct 2008 | A1 |
20080282024 | Biswas et al. | Nov 2008 | A1 |
20080294813 | Gorobets | Nov 2008 | A1 |
20090089492 | Yoon et al. | Apr 2009 | A1 |
20090125782 | Josefiak et al. | May 2009 | A1 |
20090138654 | Sutardja | May 2009 | A1 |
20090150599 | Bennett | Jun 2009 | A1 |
20090172213 | Jayachandran et al. | Jul 2009 | A1 |
20090204853 | Diggs et al. | Aug 2009 | A1 |
20100061152 | De Caro et al. | Mar 2010 | A1 |
20110191526 | Haukness et al. | Aug 2011 | A1 |
Number | Date | Country |
---|---|---|
1662886 | Aug 2005 | CN |
1985239 | Jun 2007 | CN |
Entry |
---|
Office Action dated Feb. 28, 2011 from U.S. Appl. No. 12/410,304, 11 pages. |
Office Action dated Apr. 4, 2008 received in related U.S. Appl. No. 11/480,276. |
Office Action dated Jun. 30, 2008 received in related U.S. Appl. No. 11/480,303. |
Office Action dated Apr. 7, 2011 from U.S. Appl. No. 12/350,180, 22 pages. |
Micron Technical Note, “NAND Flash 101: An Introduction to NAND Flash and How to Design It in to Your Next Product”, TN-29-19, Nov. 2006, http://download.micron.com/pdf/technotes/nand/tn2919.pdf, pp. 1-28. |
Office Action dated Nov. 16, 2010 from U.S. Appl. No. 12/410,304, 22 pages. |
Office Action dated Oct. 12, 2011 from U.S. Appl. No. 12/350,180, 22 pages. |
Notice of Panel Decision dated Aug. 18, 2011 from U.S. Appl. No. 12/410,304, 2 pages. |
Office Action dated Jun. 3, 2011 from U.S. Appl. No. 12/410,304, 4 pages. |
Brief on Appeal submitted on Sep. 19, 2011 in U.S. Appl. No. 12/410,304 in 14 pages. |
Examiner's Answer mailed on Oct. 7, 2011 received in U.S. Appl. No. 12/410,304 in 14 pages. |
Office Action dated Oct. 12, 2011 from U.S. Appl. No. 12/350,180, 12 pages. |
Office Action dated Jun. 1, 2012 for U.S. Appl. No. 12/350,180, 14 pages. |
Office Action dated Nov. 5, 2012 from U.S. Appl. No. 12/350,180, 11 pages. |
Office Action dated Oct. 8, 2013 from Chinese Patent Application No. 201010134420.0, filed Mar. 16, 2010, 16 pages. |
Office Action dated Jan. 30, 2014 from U.S. Appl. No. 12/350,180, 20 pages. |