Fullness control for media-based cache operating in a steady state

Information

  • Patent Grant
  • 9588898
  • Patent Number
    9,588,898
  • Date Filed
    Tuesday, June 2, 2015
    9 years ago
  • Date Issued
    Tuesday, March 7, 2017
    7 years ago
Abstract
A data storage system incorporating a write-caching subsystem that implements a steady-state media-based cache is described. The steady-state of the media-based cache can be obtained by directing non-sequential write commands and data received from the host device to multiple independent cache locations and, thereafter, selectively copying or moving such data between the caches so that none of the caches are either too full or too empty. In this manner, a non-sequential write command can be cached in a power-safe manner until it is efficient and/or convenient to write such data to the mainstore portion of the physical media.
Description
TECHNICAL FIELD

This disclosure relates to information storage and, more particularly, to systems and methods for managing data storage devices with a media-based cache.


BACKGROUND

Many computing systems generate or receive data that may be desirable to store for a period of time. These computing systems often utilize a data storage device for data storage and retrieval. In many cases, the data storage device receives access commands from a host device. These access commands include read commands, write commands, and so on. Oftentimes, a data communication protocol specifies particular parameters for implementing and/or servicing the access commands received by the data storage device.


Each access command serviced by the data storage device is associated with an overhead defined by the data storage device itself. Thus, on average, the data storage device performs a high number of operations per second when servicing an access command for a large amount of data. Conversely, on average, the data storage device performs a lower number of operations per second when servicing an access command for a small amount of data.





BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to representative embodiments illustrated in the accompanying figures. It should be understood that the following descriptions are not intended to limit the disclosure to one preferred embodiment. To the contrary, each is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the described embodiments as defined by the appended claims.



FIG. 1 depicts a block diagram of an example data storage device that may be configured to communicate with a host device.



FIG. 2 depicts a block diagram of a portion of a data storage device operating a media-based cache in a steady state.



FIG. 3 depicts a flow chart of sample operations for managing a data storage system implementing a write-caching subsystem incorporating a steady-state media-based cache.



FIG. 4 depicts a flow chart of sample operations of a method for caching non-sequential write commands prior to writing said commands to the physical media of a data storage system.



FIG. 5 depicts a flow chart of sample operations for managing a media-based cache in a steady state.



FIGS. 6A-6B depict a flow chart and block diagram of a data storage device incorporating a write-caching subsystem implementing a steady-state media-based cache





The use of the same or similar reference numerals in different drawings indicates similar, related, or identical items where appropriate.


DETAILED DESCRIPTION

Some embodiments described herein reference systems and methods for caching non-sequential write commands (e.g., “random writes”) received by a data storage device from a host device. In many cases, the data storage device may be a hard disk drive (“HDD”) including one or more rotating magnetic disks (the “physical media”). Such systems and methods cache non-sequential write commands within a reservation of a mainstore of the data storage device (herein, the “media-based cache” or “MBC”). Additionally, some embodiments described herein reference methods for managing the MBC in a steady-state manner, substantially independent of the type, quantity, size, or frequency of non-sequential write commands received from the host device.


The steady-state of the media-based cache can be obtained by directing non-sequential write commands and data received from the host device to multiple independent cache locations and, thereafter, selectively copying or moving such data between the caches so that none of the caches are either too full or too empty. In this manner, a non-sequential write command can be cached in a power-safe manner until it is efficient and/or convenient to write such data to the mainstore portion of the physical media.


More particularly, some embodiments described herein initially receive and cache non-sequential write commands, in time-order, in a high-speed memory (e.g., DRAM) that is power-safe (e.g., write-cache enabled data storage system with a power-safe write buffer). Thereafter, the data storage device reports to the host device that the data has been written in a power-safe manner. Subsequently received non-sequential write commands are also cached, in the order in which they are received, in the power-safe high-speed memory.


Upon determining that the power-safe high-speed memory is full, data contained therein can be written sequentially (e.g., in time-received order) to the media-based cache reservation of the physical media of the data storage device. Upon determining that the media-based cache is approaching a selected fullness threshold, at least a portion of the data contained therein can be copied to a holding pool within the high-speed memory (e.g., a portion of the DRAM, which in many cases separate from the power-safe region) and sorted and/or mapped by logical block address. At a later point, the sorted data within the holding pool can be written to the mainstore of the data storage device. In some implementations, the data in the holding pool is written to the mainstore when such a write would be convenient and/or advisable to perform (e.g., when the overhead of writing such data would be low and/or already expended by the performance of another operation).


By maintaining the fullness state of the media-based cache, the data storage device can service non-sequential write commands at a substantially constant rate, regardless of the time, order, size, or frequency with which such commands are received. In this manner, the data storage device can exhibit substantially reduced command-servicing time variability over a conventional data storage device. Such a system is generally referred to herein as a “write-caching subsystem” of a data storage device implementing a steady-state media-based cache.



FIG. 1 depicts a block diagram of an example data storage device 100 that can implement a write-caching subsystem including a steady-state media-based cache. The data storage device 100 is configured to communicate with a host device 102. The host device 102 can send an access command 104 over a connection (e.g., wireless or wired) to the data storage device 100. The access command 104 can be any number of suitable commands or instructions including, but not limited to, write commands and read commands. In many embodiments, the access command 104 can conform to a protocol 106 that defines a communication format to facilitate data transactions between the host device 102 and the data storage device 100. Upon receiving the access command 104, the data storage device 100 may service the access command 104 and, if required or requested, submit a response 108 to the host device 102. The response 108 may be in the same format and/or may conform to the same protocol as the access command 104. For example, if the host device 102 sends an access command 104 to read data, the data storage device 100 can send a response 108 conforming to the protocol 106 with the requested data.


The data storage device 100 can include a physical media 110. The physical media 110 can be implemented as one or more disks formed from a magnetic material onto (and/or into) which data can be recorded as patterns of magnetic polarity. In other embodiments, the physical media 110 can be another type of physical media such as flash storage, optical storage, tape storage, and so on.


The data storage device 100 can also include a controller 112 in communication with the physical media 110. The controller 112 can manage or coordinate the reading and/or the writing of data to the physical media 110. Additionally, the controller 112 can serve as an interface between the host device 102 and the physical media 110. For example, the controller 112 can facilitate data transactions between the physical media 110 and the host device 102 via the protocol 106.


In many cases, the controller 112 may be required to perform, monitor, or coordinate multiple discrete tasks in order to fully service an access command 104. For example, to respond to a write command, the controller 112 may translate a logical block address to a physical sector, access the physical sector on the physical media (e.g., perform a seek operation), encode the data to be written to the physical sector (e.g., error correction coding, encryption, and so on), and write the encoded data to the physical sector.


As noted above, the maximum time expense (e.g., overhead) associated with the seek operation may be substantially independent of the quantity of data to be written to the physical media 110. In other words, a large amount of data can be typically written at a higher average rate than a small amount of data because the relative proportion of total time writing to total time seeking may be substantially greater when writing a large amount of data than when writing a small amount of data. Similarly, multiple commands, whether large-size or small-size, that are physically located nearby one another on the physical media 110 may be serviced sequentially. In this manner, the “sequentiality” of a write command may be related to both the size of the command and to the location of other write commands received near-in-time (e.g., shortly before or shortly after) to the received command.


For example, the data storage device 100 may be configured to write 512B of data in 1 μs after performing a 10 ms seek operation. Accordingly, the write command is serviced in a total of 10.001 ms at an approximate rate of 51 KB written per second. In another example, the data storage device 100 can write 1 MB of data in approximately 2000 μs after performing a 10 ms seek operation. Accordingly, the write command is serviced in a total of 12 ms at an approximate rate of 83 MB written per second. In another example, the data storage device 100 can write 1 GB of data in approximately 2000 ms after performing a 10 ms seek operation. Accordingly, the write command is serviced in total of 2.010 s at an approximate rate of 512 megabytes written per second. As may be appreciated, the rate at which the data storage device 100 services write commands may become substantially constant (e.g., may not increase) as the size of write commands (e.g., the quantity of data to be written) continues to increase in proportion to the time required to seek. For example, the data storage device 100 can write 10 GB of data in approximately 20 s after performing a 10 ms seek operation. In this example, the write command is serviced in a total of 20.010 s at an approximate rate, again, of 512 megabytes written per second.


Accordingly, as used herein, the term “non-sequential write” generally refers to write operations and commands that are generally of a small size and are received at a time or in an order such that servicing of the command would require a separate seek operation and the term “sequential write” generally refers to write operations and commands of a large size. The relative and/or absolute size of non-sequential writes and sequential writes can vary from embodiment to embodiment; no particular size is required to define either type of write. However, in many embodiments, the maximum size of a non-sequential write can be defined, at least in part, on the physical structure of the data storage device 100 and/or the physical media 110. More particularly, the maximum size of a non-sequential write can depend upon the amount of available space within a cache (e.g., host queue, volatile memory cache, and so on) of the data storage device and/or the number of tracks and/or the block or sector size of the physical media 110. In some examples, the maximum size of a non-sequential write can be defined as 256 host blocks of data.



FIG. 2 depicts a block diagram of a portion of a data storage device 200 operating a media-based cache in a steady state. As with the embodiment depicted in FIG. 1, the data storage device 200 can include a host interface 202 that transacts data between a connected host device and a non-volatile physical storage 204 within the data storage device 200.


The non-volatile physical storage 204 can be implemented as a HDD including one or more rotating disks formed from a magnetic material, although such a configuration is not required. For example, the non-volatile physical storage 204 can be an SSD or an optical storage system. The non-volatile physical storage 204 can be partitioned, segmented, zoned, or sectioned in a variety of ways. For example, in some cases, the non-volatile physical storage 204 can be divided into sectors. In other examples, the non-volatile physical storage 204 can be partitioned. In many embodiments, the non-volatile physical storage 204 can be divided, either physically or logically, into two distinct portions, a mainstore 206 and a media-based cache 208.


In many cases, the host interface 202 can be coupled directly to the non-volatile physical storage 204, for example, to service a sequential write command by writing received data directly to the mainstore 206.


In other cases, the host interface 202 may cache a write command received from a host device within a write-caching subsystem 200a prior to writing that data to the mainstore 206. For example, as noted above, a write-caching subsystem such as the write-caching subsystem 200a can include multiple volatile and/or non-volatile memory locations into which data received from a host can be cached. More particularly, the write-caching subsystem 200a receives and cache non-sequential write commands, in time-order, in a high-speed memory such as the volatile memory 210 that is at least partially power-safe (e.g., write-cache enabled data storage system with a power-safe write buffer). More specifically, the volatile memory 210 can be divided, either physically or logically, into a volatile portion 212 and a power-safe region 214. Subsequently-received non-sequential write commands can also be cached within the power-safe region 214, in the time order in which they are received.


Thereafter, the write-caching subsystem 200a can determine that the volatile memory 210 is full or is approaching fullness. In response, data contained in the power-safe region 214 can be written sequentially (e.g., in time order) to the media-based cache 208. Thereafter, upon determining that the media-based cache 208 is itself approaching a selected fullness threshold, the write-caching subsystem 200a can copy at least a portion of the data in the media-based cache 208 to a holding pool within the volatile portion 212 of the volatile memory 210. Upon copying said data, the write-caching subsystem 200a also sorts and/or maps the data so that, at a later point, the sorted data within the holding pool can be written to the mainstore 206 when such a write would be convenient and/or advisable to perform. In many cases, the write-caching subsystem 200a can transfer data between the power-safe region 214, the media-based cache 208, the volatile region 212, and the mainstore 206 in order to effect a substantially steady state operation (e.g., constant or semi-constant average number of operations per section).


Expanding upon embodiments presented above, the volatile memory 210 can be implemented as a high-speed memory such as a dynamic random-access memory (herein, “DRAM”). As with the non-volatile physical storage 204 and as noted above, the volatile memory 210 can be partitioned or otherwise divided, either physically or logically, into one or more portions such as a volatile region 212 and a power-safe region 214.


The power-safe region 214 can be coupled to a separate non-volatile memory 216. The separate non-volatile memory 216 can be flash memory or any other suitable non-volatile memory type. In these embodiments, when the volatile memory 210 (and, thus by extension the power-safe region 214 thereof) experiences a power loss, a reserve power source (e.g., reserve battery, capacitive storage, energy generated from the rotation of the non-volatile physical storage 204, and so on; not shown in FIG. 2) can be used to briefly power the volatile memory 210 to effect an emergency transfer of data from the power-safe region 214 into the separate non-volatile memory 216. When power is restored, data from the separate non-volatile memory 216 can be read and thereafter written back into the volatile memory 210 or to a portion of the non-volatile physical storage 204. In this manner, data written from the host interface 202 to the power-safe region 214 can be protected from power loss. The combination of the power-safe region 214 and the separate non-volatile memory 216 define a power-safe write buffer (herein, “PSWB”) to cache non-sequential write commands from the host interface 202. The power-safe write buffer is identified in FIG. 2 as the PSWB 218.


After the host interface 202 receives a non-sequential write command and caches the command in the PSWB 218, the data storage device 200 can await the arrival of subsequent non-sequential write commands to cache until the write-caching subsystem 200a determines that the PSWB 218 is approaching fullness.


Upon determining the PSWB 218 is full, the write-caching subsystem 200a can effect a transfer of data from the PSWB 218 (e.g., the power-safe region 214) to the media-based cache 208. In many embodiments, the data can be written to the media-based cache in a sequential manner. Although FIG. 2 is illustrated with a directional signal path directly between the power-safe region 214 and the media-based cache 208, such a configuration is not required. For example, in some embodiments, the transferred data may be passed into a rotational positional ordering module (which may be controlled/executed by a controller of the data storage device) (herein, the “RPO”). The RPO 220 can determine an optimal or semi-optimal order or location (or locality range) for writing data from the power-safe region 214 to the media-based cache 208. For example, in many embodiments, the media-based cache 208 may be partitioned into a series of independently-writable partitions. In these embodiments, the RPO 220 can select a partition of the media-based cache 208 for writing data from the PSWB 218.


In many embodiments, the data contained within the PSWB 218 may be written to the media-based cache 208 (or a partition thereof) sequentially, despite that such data is not necessarily received in any particular sequential order. In these embodiments, the data may be written to the media-based cache 208 in this manner so as to effect the highest-speed transfer between the PSWB 218 and the media-based cache 208 supportable by the hardware of the non-volatile physical storage 204. In other words, transferring data from the PSWB 218 to the media-based cache 208 can be performed at a rate equivalent to (or approaching) the rate of a sequential write. Herein, the operation of transferring non-ordered and/or non-sequential data from the PSWB 218 to the media-based cache 208 is generally referred to as an “enqueuing” operation.


Upon completion of an enqueuing operation, the PSWB 218 may be prepared to accept and cache subsequently-received non-sequential write requests while other operations of the write-caching subsystem 200a are performed and/or monitored in parallel.


At a later point, the write-caching subsystem 200a can determine that the media-based cache (and/or a partition thereof) has exceeded a selected fullness threshold. In response, the write-caching subsystem 200a can copy data from the media-based cache 208 back into the volatile memory 210, specifically into the volatile region 212, into what is referred to herein as a “holding pool.” Such an operation is generally referred to herein as a “slurping” operation. In many embodiments, only a small portion (e.g., a single partition or smaller) of the media-based cache 208 is slurped at a time.


In many cases, a slurping operation can be performed or assisted by the RPO 220. For example, should the write-caching subsystem 200a determine that a slurp operation should occur, the write-caching subsystem 200a may request the RPO 220 to determine a partition or portion of the media-based cache 208 to slurp.


Additionally, upon slurping data from at least a portion of the media-based cache 208 into the volatile region 212, the write-caching subsystem 200a may also map and/or otherwise organize the data being slurped. For example, the write-caching subsystem 200a can assemble and maintain a data structure such as a self-balancing binary search tree, linked list, table, or another data structure, and the structure may organize the addresses of associated data in the media-based cache that will eventually need to be written to the mainstore. In this manner, all data within the holding pool may be mapped, organized and/or otherwise sorted, contrary to data within the media-based cache 208 in which data is received and stored in time-order.


Therefore, at a later point, the sorted data within the holding pool can be written to the mainstore 206 of the data storage device 200 when such a write would be convenient and/or advisable to perform. Herein, the operation of transferring organized data from the holding pool of within the volatile portion 212 to mainstore 206 is generally referred to as a “flushing” operation.


In many cases, a flushing operation can be performed or assisted by the RPO 220. For example, should the write-caching subsystem 200a determine that a flushing operation should occur, the write-caching subsystem 200a may submit an organized group (e.g., subset of the data within the holding pool) to the RPO 220 (herein, the “RPO pool”). The RPO 220 can select individual non-sequential write operations from the RPO pool that may be convenient and/or efficient to service at a particular time. Thereafter, the write-caching subsystem 200a can either withdraw the RPO pool back into the holding pool or the write-caching subsystem 200a can refill the RPO pool with additional non-sequential write operations from the holding pool.


In these embodiments, the write-caching subsystem 200a can selectively perform enqueuing operations, slurping operations, and flushing operations so as to provide a substantially consistent rate for servicing non-sequential write commands. In some examples, the write-caching subsystem 200a can coordinate enqueuing operations, slurping operations, and flushing operations in order to provide substantially consistent rate for servicing non-sequential write commands for a particular period of time. More specifically, the write-caching subsystem 200a can have both a standard (e.g., “normal”) rate and another accelerated rate (e.g., burst rate).


In other cases, the write-caching subsystem 200a can selectively perform enqueuing operations, slurping operations, and flushing operations based on a variety of parameters of the data storage system 200. For example, the write-caching subsystem 200a can be implemented to maintain a certain selected fullness of the media-based cache 208, selectively enqueuing, slurping, and flushing in a manner that maintains the selected fullness.


In another example, the write-caching subsystem 200a can be implemented to maintain, within a selected tolerance, a certain selected fullness of the PSWB 218, by selectively enqueuing, slurping, and flushing in a manner that maintains the selected fullness.


In another example, the write-caching subsystem 200a can be implemented to maintain, within a selected tolerance, a certain selected fullness of the holding pool of the volatile portion 212, by selectively enqueuing, slurping, and flushing in a manner that maintains the selected fullness.


In another example, the write-caching subsystem 200a can be implemented to maintain, within a selected tolerance, a selected minimum non-sequential write servicing rate, by selectively enqueuing, slurping, and flushing in a manner that maintains the selected minimum rate.


In another example, the write-caching subsystem 200a can be implemented to maintain, within a selected tolerance, a selected minimum (or average) input/output operations per second (herein, “IOPS”) rating, by selectively enqueuing, slurping, and flushing in a manner that maintains the selected minimum rate. In many cases, the IOPS rating of a data storage device may be obtained by subjecting the data storage device to various tests in which sequential write, non-sequential write, sequential read, non-sequential read, and other operations are requested by a host device. Thereafter, the total number of operations performed by the data storage device can be time-averaged to obtain an IOPS rating therefor.


In another example, the write-caching subsystem 200a can be implemented to maintain, within a selected tolerance, a selected maximum (or time-average) IOPS variability rating, by selectively enqueuing, slurping, and flushing in a manner that maintains the selected average variability.


In another example, the write-caching subsystem 200a can be implemented to maintain a selected number of total non-sequential write commands currently within the write-caching subsystem 200a that have not yet been flushed to the mainstore 208. Herein, the sum total of such commands is referred to as “total writes-in-flight.” In some embodiments, the write-caching subsystem 200a can be implemented to maintain, within a selected tolerance, an average total writes-in-flight by selectively enqueuing, slurping, and flushing in a manner that maintains the selected average.


In yet further embodiments, the write-caching subsystem 200a can be implemented to maintain more than one of the examples provided above by selectively enqueuing, slurping, and flushing. For example, the write-caching subsystem 200a can be configured to maintain a selected fullness of the media-based cache in addition to maintaining a selected average total writes-in-flight.



FIG. 3 depicts a flow chart of sample operations for managing a data storage system implementing a write-caching subsystem incorporating a steady-state media-based cache. The method begins at operation 300 in which the data storage system can operate. At operation 302, the data storage system receives a write command from a host device. At operation 304, the data storage system determines whether the received write command is sufficiently small to be considered a non-sequential write command that can be cached within the write-caching subsystem incorporating the steady-state media-based cache. More specifically, the data storage system determines whether the length of the command write command received is below or equal to a maximum command length. If the command received is less than the maximum command length, the method continues to operation 306 at which the data storage system identifies the command as eligible for caching within the media-based cache. Alternatively, should the command be longer than the maximum command length, the method continues to operation 308 at which the data storage system identifies the command as ineligible for caching.


In many embodiments, a method such as depicted in the flow chart of FIG. 3 is performed or otherwise carried out, either partially or entirely, by a host interface, such as the host interface 202 depicted in FIG. 2. In other examples, the method such as depicted is performed or otherwise carried out, either partially or entirely, by a controller such as the controller 112 depicted in FIG. 1.



FIG. 4 depicts a flow chart of sample operations of a method for caching non-sequential write commands prior to writing said commands to the physical media of a data storage system. The method begins at operation 400 in which a data storage system implementing a write-caching subsystem can operate. At operation 402, the data storage system receives a non-sequential write that is eligible for caching within the write-caching system. At operation 404, the non-sequential write instruction is stored within a PSWB associated with the data storage system. After operation 404, one or more operations can occur.


For example, after the operation 404, the data storage system reports to a host device that the non-sequential write command has been fully serviced at operation 406. Additionally, after operation 404, the method continues to operation 408 in which the non-sequential write command is enqueued within a media-based cache. At a later time, the method continues at operation 410 in which the non-sequential write command is slurped from the media-based cache into a volatile memory. At a later time, the method continues to operation 412 in which the non-sequential write command is flushed from the volatile memory to a mainstore of the data storage system. In many cases, one or more of the operations of the method depicted in FIG. 4 can be performed with the assistance or oversight of an RPO. For example, in some embodiments, the enqueuing operation performed at operation 408 can be performed with the assistance or oversight of the RPO. Additionally, in some embodiments, the slurping operation performed at operation 410 can be performed with the assistance or oversight of the RPO. Further, in some embodiments, the flush operation performed at operation 412 can be performed with the assistance or oversight of the RPO.


As with other embodiments described and depicted herein, the method described above can be performed or otherwise carried out, either partially or entirely, by a host interface, such as the host interface 202 depicted in FIG. 2. In other examples, the method depicted in FIG. 4 can be performed or otherwise carried out, either partially or entirely, by a controller such as the controller 112 depicted in FIG. 1.



FIG. 5 depicts a flow chart of sample operations for managing write-caching subsystem with a media-based cache maintained in a steady state, such as depicted in FIG. 2. The method is generally presented as a semi-hierarchical decision tree including five major decision branch paths. To facilitate an understanding of the method depicted, the five major decision branches are described below from left to right. The method begins at operation 500.


The first major decision branch of the method for managing the media-based cache in a steady state as depicted in FIG. 5 relates to enqueueing operations as described herein. The decision branch begins from operation 500 at operation 502 in which the write-caching subsystem determines whether the PSWB of the write-caching subsystem is full. In many cases, the PSWB can be determined to be full when the PSWB no longer contains addressable space to cache a single subsequent non-sequential write command. In such a case, the operation 502 continues upon determining that the PSWB no longer contains additional addressable space. In other embodiments, the operation 502 continues after the data storage system receives a subsequent non-sequential write command (e.g., in a host queue) after the PSWB is already entirely full.


Upon determining that the PSWB is full, the first major decision branch of the method continues to operation 504 in which the write-caching subsystem determines whether the media-based cache of the write-caching subsystem has exceeded a particular selected fullness threshold. As noted with respect to other embodiments described herein, the selected fullness threshold can vary from embodiment to embodiment.


Upon determining that the media-based cache has not exceeded the selected threshold (and, therefore, contains enough space to enqueue commands within the PSWB), the first major decision branch of the method continues to operation 506 in which the RPO selects a particular partition (or location) of the media-based cache into which data from the PSWB should be enqueued. Thereafter, at operation 508, the write-caching subsystem can enqueue (e.g., move) data of the PSWB within the media-based cache. In some embodiments, the entire content of the PSWB can be enqueued within the media-based cache. In other embodiments, only a portion of the content of the PSWB can be enqueued. Thereafter, the first major decision branch of the method proceeds to operation 510, which, in turn, loops back to operation 500 to define a repeating operational loop.


The second major decision branch of the method for managing the media-based cache in a steady state as depicted in FIG. 5 relates to slurping operations as described herein. The decision branch begins, either from the NO branch of the determination in operation 504 or operation 502, at operation 512 in which the write-caching subsystem determines whether the holding pool can slurp new commands from the media-based cache.


Upon determining that the holding pool contains space to slurp commands from the media-based cache, the second major decision branch of the method continues to operation 514 in which the write-caching subsystem determines whether the media-based cache contains commands that are able to be slurped into the holding pool.


Upon determining that the media-based cache contains commands that are able to be slurped into the holding pool, the second major decision branch of the method continues to operation 516, in which the RPO selects a particular partition (or location) of the media-based cache from data may be slurped. Thereafter, at operation 518, the write-caching subsystem slurps (e.g., copy) data from the RPO-selected portion of the media-based cache to the holding pool. In some embodiments, the entire content of an RPO-selected media-based cache partition can be slurped. In other embodiments, only a portion of the content of the RPO-selected media-based cache partition can be slurped. Thereafter, the second major decision branch of the method proceeds to operation 510, which as noted above, loops back to operation 500 to define a repeating operational loop.


The third major decision branch of the method for managing the media-based cache in a steady state as depicted in FIG. 5 relates to flushing operations as described herein. The decision branch begins, either from the NO branch of the determination in operation 512 or operation 514, at operation 520 in which the write-caching subsystem determines whether the media-based cache of the write-caching subsystem has exceeded a particular selected fullness threshold. As noted with respect to other embodiments described herein, the selected fullness threshold can vary from embodiment to embodiment.


Upon determining that the media-based cache of the write-caching subsystem has exceeded the selected fullness threshold, the third major decision branch of the method continues to operation 522 in which the RPO selects one or more non-sequential write commands from the RPO pool to flush to mainstore. Next at operation 524, a flush may be performed to the RPO-selected non-sequential write commands from the RPO pool. Thereafter, at operation 526, the RPO pool can be refilled from the holding pool. In some embodiments, the write-caching subsystem selects non-sequential write commands within the holding pool to fill the RPO pool. In other embodiments, the RPO selects non-sequential write commands within of the holding pool to refill the RPO pool. Thereafter, at operation 528, the write-caching subsystem determines whether the holding pool and the RPO pool are non-empty (e.g., still contain flushable non-sequential write operations). Additionally, operation 528 determines whether a selected timeout has elapsed. Should the write-caching subsystem determine that the holding pool and the RPO pool are non-empty and that the timeout has not yet elapsed, the third major decision branch of the method can return to operation 522.


Alternatively, if it is determined that a timeout has occurred or all flush writes are completed, the third major decision branch of the method proceeds to operation 510, which as noted above, loops back to operation 500 to define a repeating operational loop. In many embodiments, the timeout may be selected (or adjusted) by the write-caching subsystem in order to facilitate steady state operation of the media-based cache.


The fourth major decision branch of the method for managing the media-based cache in a steady state as depicted in FIG. 5 relates to non-caching operations of the data storage system as described herein. The decision branch begins, from the NO branch of the determination in operation 520, at operation 530 in which the write-caching subsystem can determine whether a non-cacheable command (e.g., not cached or managed by the write-caching subsystem) is available for processing. Non-cacheable commands can include a SATA flush request, a S.M.A.R.T. command, an identify command, a drive configure command and so on.


Thereafter, at operation 532, the RPO selects a command from a pool of non-cacheable commands. Next at operation 536, the RPO-selected command can be serviced. Thereafter, the fourth major decision branch of the method proceeds to operation 510, which as noted above, loops back to operation 500 to define a repeating operational loop.


The fifth major decision branch of the method for managing the media-based cache in a steady state as depicted in FIG. 5 relates to idle operations of the data storage system as described herein. The decision branch begins, from the NO branch of the determination in operation 530, at operation 538 in which the write-caching subsystem determines whether to perform idle activities such as enqueuing, slurping, or flushing while the host is idle. Thereafter, the fifth major decision branch of the method proceeds to operation 510, which as noted above, loops back to operation 500 to define a repeating operational loop.


As with other embodiments described and depicted herein, the method described above can be performed or otherwise carried out, either partially or entirely, by a host interface, such as the host interface 202 depicted in FIG. 2. In other examples, the method depicted in FIG. 5 can be performed or otherwise carried out, either partially or entirely, by a controller such as the controller 112 depicted in FIG. 1.



FIGS. 6A-6B depict a flow chart and block diagram of a data storage device incorporating a write-caching subsystem implementing a steady-state media-based cache. The method of FIG. 6A begins at operation 600 in which a write command is received from a host. Next, at operation 602, an aggregate fullness (or an aggregate fullness status) of a write-caching subsystem is determined. Next at operation 604, an action to be performed (e.g., enqueuing, slurping, flushing, servicing a non-cacheable command, servicing a read command, servicing another command, and so on) is selected based on the determined aggregate fullness. Next, at operation 606, the selected action is performed.


In many embodiments, the method of FIG. 6A can be implemented by an HDD data storage device, such as depicted by the flow chart of FIG. 6B. As one example, the data storage device can include a processor 608 that can perform, monitor, or coordinate one or more of the operations of the data storage device. The processor 608 can be connected to a voice coil motor 610 (“VCM”) that controls the location of a read head and a write head along a rotatable disk 612 that is formed from a magnetic material. The processor 608 may also communicate with one or more host devices 614.


Some embodiments described herein relate to data storage systems and data storage devices and methods of operating the same. It should be appreciated that the various embodiments described herein, as well as functionality, operation, components, and capabilities thereof may be combined with other elements as necessary, and so any physical, functional, or operational discussion of any element or feature is not intended to be limited solely to a particular embodiment to the exclusion of others. For example, the host device 102 as depicted in FIG. 1 may be any suitable electronic device such as a laptop computer, desktop computer, server, cellular phone, tablet computer, and so on. In some embodiments, the host device 102 can be implemented as a system of individual electronic devices such as a network of servers. Additionally, although many embodiments described herein reference hard disk drives (“HDD”) containing one or more magnetic recording disks, other physical media and data storage devices such as solid state drives (“SSD”), flash storage devices, tape storage devices, hybrid storage devices, optical storage devices, and the like may be equally applicable to one, many, or all embodiments described herein or variants thereof.


As another example, the controller 112 of FIG. 1 can be implemented as an electrical circuit, as an integrated circuit, as a series of instructions (e.g., firmware) executed by a processor 114, or as a combination thereof. In many embodiments, the controller 112 can perform, monitor, and/or coordinate one or more operations of the data storage device 100 depicted in FIG. 1. Similarly, the host interface 202 of FIG. 2 can be implemented as an electrical circuit, as an integrated circuit, as a series of instructions executed by a processor, or as a combination thereof. The host interface 202 can contain and/or can be placed in communication with additional hardware components that assist in the programming and/or execution of commands that the data storage device 200 is configured to perform.


As another example, the mainstore 206 of FIG. 2 can, in some embodiments, represent the majority of the storage space provided by the non-volatile physical storage 204. The size of the mainstore 206 can be reported to a host device as the amount of available storage space provided by the data storage device 200. In this manner, the portion of the non-volatile physical storage 204 reserved for the media-based cache 208 may not, in all embodiments, be reported to a host device. In this manner, the media-based cache 208 can be a distinct and reserved portion of the non-volatile physical storage 204. Further, in some cases, the media-based cache 208 may occupy a selected physical location on the physical media of the non-volatile physical storage 204. For example, in one embodiment, the media-based cache 208 can be an outer track of the non-volatile physical storage.


In still further embodiments, the media-based cache 208 may be located in a physical location so as to provide a statistically advantageous position to which (or from which) the data storage device 200 can read. More particularly, in some embodiments, the media-based cache 208 can be disposed in a central track of the physical media of the non-volatile physical storage 204. In this configuration, the distance from the media-based cache 208 to any other location in the non-volatile physical storage 204 (e.g., any location within the mainstore 206) can be, on average, reduced. In these embodiments, transfer of data between the media-based cache 208 and the mainstore 206 may be more easily accomplished because of the reduced physical distance between the media-based cache 208 and the mainstore 206.


In one example, access commands can be received at the host interface 202 and thereafter placed into a command queue. In this manner, the host interface 206 and the command queue cooperate to enable the data storage device to receive and process more than one host command at a time.


The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the described embodiments. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the described embodiments. Thus, the foregoing descriptions of the specific embodiments described herein are presented for purposes of illustration and description. They are not meant to be exhaustive or to limit the embodiments to the precise forms disclosed. It will be apparent to one of ordinary skill in the art that many modifications and variations are possible in view of the above teachings. In particular, any features described with respect to one embodiment may also be used in some embodiments, where compatible. Likewise, the features of the different embodiments may be exchanged, substituted, or omitted where compatible and appropriate.


Many embodiments of the foregoing disclosure may include or may be described in relation to various methods of operation, use, manufacture, and so on. Notably, the operations of methods presented herein are meant only to be exemplary and, accordingly, are not necessarily exhaustive. For example an alternate operation order, or fewer or additional steps may be required or desired for particular embodiments.

Claims
  • 1. A method of caching write commands received by a data storage system from a host, the data storage system comprising a physical media and a write-caching subsystem, the method comprising: receiving a write command from a host;determining an aggregate fullness status of the write-caching subsystem, the write-caching subsystem comprising a volatile memory and a media-based cache, the aggregate fullness based on a fullness status of at least a portion of the media-based cache;selecting, based on a state of the write-caching subsystem, an action to be performed, the action selected from: reducing the aggregate fullness status of the write-caching subsystem; andservicing a command from the host that is not managed by the write-caching subsystem; andperforming the selected action.
  • 2. The method of claim 1, wherein the fullness status is a first fullness status and an aggregate fullness status is further based on a second fullness status of at least a portion of the volatile memory.
  • 3. The method of claim 1, wherein the operation of selecting an action to be performed is based in part on a fullness status of the media-based cache.
  • 4. The method of claim 1, wherein performing the action to reduce the fullness of the write-caching subsystem comprises moving data within the media-based cache to a primary storage portion in the physical media.
  • 5. The method of claim 1, wherein the volatile memory comprises: a first portion for buffering write data to be written to the media-based cache; anda second portion for storing data copied from the media-based cache to be written to a primary storage portion in the physical media.
  • 6. The method of claim 5, wherein performing the action to reduce the fullness of the write-caching subsystem comprises moving data from the second portion of the volatile memory to a primary storage portion in the physical media.
  • 7. The method of claim 5, wherein the operation of selecting an action to be performed is based in part on a determination of whether to perform an action to modify the fullness of the write-caching subsystem.
  • 8. The method of claim 7, wherein performing an action to modify the fullness of the write-caching subsystem comprises enqueuing data of write commands from the first portion of the volatile memory into the media-based cache.
  • 9. The method of claim 5, wherein the operation of selecting an action to be performed is based in part on a minimum batch number of write commands to be selected for rotational position optimization.
  • 10. The method of claim 9, wherein the operation of selecting an action to be performed is based in part on determining whether to perform an action to read data from the media-based cache into the second portion of the volatile memory.
  • 11. The method of claim 9, wherein the selected write commands are selected based on a locality range of write locations on the physical media.
  • 12. A data storage system comprising: a host interface;a host command cache in communication with the host interface;a physical media comprising: a media-based cache; anda mainstore;a write cache subsystem in communication with the host command cache, the write cache subsystem comprising: a volatile memory comprising: a first portion for buffering write data to be written to the media-based cache; anda second portion for storing data copied from the media-based cache to be written to the mainstore; anda processor coupled to the host interface, the write-caching subsystem, and the physical media, the processor configured to perform the operations of: receiving a write command from a host;determining an aggregate fullness status of the write-caching subsystem the aggregate fullness based on a fullness status of at least a portion of the media-based cache;selecting, based on a state of the write-caching subsystem, an action to be performed, the action selected from: reducing the aggregate fullness status of the write-caching subsystem; andservicing a command from the host that is not managed by the write-caching subsystem; andperforming the selected action.
  • 13. The data storage system of claim 12, wherein the processor is further configured to perform rotational positional ordering on operations related to the write cache subsystem.
  • 14. The data storage system of claim 12, wherein the operation of moving the write command from the first portion of the volatile memory to the media-based cache is performed by the processor upon determining that the first portion of the write cache subsystem exceeds a selected fullness threshold.
  • 15. The data storage system of claim 12, wherein the physical media comprises at least one rotating magnetic disk.
  • 16. The data storage system of claim 12, wherein the write cache subsystem comprises a dynamic random-access memory.
  • 17. The data storage system of claim 12, wherein the media-based cache is partitioned.
  • 18. A method of writing data to a physical media of a data storage system, the method comprising: determining that data to be written is eligible for storage within a media-based cache within the physical media;storing the data in a power-safe data cache;writing the data sequentially from the power-safe data cache to the media-based cache;copying the data from the media-based cache to a data holding cache on a volatile memory; andflushing the data holding cache to the physical media.
  • 19. The method of claim 18, wherein the operation of determining that the data is eligible for storage within a media-based cache within the physical media comprises determining that the data is shorter than a selected minimum length.
  • 20. The method of claim 18, wherein the operation of writing the data sequentially from the power-safe data cache to the media-based cache is performed by determining that the power-safe data cache exceeds a selected fullness threshold.
US Referenced Citations (451)
Number Name Date Kind
5222224 Flynn Jun 1993 A
6018789 Sokolov et al. Jan 2000 A
6065095 Sokolov et al. May 2000 A
6078452 Kittilson et al. Jun 2000 A
6081447 Lofgren et al. Jun 2000 A
6092149 Hicken et al. Jul 2000 A
6092150 Sokolov et al. Jul 2000 A
6094707 Sokolov et al. Jul 2000 A
6105104 Guttmann et al. Aug 2000 A
6111717 Cloke et al. Aug 2000 A
6145052 Howe et al. Nov 2000 A
6175893 D'Souza et al. Jan 2001 B1
6178056 Cloke et al. Jan 2001 B1
6191909 Cloke et al. Feb 2001 B1
6195218 Guttmann et al. Feb 2001 B1
6205494 Williams Mar 2001 B1
6208477 Cloke et al. Mar 2001 B1
6223303 Billings et al. Apr 2001 B1
6230233 Lofgren et al. May 2001 B1
6246346 Cloke et al. Jun 2001 B1
6249393 Billings et al. Jun 2001 B1
6256695 Williams Jul 2001 B1
6262857 Hull et al. Jul 2001 B1
6263459 Schibilla Jul 2001 B1
6272694 Weaver et al. Aug 2001 B1
6278568 Cloke et al. Aug 2001 B1
6279089 Schibilla et al. Aug 2001 B1
6289484 Rothberg et al. Sep 2001 B1
6292912 Cloke et al. Sep 2001 B1
6310740 Dunbar et al. Oct 2001 B1
6317850 Rothberg Nov 2001 B1
6327106 Rothberg Dec 2001 B1
6337778 Gagne Jan 2002 B1
6369969 Christiansen et al. Apr 2002 B1
6384999 Schibilla May 2002 B1
6388833 Golowka et al. May 2002 B1
6405342 Lee Jun 2002 B1
6408357 Hanmann et al. Jun 2002 B1
6408406 Parris Jun 2002 B1
6411452 Cloke Jun 2002 B1
6411458 Billings et al. Jun 2002 B1
6412083 Rothberg et al. Jun 2002 B1
6415349 Hull et al. Jul 2002 B1
6425128 Krapf et al. Jul 2002 B1
6441981 Cloke et al. Aug 2002 B1
6442328 Elliott et al. Aug 2002 B1
6445524 Nazarian et al. Sep 2002 B1
6449767 Krapf et al. Sep 2002 B1
6453115 Boyle Sep 2002 B1
6470420 Hospodor Oct 2002 B1
6480020 Jung et al. Nov 2002 B1
6480349 Kim et al. Nov 2002 B1
6480932 Vallis et al. Nov 2002 B1
6483986 Krapf Nov 2002 B1
6487032 Cloke et al. Nov 2002 B1
6490635 Holmes Dec 2002 B1
6493173 Kim et al. Dec 2002 B1
6499083 Hamlin Dec 2002 B1
6519104 Cloke et al. Feb 2003 B1
6525892 Dunbar et al. Feb 2003 B1
6545830 Briggs et al. Apr 2003 B1
6546489 Frank, Jr. et al. Apr 2003 B1
6550021 Dalphy et al. Apr 2003 B1
6552880 Dunbar et al. Apr 2003 B1
6553457 Wilkins et al. Apr 2003 B1
6578106 Price Jun 2003 B1
6580573 Hull et al. Jun 2003 B1
6594183 Lofgren et al. Jul 2003 B1
6600620 Krounbi et al. Jul 2003 B1
6601137 Castro et al. Jul 2003 B1
6603622 Christiansen et al. Aug 2003 B1
6603625 Hospodor et al. Aug 2003 B1
6604220 Lee Aug 2003 B1
6606682 Dang et al. Aug 2003 B1
6606714 Thelin Aug 2003 B1
6606717 Yu et al. Aug 2003 B1
6611393 Nguyen et al. Aug 2003 B1
6615312 Hamlin et al. Sep 2003 B1
6639748 Christiansen et al. Oct 2003 B1
6647481 Luu et al. Nov 2003 B1
6654193 Thelin Nov 2003 B1
6657810 Kupferman Dec 2003 B1
6661591 Rothberg Dec 2003 B1
6665772 Hamlin Dec 2003 B1
6687073 Kupferman Feb 2004 B1
6687078 Kim Feb 2004 B1
6687850 Rothberg Feb 2004 B1
6690523 Nguyen et al. Feb 2004 B1
6690882 Hanmann et al. Feb 2004 B1
6691198 Hamlin Feb 2004 B1
6691213 Luu et al. Feb 2004 B1
6691255 Rothberg et al. Feb 2004 B1
6693760 Krounbi et al. Feb 2004 B1
6694477 Lee Feb 2004 B1
6697914 Hospodor et al. Feb 2004 B1
6704153 Rothberg et al. Mar 2004 B1
6708251 Boyle et al. Mar 2004 B1
6710951 Cloke Mar 2004 B1
6711628 Thelin Mar 2004 B1
6711635 Wang Mar 2004 B1
6711660 Milne et al. Mar 2004 B1
6715044 Lofgren et al. Mar 2004 B2
6724982 Hamlin Apr 2004 B1
6725329 Ng et al. Apr 2004 B1
6735650 Rothberg May 2004 B1
6735693 Hamlin May 2004 B1
6744772 Eneboe et al. Jun 2004 B1
6745283 Dang Jun 2004 B1
6751402 Elliott et al. Jun 2004 B1
6757481 Nazarian et al. Jun 2004 B1
6772281 Hamlin Aug 2004 B2
6781826 Goldstone et al. Aug 2004 B1
6782449 Codilian et al. Aug 2004 B1
6791779 Singh et al. Sep 2004 B1
6792486 Hanan et al. Sep 2004 B1
6799274 Hamlin Sep 2004 B1
6811427 Garrett et al. Nov 2004 B2
6826003 Subrahmanyam Nov 2004 B1
6826614 Hanmann et al. Nov 2004 B1
6832041 Boyle Dec 2004 B1
6832929 Garrett et al. Dec 2004 B2
6845405 Thelin Jan 2005 B1
6845427 Atai-Azimi Jan 2005 B1
6850443 Lofgren et al. Feb 2005 B2
6851055 Boyle et al. Feb 2005 B1
6851063 Boyle et al. Feb 2005 B1
6853731 Boyle et al. Feb 2005 B1
6854022 Thelin Feb 2005 B1
6862660 Wilkins et al. Mar 2005 B1
6880043 Castro et al. Apr 2005 B1
6882486 Kupferman Apr 2005 B1
6884085 Goldstone Apr 2005 B1
6888831 Hospodor et al. May 2005 B1
6892217 Hanmann et al. May 2005 B1
6892249 Codilian et al. May 2005 B1
6892313 Codilian et al. May 2005 B1
6895455 Rothberg May 2005 B1
6895500 Rothberg May 2005 B1
6898730 Hanan May 2005 B1
6910099 Wang et al. Jun 2005 B1
6928470 Hamlin Aug 2005 B1
6931439 Hanmann et al. Aug 2005 B1
6934104 Kupferman Aug 2005 B1
6934713 Schwartz et al. Aug 2005 B2
6940873 Boyle et al. Sep 2005 B2
6943978 Lee Sep 2005 B1
6948165 Luu et al. Sep 2005 B1
6950267 Liu et al. Sep 2005 B1
6954733 Ellis et al. Oct 2005 B1
6961814 Thelin et al. Nov 2005 B1
6965489 Lee et al. Nov 2005 B1
6965563 Hospodor et al. Nov 2005 B1
6965966 Rothberg et al. Nov 2005 B1
6967799 Lee Nov 2005 B1
6968422 Codilian et al. Nov 2005 B1
6968450 Rothberg et al. Nov 2005 B1
6973495 Milne et al. Dec 2005 B1
6973570 Hamlin Dec 2005 B1
6976190 Goldstone Dec 2005 B1
6983316 Milne et al. Jan 2006 B1
6986007 Procyk et al. Jan 2006 B1
6986154 Price et al. Jan 2006 B1
6995933 Codilian et al. Feb 2006 B1
6996501 Rothberg Feb 2006 B1
6996669 Dang et al. Feb 2006 B1
7002926 Eneboe et al. Feb 2006 B1
7003674 Hamlin Feb 2006 B1
7006316 Sargenti, Jr. et al. Feb 2006 B1
7009820 Hogg Mar 2006 B1
7023639 Kupferman Apr 2006 B1
7024491 Hanmann et al. Apr 2006 B1
7024549 Luu et al. Apr 2006 B1
7024614 Thelin et al. Apr 2006 B1
7027716 Boyle et al. Apr 2006 B1
7028174 Atai-Azimi et al. Apr 2006 B1
7031902 Catiller Apr 2006 B1
7046465 Kupferman May 2006 B1
7046488 Hogg May 2006 B1
7050252 Vallis May 2006 B1
7054937 Milne et al. May 2006 B1
7055000 Severtson May 2006 B1
7055167 Masters May 2006 B1
7057836 Kupferman Jun 2006 B1
7062398 Rothberg Jun 2006 B1
7075746 Kupferman Jul 2006 B1
7076604 Thelin Jul 2006 B1
7082494 Thelin et al. Jul 2006 B1
7088538 Codilian et al. Aug 2006 B1
7088545 Singh et al. Aug 2006 B1
7092186 Hogg Aug 2006 B1
7095577 Codilian et al. Aug 2006 B1
7099095 Subrahmanyam et al. Aug 2006 B1
7106537 Bennett Sep 2006 B1
7106947 Boyle et al. Sep 2006 B2
7110202 Vasquez Sep 2006 B1
7111116 Boyle et al. Sep 2006 B1
7114029 Thelin Sep 2006 B1
7120737 Thelin Oct 2006 B1
7120806 Codilian et al. Oct 2006 B1
7126776 Warren, Jr. et al. Oct 2006 B1
7129763 Bennett et al. Oct 2006 B1
7133600 Boyle Nov 2006 B1
7136244 Rothberg Nov 2006 B1
7146094 Boyle Dec 2006 B1
7149046 Coker et al. Dec 2006 B1
7150036 Milne et al. Dec 2006 B1
7155616 Hamlin Dec 2006 B1
7171108 Masters et al. Jan 2007 B1
7171110 Wilshire Jan 2007 B1
7194576 Boyle Mar 2007 B1
7200698 Rothberg Apr 2007 B1
7205805 Bennett Apr 2007 B1
7206497 Boyle et al. Apr 2007 B1
7215496 Kupferman et al. May 2007 B1
7215771 Hamlin May 2007 B1
7237054 Cain et al. Jun 2007 B1
7240161 Boyle Jul 2007 B1
7249365 Price et al. Jul 2007 B1
7263709 Krapf Aug 2007 B1
7274639 Codilian et al. Sep 2007 B1
7274659 Hospodor Sep 2007 B2
7275116 Hanmann et al. Sep 2007 B1
7280302 Masiewicz Oct 2007 B1
7292774 Masters et al. Nov 2007 B1
7292775 Boyle et al. Nov 2007 B1
7296284 Price et al. Nov 2007 B1
7302501 Cain et al. Nov 2007 B1
7302579 Cain et al. Nov 2007 B1
7318088 Mann Jan 2008 B1
7319806 Willner et al. Jan 2008 B1
7325244 Boyle et al. Jan 2008 B2
7330323 Singh et al. Feb 2008 B1
7346790 Klein Mar 2008 B1
7366641 Masiewicz et al. Apr 2008 B1
7369340 Dang et al. May 2008 B1
7369343 Yeo et al. May 2008 B1
7372650 Kupferman May 2008 B1
7380147 Sun May 2008 B1
7392340 Dang et al. Jun 2008 B1
7404013 Masiewicz Jul 2008 B1
7406545 Rothberg et al. Jul 2008 B1
7415571 Hanan Aug 2008 B1
7436610 Thelin Oct 2008 B1
7437502 Coker Oct 2008 B1
7440214 Ell et al. Oct 2008 B1
7451344 Rothberg Nov 2008 B1
7461202 Forrer, Jr. et al. Dec 2008 B2
7471483 Ferris et al. Dec 2008 B1
7471486 Coker et al. Dec 2008 B1
7486060 Bennett Feb 2009 B1
7496493 Stevens Feb 2009 B1
7518819 Yu et al. Apr 2009 B1
7526184 Parkinen et al. Apr 2009 B1
7539924 Vasquez et al. May 2009 B1
7543117 Hanan Jun 2009 B1
7551383 Kupferman Jun 2009 B1
7562282 Rothberg Jul 2009 B1
7577973 Kapner, III et al. Aug 2009 B1
7596797 Kapner, III et al. Sep 2009 B1
7599139 Bombet et al. Oct 2009 B1
7619841 Kupferman Nov 2009 B1
7647544 Masiewicz Jan 2010 B1
7649704 Bombet et al. Jan 2010 B1
7653927 Kapner, III et al. Jan 2010 B1
7656603 Xing Feb 2010 B1
7656763 Jin et al. Feb 2010 B1
7657149 Boyle Feb 2010 B2
7672072 Boyle et al. Mar 2010 B1
7673075 Masiewicz Mar 2010 B1
7688540 Mei et al. Mar 2010 B1
7724461 McFadyen et al. May 2010 B1
7725584 Hanmann et al. May 2010 B1
7730295 Lee Jun 2010 B1
7760458 Trinh Jul 2010 B1
7768776 Szeremeta et al. Aug 2010 B1
7804657 Hogg et al. Sep 2010 B1
7813954 Price et al. Oct 2010 B1
7827320 Stevens Nov 2010 B1
7839588 Dang et al. Nov 2010 B1
7843660 Yeo Nov 2010 B1
7852596 Boyle et al. Dec 2010 B2
7859782 Lee Dec 2010 B1
7872822 Rothberg Jan 2011 B1
7898756 Wang Mar 2011 B1
7898762 Guo et al. Mar 2011 B1
7900037 Fallone et al. Mar 2011 B1
7907364 Boyle et al. Mar 2011 B2
7929234 Boyle et al. Apr 2011 B1
7933087 Tsai et al. Apr 2011 B1
7933090 Jung et al. Apr 2011 B1
7934030 Sargenti, Jr. et al. Apr 2011 B1
7940491 Szeremeta et al. May 2011 B2
7944639 Wang May 2011 B1
7945727 Rothberg et al. May 2011 B2
7949564 Hughes et al. May 2011 B1
7974029 Tsai et al. Jul 2011 B2
7974039 Xu et al. Jul 2011 B1
7982993 Tsai et al. Jul 2011 B1
7984200 Bombet et al. Jul 2011 B1
7990648 Wang Aug 2011 B1
7992179 Kapner, III et al. Aug 2011 B1
8004785 Tsai et al. Aug 2011 B1
8006027 Stevens et al. Aug 2011 B1
8014094 Jin Sep 2011 B1
8014977 Masiewicz et al. Sep 2011 B1
8019914 Vasquez et al. Sep 2011 B1
8040625 Boyle et al. Oct 2011 B1
8078943 Lee Dec 2011 B1
8079045 Krapf et al. Dec 2011 B2
8082433 Fallone et al. Dec 2011 B1
8085487 Jung et al. Dec 2011 B1
8089719 Dakroub Jan 2012 B1
8090902 Bennett et al. Jan 2012 B1
8090906 Blaha et al. Jan 2012 B1
8091112 Elliott et al. Jan 2012 B1
8094396 Zhang et al. Jan 2012 B1
8094401 Peng et al. Jan 2012 B1
8116020 Lee Feb 2012 B1
8116025 Chan et al. Feb 2012 B1
8134793 Vasquez et al. Mar 2012 B1
8134798 Thelin et al. Mar 2012 B1
8139301 Li et al. Mar 2012 B1
8139310 Hogg Mar 2012 B1
8144419 Liu Mar 2012 B1
8145452 Masiewicz et al. Mar 2012 B1
8149528 Suratman et al. Apr 2012 B1
8154812 Boyle et al. Apr 2012 B1
8159768 Miyamura Apr 2012 B1
8161328 Wilshire Apr 2012 B1
8164849 Szeremeta et al. Apr 2012 B1
8174780 Tsai et al. May 2012 B1
8190575 Ong et al. May 2012 B1
8194338 Zhang Jun 2012 B1
8194340 Boyle et al. Jun 2012 B1
8194341 Boyle Jun 2012 B1
8201066 Wang Jun 2012 B1
8271692 Dinh et al. Sep 2012 B1
8279550 Hogg Oct 2012 B1
8281218 Ybarra et al. Oct 2012 B1
8285923 Stevens Oct 2012 B2
8289656 Huber Oct 2012 B1
8305705 Roohr Nov 2012 B1
8307156 Codilian et al. Nov 2012 B1
8310775 Boguslawski et al. Nov 2012 B1
8315006 Chahwan et al. Nov 2012 B1
8316263 Gough et al. Nov 2012 B1
8320067 Tsai et al. Nov 2012 B1
8324974 Bennett Dec 2012 B1
8332695 Dalphy et al. Dec 2012 B2
8341337 Ong et al. Dec 2012 B1
8350628 Bennett Jan 2013 B1
8356184 Meyer et al. Jan 2013 B1
8370683 Ryan et al. Feb 2013 B1
8375225 Ybarra Feb 2013 B1
8375274 Bonke Feb 2013 B1
8380922 DeForest et al. Feb 2013 B1
8390948 Hogg Mar 2013 B2
8390952 Szeremeta Mar 2013 B1
8392689 Lott Mar 2013 B1
8407393 Yolar et al. Mar 2013 B1
8413010 Vasquez et al. Apr 2013 B1
8417566 Price et al. Apr 2013 B2
8421663 Bennett Apr 2013 B1
8422172 Dakroub et al. Apr 2013 B1
8427771 Tsai Apr 2013 B1
8429343 Tsai Apr 2013 B1
8433937 Wheelock et al. Apr 2013 B1
8433977 Vasquez et al. Apr 2013 B1
8458526 Dalphy et al. Jun 2013 B2
8462466 Huber Jun 2013 B2
8467151 Huber Jun 2013 B1
8489841 Strecke et al. Jul 2013 B1
8493679 Boguslawski et al. Jul 2013 B1
8498074 Mobley et al. Jul 2013 B1
8499198 Messenger et al. Jul 2013 B1
8512049 Huber et al. Aug 2013 B1
8514506 Li et al. Aug 2013 B1
8531791 Reid et al. Sep 2013 B1
8554741 Malina Oct 2013 B1
8560759 Boyle et al. Oct 2013 B1
8565053 Chung Oct 2013 B1
8576511 Coker et al. Nov 2013 B1
8578100 Huynh et al. Nov 2013 B1
8578242 Burton et al. Nov 2013 B1
8589773 Wang et al. Nov 2013 B1
8593753 Anderson Nov 2013 B1
8595432 Vinson et al. Nov 2013 B1
8599510 Fallone Dec 2013 B1
8601248 Thorsted Dec 2013 B2
8611032 Champion et al. Dec 2013 B2
8612650 Carrie et al. Dec 2013 B1
8612706 Madril et al. Dec 2013 B1
8612798 Tsai Dec 2013 B1
8619383 Jung et al. Dec 2013 B1
8621115 Bombet et al. Dec 2013 B1
8621133 Boyle Dec 2013 B1
8626463 Stevens et al. Jan 2014 B2
8630052 Jung et al. Jan 2014 B1
8630056 Ong Jan 2014 B1
8631188 Heath et al. Jan 2014 B1
8634158 Chahwan et al. Jan 2014 B1
8635412 Wilshire Jan 2014 B1
8640007 Schulze Jan 2014 B1
8654619 Cheng Feb 2014 B1
8661193 Cobos et al. Feb 2014 B1
8667248 Neppalli Mar 2014 B1
8670205 Malina et al. Mar 2014 B1
8683295 Syu et al. Mar 2014 B1
8683457 Hughes et al. Mar 2014 B1
8687306 Coker et al. Apr 2014 B1
8693133 Lee et al. Apr 2014 B1
8694841 Chung et al. Apr 2014 B1
8699159 Malina Apr 2014 B1
8699171 Boyle Apr 2014 B1
8699172 Gunderson et al. Apr 2014 B1
8699175 Olds et al. Apr 2014 B1
8699185 Teh et al. Apr 2014 B1
8700850 Lalouette Apr 2014 B1
8743502 Bonke et al. Jun 2014 B1
8749910 Dang et al. Jun 2014 B1
8751699 Tsai et al. Jun 2014 B1
8755141 Dang Jun 2014 B1
8755143 Wilson et al. Jun 2014 B2
8756361 Carlson et al. Jun 2014 B1
8756382 Carlson et al. Jun 2014 B1
8769593 Schwartz et al. Jul 2014 B1
8773802 Anderson et al. Jul 2014 B1
8780478 Huynh et al. Jul 2014 B1
8782334 Boyle et al. Jul 2014 B1
8793532 Tsai et al. Jul 2014 B1
8797669 Burton Aug 2014 B1
8799977 Kapner, III et al. Aug 2014 B1
8819375 Pruett et al. Aug 2014 B1
8825976 Jones Sep 2014 B1
8825977 Syu et al. Sep 2014 B1
20030126232 Mogul Jul 2003 A1
20040230739 Tsern Nov 2004 A1
20090113702 Hogg May 2009 A1
20100058000 Moyer Mar 2010 A1
20100146205 Baum et al. Jun 2010 A1
20100306551 Meyer et al. Dec 2010 A1
20110119442 Haines May 2011 A1
20110226729 Hogg Sep 2011 A1
20120159042 Lott et al. Jun 2012 A1
20120275050 Wilson et al. Nov 2012 A1
20120281963 Krapf et al. Nov 2012 A1
20120324980 Nguyen et al. Dec 2012 A1
20140201424 Chen et al. Jul 2014 A1
20140205012 Lee Jul 2014 A1
20140281229 Kowles Sep 2014 A1
20150143043 Maddah-Ali May 2015 A1