This disclosure relates to data storage systems, such as non-volatile data storage systems, for computer systems. More particularly, the disclosure relates to managing system data using garbage collection and logs.
Data storage systems can utilize various types of system data, such as logical to physical address mapping data, which associates logical addresses used by a host system for accessing stored data with corresponding physical locations in a data storage system where data is stored. Due to updates of data stored in the data storage system, certain stored data may become outdated, thereby causing entries in the mapping table corresponding to such outdated data to become invalid. In such circumstances, the data storage system can generate updated mapping table entries. However, various problems arise with maintenance of system data, such as increased write amplification associated with storing system data, compromised coherency, and the like. Accordingly, it is desirable to provide mechanisms that efficiently handle maintenance of system data.
Systems and methods that embody the various features of the invention will now be described with reference to the following drawings, in which:
While certain embodiments are described, these embodiments are presented by way of example only, and are not intended to limit the scope of protection. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the scope of protection.
Data storage systems can utilize various types of system data, including logical to physical address mapping data, superblock data, invalid page data, invalid counter data, wear level data, and so on. Logical to physical address mapping table can comprise information associating logical addresses used by a host system to store data in a data storage system and/or access stored data with physical locations in the data storage system where data is stored. Superblock table can comprise information concerning logical groupings of non-volatile memory blocks into superblocks. Superblocks can be utilized to achieve parallel or concurrent execution of multiple storage access operations. In one embodiment, each block in a superblock can be on one die of a group of non-volatile memory dies. Other configurations are possible in other embodiments. For example, a superblock can be comprised of blocks from various planes in a group of dies. In another embodiment, multiple blocks within a superblock may be on the same die/plane. Invalid page table can comprise information concerning locations of non-volatile memory pages that are defective or unreliable (e.g., those pages where data should not be stored). Invalid counter table can comprise information corresponding to the associations of invalid pages with particular superblocks. Wear level table can comprise erase counts for non-volatile memory locations (e.g., partial pages, pages, blocks, groups of blocks, etc.).
In operation, a copy of such system data (e.g., a logical to physical address mapping table) may be stored in a volatile memory such as DRAM and the copy can be used to track changes to the data storage system. This volatile copy of the system data (e.g., copy of the system data stored in the volatile memory) is reconstructed at power-up, and to ensure power-safety, the data storage system needs to maintain sufficient information in the non-volatile memory to enable a complete and accurate reconstruction. Thus, embodiments of the present invention are directed to efficiently updating the non-volatile copy of the system data (e.g., copy of the system data stored in the non-volatile memory) to ensure data coherency (e.g., to ensure that the volatile and non-volatile copies of the system data are synchronized).
Some embodiments of the present invention generate and update the non-volatile version of the system data quickly and efficiently while reducing write amplification associated with managing of system data. Advantageously, coherency of system data updates is also ensured. In addition, changes to system data (e.g., the non-volatile copy of the system data) can be committed atomically (e.g., together or at same or substantially same time) even in cases when sections or parts of system data being updated are stored in different memory regions. Further, start-up performance (e.g., start-up time) of the data storage system is improved because reconstruction time of system data is reduced. In addition, segmenting of system data (e.g., storing and loading segments of system data) is allowed, and efficiency of loading segments of system data is increased.
In some embodiments of the present invention, system data stored in the volatile memory (e.g., logical to physical address mapping table) is updated based on changes to data stored in a data storage system (e.g., changes as a result of host system activity). Logs are used to track changes to the mapping table in volatile memory and such logs are periodically saved to the non-volatile memory. The version or copy of the mapping table stored in the non-volatile memory is generally updated at a later time with the information in saved logs. When changes to stored data are scattered (e.g., changes are made to random memory locations), it may be beneficial to generate and accumulate more logs reflecting changes needed to be made to the copy of the system data stored in the non-volatile memory. As another example, when changes to stored data are substantially consolidated (e.g., changes are made to consecutive memory locations), it may be beneficial to update the copy of the system data stored in the non-volatile memory more frequently. Basing changes on the activity of a host system can reduce write amplification associated with updating system data in the non-volatile memory.
In some embodiments of the present invention, executing storage access commands received from a host system (or generated internally) can cause the copy of the system data stored in the non-volatile memory array to become outdated. Logs reflecting or recording changes to the system data in the non-volatile memory can be selectively generated and used to update the copy of the system data in the non-volatile memory. In response to determining that the execution of storage access commands causes related system data in the non-volatile memory to become outdated, system data in the non-volatile memory can be updated at a first update frequency so that at least some system data becomes current or up-to-date. In response to determining that the execution of the storage access commands does not cause related system data in the non-volatile memory to become outdated, system data in the non-volatile memory can be updated at a second update frequency. The first update frequency can be different (e.g., lower) than the second update frequency. No logs may be generated when it is determined that the execution of the storage access commands causes related system data to become outdated.
In some embodiments of the present invention, changes to the system data can be consolidated before updating the system data. For example, a number of log pages can be generated before committing changes reflected by the log pages to the non-volatile copy of the system data. In some embodiments, changes reflected in the log pages can be consolidated into new log pages. Write amplification can be reduced if two or more log pages reflect updates to the same system data region (e.g., same non-volatile memory page where system data is stored). Reconstruction time of system data can be reduced.
In some embodiments of the present invention, in response to determining that a number of generated logs exceeds a threshold, system data stored in the non-volatile memory can be updated based on the generated logs. After changes reflected in the logs have been applied or committed to the non-volatile copy of the system data, logs can be garbage collected. In some embodiments, the non-volatile copy of the system data is updated based on changes recorded in a set of logs that have been generated before other logs.
In some embodiments of the present invention, changes to the system data stored in the volatile memory are recorded in logs. Non-volatile copy of the system data can be updated after a log has been generated. This can improve performance and/or reduce write amplification by making updates to the non-volatile copy of the system data as recorded in a log at the same time or at substantially same time. Further, the volatile copy of the system data can be quickly reconstructed by making the updates recorded in the log that was generated last. After changes reflected in the log have been applied to system data, the log can be garbage collected.
In some embodiments, non-volatile memory array 150 can be partitioned into dies, planes, channels, blocks, pages, and so on. In certain embodiments, a non-volatile memory page (e.g., NAND page) can refer to a smallest grouping of memory cells (or smallest granularity) that can be programmed or written in a single operation or as a unit (e.g., atomically). In some embodiments, the data storage system 120 can also comprise other types of storage, such as one or more magnetic storage modules 160, which can comprise volatile cache (e.g., DRAM) 162 and non-volatile magnetic media 164.
The controller 130 can be configured to receive data and/or storage access commands from a storage interface module 112 (e.g., a device driver) of a host system 110. Storage access commands communicated by the storage interface 112 can include write data and read data commands issued by the host system 110. Read and write commands can specify a logical address (e.g., LBA) used to access the data storage system 120. The controller 130 can execute the received commands in the non-volatile memory array 150 and/or other storage modules, such as magnetic storage 160.
Data storage system 120 can store data communicated by the host system 110. In other words, the data storage system 120 can act as memory storage for the host system 110. To facilitate this function, the controller 130 can implement a logical interface. The logical interface can present to the host system 110 data storage system's memory as a set of logical addresses (e.g., contiguous address) where host data can be stored. Internally, the controller 130 can map logical addresses to various physical locations or addresses in the non-volatile memory array 150 and/or other storage modules. Such mapping can be performed using a logical to physical address mapping table.
The controller 130 includes a system data manager module 132 for generating, maintaining, updating, storing, etc. various types of system data. Such system data can include one or more of the following: logical to physical address mapping data, superblock data, invalid page data, invalid counter data, wear level data, and so on. Although logical to physical address mapping table system data example is used in some parts of this disclosure, approaches disclosed herein are applicable to other types of system data organized as tables or using other suitable data structures.
The controller 130 also includes volatile memory 134 where various types of data can be stored, such as system data and/or logs 136. It may be advantageous to store a working copy of the system data in the volatile memory 134 for faster access (e.g., particularly data that is frequently used by the controller 130 and/or the data storage system 120). In some embodiments, volatile memory 134 can be located outside the controller 130. A back-up copy of the system data can be stored in the non-volatile memory array 150.
Garbage Collection and Logs
In some embodiments, system data can include more or less table pages may be stored than is depicted in
In block 304, the process 300 can generate a log (or logs) that records changes to the volatile copy of the system data caused by the execution of storage access operations. In one embodiment, the process 300 can store the log in the non-volatile memory array 150 (e.g., a non-volatile memory page) so that the log can be maintained in case there is a loss and/or interruption of power, a reboot or restart, etc. of the data storage system 120. In block 306, the process 300 can make updates to the system data stored in the non-volatile memory array based on the changes recorded in the log. In other words, the process 300 can synchronize the volatile and non-volatile copies of the system data. In one embodiment, all changes recorded in the log can be applied to the system data. In another embodiment, changes recorded in the log page can be partially applied to the system data (e.g., some changes can be applied at a certain time and other changes can be applied at a later time). Applying changes based on the log page can reduce write amplification associated with managing of system data and provide coherency because updates to system data may be consolidated. Storing the log persistently (e.g., in non-volatile memory 150) provides power-safety. Further, this can also reduce reconstruction time of the volatile copy of the system data following system restart and/or reduce the start-up time because system data can be updated using information from a limited number of new log pages (such as one new log page). In block 308, the process 300 can perform garbage collection of the log. In other words, the process 300 can mark the log as invalid (or indicate in some other manner that indicates that the log does not contain information that needs to be maintained) so that the log can be overwritten in the future.
In block 406, the process 400 can determine whether the number of generated logs reaches or exceeds the threshold. If the number of generated logs does not reach or exceed the threshold, the process 400 returns to block 402. Otherwise, in block 408, the process 400 can update the non-volatile copy of the system data based on the generated logs. In one embodiment, the process 400 can update the non-volatile copy of the system data based on changes recorded in all new logs that have been generated since the last update of system data took place. In another embodiment, the process 400 can update the non-volatile copy of the system data based on a subset of new logs that have been generated since the last update of system data took place. For example, the process 400 can update the non-volatile copy of the system data based on a number of new logs that have been generated first (e.g., since the last update of system data). In other embodiments, the process 400 can select other subsets of logs. After changes recorded in the logs have been applied to the system data, the process 400 can perform garbage collection of those logs, as is depicted in block 410. Garbage collection can be performed in the manner explained above or any other suitable manner.
Time for reconstruction of the volatile copy of the system data and start-up time of the data storage system 120 can thus be reduced. In one embodiment, reconstruction of system data may involve applying changes recorded in those logs that are valid, which can be bounded by a threshold Tv (e.g., 2, 4, 5, 10, 12, 16, 18, and so on) of block 406 and threshold Ti (e.g., 2, 4, 5, 10, 12, 16, 18, and so on) of block 408. That is, changes recorded in at most Tv log pages may need to be applied to system data in order to reconstruct the system data.
In block 506, the process 500 determines whether execution of storage access commands causes related system data to become outdated. If the process 500 determines that execution of storage access commands causes related system data to become outdated, in block 508 the process can update the non-volatile copy of the system data at a first update frequency or rate. In one embodiment, when execution of storage access commands causes related system data to become outdated, consolidated changes are being made to data stored in the data storage system 120. For example, the host system 110 may be storing data sequentially (such as when storing a media file), which causes data stored in consecutive or substantially consecutive logical addresses to be updated. When such consolidated changes are being made, write amplification associated with managing of system data and system performance can be improved when the non-volatile copy of the system data is updated more frequently. For example, fewer logs may be generated in such case.
In one embodiment, generation of logs can be bypassed, such as when the process 500 determines that execution of storage access commands causes related system data to become outdated. In such case, changes to system data are consolidated, and it may be more efficient to update system data in the non-volatile memory array 150 directly without generating logs. This can reduce write amplification (e.g., because no logs are written to the non-volatile memory array). In one embodiment, metadata can be generated and utilized to invalidate earlier generated logs recording changes to system data that is being updated directly. In other words, changes reflected in some log pages become outdated when generation of logs is bypassed and system data is updated directly. For example, metadata can include identification of a most recently generated log page that comprises outdated system data changes. This log page along with all earlier generated log pages recording outdated changes to system data can be invalidated.
If the process 500 determines that execution of storage access commands does not cause related system data to become outdated, in block 510 the process can update the non-volatile copy of the system data at a second update frequency or rate (which can be, for example, lower than the first update frequency). In one embodiment, when execution of storage access commands does not cause related system data to become outdated, scattered or random changes are being made to data stored in the data storage system 120. For example, the host system 110 may be randomly updating stored data (e.g., as part of performing periodic logging), which causes data stored in non-consecutive or substantially non-consecutive logical addresses to be updated. When such scattered changes are being made, write amplification associated with managing of system data and system performance can be improved by waiting and consolidating more changes to the system data (e.g., because changes may correspond to the same system data entry). For example, more log pages can be generated, and the update frequency can be lower than in the consolidated or sequential case. As another example, changes reflected in already generated log pages can be consolidated into new log pages (and already generated log pages can be invalidated). In one embodiment, the update frequency can be the same as or higher than in the consolidated or sequential case.
In one embodiment, the process 500 can perform updates to the non-volatile copy of the system data based on all new logs generated since the last update of system data took place. In another embodiment, the process 500 can commit changes to system data when a certain number of logs have been generated, as is explained above. This number can be a preset number or can be dynamically determined and/or adjusted. Waiting to consolidate updates to system data can reduce write amplification and ensure coherency. For example, the number of updates to the non-volatile copy of the system data can be reduced because logs can comprise multiple changes corresponding to the same system data entry, and only the latest changes can be applied. Further, updating system data after a certain number of logs have been generated can improve reconstruction time of the volatile copy of the system data and reduce start-up time (e.g., because amount of changes to system data is bounded by the number of new logs).
In addition, the process 500 can update system data based on a subset of new logs that have been generated since the last update of system data took place. For example, as described above, the process 500 can update system data based on a number of logs that have been generated first (e.g., since the last update of system data). In other embodiments, the process 500 can select other subsets of logs. After changes recorded in the logs have been applied to system data, the process 500 can perform garbage collection of those logs. Garbage collection can be performed in the manner explained above or any other suitable manner.
Conclusion
To increase the efficiency of managing and storing system data in non-volatile memory and to reduce write amplification and reconstruction time, a data storage system can utilize garbage collection and logs. Write amplification associated with managing and storing system data in the non-volatile memory can be reduced at least in part because changes to the system data can be consolidated and/or because updates to the system data stored in the non-volatile memory can be performed earlier or later depending on host system (or data storage system) activity. Time associated with a reconstruction of the system data can be reduced because the data storage system can set an upper bound on the number of valid new logs reflecting changes to the system data.
Other Variations
Those skilled in the art will appreciate that additional system components can be utilized, and disclosed system components can be combined or omitted. In some embodiments system data can be generated without being stored in the non-volatile memory. In addition, while in some embodiments a table page or log page refers to a quantity of mapping table data that fits into a page of non-volatile memory, other storage arrangements can be utilized. Although logical to physical address mapping table system data example is used throughout this disclosure, approaches disclosed herein are applicable to other types of system data organized as tables or using other suitable data structures. System data can be stored in other non-volatile memory modules, such as magnetic storage. For example, in one embodiment where the data storage system is a shingled magnetic device, which may use address indirection that require maintenance of mapping table data, the system data may be saved in the magnetic storage 160 in the manners described herein. The actual steps taken in the disclosed processes, such as the processes illustrated in
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the protection. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the protection. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the protection. For example, the systems and methods disclosed herein can be applied to hard disk drives, hybrid hard drives, and the like. In addition, other forms of storage (e.g., DRAM or SRAM, battery backed-up volatile DRAM or SRAM devices, EPROM, EEPROM memory, etc.) may additionally or alternatively be used. As another example, the various components illustrated in the figures may be implemented as software and/or firmware on a processor, ASIC/FPGA, or dedicated hardware. Also, the features and attributes of the specific embodiments disclosed above may be combined in different ways to form additional embodiments, all of which fall within the scope of the present disclosure. Although the present disclosure provides certain preferred embodiments and applications, other embodiments that are apparent to those of ordinary skill in the art, including embodiments which do not provide all of the features and advantages set forth herein, are also within the scope of this disclosure. Accordingly, the scope of the present disclosure is intended to be defined only by reference to the appended claims.
Number | Name | Date | Kind |
---|---|---|---|
5835955 | Dornier et al. | Nov 1998 | A |
6615223 | Shih et al. | Sep 2003 | B1 |
6728826 | Kaki et al. | Apr 2004 | B2 |
6856556 | Hajeck | Feb 2005 | B1 |
7126857 | Hajeck | Oct 2006 | B2 |
7212440 | Gorobets | May 2007 | B2 |
7430136 | Merry, Jr. et al. | Sep 2008 | B2 |
7447807 | Merry et al. | Nov 2008 | B1 |
7502256 | Merry, Jr. et al. | Mar 2009 | B2 |
7502259 | Gorobets | Mar 2009 | B2 |
7509441 | Merry et al. | Mar 2009 | B1 |
7596643 | Merry, Jr. et al. | Sep 2009 | B2 |
7653778 | Merry, Jr. et al. | Jan 2010 | B2 |
7685337 | Merry, Jr. et al. | Mar 2010 | B2 |
7685338 | Merry, Jr. et al. | Mar 2010 | B2 |
7685374 | Diggs et al. | Mar 2010 | B2 |
7702868 | Hardman | Apr 2010 | B1 |
7711923 | Rogers et al. | May 2010 | B2 |
7733712 | Walston et al. | Jun 2010 | B1 |
7765373 | Merry et al. | Jul 2010 | B1 |
7898855 | Merry, Jr. et al. | Mar 2011 | B2 |
7912991 | Merry et al. | Mar 2011 | B1 |
7913061 | Gorobets et al. | Mar 2011 | B2 |
7936603 | Merry, Jr. et al. | May 2011 | B2 |
7962792 | Diggs et al. | Jun 2011 | B2 |
8078918 | Diggs et al. | Dec 2011 | B2 |
8090899 | Syu | Jan 2012 | B1 |
8095851 | Diggs et al. | Jan 2012 | B2 |
8108692 | Merry et al. | Jan 2012 | B1 |
8122185 | Merry, Jr. et al. | Feb 2012 | B2 |
8127048 | Merry et al. | Feb 2012 | B1 |
8135903 | Kan | Mar 2012 | B1 |
8151020 | Merry, Jr. et al. | Apr 2012 | B2 |
8161227 | Diggs et al. | Apr 2012 | B1 |
8166245 | Diggs et al. | Apr 2012 | B2 |
8194340 | Boyle et al. | Jun 2012 | B1 |
8194341 | Boyle | Jun 2012 | B1 |
8243525 | Kan | Aug 2012 | B1 |
8254172 | Kan | Aug 2012 | B1 |
8261012 | Kan | Sep 2012 | B2 |
8296625 | Diggs et al. | Oct 2012 | B2 |
8312207 | Merry, Jr. et al. | Nov 2012 | B2 |
8316176 | Phan et al. | Nov 2012 | B1 |
8341339 | Boyle et al. | Dec 2012 | B1 |
8375151 | Kan | Feb 2013 | B1 |
8392635 | Booth et al. | Mar 2013 | B2 |
8397107 | Syu et al. | Mar 2013 | B1 |
8407449 | Colon et al. | Mar 2013 | B1 |
8423722 | Deforest et al. | Apr 2013 | B1 |
8433858 | Diggs et al. | Apr 2013 | B1 |
8443167 | Fallone et al. | May 2013 | B1 |
8447920 | Syu | May 2013 | B1 |
8458435 | Rainey, III et al. | Jun 2013 | B1 |
8478930 | Syu | Jul 2013 | B1 |
8489854 | Colon et al. | Jul 2013 | B1 |
8503237 | Horn | Aug 2013 | B1 |
8521972 | Boyle et al. | Aug 2013 | B1 |
8549236 | Diggs et al. | Oct 2013 | B2 |
8582223 | Garani et al. | Nov 2013 | B1 |
8583835 | Kan | Nov 2013 | B1 |
8601311 | Horn | Dec 2013 | B2 |
8601313 | Horn | Dec 2013 | B1 |
8612669 | Syu et al. | Dec 2013 | B1 |
8612804 | Kang et al. | Dec 2013 | B1 |
8615681 | Horn | Dec 2013 | B2 |
8638602 | Horn | Jan 2014 | B1 |
8639872 | Boyle et al. | Jan 2014 | B1 |
8683113 | Abasto et al. | Mar 2014 | B2 |
8700834 | Horn et al. | Apr 2014 | B2 |
8700950 | Syu | Apr 2014 | B1 |
8700951 | Call et al. | Apr 2014 | B1 |
8706985 | Boyle et al. | Apr 2014 | B1 |
8707104 | Jean | Apr 2014 | B1 |
8713066 | Lo et al. | Apr 2014 | B1 |
8713357 | Jean et al. | Apr 2014 | B1 |
8719531 | Strange et al. | May 2014 | B2 |
8724422 | Agness et al. | May 2014 | B1 |
8725931 | Kang | May 2014 | B1 |
8745277 | Kan | Jun 2014 | B2 |
8751728 | Syu et al. | Jun 2014 | B1 |
8769190 | Syu et al. | Jul 2014 | B1 |
8769232 | Suryabudi et al. | Jul 2014 | B2 |
8775720 | Meyer et al. | Jul 2014 | B1 |
8782327 | Kang et al. | Jul 2014 | B1 |
8788778 | Boyle | Jul 2014 | B1 |
8788779 | Horn | Jul 2014 | B1 |
8788880 | Gosla et al. | Jul 2014 | B1 |
8793429 | Call et al. | Jul 2014 | B1 |
8966205 | Lo et al. | Feb 2015 | B1 |
8984247 | Lo et al. | Mar 2015 | B1 |
20020184436 | Kim et al. | Dec 2002 | A1 |
20030065899 | Gorobets | Apr 2003 | A1 |
20040210706 | In et al. | Oct 2004 | A1 |
20050166028 | Chung et al. | Jul 2005 | A1 |
20080082775 | Gorobets | Apr 2008 | A1 |
20080098195 | Cheon et al. | Apr 2008 | A1 |
20090150599 | Bennett | Jun 2009 | A1 |
20100106897 | Yoshimura | Apr 2010 | A1 |
20100174849 | Walston et al. | Jul 2010 | A1 |
20100180068 | Matsumoto et al. | Jul 2010 | A1 |
20100250793 | Syu | Sep 2010 | A1 |
20100262799 | Lasser et al. | Oct 2010 | A1 |
20110029720 | Lu | Feb 2011 | A1 |
20110099323 | Syu | Apr 2011 | A1 |
20110161621 | Sinclair et al. | Jun 2011 | A1 |
20110173395 | Bhattacharjee et al. | Jul 2011 | A1 |
20110283049 | Kang et al. | Nov 2011 | A1 |
20110296123 | Adler et al. | Dec 2011 | A1 |
20110307651 | Wong | Dec 2011 | A1 |
20120110258 | Lakey et al. | May 2012 | A1 |
20120239860 | Atkisson et al. | Sep 2012 | A1 |
20120260020 | Suryabudi et al. | Oct 2012 | A1 |
20120265924 | Purdy et al. | Oct 2012 | A1 |
20120278531 | Horn | Nov 2012 | A1 |
20120284460 | Guda | Nov 2012 | A1 |
20120324191 | Strange et al. | Dec 2012 | A1 |
20130132638 | Horn et al. | May 2013 | A1 |
20130145106 | Kan | Jun 2013 | A1 |
20130166819 | Yerushalmi et al. | Jun 2013 | A1 |
20130185508 | Talagala et al. | Jul 2013 | A1 |
20130290793 | Booth et al. | Oct 2013 | A1 |
20140059405 | Syu et al. | Feb 2014 | A1 |
20140101369 | Tomlin et al. | Apr 2014 | A1 |
20140115427 | Lu | Apr 2014 | A1 |
20140133220 | Danilak et al. | May 2014 | A1 |
20140136753 | Tomlin et al. | May 2014 | A1 |
20140149826 | Lu et al. | May 2014 | A1 |
20140157078 | Danilak et al. | Jun 2014 | A1 |
20140181432 | Horn | Jun 2014 | A1 |
20140223255 | Lu et al. | Aug 2014 | A1 |
20160048352 | Lo et al. | Feb 2016 | A1 |
Entry |
---|
U.S. Appl. No. 13/478,036, filed May 22, 2012, to Lo, et al., 22 pages. |