Patent Application
20100306236
Referring to
An archiving file system may have the additional capability to maintain, for each file or directory, a multiplicity of copies on a plurality of storage devices, either randomly or sequentially accessible, as well as possibly having the capability to maintain and preserve a multiplicity of incarnations. Storage may be stratified into primary storage 24 and secondary storage 26, with the storage blocks 14, 16 residing in the primary storage 24. The data associated with a given file need not be contained in the primary storage 24. It is possible for it to be resident in the secondary storage 26 only. The secondary storage 26 may encompass such technologies as magnetic disk, magnetic tape, non-volatile memories, optical disk and tape, CD-ROM, WORM, etc. Risk of data loss may be managed through the use of the secondary storage 26.
A data policy management system includes one or more computers configured to execute an archiving file system, a database system, at least one asynchronous update process, and at least one data policy manager process. The archiving file system is configured to inform the at least one asynchronous update process of nodes that have been updated. The at least one asynchronous update process is configured to acquire current information contained within the nodes that has been updated, and to update data contained within the database system to reflect the acquired information. The at least one data policy manager process is configured to query the database system, and to enforce a set of data policies upon the archiving file system based on results of the query.
A method for managing data includes identifying nodes of an archiving file system executing on one or more computers that have been updated, acquiring time ordered node state change events within the archiving file system, storing the node state change events, and reading the stored node state change events. The method further includes acquiring current information contained within the nodes that has been updated, updating data contained within a database system executing on the one or more computers to reflect the acquired information, querying the database system, and enforcing data policies upon the archiving file system based on the results of the query.
A computer storage medium has information stored thereon for directing one or more computers to (i) identify nodes of an archiving file system that have been updated, (ii) acquire time ordered node state change events within the archiving file system, (iii) store the node state change events, (iv) read the stored node state change events, (v) acquire current information contained within the nodes that has been updated, (vi) update data contained within a database system to reflect the acquired information, (vii) query the database system, and (viii) enforce data policies upon the archiving file system based on the results of the query.
While example embodiments in accordance with the invention are illustrated and disclosed, such disclosure should not be construed to limit the invention. It is anticipated that various modifications and alternative designs may be made without departing from the scope of the invention.
A data policy may be a mechanism that provides governance over data that is contained within an archiving file system, either directly through active nodes or indirectly through nodes that had at some temporal moment existed as an active node in the file system. Data policies may control the life span and retention requirements of such data, as well as govern the storage requirements and residency of such data. For example, a data policy may define the minimum number of secondary storage copies that a file must have before it is considered to be safe. Other policies may determine the retention conditions that a file must reside in primary storage. Further policies may define the life span of the data.
The proper operation of an archiving file system may require interrogation of the file system to make decisions regarding the data contained within the given file system. The I/O needed to make these decisions may be unproductive as it detracts from user initiated (productive) I/O including reading, writing or migrating data between primary and secondary storage. A data policy may be used as the instrument of governance to determine a specific data copy's lifespan and storage residency requirements. The management of this data policy may require interrogation of the file system's current state. That is, as files are created, deleted modified, etc., their attributes are changed, and the state of the file system is also changed.
During the governance of data represented by a file system, a given data item may have one of four states (for a given data item associated with a given file, these four states may be a function of file system state for the file, and the data policies that govern the file): (1) The active state—if a file is represented as a node in the current file system, then it along with its current incarnation of data is considered to be active; (2) The dormant state—if a given data incarnation is no longer actively accessible through the current file system (either, for example, by no longer being represented by a node in the file system because the file or directory has been deleted, or due to later incarnation where the node can no longer directly access the data as it resides on secondary storage), then the data is considered to be dormant; (3) The expired state—if a given data incarnation is no longer accessible through the file system and no longer meets the retention policies as expressed by the data policy, then the data is considered to be expired; and, (4) The recycled state—if the data has been exactly copied to a different unit of secondary storage (where a unit of storage may be a plurality of storage devices), then the data is considered to be recycled. This instance of the data theoretically remains available until the unit of storage it resides on is physically over written or destroyed.
Current archiving file systems may have difficulty enforcing a data policy. The task of managing a data policy may require repeated and extensive interrogation of the file and directory nodes, and the directory lists. As mentioned above, computation and I/O incurred by this interrogation may be unproductive and detract from productive computation and I/O related to reading and writing data in the file system. Furthermore, current file systems may have difficulty enforcing data policies at the data level. Policy decisions may be based on the time an archive copy is created, rather than on the creation time or modification time of the data. Furthermore, file system knowledge of the secondary storage of past incarnations of a given data element may be lost. Recovery of such knowledge may require the extensive task of reading all secondary storage associated with the file system.
Advanced backup systems may focus on the backing-up and selective restoral aspects of data management. Other backup systems may focus on the ability to restore a file or set of files should they become destroyed or corrupted. These systems, however, may either have difficulty enacting a data policy, or ignore it all together. While many such systems maintain an inventory (some of which utilize a database) of copies that have been made, this information is not coupled with the file system. The purpose of such inventories is to answer the query “given a file's name, what restorable copies of this file are available?” The backup inventory is not synchronized with the current state of the file system. For example, suppose that a file has been renamed where one of the symbolic names contained within the path of the file has changed. To such a backup system, such a file becomes a completely new entity.
Node information is stored using a database in certain file systems. Such systems rely on immediate synchronous update of the database as the state of the file system changes, i.e., files are created, modified, deleted, symbolic names changed, etc. Without immediate update, these file systems may block subsequent access to the file system.
In certain embodiments, the management of a multiplicity of data policies is separated from an archiving file system. An adjacent database may be employed to mirror (shadow) the state of the file system. The database may be interrogated by a data policy management mechanism, thereby minimizing unproductive I/O upon the file system for the purposes of data governance. A logging mechanism may be used to monitor changes in the state of the file system. An updating mechanism may maintain synchronization between the file system and the adjacent database.
The data policy management may be capable of operating while the adjacent database is inconsistent with the state of the file system. The inconsistency between the adjacent database and the file system may be referred to as a window of inconsistency. The acceptable window of inconsistency may depend on the nature of the query being made on behalf of the data policy manager. The governance of the multiplicity of data policies may be performed within varying windows of inconsistency. The file system, however, may be the final authority. Furthermore, the database may be distributed across a network and need not reside on the system that hosts the file system.
Referring now to
The file system 30 may include a multiplicity of files and directories, and manage storage on a primary storage device and a plurality of secondary storage devices. The file system 30 may also include a host processor(s), and a hierarchy of memories used for the transport of data within the file system 30, and among the primary and secondary storage devices.
The data management system 28 may further include a logger 34 (e.g., a logger process), updater 36 (e.g., updating process), and data policy manager 38 (e.g., a policy manager process). The logger 34 may extract events from the file system 30 in a manner that preserves their order of occurrence. These events may be stored in an event log 40. The updater 36 may update the database 32 to reflect changes made in the file system 30. The data policy manager 38 may interact with the adjacent database 32, and initiate actions to enforce the specified data policies. For some events, the database 32 may already be current with the file system 30 due to the latency of the time the event occurred to the time it is read from the event log 34, and processed by the updater 36, because an earlier event for the node triggered the update where the information was inconsistent.
The adjacent database 32, in some embodiments, may include a database engine such as MySQL, related storage devices that host the data associated with this database, a client application program interface that connects the updater 36 and data policy manager 38 to the database engine, and a set of tables discussed in more detail below. The tables discussed below mirror the relevant information contained within the file system 30. While the tables mirror relevant information in the file system 30, the data associated/accumulated in the database may, over time, contain information beyond that contained in the current state of the file system 30. For example if a file is deleted, the file system 30 may no longer know of the file, whereas the database 32 may contain the history of this file, when created, when deleted, any archive copies residing in secondary storage, etc.
Referring now to
Referring now to
Should the event buffer 56 at any given time have only one remaining entry at the time of the event, a lost event (as mentioned above) may be placed into the buffer 56 and the actual event may be lost. Should the buffer 56 be full at the time, no action may be taken. The event is not recorded and may be considered lost. The time stamp associated with this event marks the start time of lost events. When the buffer 56 has been emptied by the logger 34, the time stamp of the next event marks the time that event logging has resumed. Lost activity may be discovered by the sequential scanning of all nodes for nodes that have a change time after the lost event time stamp and before the time stamp of the following recorded event. The following example node update algorithm may be applied:
1. Get node information from database.
2. If node entry not found, then go to NEW (4).
3. Build lists from database for name and archive.
4. NEW: Read node data from the archiving file system.
5. If node information is not available from 4 and node entry not found in database, then the file is transitory.
6. If node data for this temporal version of the node is not available, DELETE entry as follows:
a. Get name entry for parent directory from database.
b. Search name list (built in 3) for entry that matches name path determined in 6a.
c. Mark entry as deleted.
d. If all entries in name list marked deleted, mark node as deleted.
7. Determine path of parent directory from database.
8. If new node, INSERT node into database, else UPDATE database as follows:
a. If size in node < > size in database then update.
b. If creation time in node < > create time in database then update.
c. If modification time in node < > modification time in database then update.
d. If user id in node < > user id in database then update.
e. If group id in node < > group id in database then update.
f. Update any other fields contained in the node data and tracked in the database.
9. Scan parent directory and build list of all entries that match node identifier. This list is to known as the object list.
10. If name list (built in 3) is empty, then INSERT name into database, go to 15.
11. Mark each entry in name list where the path name does not match the path name (as determined in 7) for this file.
12. For entry in the path list which matches in path, then mark each entry in the name list where the object name matches a name in the object list (built in 9) and remove matching entry from object list.
13. If object list is empty, then INSERT name into database and go to 15. (After execution of 11 and 12, if there is an item in the object list, then the name entry being worked has either been renamed or a new entry created. Furthermore, all marked name list entries are eliminated from consideration since they either did not match in path or a one-for-one match was found between a name in the object list and an object name in the name list.)
14. Use first remaining entry in the object list and UPDATE first unmarked name list entry replacing object name with first remaining entry in object list.
15. For each archive entry identified in the file system node information, do
a. If archive entry not in archive list (built in 3), then INSERT archive entry.
16. For each archive entry UPDATE copy stale status as follows:
a. If modification time in node does not match modification time associated with entry in archive list, then archive entry is considered to be stale.
Special consideration may be made to the algorithm expressed above to accommodate UNIX style symbolic links. The following additional steps may be needed:
1. After 4, build link list from database if file type is symbolic link.
a. Read link value string.
b. If link list empty, then INSERT link into database, else UPDATE database as follows:
Referring now to
Name Table (path table):
Each node in the file system 30 may be identified with a unique number. Certain archiving file systems 30, such as Sun Microsystems' SAM-QFS, uniquely identify each node and each temporal instance or generation of each node. In certain embodiments described herein, each update interrogates node information 70 contained within the file system 30, and a directory that instances a node 72. The file system 30, in the embodiment of
A rename event occurs when the symbolic name has changed or the file has been moved from one directory to another. In the later case, the parent nodes of the origin directory and destination directory must be reported in the event buffer 56 illustrated in
1. Read node information from the file system.
2. If node information not available, then file has been deleted and the rename is lost, exit.
3. Build name list from database.
4. Determine path of source and target parent directories from database.
5. Scan source and target parent directories and build object lists of all entries that match node identifier.
6. Scan name list to eliminate entries from the name list and corresponding object lists based on the following conditions:
a. If the path matches the target path and the object name is found in the target object list (built in 5), then eliminate name list entry and entry in target object list.
b. If the path matches the source path and the object name is found in the source object list (built in 5), then eliminate name list entry and entry in source object list.
c. If the path does not match the target path or the source path, then eliminate name list entry. (The remaining entry in the name list should be the entry that is being renamed. The target object list should have at least one entry remaining, and in most cases the source object list should be empty. It is possible that subsequent file system operations may have created additional files linked to this node.)
7. If the node is a directory, then update database as follows:
a. For each name entry in database in which the path leading matches the source name as determined from the remaining name list entry resulting from execution of 6, replace the leading path with the directory's new path name which is the concatenation of the target directory path with the target object name.
8. UPDATE name entry in database as follows:
a. If rename involves changing directories, then replace path in name entry with path of target parent directory.
b. Replace object in name entry with target object name.
9. Proceed to update node as described above.
For the case where only the symbolic name is changing, the event may be recorded with only one entry. The event parameter identifies the nature of the rename. Possibilities for rename include (i) rename where only the symbolic name for the file is changed (the file does not change directories), and (ii) rename where the file is moved from one directory to another. It is the second case where two events may appear. The first event identifies the parent directory of origin (the source) and the second event identifies the destination parent directory (the target). The symbolic name may change as part of this move.
Referring now to
Policies may include secondary storage disposition, data lifespan and retention enforcement, and secondary storage recycling. Informative queries of the database 32 may also be made including complete temporal file history, secondary storage utilization, secondary storage contents, and the construction of inventories for specific units of secondary storage. To respond to these queries, the database 32 need not be fully synchronized with the file system 30 illustrated in
As apparent to those of ordinary skill, the algorithms, etc. disclosed herein may be deliverable to a processing device in many forms including, but not limited to, (i) information permanently stored on non-writable storage media such as ROM devices and (ii) information alterably stored on writeable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media. The algorithms, etc. may also be implemented in a software executable object. Alternatively, the algorithms, etc. may be embodied in whole or in part using suitable hardware components, such as Application Specific Integrated Circuits (ASICs), state machines, controllers or other hardware components or devices, or a combination of hardware, software and firmware components.
While embodiments of the invention have been illustrated and described, it is not intended that these embodiments illustrate and describe all possible forms of the invention. Rather, the words used in the specification are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the invention.
