Patent Application
20070294310
The accompanying drawings, which are incorporated in and constitute a part of this specification exemplify the embodiments of the present invention and, together with the description, serve to explain and illustrate principles of the inventive technique. Specifically:
In the following detailed description, reference will be made to the accompanying drawing(s), in which identical functional elements are designated with like numerals. The aforementioned accompanying drawings show by way of illustration, and not by way of limitation, specific embodiments and implementations consistent with principles of the present invention. These implementations are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other implementations may be utilized and that structural changes and/or substitutions of various elements may be made without departing from the scope and spirit of present invention. The following detailed description is, therefore, not to be construed in a limited sense. Additionally, the various embodiments of the invention as described may be implemented in the form of a software running on a general purpose computer, in the form of a specialized hardware, or combination of software and hardware.
According to an exemplary embodiment of the present invention, an inventive FCAS system includes a file system configured to store the Fixed Content data and a database system configured to manage the stored Fixed Content and its metadata along with other related information like retention and management policy. Additionally, the inventive FCAS includes data integrity resolving means. The data integrity resolving means executes a file system recovery procedure and a database recovery procedure. The aforesaid two recovery procedures are executed separately, whereupon the data integrity resolving means compares the corresponding lists of entries in both the file system and the database from viewpoint of pointers. Subsequently, the data integrity resolving means selects and recovers only entries having the complete file and metadata pointers and extracts and stores all pointer-incomplete entries, files and metadata, in temporary memory area. The inventive FCAS also contains a recovery point extracting means, which, upon a request from the archive software, extracts and returns to the archiving software a recovery point as well as other recovery-related information. Specifically, the aforesaid recovery point extracting means may furnish the content identifier (ID) and/or the timestamp of the recovery point. In an embodiment of the invention, the recovery point is the very last entry corresponding to pointer complete files and metadata. In addition to the recovery point, the recovery point extracting means may furnish to the arching software the file identifiers (ID) of pointer incomplete files, attributes of pointer incomplete metadata as well as the timestamps of files and metadata, which the archiving software may utilize to confirm that the utilized data is the most recent.
The host 10 and the storage 20 communicate through storage network 92, which may include one or more network switches. Exemplary embodiments of the storage network 92 include, without limitation, FibreChannel, ESCON, FICON, and IP networks, implementing the iSCSI, NFS, CIFS, HTTP, WebDAV, and/or any other IP-based storage access protocols.
The host 30 and the FCAS 40 also communicate through a storage network, which is designated in
The host 10 can be implemented based on any suitable computer platform including, without limitation, mainframe, UNIX based or Windows based. Examples of the application software 11 include database, email, file server and other similar applications.
The storage system 20 can be implemented based on any known storage system architecture, including, without limitation, a block-based external storage system architecture, NAS (Network Attached Storage) architecture and the like. Examples of commercially-available implementations of the storage 20 include, without limitation, Hitachi TagmaStore USP, NSC, AMS and WMS. The implementation of the storage network 92 may vary depending on the details of the implementation of the storage system 20. Additionally, the storage system 20 may include replication means 21 configured to facilitate the replication of application data 22 stored in the storage system 20 to a remote storage system.
The host 30 may also be implemented based on a variety of existing computer platforms. The archiving software 31 executing on the host 30 communicates with the application software 11 executing on the host 10, facilitates the archiving of the application data 21 and stores archive data, consisting of fixed content 62, its metadata 63 and other related data, to the inventive FCAS 40. Examples of the archive software 31 include email archive software, database archive software, file archive software, and the like. The system architecture depicted in
In one embodiment of the invention, the archiving software 31 may include means 32 configured to issue a query to the FCAS system 40, requesting the FCAS system 40 to furnish the information on the recovery point to the archiving software 31.
An exemplary embodiment of the inventive FCAS system 40 shown in
The aforesaid gateway 50 and the storage 60 may communicate through storage network 94, which may be implemented using one or more network switches. The set of exemplary embodiments of the storage network 94 may be the same as the embodiment of the storage network 92. In one embodiment of the invention, FibreChannel and/or iSCSI networking interconnects are utilized for purposes of implementing the storage network 94. Additionally, in one system implementation, the storage network 94 is not accessible from the outside of the FCAS system 40.
The gateway 50 provides object-based interface and may be implemented using specialized software executing on one of the host computers of the system. The FCAS software may implement any required or desired FCAS functionalities including, without limitation, data retention management, WORM (Write Once Read Many), shredding, namespace management, auditing and the like.
The storage system 60 can be based on any existing storage system architecture, such as a block based external storage system, NAS (network Attached Storage) and the like. Commercially available products, which may be used in implementing the storage system 60 include, without limitation, Hitachi TagmaStore USP, NSC, AMS and WMS. The storage system 60 stores and manages the fixed content 62, its metadata 63 and any other related data (not shown).
The storage system 60 includes replication means 61, which is leveraged to protect the fixed content 62, its metadata 63 and other related data by replicating the respective information.
The embodiment of the FCAS Gateway 50 shown in
The FCAS gateway 50 additionally includes Recovery Point Extracting Means 51. In one embodiment of the invention, the archive data is protected by way of replication, wherein the archive data 22 is copied and stored in any storage means. Therefore, the archive data can be re-generated by the archive software 31 as long as the archive software is aware of the corresponding recovery point. The Recovery Point Extracting Means 51 determines the recovery point and returns it to the archive software 31 upon a request from the recovery point inquiring means 32.
The storage system 20a may additionally incorporate data and metadata replication means 21a, as well as other functions and platforms, including, without limitation, replication management software, administrator's management process and skills, and network infrastructures for remote replication.
In this system configuration, the application software 11 and the archive software 31, together with the FCAS gateway 50 share the same storage system 20a, thereby achieving the consolidation of the utilized storage devices. Because no additional separate storage system is required, the initial costs of setting up the system depicted in
The system configuration shown in
It should be noted that not all the data needs to be stored at the external storage units 81, 82 and 83, but some of the data may be stored in a storage area in the storage virtualization system 70. For example, the application data 72 may require a high storage system performance and, therefore, this data should be stored in the storage virtualization system 70. The metadata 73 may impose the same performance requirements as the application data 72, because various search functions use these metadata. Therefore, the metadata may also need to be stored in a high performance storage virtualization system 70. However, the fixed content 73, itself may not require advanced storage performance characteristics, but require low a TCO (total cost of ownership). Therefore, such fixed content may be stored in cheap external storage units. The aforesaid concept is usually called as HSM (Hierarchical Storage Management) or even ILM/DLM (Information/Data Lifecycle Management).
The storage virtualization system 70 will now be described. Recently, the virtualization has become one of popular technologies in the storage industry. The term virtualization is utilized to characterize an act of integrating one or more (back end) services or functions with additional (front end) functionality for the purpose of providing useful abstractions. Typically, virtualization hides some of the back end complexity, or adds or integrates new functionality with existing back end services. Examples of virtualization are the aggregation of multiple instances of a service into one virtualized service, or adding security to an otherwise insecure service. Virtualization can be nested or applied to multiple layers of a system.
Virtualization system is a system that incorporates the facilities for providing the virtualization functionality. Exemplary embodiments of commercially available virtualization systems, include, without limitation, Hitachi USP (Universal Storage Platform) having a virtualization engine called Universal Volume Manager, IBM SVC (SAN Volume Controller), EMC Invista, CISCO MDS, and the lime. It should be noted that some available virtualization systems, such as Hitachi USP, themselves incorporate an integral storage system.
Other suitable exemplary virtualization systems are described in Japanese patents No. 10-283272 and 2004-5370, which are incorporated herein by reference in their entirety.
In-system replication also can be consolidated (not shown in the figures). Not only the application data but also the archive data are copied from the primary volumes to secondary volumes, by using in-system replication means, the replicated data may be also copied to outside storage media like tapes and, thus, all the data are protected. Because the replication platform is shared among the production system (application) and the archive system, users can effectively eliminate operating costs as well as initial deployment costs.
File location column 312 identifies the storage location where the file corresponding to a specific fixed content instance is stored. In the example shown in
The value in the application type column 313 indicates the type of the application that generated the original fixed content data. This attribute is used, for example, when the fixed content is reused in the future.
The description column 314 briefly describes the information contained in the fixed content. This attribute is useful, for example, in searching for appropriate fixed content instance. The application type attribute 313 and the description attribute 314 are examples of fixed content metadata.
Retention period attribute 315 specifies how long the fixed content will be preserved. The metadata table may further specify a process, which is utilized for, example, to dispose of the fixed content, when the fixed content retention period expires. For example, after the prescribed retention period expires, the fixed content may be subject to deletion and/or shredding. The retention period attribute 315 is an example of fixed content management policies.
As the other attributes, each metadata instance may have an associated timestamp indicating when the metadata was last updated. This timestamp may be accessible by the archive software as will be described in detail below. The rows 302 and 303 of table 212 shown in
The inventive process for resolving data integrity issues may be initiated by an administrator, archiving software, the FCAS system itself as well as other system components. Furthermore, the inventive process may be initiated from any location, where the replicated data is stored, including for example, a secondary data storage site.
Now, the process illustrated in
As shown in
At step 411, the inventive FCAS recovers its file system. Recovering the file system is a well known technology to persons of skill in the art. The exact manner of the recovery procedure depends on the implementation of the storage system(s) involved. Therefore, the details of the implementation of the file system recovery are not essential to the concept of the present invention. For example, during the aforesaid file system recovery step 411, the file 541 that has been written last is checked for completeness. More specifically, the contents of the file 541 may be checked for data integrity. If data integrity problems are encountered, the file 541 is withdrawn. The described file system recovery step is sometimes referred to as File System Crash Recovery or fsck (file system check)
At step 412, the FCAS recovers its database. Recovering the database is also a well known technology. The exact manner of the database recovery procedure depends on the implementation of the data system(s) involved. Therefore, the details of the implementation of the database system recovery are not essential to the concept of the present invention. For example, during the database recovery step 412, the metadata entry 542, which appears in the database log subsequently to a checkpoint (not shown), is checked for completeness and/or the integrity of the data that it contains. If any data completeness and/or integrity problems are discovered, the entry 542 is withdrawn. The described database recovery step is sometimes referred to as Database Crash Recovery.
At step 413, the inventive FCAS system compares pointer information associated with each of the entries in the list of files stored by the file system and the list of metadata records stored in the database. During this step, the system checks for existence of any files that are not pointed to by at least one metadata record. In addition, the system checks for any metadata records that do not point to at least one file and/or for any metadata records, which point to files that do not exist. The files and metadata identified at step 413 are referred to as pointer incomplete and designated in
At step 414, the inventive FCAS system selects and recovers only the file and metadata entries having complete pointer information. In other words, any pointer incomplete entries are deleted or moved to a temporary memory space. The removed entries are logically incomplete and need to be archived again.
At step 415, the inventive FCAS extracts and stores all pointer incomplete entries, including files and metadata, in the temporary memory area. Those data may be accessed and re-used by recovery point extracting means 51 as well as other system components.
The inventive procedure for obtaining a recovery point is described with reference to
In another example, not illustrated in the figures, the archiving software may query the inventive FCAS system for the information on the pointer incomplete files and/pr pointer incomplete metadata shown in
With reference to
For example, if the discrepancy between the timestamp of a fixed content file and the timestamp of the associated metadata's is greater than a predetermined threshold value, the archiving software may determine that the last update of either the file or the metadata has failed and that the currently available data is outdated and needs to be updated again.
An exemplary system architecture of the second embodiment of the invention is shown in
The computer platform 1201 may generally include a processor 1205 for handling various information and performing other computational and control tasks. The computer platform 1201 may also include a volatile storage 1206, such as a random access memory (RAM) or other similar dynamic storage device for storing various information as well as instructions to be executed by processor 1205. The volatile storage 1206 also may be used for storing temporary variables or other intermediate information during execution of instructions by processor 1205. Computer platform 1201 may further include a read only memory (ROM or EPROM) 1207 or other static storage device for storing static information and instructions for processor 1205, such as basic input-output system (BIOS), as well as various system configuration parameters. A persistent storage device 1208, such as a magnetic disk, optical disk, or solid-state flash memory device may be provided for storing information and instructions. Upon start of the computer platform 1201, it may be configured to automatically load and execute the instructions stored in the storage devices 1207 and/or 1208.
The aforementioned processor 1205, as well as storage devices 1206, 1207 and 1208 may be interconnected using a data bus 1204 facilitating exchange of data among various elements of the computer system 1201. The data bus 1204 may be implemented using any known computer interconnect mechanism, including, without limitation, PCI, SCSI, Infiniband, etc.
When the computer platform 1201 is utilized to implement the host 10 and host 30, it may be coupled via bus 1204 to a display 1209, such as a cathode ray tube (CRT), plasma display, or a liquid crystal display (LCD), for displaying information to a system administrator or user of the computer platform 1201. An input device 1210, including alphanumeric and other keys, is coupled to bus 1201 for communicating information and command selections to processor 1205. Another type of user input device is cursor control device 1211, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 1204 and for controlling cursor movement on display 1209. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
When the computer platform 1201 is used to implement the storage systems 20 and/or 60, as well as the storage virtualization system 70, one or more external storage units 1212 may be connected to the computer platform 1201 via bus 1204 to implement storage units 22, 62, 63 and 81-83 for storing various data and metadata. Additionally or alternatively, the aforesaid storage units may be implemented using persistent storage 1208 or network storage 1219 or 1222. In an embodiment of the computer system 1200, the external removable storage device 1212 may be used to facilitate exchange of data with other computer systems.
According to one embodiment of the invention, the functions of the host 10, host 30, FCAS systems 40/50, storage system 20 and/or storage virtualization system 70 are performed by an implementation of the computer platform 1201 in response to the processor 1205 executing one or more sequences of one or more instructions contained in the volatile memory 1206. Such instructions may be read into volatile memory 1206 from another computer-readable medium, such as persistent storage device 1208. Execution of the sequences of instructions contained in the volatile memory 1206 causes processor 1205 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.
The term “computer-readable medium” as used herein refers to any medium that is capable of providing, directly or indirectly, instructions to processor 1205 for execution. The computer-readable medium is just one example of a machine-readable medium, which may carry instructions for implementing any of the methods and/or techniques described herein. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 1208. Volatile media includes dynamic memory, such as volatile storage 1206. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise data bus 1204. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, a flash drive, a memory card, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.
Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to processor 1205 for execution. For example, the instructions may initially be carried on a magnetic disk from a remote computer. Alternatively, a remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 1200 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on the data bus 1204. The bus 1204 carries the data to the volatile storage 1206, from which processor 1205 retrieves and executes the instructions. The instructions received by the volatile memory 1206 may optionally be stored on persistent storage device 1208 either before or after execution by processor 1205. The instructions may also be downloaded into the computer platform 1201 via Internet using a variety of network data communication protocols well known in the art.
The aforesaid host 10 and host 30 are designed to operate on a computer network 91. To this end, the computer platform 1201 also includes a communication interface, such as network interface card 1213 coupled to the data bus 1204. Communication interface 1213 provides a two-way data communication coupling to a network link 1214 that is connected to a local network 1215. For example, communication interface 1213 may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 1213 may be a local area network interface card (LAN NIC) to provide a data communication connection to a compatible LAN. Wireless links, such as well-known 802.11a, 802.11b, 802.11 g and Bluetooth may also used for network implementation. In any such implementation, communication interface 1213 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 1213 typically provides data communication through one or more networks to other network resources. For example, network link 1214 may provide a connection through local network 1215 to a host computer 1216, or an additional network storage/server 1222. Additionally or alternatively, the network link 1213 may connect through gateway/firewall 1217 to the wide-area or global network 1218, such as an Internet. Thus, the computer platform 1201 can access network resources located anywhere on the Internet 1218, such as a remote network storage/server 1219. On the other hand, the computer platform 1201 may also be accessed by clients located anywhere on the local area network 1215 and/or the Internet 1218. The network clients 1220 and 1221 may themselves be implemented based on the computer platform similar to the platform 1201.
Local network 1215 and the Internet 1218 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 1214 and through communication interface 1213, which carry the digital data to and from computer platform 1201, are exemplary forms of carrier waves transporting the information.
Computer platform 1201 can send messages and receive data, including program code, through the variety of network(s) including Internet 1218 and LAN 1215, network link 1214 and communication interface 1213. In the Internet example, when the system 1201 acts as a network server, it might transmit a requested code or data for an application program running on client(s) 1220 and/or 1221 through Internet 1218, gateway/firewall 1217, local area network 1215 and communication interface 1213. Similarly, it may receive code from other network resources.
The received code may be executed by processor 1205 as it is received, and/or stored in persistent or volatile storage devices 1208 and 1206, respectively, or other non-volatile storage for later execution. In this manner, computer system 1201 may obtain application code in the form of a carrier wave.
Finally, it should be understood that processes and techniques described herein are not inherently related to any particular apparatus and may be implemented by any suitable combination of components. Further, various types of general purpose devices may be used in accordance with the teachings described herein. It may also prove advantageous to construct specialized apparatus to perform the method steps described herein. The present invention has been described in relation to particular examples, which are intended in all respects to be illustrative rather than restrictive. Those skilled in the art will appreciate that many different combinations of hardware, software, and firmware will be suitable for practicing the present invention. For example, the described software may be implemented in a wide variety of programming or scripting languages, such as Assembler, C/C++, perl, shell, PHP, Java, etc.
Moreover, other implementations of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. Various aspects and/or components of the described embodiments may be used singly or in any combination in the data search and retrieval system. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
