Patent Application
20070214105
An embodiment of the invention is generally directed to electronic data storage systems that have relatively high capacity, performance and data availability, and more particularly to ones that are scalable with respect to adding storage capacity and clients. Other embodiments are also described and claimed.
In today's information intensive environment, there are many businesses and other institutions that need to store huge amounts of digital data. These include entities such as large corporations that store internal company information to be shared by thousands of networked employees; online merchants that store information on millions of products; and libraries and educational institutions with extensive literature collections. A more recent need for the use of large-scale data storage systems is in the broadcast television programming market. Such businesses are undergoing a transition, from the older analog techniques for creating, editing and transmitting television programs, to an all-digital approach. Not only is the content (such as a commercial) itself stored in the form of a digital video file, but editing and sequencing of programs and commercials, in preparation for transmission, are also digitally processed using powerful computer systems. Other types of digital content that can be stored in a data storage system include seismic data for earthquake prediction, and satellite imaging data for mapping.
A powerful data storage system referred to as a media server is offered by Omneon Video Networks of Sunnyvale, Calif. (the assignee of this patent application). The media server is composed of a number of software components that are running on a network of server machines. The server machines have mass storage devices such as rotating magnetic disk drives that store the data. The server accepts requests to create, write or read a file, and manages the process of transferring data into one or more disk drives, or delivering requested read data from them. The server keeps track of which file is stored in which drives. Requests to access a file, i.e. create, write, or read, are typically received from what is referred to as a client application program that may be running on a client machine connected to the server network. For example, the application program may be a video editing application running on a workstation of a television studio, that needs a particular video clip (stored as a digital video file in the system).
Video data is voluminous, even with compression in the form of, for example, Motion Picture Experts Group (MPEG) formats. Accordingly, data storage systems for such environments are designed to provide a storage capacity of tens of terabytes or greater. Also, high-speed data communication links are used to connect the server machines of the network, and in some cases to connect with certain client machines as well, to provide a shared total bandwidth of one hundred Gb/second and greater, for accessing the system. The storage system is also able to service accesses by multiple clients simultaneously.
To help reduce the overall cost of the storage system, a distributed architecture is used. Hundreds of smaller, relatively low cost, high volume manufactured disk drives (currently each unit has a capacity of one hundred or more Gbytes) may be networked together, to reach the much larger total storage capacity. However, this distribution of storage capacity also increases the chances of a failure occurring in the system that will prevent a successful access. Such failures can happen in a variety of different places, including not just in the system hardware (e.g., a cable, a connector, a fan, a power supply, or a disk drive unit), but also in software such as a bug in a particular client application program. Storage systems have implemented redundancy in the form of a redundant array of inexpensive disks (RAID), so as to service a given access (e.g., make the requested data available), despite a disk failure that would have otherwise thwarted that access. The systems also allow for rebuilding the content of a failed disk drive, into a replacement drive.
A storage system should also be scalable, to easily expand to handle larger data storage requirements as well as an increasing client load, without having to make complicated and extensive hardware and software replacements.
The embodiments of the invention are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” embodiment of the invention in this disclosure are not necessarily to the same embodiment, and they mean at least one.
An embodiment of the invention is a data storage system that may better achieve demanding requirements of capacity, performance and data availability, with a more scalable architecture.
The system 102 can be accessed using client machines or a client network that can take a variety of different forms. For example, content files (in this example, various types of digital media files including MPEG and high definition (HD)) can be requested to be stored, by a media server 104. As shown in
The OCL system provides a relatively high performance, high availability storage subsystem with an architecture that may prove to be particularly easy to scale as the number of simultaneous client accesses increase or as the total storage capacity requirement increases. The addition of media servers 104 (as in
An embodiment of the invention is an OCL system that is designed for non-stop operation, as well as allowing the expansion of storage, clients and networking bandwidth between its components, without having to shutdown or impact the accesses that are in process. The OCL system preferably has sufficient redundancy such that there is no single point of failure. Data stored in the OCL system has multiple replications, thus allowing for a loss of mass storage units (e.g., disk drive units) or even an entire server, without compromising the data. In contrast to a typical RAID system, a replaced drive unit of the OCL system need not contain the same data as the prior (failed) drive. That is because by the time a drive replacement actually occurs, the pertinent data (file slices stored in the failed drive) had already been saved elsewhere, through a process of file replication that had started at the time of file creation. Files are replicated in the system, across different drives, to protect against hardware failures. This means that the failure of any one drive at a point in time will not preclude a stored file from being reconstituted by the system, because any missing slice of the file can still be found in other drives. The replication also helps improve read performance, by making a file accessible from more servers.
In addition to mass storage unit failures, the OCL system may provide protection against failure of any larger, component part or even a complete component (e.g., a metadata server, a content server, and a networking switch). In larger systems, such as those that have three or more groups of servers arranged in respective endosures or racks as described below, there is enough redundancy such that the OCL system should continue to operate even in the event of the failure of a complete enclosure or rack.
Referring now to
The file system driver or FSD is software that is installed on a client machine, to present a standard file system interface, for accessing the OCL system. On the other hand, the software development kit or SDK allows a software developer to access the OCL directly from an application program. This option also allows OCL-specific functions, such as the replication factor setting to be described below, to be available to the user of the client machine.
In the OCL system, files are typically divided into slices when stored across multiple content servers. Each content server program runs on a different machine having its own set of one or more local disk drives. This is the preferred embodiment of a storage element for the system. Thus, the parts of a file are spread across different disk drives, i.e. in different storage elements. In a current embodiment, the slices are preferably of a fixed size and are much larger than a traditional disk block, thereby permitting better performance for large data files (e.g., currently 8 Mbytes, suitable for large video and audio media files). Also, files are replicated in the system, across different drives, to protect against hardware failures. This means that the failure of any one drive at a point in time will not predude a stored file from being reconstituted by the system, because any missing slice of the file can still be found in other drives. The replication also helps improve read performance, by making a file accessible from more servers. To keep track of what file is stored where (or where are the slices of a file stored), each metadata server program has knowledge of metadata (information about files) which includes the mapping between the file name of a newly created or previously stored file, and its slices, as well as the identify of those storage elements of the system that actually contain the slices.
The metadata server determines which of the content servers are available to receive the actual content or data for storage. The metadata server also performs load balancing, that is determining which of the content servers should be used to store a new piece of data and which ones should not, due to either a bandwidth limitation or a particular content server filling up. To assist with data availability and data protection, the file system metadata may be replicated multiple times. For example, at least two copies may be stored on each metadata server machine (and, for example, one on each hard disk drive unit). Several checkpoints of the metadata should be taken at regular time intervals. A checkpoint is a point in time snapshot of the file system or data fabric that is running in the system, and is used in the event of system recovery. It is expected that on most embodiments of the OCL system, only a few minutes of time may be needed for a checkpoint to occur, such that there should be minimal impact on overall system operation.
In normal operation, all file accesses initiate or terminate through a metadata server. The metadata server responds, for example, to a file open request, by returning a list of content servers that are available for the read or write operations. From that point forward, client communication for that file (e.g., read; write) is directed to the content servers, and not the metadata servers. The OCL SDK and FSD, of course, shield the client from the details of these operations. As mentioned above, the metadata servers control the placement of files and slices, providing a balanced utilization of the content servers.
Although not shown in
The connections between the different components of the OCL system, that is the content servers and the metadata servers, should provide the necessary redundancy in the case of a system interconnect failure. See
One or more networking switches (e.g., Ethernet switches, Infiniband switches) are preferably used as part of the system interconnect. Such a device automatically divides a network into multiple segments, acts as a high-speed, selective bridge between the segments, and supports simultaneous connections of multiple pairs of computers which may not compete with other pairs of computers for network bandwidth. It accomplishes this by maintaining a table of each destination address and its port. When the switch receives a packet, it reads the destination address from the header information in the packet, establishes a temporary connection between the source and destination ports, sends the packet on its way, and may then terminate the connection.
A switch can be viewed as making multiple temporary crossover cable connections between pairs of computers. High-speed electronics in the switch automatically connect the end of one cable (source port) from a sending computer to the end of another cable (destination port) going to the receiving computer, for example on a per packet basis. Multiple connections like this can occur simultaneously.
In the example topology of
In this example, there are three metadata servers each being connected to the 1 Gb Ethernet switches over separate interfaces. In other words, each 1 Gb Ethernet switch has at least one connection to each of the three metadata servers. In addition, the networking arrangement is such that there are two private networks referred to as private ring 1 and private ring 2, where each private network has the three metadata servers as its nodes. The metadata servers are connected to each other with a ring network topology, with the two ring networks providing redundancy. The metadata servers and content servers are preferably connected in a mesh network topology as described here. An example physical implementation of the embodiment of
Turning now to
Each content server machine or storage element may have one or more local mass storage units, e.g. rotating magnetic disk drive units, and its associated content server program manages the mapping of a particular slice onto its one or more drives. The file system or data fabric implements file redundancy by replication. In the preferred embodiment, replication operations are controlled at the slice level. The content servers communicate with one another to achieve slice replication and obtaining validation of slice writes from each other, without involving the client.
In addition, since the file system or data fabric is distributed amongst multiple machines, the file system uses the processing power of each machine (be it a content server, a client, or a metadata server machine) on which it resides. As described below in connection with the embodiment of
Still referring to
According to an embodiment of the invention, the amount of replication (also referred to as “replication factor”) is associated individually with each file. All of the slices in a file preferably share the same replication factor. This replication factor can be varied dynamically by the user. For example, the OCL system's application programming interface (API) function for opening a file may include an argument that specifies the replication factor. This fine grain control of redundancy and performance versus cost of storage allows the user to make decisions separately for each file, and to change those decisions over time, reflecting the changing value of the data stored in a file. For example, when the OCL system is being used to create a sequence of commercials and live program segments to be broadcast, the very first commercial following a halftime break of a sports match can be a particularly expensive commercial. Accordingly, the user may wish to increase the replication factor for such a commercial file temporarily, until after the commercial has been played out, and then reduce the replication factor back down to a suitable level once the commercial has aired.
According to another embodiment of the invention, the content servers in the OCL system are arranged in groups. The groups are used to make decisions on the locations of slice replicas. For example, all of the content servers that are physically in the same equipment rack or enclosure may be placed in a single group. The user can thus indicate to the system the physical relationship between content servers, depending on the wiring of the server machines within the enclosures. Slice replicas are then spread out so that no two replicas are in the same group of content servers. This allows the OCL system to be resistant against hardware failures that may encompass an entire rack.
Replication
Replication of slices is preferably handled internally between content servers. Clients are thus not required to expend extra bandwidth writing the multiple copies of their files. In accordance with an embodiment of the invention, the OCL system provides an acknowledgment scheme where a client can request acknowledgement of a number of replica writes that is less than the actual replication factor for the file being written. For example, the replication factor may be several hundred, such that waiting for an acknowledgment on hundreds of replications would present a significant delay to the client's processing. This allows the client to tradeoff speed of writing versus certainty of knowledge of the protection level of the file data. Clients that are speed sensitive can request acknowledgement after only a small number of replicas have been created. In contrast, clients that are writing sensitive or high value data can request that the acknowledgement be provided by the content servers only after all specified number of replicas have been created.
Intelligent Slices
According to an embodiment of the invention, files are divided into slices when stored in the OCL system. In a preferred case, a slice can be deemed to be an intelligent object, as opposed to a conventional disk block or stripe that is used in a typical RAID or storage area network (SAN) system. The intelligence derives from at least two features. First, each slice may contain information about the file for which it holds data. This makes the slice self-locating. Second, each slice may carry checksum information, making it self-validating. When conventional file systems lose metadata that indicates the locations of file data (due to a hardware or other failure), the file data can only be retrieved through a laborious manual process of trying to piece together file fragments. In accordance with an embodiment of the invention, the OCL system can use the file information that are stored in the slices themselves, to automatically piece together the files. This provides extra protection over and above the replication mechanism in the OCL system. Unlike conventional blocks or stripes, slices cannot be lost due to corruption in the centralized data structures.
In addition to the file content information, a slice also carries checksum information that may be created at the moment of slice creation. This checksum information is said to reside with the slice, and is carried throughout the system with the slice, as the slice is replicated. The checksum information provides validation that the data in the slice has not been corrupted due to random hardware errors that typically exist in all complex electronic systems. The content servers preferably read and perform checksum calculations continuously, on all slices that are stored within them. This is also referred to as actively checking for data corruption. This is a type of background checking activity which provides advance warning before the slice data is requested by a client, thus reducing the likelihood that an error will occur during a file read, and reducing the amount of time during which a replica of the slice may otherwise remain corrupted.
Scalable Network Topology
In accordance with another embodiment of the invention, a multi-node computer system, such as the OCL data storage system, has a physical network topology as depicted in
Each of the internal packet switches 510 is communicatively coupled to an external packet switch 512_1. In this example, the external switch 512 has sixteen ports that are in use by eight server groups (two by each group). The external switch 512 has additional ports (not shown) that are communicatively coupled to client machines (not shown), to give client access to the storage system. Note that in this topology, client access to a data storage server is through that storage server's associated internal packet switch 510 and the external switch 512. The data storage servers of each server group 508 communicate, at a physical layer, with their respective internal packet switch 510, and not the external switch 512.
For redundancy, an additional external switch 512_2 may be added to the system as shown. In that case, there is a further redundant link 513 that connects each internal switch 510, to the external switch 512_2, e.g. through a further pair of ports that are connected via a pair of cables to a respective pair of ports in the external switch 512_2. The provision of the second external switch 512_2, in addition to providing redundant client access to the data storage servers (where once again for clarity,
It should be noted that each of the internal switches 510 and external switches 512 is preferably in a separate enclosure containing its own power supply, processor, memory, ports and packet forwarding table. As an alternative, each internal switch 510 may be a separate pair of switch enclosures that are communicatively coupled to each other by a pair of multi Gb Ethernet cables. Thus, each internal switch 510 may be composed of one or more separate switch enclosures. Each switch enclosure may be a 1U height stackable switch unit that can be mounted in a standard telecommunications rack or equipment cabinet.
To illustrate the scalability of the network topology shown in
Thus, it can be seen that the network topology, in accordance with an embodiment of the invention shown in
To illustrate the scalability of the network topology of
In operation 1316, a number of upgrade server groups and a number of upgrade internal packet switches are provided. See for example
Turning now to
Each adapter switch 708 is connected by a separate pair of high bandwidth (e.g., 10 Gb Ethernet) links to both of the internal switches 510_1_A, 510_1_B. Each of these switches has one or more high bandwidth ports that are connected to an external switch (e.g., external switch 512 or 612).
In the embodiment of
Turning now to
To illustrate the scalability of the topology depicted in
Next, assume that the system is upgraded with four additional dusters 904_5, . . . 904_8. Each additional duster has eight ports that are to be communicatively coupled to the existing external packet switches of the system, as shown in
Turning now to
Next, assume that the system is to be upgraded by about fifty percent, that is a single upgrade rack is to be added (see
Turning now to
The above discussion regarding the physical connections of the different network topologies assumes that the software running in the data storage server machines is aware of how to access, e.g. via respective IP addresses, other data storage server machines in the system through the packet switching interconnect. Well known algorithms may be used to make each node of the system aware of the addresses of other nodes in the system. In addition, routing and/or forwarding tables within the internal and external switches can be populated using known routing algorithms, to avoid problem routes and instead select the most efficient path to deliver a packet from its source to its indicated destination address.
An embodiment of the invention may be a machine readable medium having stored thereon instructions which program one or more processors to perform some of the operations described above. In other embodiments, some of these operations might be performed by specific hardware components that contain hardwired logic. Those operations might alternatively be performed by any combination of programmed computer components and custom hardware components.
A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), not limited to Compact Disc Read-Only Memory (CD-ROMs), Read-Only Memory (ROMs), Random Access Memory (RAM), Erasable Programmable Read-Only Memory (EPROM), and a transmission over the Internet.
The invention is not limited to the specific embodiments described above. For example, although the OCL system was described with a current version that uses only rotating magnetic disk drives as the mass storage units, alternatives to magnetic disk drives are possible, so long as they can meet the needed speed, storage capacity, and cost requirements of the system. Accordingly, other embodiments are within the scope of the claims.
