Patent Application
20060198386
1. Field of the Invention
The present invention relates in general to the field of information handling system clusters, and more particularly to a system and method for distributed information handling system cluster active-active master node.
2. Description of the Related Art
As the value and use of information continues to increase, individuals and businesses seek additional ways to process and store information. One option available to users is information handling systems. An information handling system generally processes, compiles, stores, and/or communicates information or data for business, personal, or other purposes thereby allowing users to take advantage of the value of the information. Because technology and information handling needs and requirements vary between different users or applications, information handling systems may also vary regarding what information is handled, how the information is handled, how much information is processed, stored, or communicated, and how quickly and efficiently the information may be processed, stored, or communicated. The variations in information handling systems allow for information handling systems to be general or configured for a specific user or specific use such as financial transaction processing, airline reservations, enterprise data storage, or global communications. In addition, information handling systems may include a variety of hardware and software components that may be configured to process, store, and communicate information and may include one or more computer systems, data storage systems, and networking systems.
Information handling systems typically are used as discrete units that operate independently to process and store information. Increasingly, information handling systems are interfaced with each other through networks, such as local area networks that have plural client information handling systems supported with one or more central server information handling systems. For instance, businesses often interconnect employee information handling systems through a local area network in order to centralize the storage of documents and communication by e-mail. As another example, a web site having substantial traffic sometimes coordinates requests to the web site through plural information handling systems that cooperate to respond to the requests. As requests arrive at a managing server, the requests are allocated to supporting servers that handle the requests, typically in an ordered-queue. More recently, information handling systems have been interfaced as High Performance Computing Clusters (HPCC) in which plural information handling systems perform complex operations by combining their processing power under the management of a single master node information handling system. The master node assigns tasks to information handling systems of its cluster, such as distributing jobs, handling all file input/output, and managing computing nodes, so that multiple information handling systems execute an application much like a supercomputer, such as a weather prediction application.
One difficulty that arises with coordination of plural information handling systems is that failure of a managing information handling system often results in failure of managed information handling systems due to an inability to access the managed information handling systems. A so-called single point of failure (SPOF) is especially undesirable when high-availability is critical. A related difficulty sometimes results from overloading of a managing information handling system when a large number of transactions are simultaneously initiated or otherwise coordinated through the managing information handling system. To avoid or at least reduce the impact of a failure of a managing node, various architectures use varying degrees of redundancy. Various Linux projects, such as Linux-HA and Linux Virtual Server, provide a failover policy in a Linux cluster so that assignment of tasks continues on a node-by-node basis in the event of a managing node failure, however these projects will not work with an HPCC architecture in which tasks are allocated to multiple information handling systems. Load Sharing Facility from Platform Computing Inc. and High Availability Open Source Cluster Application Resources (HA-OSCAR) are a job management applications that run on a HPCC master node to provide an active-standby master node architecture in which a standby master node recovers operations in the event of a failed master node. However, the active-standby HPCC architecture disrupts management of computing nodes during the transition from a standby to an active state and typically loses tasks in progress.
Therefore a need has arisen for a system and method which provides an active-active HPCC master node architecture.
In accordance with the present invention, a system and method are provided which substantially reduce the disadvantages and problems associated with previous methods and systems for managing information handling system clusters. A distributed active-active master node architecture supports simultaneous management of computing node resources by plural master nodes for improved management and reliability of clustered computing nodes.
More specifically, plural master nodes of a High Performance Computing Cluster (HPCC) interface with each other and common storage to manage assignment and performance of computing job requests. A resource manager associated with each master node determines computing resources of computing nodes that are desired to perform a job request. A job scheduler reserves the desired computing resources in storage common to the plural master nodes and confirms that a conflict does not exist for the resources in a reservation or assignment by another master node. Once the availability of desired resources is confirmed, the resource manager assigns and manages the resources to perform the job request. During operation of a job request by a master node, failure managers associated with the other master nodes monitor the operation of the master node to detect a failure. Upon detection of a failed master node, the jobs under management by that master node are assigned to an operating master node by reference to the common storage.
The present invention provides a number of important technical advantages. One example of an important technical advantage is that plural master nodes of a HPCC information handling system simultaneously manage computing resources of common computing nodes. The availability of plural master nodes reduces the risk of a slow down of computing jobs caused by a bottleneck at a master node. Plural master nodes also reduces the risk of a failure of the information handling system by avoiding the single point of failure of a single master node. The impact from a failed master is reduced since the use of common storage by the master nodes allows an operating master node to recover jobs associated with the failed master node without the loss of information associated with the computing job.
The present invention may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference number throughout the several figures designates a like or similar element.
A High Performance Computing Cluster (HPCC) information handling system has the computing resources of plural computing nodes managed with plural active master nodes. For purposes of this disclosure, an information handling system may include any instrumentality or aggregate of instrumentalities operable to compute, classify, process, transmit, receive, retrieve, originate, switch, store, display, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data for business, scientific, control, or other purposes. For example, an information handling system may be a personal computer, a network storage device, or any other suitable device and may vary in size, shape, performance, functionality, and price. The information handling system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and/or other types of nonvolatile memory. Additional components of the information handling system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, a mouse, and a video display. The information handling system may also include one or more buses operable to transmit communications between the various hardware components.
Referring now to
Resource managers 18 assign and manage jobs with computing nodes 14 applied as a HPCC configuration, however, allocation of computing resources between jobs is further managed by a reservation system enforced on resource managers 12 by a job scheduler 20. Job scheduler 20 uses a token system to reserve desired computing resources so that different resource managers 18 do not attempt to simultaneously use the same computing resources. For instance, when a job request is received from user interface 20, resource manager determines the desired computing resources and requests an assignment of the resources from job scheduler 20. Job scheduler 20 saves tokens for the desired resources in a token table 24 of a storage 22 based on the currently assigned computing resources of a job table 26. Job scheduler 20 waits a predetermined time and then confirms that another job scheduler 20 has not reserved tokens for the desired computing resources or otherwise assigned the desired computing resources to a job in job table 26. Once job scheduler 20 confirms that the reserved computing resources remain available, resource manager 12 is allowed to assign the computing resources as a HPCC configuration. In order to avoid conflicting use of computing resources of computing nodes 14, storage 22 is common to all master nodes 12 and all storage related caches are disabled to avoid potential cache coherence difficulties.
The availability of plural master nodes 12 improves HPCC performance by avoiding bottlenecks at the management of computing nodes 14. In addition, the availability of plural master nodes reduces the risk of failure of a job by allowing an operating master node 12 to recover jobs managed by a failed master node 12. A failure manager running on each master node 12 monitors communication from the other master nodes to detect a failure. For instance, failure manager 28 monitors communications across switch 16 to detect messages having the network address of other master nodes 12 and determines that a master node 12 has failed if no communications are detected with the address of the master node for a predetermined time period. For instance, failure manager 28 attempts to detect and recover a failed master node 12 within three to eight seconds of a failure, with eight seconds exceeding the Remote Procedure Call (RPC) timeout used for NFS access so that no file access will be lost. Upon detection of a failed master node 12, failure manager 28 recovers the failure by assuming jobs in job table 26 that are associated with the failed master node. The use of redundant storage 22 that is common to all master nodes ensures that consistency of data is maintained during recovery of jobs associated with a failed master node.
Referring now to
Although the present invention has been described in detail, it should be understood that various changes, substitutions and alterations can be made hereto without departing from the spirit and scope of the invention as defined by the appended claims.
