Patent Application
20060100981
1. Technical Field
This invention generally relates to data processing, and more specifically relates to networked computer systems.
2. Background Art
Since the dawn of the computer age, computer systems have become indispensable in many fields of human endeavor including engineering design, machine and process control, and information storage and access. In the early days of computers, companies such as banks, industry, and the government would purchase a single computer which satisfied their needs, but by the early 1950's many companies had multiple computers and the need to move data from one computer to another became apparent. At this time computer networks began being developed to allow computers to work together.
Networked computers are capable of performing tasks that no single computer could perform. In addition, networks allow low cost personal computer systems to connect to larger systems to perform tasks that such low cost systems could not perform alone. Most companies in the United States today have one or more computer networks. The topology and size of the networks may vary according to the computer systems being networked and the design of the system administrator. It is very common, in fact, for companies to have multiple computer networks. Many large companies have a sophisticated blend of local area networks (LANs) and wide area networks (WANs) that effectively connect most computers in the company to each other.
With multiple computers hooked together on a network, it soon became apparent that networked computers could be used to complete tasks by delegating different portions of the task to different computers on the network, which can then process their respective portions in parallel. In one specific configuration for shared computing on a network, the concept of a computer “cluster” has been used to define groups of computer systems on the network that can work in parallel on different portions of a task.
Clusters of computer systems have also been used to provide high-reliability services. The high reliability is provided by allowing services on a server that fails to be moved to a server that is still alive. This type of fault-tolerance is very desirable for many companies, such as those that do a significant amount of e-commerce. In order to provide high-reliability services, there must be some mechanism in place to detect when one of the servers in the cluster becomes inoperative. One known way to determine whether all the servers in a cluster are operative is to have each server periodically issue a message to the other servers indicating that the server that sent the message is still alive and well. These types of messages are commonly referred to in the art as “heartbeats” because as long as the messages continue (i.e., as long as the heart is still beating), we know the server is still alive.
In the prior art, when a server becomes invisible due to lack of a heartbeat, a server in the cluster that is designated as a manager assumes the server that no longer has a heartbeat has failed. As a result, the manager must provide the resources that were on the failed server on another server in the cluster. Note, however, that the absence of a heartbeat does not always mean a server is dead. For example, a server may not provide a heartbeat because it may be temporarily unresponsive due to trashing, swapping, network floods, etc. If the server is not giving heartbeats but is still alive, there exists the possibility that the server may once again become responsive and start providing heartbeats. If the manager has already assumed the server has failed, and has provided the server's services on another server, we now have two servers that try to provide the same services. This creates a problem in administrating the cluster. One way to deal with this problem is to monitor data for a service to make sure that two servers don't try to access the same data for the same service. However, this is complex and inefficient. Without a mechanism for assuring that services in a computer cluster are not duplicated when a server failure is detected, the computer industry will continue to suffer from inadequate and inefficient ways of handling a failed server in a computer cluster.
An apparatus and method provide a quorum-based server power-down mechanism that allows a manager in a computer cluster to power-down unresponsive servers in a manner that assures that an unresponsive server does not become responsive again. In order for a manager in a cluster to power down servers in the cluster, the cluster must have quorum, meaning that a majority of the computers in the cluster must be responsive. If the cluster has quorum, and if the manager server did not fail, the manager causes the failed server(s) to be powered down. If the manager server did fail, the new manager causes all unresponsive servers in the cluster to be powered down. If the power-down is successful, the resources on the failed server(s) may be failed over to other servers in the cluster that were not powered down. If the power-down is not successful, the cluster is disabled.
The foregoing and other features and advantages of the invention will be apparent from the following more particular description of preferred embodiments of the invention, as illustrated in the accompanying drawings.
The preferred embodiments of the present invention will hereinafter be described in conjunction with the appended drawings, where like designations denote like elements, and:
According to preferred embodiments of the present invention, a quorum-based server power-down mechanism in a computer cluster assures that an unresponsive server in the cluster is powered-down before the resources are failed over to one or more other responsive servers. The power-down mechanism is quorum-based, meaning that only a cluster that includes a majority of the servers in the cluster may perform power-down operations. By powering down failed servers, the preferred embodiments assure that a failed system does not become responsive again.
The prior art provides a way for a server in a cluster to determine when it has become unresponsive, and to know it needs to shut down. Method 400 in
The check for a majority partition is necessary because different partitions will realize that we have partitioned in an asynchronous manner. If no partition had a majority then each partition will panic any servers with resources active. While method 400 in
Referring now to
Service processor interface 180 preferably connects the computer system 100 to a separate service processor 182. Service processor 182 preferably includes a server power-down mechanism 184 that allows servers coupled to the service processor to be individually powered-down. Service processor 182 typically provides an interface that allows a computer system (such as 100) to command the service processor to power down another computer system in the cluster. In addition, service processor 180 can terminate a single process on another machine when servers in the cluster are processes rather than physical boxes or logical partitions.
Main memory 120 in accordance with the preferred embodiments contains data 121, an operating system 122, and a cluster engine 123. Data 121 represents any data that serves as input to or output from any program in computer system 100. Operating system 122 is a multitasking operating system known in the industry as OS/400; however, those skilled in the art will appreciate that the spirit and scope of the present invention is not limited to any one operating system. Cluster engine 123 provides for communication between computer systems in a cluster. Cluster engine 123 includes many features and mechanisms that are known in the art that support cluster communications but are not shown in
In computer system 100 of
Computer system 100 utilizes well known virtual addressing mechanisms that allow the programs of computer system 100 to behave as if they only have access to a large, single storage entity instead of access to multiple, smaller storage entities such as main memory 120 and DASD device 155. Therefore, while data 121, operating system 122, and cluster engine 123 are shown to reside in main memory 120, those skilled in the art will recognize that these items are not necessarily all completely contained in main memory 120 at the same time. It should also be noted that the term “memory” is used herein to generically refer to the entire virtual memory of computer system 100, and may include the virtual memory of other computer systems coupled to computer system 100.
Processor 110 may be constructed from one or more microprocessors and/or integrated circuits. Processor 110 executes program instructions stored in main memory 120. Main memory 120 stores programs and data that processor 110 may access. When computer system 100 starts up, processor 110 initially executes the program instructions that make up operating system 122. Operating system 122 is a sophisticated program that manages the resources of computer system 100. Some of these resources are processor 110, main memory 120, mass storage interface 130, display interface 140, network interface 150, system bus 160, and service processor interface 180.
Although computer system 100 is shown to contain only a single system bus, those skilled in the art will appreciate that the present invention may be practiced using a computer system that has multiple buses. In addition, the interfaces that are used in the preferred embodiment each include separate, fully programmed microprocessors that are used to off-load compute-intensive processing from processor 110. However, those skilled in the art will appreciate that the present invention applies equally to computer systems that simply use I/O adapters to perform similar functions.
Display interface 140 is used to directly connect one or more displays 165 to computer system 100. These displays 165, which may be non-intelligent (i.e., dumb) terminals or fully programmable workstations, are used to allow system administrators and users to communicate with computer system 100. Note, however, that while display interface 140 is provided to support communication with one or more displays 165, computer system 100 does not necessarily require a display 165, because all needed interaction with users and other processes may occur via network interface 150.
Network interface 150 is used to connect other computer systems and/or workstations (e.g., 175 in
At this point, it is important to note that while the present invention has been and will continue to be described in the context of a fully functional computer system, those skilled in the art will appreciate that the present invention is capable of being distributed as a program product in a variety of forms, and that the present invention applies equally regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of suitable signal bearing media include: recordable type media such as floppy disks and CD RW (e.g., 195 of
Referring to
Referring to
If the power-down operation succeeded (step 560=YES)), the resources on the failed server(s) may be failed over to servers in the cluster that are still responsive (step 570). The concept of failing over resources from a dead server to a live server in the cluster is well-known in the art, and therefore need not be discussed in further detail here. The failing over of resources is the process of making these same resources available on a different server in the cluster. This is the very nature of one specific way to provide highly-reliable services, using multiple servers that can take over for each other when one of the servers fails. If the power-down operation did not succeed (step 560=NO), the cluster is disabled (step 580). The preferred embodiments depend on the service processor doing its job of powering down a selected server when the quorum-based server power-down mechanism sends the command to power down the selected server. If the service processor is unable to perform its power-down function, this means there is a problem with the service processor itself or something else that requires intervention by a system administrator. Thus, once a cluster is disabled in step 580, a system administrator is preferably notified of the problem so the system administrator can take appropriate action to correct the problem.
With an understanding of method 500 in
One skilled in the art will appreciate that many variations are possible within the scope of the present invention. Thus, while the invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that these and other changes in form and details may be made therein without departing from the spirit and scope of the invention. For example, while a known service processor is shown as one possible mechanism for powering down servers, other mechanisms could also be used within the scope of the preferred embodiments. For example, addressable power strips could be used that are capable of receiving commands, and shutting off power to a particular plug in the power strip or to the entire power strip. Any mechanism for putting a server in an unresponsive state until some step of intervention is taken falls within the scope of the term “service processor” as used herein. In addition, the servers recited herein may reside within logical partitions, which means that the power down of a server in a logical partition implies simply shutting down the logical partition.
