1. Field of the Invention
The present invention is related to handling of event notifications in large-scale distributed computer systems, and more specifically to handling of registration for remote event notifications in processing mode clusters.
2. Description of Related Art
In large-scale distributed computer systems, such as those using distributed software models to perform tasks, multiple nodes provide independent execution of sub-tasks. The nodes are typically organized into clusters in order to perform specific tasks and/or as a physical organization of a processing system, for example, clustering by processor affinity. In order to keep such a system operational, and further, to provide indication of events occurring at one node that either require a reaction from another node or indicate to the other node that either an erroneous operating condition has occurred, or that a phase of processing is complete. In particular, event notification and event logging are operations used to indicate system health to system administrators or software applications, including operating systems components.
Health monitoring techniques employed in distributed processing systems perform an important function in that connections to other nodes must be reliable and all of the active nodes that have been assigned tasks need to perform those tasks in order to ensure that the totality of the processing requirements are met, and in a timely fashion. The health of a node-based distributed processing system is typically monitored by: 1) a heartbeat messaging system, which passes messages between the nodes and a central monitoring component; and 2) an event notification system that signals interested nodes when events occur on other nodes. Event notification systems in node-based distributed processing systems typically require an interested application (a consumer) to register to receive event notifications either with a centralized event manager, or with the processes or objects that generate the events (an event producer).
The invention provides event notification methods that provide local and remote event notification to event consumers in systems that require a local consumer for event notification. The methods always provide a local consumer at a node for which a remote notification is needed, where the remote notification is directed to a node having an interested consumer.
In one embodiment of the invention, event consumers register locally to receive event notifications for a particular event type and specify whether or not they are to receive notifications when the event type occurs on a remote node using an indication that the event of interest is to be handled as a cluster event. A cluster event is an event for which the event consumer should be notified if the event occurs on any node of the cluster. If the event of interest is a cluster event, then event consumers are started on all nodes and also register the event as a cluster event. When the event manager receives a request to register for a cluster event, when the event occurs, the event notification is sent to all other nodes. Otherwise, if the event is not a cluster event, the event notification is only provided to the local consumer(s) and is not propagated among the nodes.
In another embodiment of the invention, event consumers register to receive event notifications for a particular event type and specify from which nodes they are to receive notifications. If a node registers to receive an event from a remote node and there is no local consumer for the event on the remote node, a listener thread registers as a local consumer on the remote node. Otherwise the listener thread monitors communication of the event from the event producer to the local consumer on the remote node. When the event occurs, the listener thread on the node that generated the event sends an event notification to only the remote nodes that have registered to receive the event.
The event notification methods can be implemented in a mountable file system in which file (or device) open, read and write requests are received from event consumers and interpreted as event registrations by the file system implementation, providing a facile and portable mechanism for providing event notifications between nodes in a distributed processing system. The files within this file system represent events that event consumers may monitor.
The foregoing and other objectives, features, and advantages of the invention will be apparent from the following, more particular, description of the preferred embodiments of the invention, as illustrated in the accompanying drawings.
The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives, and advantages thereof, will best be understood by reference to the following detailed description of the invention when read in conjunction with the accompanying Figures, wherein like reference numerals indicate like components, and:
The present invention encompasses techniques for event notification in clusters of nodes within distributed computing systems. In particular, embodiments of the invention provide remote event notification techniques that may be used within systems that require local event consumers in order for an event producer to generate event notifications and/or for the event notification framework to recognize the event notifications. According to one embodiment of the invention, a local event consumer is started on every node when an event type is designated as a cluster event. The starting of the local event consumers is the responsibility of the user, who starts the monitoring of the event on every node according to the particular service or application that implements the monitoring. If the event type is a cluster event, each node sends event notifications to all of the other nodes when an event of that type occurs on the node. According to another embodiment of the invention, nodes are informed of a subset of interested nodes that should receive event notifications for an event type, and if no local consumer for each particular event type is present on a given node, a listener thread registers as a local event consumer.
Referring now to
Referring now to
While event notification in accordance with embodiments of the present invention may be implemented using a variety of event notification mechanisms, such as direct object-to-object interfaces, using operating system-maintained signaling object such as semaphores, in the depicted embodiment, the event notification is provided by a special file system that implements an event manager. Event consumers, i.e., the applications or objects interested in receiving event notifications, register to receive event notifications by calling file operation functions/methods on an interface to the file system. Event producers notify the file system of events via a separate application programming interface (API) provided by the event manager. The event manager then provides the event notifications to the event consumers interested in the particular events represented by the event notifications. In the depicted embodiment, the file system is a kernel extension, which facilitates availability and uniformity of the event manager at each node in the system. Further details of a file-system based event manager are provided in U.S. Patent Application Publication U.S. 200901991051, which is incorporated herein by reference. The event notification system the “Autonomic Health Advisor File System” (AHAFS) disclosed in the above-incorporated U.S. patent application does not provide direct event notification between the nodes, so a separate layer is used for inter-node event communication. The generic methods open( ) and write( ) described below are understood to include specific file operations fopen( ) and fwrite( ) as alternatives, or other operations provided in a file system interface that can be used to provide the same type of information flow.
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
As noted above, the present invention may be embodied as a system, method, and/or a computer program product. A computer program product may be embodied in firmware, an image in system memory or another memory/cache, stored on a fixed or re-writable media such as an optical disc having computer-readable code stored thereon. Any combination of one or more computer readable medium(s) may be used to store the program instructions in accordance with an embodiment of the invention. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In the context of the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. Further, while the illustrative embodiment is directed to an AIX (AIX is a trademark of IBM) or other type of UNIX operating system (UNIX is a trademark of The Open Group), in which the event notification is provided by a mountable file system provided as a kernel extension, it is understood that the techniques of the present invention can be applied in event monitoring systems executing under any operating system and can be implemented using proprietary or standardized signaling interfaces as an alternative to the file system interface provided in the depicted embodiments.
While the invention has been particularly shown and described with reference to the preferred embodiments thereof, it will be understood by those skilled in the art that the foregoing and other changes in form and details may be made therein without departing from the spirit and scope of the invention.
The Present Application is a Continuation of U.S. patent application Ser. No. 12/879,414, filed on Sep. 10, 2010 and thus claims priority thereto under 35 U.S.C. 120.
Number | Name | Date | Kind |
---|---|---|---|
6038563 | Bapat et al. | Mar 2000 | A |
6064656 | Angal et al. | May 2000 | A |
6185613 | Lawson et al. | Feb 2001 | B1 |
6230160 | Chan et al. | May 2001 | B1 |
6411967 | Van Renesse | Jun 2002 | B1 |
6591317 | Schzukin et al. | Jul 2003 | B1 |
6631363 | Brown et al. | Oct 2003 | B1 |
6754704 | Prorock | Jun 2004 | B1 |
6778504 | Santry et al. | Aug 2004 | B2 |
6862619 | Sugauchi et al. | Mar 2005 | B1 |
6983324 | Block et al. | Jan 2006 | B1 |
7051105 | Lauzon et al. | May 2006 | B2 |
7058957 | Nguyen | Jun 2006 | B1 |
7542437 | Redi et al. | Jun 2009 | B1 |
7571230 | Gissel et al. | Aug 2009 | B2 |
7650404 | Parupudi et al. | Jan 2010 | B2 |
7657450 | Amit et al. | Feb 2010 | B2 |
7664125 | Bauer et al. | Feb 2010 | B1 |
7747717 | Colrain | Jun 2010 | B2 |
7856480 | Muchow | Dec 2010 | B2 |
7913105 | Ganesh et al. | Mar 2011 | B1 |
8108715 | Agarwal | Jan 2012 | B1 |
8161053 | Khan et al. | Apr 2012 | B1 |
8384549 | Lemmon | Feb 2013 | B2 |
8433760 | Ganapathy et al. | Apr 2013 | B2 |
8484472 | Sherkin et al. | Jul 2013 | B2 |
20020083030 | Yang et al. | Jun 2002 | A1 |
20020178275 | Hein et al. | Nov 2002 | A1 |
20030061340 | Sun et al. | Mar 2003 | A1 |
20030088831 | Bauer et al. | May 2003 | A1 |
20030093516 | Parsons et al. | May 2003 | A1 |
20030105801 | Tse et al. | Jun 2003 | A1 |
20030225840 | Glassco et al. | Dec 2003 | A1 |
20030233594 | Earl | Dec 2003 | A1 |
20040008727 | See et al. | Jan 2004 | A1 |
20040030775 | Lauzon et al. | Feb 2004 | A1 |
20040064835 | Bellwood et al. | Apr 2004 | A1 |
20040088716 | Solter | May 2004 | A1 |
20040172467 | Wechter et al. | Sep 2004 | A1 |
20050065953 | Bower et al. | Mar 2005 | A1 |
20050076145 | Ben-Zvi et al. | Apr 2005 | A1 |
20050152396 | Pichna et al. | Jul 2005 | A1 |
20050234929 | Ionescu et al. | Oct 2005 | A1 |
20060031282 | Tuttle et al. | Feb 2006 | A1 |
20060050629 | Saika | Mar 2006 | A1 |
20060271676 | Talayco et al. | Nov 2006 | A1 |
20070041328 | Bell | Feb 2007 | A1 |
20070140243 | Eastham | Jun 2007 | A1 |
20070226182 | Sobotka et al. | Sep 2007 | A1 |
20070282959 | Stern | Dec 2007 | A1 |
20080008106 | Boberg et al. | Jan 2008 | A1 |
20080077635 | Sporny et al. | Mar 2008 | A1 |
20080183857 | Barfield et al. | Jul 2008 | A1 |
20080317050 | Xiong et al. | Dec 2008 | A1 |
20090043887 | Coekaerts | Feb 2009 | A1 |
20090138808 | Moromisato et al. | May 2009 | A1 |
20090199051 | Jann et al. | Aug 2009 | A1 |
20100077310 | Karachale et al. | Mar 2010 | A1 |
20100094922 | Sathish | Apr 2010 | A1 |
20100099447 | Boberg et al. | Apr 2010 | A1 |
20100113072 | Gibson et al. | May 2010 | A1 |
20100153528 | Pearson et al. | Jun 2010 | A1 |
20100223492 | Farrugia et al. | Sep 2010 | A1 |
20100274885 | Yoo et al. | Oct 2010 | A1 |
20100281304 | Moyer et al. | Nov 2010 | A1 |
20100290469 | Assarpour | Nov 2010 | A1 |
20100332277 | Dentzer et al. | Dec 2010 | A1 |
20110035462 | Akella | Feb 2011 | A1 |
20110041140 | Harm et al. | Feb 2011 | A1 |
20110093743 | Arcese et al. | Apr 2011 | A1 |
20110145639 | Farahmand et al. | Jun 2011 | A1 |
20110202500 | Warn et al. | Aug 2011 | A1 |
20110274053 | Baik et al. | Nov 2011 | A1 |
20120036250 | Vaswani et al. | Feb 2012 | A1 |
20120047257 | Hauser | Feb 2012 | A1 |
20120144018 | Fried et al. | Jun 2012 | A1 |
20120203897 | Mishra et al. | Aug 2012 | A1 |
20120203899 | Ganapathy et al. | Aug 2012 | A1 |
20130042001 | Gould et al. | Feb 2013 | A1 |
Entry |
---|
Office Action in U.S. Appl. No. 12/879,414 mailed on Apr. 30, 2013, 20 pages (pp. 1-20 in pdf). |
Number | Date | Country | |
---|---|---|---|
20120198478 A1 | Aug 2012 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 12879414 | Sep 2010 | US |
Child | 13427195 | US |