1. Technical Field
The invention relates generally to monitoring a problem condition, and more particularly, to a communications protocol implementation that performs self-health monitoring of one or more problem conditions.
2. Background Art
A systems network architecture (SNA) network provides high availability for mainframe systems, such as a zSeries eServer offered by International Business Machines Corp. of Armonk, N.Y. (IBM). Operating systems, such as IBM's z/OS exploit features of the SNA network to provide high performance for applications executing in a mainframe system. However, workloads processed by these mainframe systems are increasingly being driven by client requests flowing over an internet protocol (IP) network infrastructure. As a result, a lot of emphasis has been placed on ensuring that the z/OS IP network infrastructure delivers the same high availability attributes as those provided by the SNA network.
The use of a dynamic virtual IP address (DVIPA) is an important virtualization technology that assists in providing high availability z/OS solutions using IP networks in a cluster system (sysplex) environment. DVIPA provides an ability to separate the association of an IP address with a physical network adapter interface. To this extent, DVIPA can be viewed as a virtual destination that is not bound to a particular system/network interface, and therefore is not bound to any failure of any particular system/network interface. This results in a highly flexible configuration that provides the high availability on which many z/OS solutions depend.
DVIPA can be deployed using one of various configurations. Each configuration provides protection against a failure of a system, network interface and/or application. For example, in multiple application-instance DVIPA, a set of applications executing in the same z/OS image are represented by a DVIPA. This DVIPA allows clients to reach these applications over any network interface attached to the z/OS image and allows for automatic rerouting of traffic around a failure in a particular network interface. Additionally, should the primary system fail or enter a planned outage, the DVIPA can be automatically moved to another system in the sysplex. Further, a unique application-instance DVIPA can be associated with a particular application instance in the sysplex. In this case, the DVIPA can be dynamically moved to any system in the sysplex on which the application is executing. This DVIPA provides automatic recovery in scenarios where a particular application or system fails. In particular, a new instance of the application running on another system can trigger the DVIPA to be moved to the other system, allowing client requests to continue to be able to reach the application. Still further, a distributed DVIPA represents a cluster of one or more applications executing on various systems within a sysplex. In this case, new client transmission control protocol (TCP) connection requests can be load balanced across application instances active anywhere in the sysplex, thereby providing protection against the failure of any system, network interface and/or application in the sysplex, while also providing an ability to deploy a highly scalable solution within the sysplex.
As a result, DVIPA provides high availability TCP/IP communications to an application running in a sysplex environment even when a major component, such as a hardware system, an operating system, a TCP/IP protocol stack, a network adapter or an application, fails. In these situations, the failure is automatically detected and recovery action is automatically initiated, ensuring that client requests continue to be processed successfully. However, other problem conditions, apart from the failure of a major component, can prevent client requests from being processed successfully.
To this extent, a need exists for an improved communications protocol implementation that monitors one or more problem conditions.
The invention provides a solution for monitoring one or more problem conditions in a communications protocol implementation. The communications protocol implementation includes an internal monitor thread that monitors one or more resources for problem condition(s). The internal monitor thread sets a problem flag based on a problem condition being present. A control process in the communications protocol implementation that controls a resource includes a problem monitor that resets the problem flag when the problem condition is cleared. To this extent, the problem monitor provides a check against the internal monitor thread. In one embodiment, the internal monitor thread is periodically executed, and only sets the problem flag after the problem condition has been present for a problem time period. Further, the internal monitor thread can take action in response to the problem condition only after the problem flag has been set for at least two consecutive executions. Additionally, the internal monitor thread can monitor the health of one or more external communication processes that are utilized by the communications protocol implementation using, for example, a heartbeat signal. In this manner, the internal monitor thread and the problem monitor provide the communications protocol implementation with the ability to perform self-health monitoring. Further, an external monitor can monitor critical functions of the communications protocol implementation to provide an external check on the health of the communications protocol implementation.
A first aspect of the invention provides a method of monitoring a set of problem conditions in a communications protocol implementation, the method comprising: controlling a resource exploited by the communications protocol implementation with a control process, wherein the control process includes a problem monitor for a first problem condition that is associated with the resource; and monitoring the resource for the first problem condition with an internal monitor thread, wherein the internal monitor thread sets a problem flag based on the first problem condition being present and the problem monitor resets the problem flag when the first problem condition is cleared.
A second aspect of the invention provides a system for monitoring a set of problem conditions in a communications protocol implementation, the system comprising: a set of control processes, wherein each control process controls a resource exploited by the communications protocol implementation, and wherein each control process includes a problem monitor for a first problem condition that is associated with the resource; and an internal monitor thread for monitoring the resource for the first problem condition, wherein the internal monitor thread sets a problem flag based on the first problem condition being present and the problem monitor resets the problem flag when the first problem condition is cleared.
A third aspect of the invention provides a communications protocol implementation comprising: a set of control processes, wherein each control process controls a resource exploited by a protocol, and wherein each control process includes a problem monitor for a first problem condition that is associated with the resource; and an internal monitor thread for monitoring the resource for the first problem condition, wherein the internal monitor thread sets a problem flag based on the first problem condition being present and the problem monitor resets the problem flag when the first problem condition is cleared.
A fourth aspect of the invention provides a system for processing messages in a communications protocol, the system comprising: a protocol implementation that includes: a set of control processes, wherein each control process controls a resource exploited by the protocol, and wherein each control process includes a problem monitor for a first problem condition that is associated with the resource; and an internal monitor thread for monitoring the resource for the first problem condition, wherein the internal monitor thread sets a problem flag based on the first problem condition being present and the problem monitor resets the problem flag when the first problem condition is cleared; an external monitor that monitors message processing by the protocol implementation, wherein the external monitor detects a second problem condition; and an external communication process utilized by the protocol implementation, wherein the internal monitor further monitors the external communication process.
A fifth aspect of the invention provides a computer-readable medium that includes computer program code to enable a computer infrastructure to process messages in a communications protocol, the computer-readable medium comprising computer program code for performing at least some of the method steps described herein.
A sixth aspect of the invention provides a method of generating a system for processing messages in a communications protocol, the method comprising: obtaining a computer infrastructure; and deploying means for performing at least some of the steps described herein to the computer infrastructure.
The illustrative aspects of the present invention are designed to solve the problems herein described and other problems not discussed, which are discoverable by a skilled artisan.
These and other features of this invention will be more readily understood from the following detailed description of the various aspects of the invention taken in conjunction with the accompanying drawings that depict various embodiments of the invention, in which:
It is noted that the drawings of the invention are not to scale. The drawings are intended to depict only typical aspects of the invention, and therefore should not be considered as limiting the scope of the invention. In the drawings, like numbering represents like elements between the drawings.
As indicated above, the invention provides a solution for monitoring one or more problem conditions in a communications protocol implementation. The communications protocol implementation includes an internal monitor thread that monitors one or more resources for problem condition(s). The internal monitor thread sets a problem flag based on a problem condition being present. A control process in the communications protocol implementation that controls a resource includes a problem monitor that resets the problem flag when the problem condition is cleared. To this extent, the problem monitor provides a check against the internal monitor thread. In one embodiment, the internal monitor thread is periodically executed, and only sets the problem flag after the problem condition has been present for a problem time period. Further, the internal monitor thread can take action in response to the problem condition only after the problem flag has been set for at least two consecutive executions. Additionally, the internal monitor thread can monitor the health of one or more external communication processes that are utilized by the communications protocol implementation using, for example, a heartbeat signal. In this manner, the internal monitor thread and the problem monitor provide the communications protocol implementation with the ability to perform self-health monitoring. Further, an external monitor can monitor critical functions of the communications protocol implementation to provide an external check on the health of the communications protocol implementation.
Turning to the drawings,
Communications between application 34 and one or more nodes (e.g., computing devices, applications, etc.) connected to IP network infrastructure 16 use a particular communications protocol. For example, common communication protocols comprise the transmission control protocol (TCP), and the internet protocol (IP), which together are commonly used to enable communication over public and/or private networks. IP network infrastructure 16 can comprise any combination of one or more types of networks (e.g., the Internet, a wide area network, a local area network, a virtual private network, etc.). Further, communication over IP network infrastructure 16 can utilize any combination of various wired/wireless transmission techniques and/or communication links. While shown and discussed herein with reference to the TCP/IP protocol as an illustrative embodiment, it is understood that the invention is not limited to TCP/IP protocol, and any type of communications protocol can be used.
The communications protocol defines how messages are created and subsequently processed by the sender and receiver. For example, the communications protocol defines a format for messages, specifies how endpoints are identified, specifies how data is stored, and the like. In order to process messages in a particular communications protocol, an operating system 32 generally includes an implementation of the communications protocol. When the communications protocol is implemented using a hierarchy of software layers, the communications protocol implementation is typically referred to as a “protocol stack”. To this extent, operating system 32 is shown including a TCP/IP stack 40 that provides support for sending and receiving messages in the TCP and IP protocols. Additionally, operating system 32 can include one or more additional systems that can be utilized and shared by multiple communications protocol implementations while processing messages.
TCP/IP stack 40 enables operating system 32 to process messages in the TCP and IP protocols by performing some or all of the process steps of the invention. To this extent, TCP/IP stack 40 is shown including a message system 42, a profile system 44, an internal monitor thread 46 and a set (one or more) of control processes 48, each of which includes a problem monitor 50. Operation of each of these systems is discussed further below. However, it is understood that some of the various systems shown in
Regardless, the invention provides a communications protocol implementation, such as TCP/IP stack 40, that monitors a set (one or more) of problem conditions in the communications protocol implementation.
While processing TCP/IP message 60 and/or message data 62, message system 42 can exploit one or more resources 52 on server 14 (
When exploiting a resource 52, TCP/IP stack 40 can incur one or more problem conditions. Using IBM's SNA sysplex environment and TCP/IP communications in z/OS operating system as an illustrative environment, a problem condition can arise with the availability of the virtual telecommunications access method (VTAM) address space. The VTAM address space is exploited by TCP/IP stack 40 when performing various processing for a z/OS communication server network attachment. When the VTAM address space is not available to TCP/IP stack 40 for a prolonged period, TCP/IP processing, including any DVIPA operations, will be adversely impacted. Additionally, one or more problem conditions, such as a critical shortage, can occur with other storage resources including, for example, communication storage manager (CSM) storage, extended common storage area (ECSA), TCP/IP private storage, etc.
Similarly, TCP/IP stack 40 exploits a cross-system coupling facility (XCF) route when communicating with other systems in the sysplex. When no XCF route is available, server 14 (
In any event, TCP/IP stack 40 can include a set (one or more) of control processes 48, each of which controls a unique resource 52 exploited by message system 42. Control process 48 can manage obtaining resource 52, exploiting resource 52 (e.g., reading/writing data from/to resource 52), relinquishing resource 52, and the like, in a known manner. Additionally, TCP/IP stack 40 can include an internal monitor thread 46 that monitors resource(s) 52 for one or more problem conditions. Internal monitor thread 46 can execute periodically, and monitor several resources 52 and/or problem conditions for each resource 52. Internal monitor thread 46 can set a problem flag 66 that is unique to each problem condition and resource 52 combination based on the problem condition being present.
Additionally, control process 48 can include a problem monitor 50 for each monitored problem condition that corresponds to the resource 52 that is controlled by control process 48. When problem monitor 50 detects that the corresponding problem condition has been cleared, problem monitor 50 can reset the problem flag 66 for the problem condition and resource 52 combination. Problem flag 66 can be implemented in any known manner. For example, problem flag 66 can comprise a designated shared memory location/portion of a memory location (e.g., a bit). In this case, problem monitor 50 and internal monitor thread 46 can read and/or write to problem flag 66 using uninterruptible operations, semaphores, or the like.
In one embodiment, prior to initiating a recovery action, internal monitor thread 46 can first determine whether the problem condition has persisted for at least a predefined problem time period. To this extent, internal monitor thread 46 can further track a time period that the problem condition has persisted using any solution. The problem time period can be fixed or can be configured by a user/system. In the latter case, the problem time period can be defined in a protocol implementation profile 64. For example, TCP/IP stack 40 can include a profile system 44 for managing protocol implementation profile 64. Profile system 44 can generate a user interface or the like that enables a user to define the one or more profile settings (e.g., the problem time period), can read and/or process profile setting data, can receive and/or generate profile setting data, can write profile setting data to protocol implementation profile 64, and/or the like.
In any event, profile system 44 can obtain protocol implementation profile 64 and provide profile setting data to other systems in TCP/IP stack 40. To this extent, profile system 44 can obtain the problem time period from protocol implementation profile 64 and provide it to internal monitor thread 46. In one embodiment, internal monitor thread 46 is periodically executed based on the problem time period. For example, internal monitor thread 46 could be executed four times during the problem time period (e.g., every fifteen seconds when the problem time period is set to sixty seconds). When internal monitor thread 46 is monitoring multiple problem conditions, the same problem time period can be used for all of the problem conditions. Alternatively, different problem time periods could be defined for different problem conditions. In the latter case, the frequency with which internal monitor thread 46 is executed can be determined based on the shortest problem time period. Alternatively, multiple internal monitor threads 46 can be used, each of which monitors a unique set of related problem conditions (e.g., all problem conditions having the same problem time period).
When, in step S2, internal monitor thread 46 determines that the problem condition is present, then in step S5, internal monitor thread 46 determines whether the problem condition has persisted for the problem time period. For example, internal monitor thread 46 can subtract the last time stored in step S3 from the current time to determine if the difference exceeds the problem time period. If the problem condition has not persisted for at least the problem time period, then flow continues to step S4. When the problem condition has persisted for the problem time period, then in step S6, internal monitor thread 46 can determine if problem flag 66 has been set. When problem flag 66 is not set, then in step S7, internal monitor thread 46 sets problem flag 66 and flow continues to step S4.
Turning to
Since internal monitor thread 46 only processes each resource 52 once during each execution, problem flag 66 will be set for at least the time period between consecutive executions of internal monitor thread 46 before any action is taken. This enables problem monitor 50 to act as a check against the false identification of a problem condition by internal monitor thread 46, e.g., when a problem condition occurs for only a brief period of time. Returning to
To this extent, in step S8, internal monitor thread 46 can issue one or more eventual action messages. Each eventual action message can include data on the particular problem condition that was detected, and can be sent to, for example, a console for the sysplex, another system executing within operating system 32 (
It is understood that the method steps of
Returning to
In addition to monitoring problems, such as availability, of resources 52, internal monitor thread 46 can monitor one or more external communication processes that are utilized by TCP/IP stack 40 during message processing. In this case, internal monitor thread 46 can determine a health of the external communication process(es). For example, message system 42 can use a routing daemon 54 when implementing certain DVIPA functionality. Routing daemon 54 can periodically send a “heartbeat” signal that is received by internal monitor thread 46. When internal monitor thread 46 does not receive the heartbeat signal for a certain period of time (e.g., the problem time period plus one additional execution), then internal monitor thread 46 can identify it as a problem condition and respond accordingly (e.g., issue eventual action message(s)). To this extent, internal monitor thread 46 can set a problem flag 66 for routing daemon 54 as discussed herein. Further, while not shown, TCP/IP stack 40 can include a control process 48 that controls the external communication process, such as routing daemon 54 and can reset the problem flag 66 when the problem condition is cleared (e.g., the heartbeat signal is received).
While shown and described herein as a method and system for monitoring a set of problem conditions in a communications protocol implementation, it is understood that the invention further provides various alternative embodiments. For example, in one embodiment, the invention provides a computer-readable medium that includes computer program code to enable a computer infrastructure to monitor a set of problem conditions in a communications protocol implementation. To this extent, the computer-readable medium includes program code, such as TCP/IP stack 40 (
In still another embodiment, the invention provides a method of generating a system for monitoring a set of problem conditions in a communications protocol implementation. In this case, a computer infrastructure, such as environment 10 (
As used herein, it is understood that the terms “program code” and “computer program code” are synonymous and mean any expression, in any language, code or notation, of a set of instructions intended to cause a computing device having an information processing capability to perform a particular function either directly or after any combination of the following: (a) conversion to another language, code or notation; (b) reproduction in a different material form; and/or (c) decompression. To this extent, program code can be embodied as one or more types of program products, such as an application/software program, component software/a library of functions, an operating system, a basic I/O system/driver for a particular computing and/or I/O device, and the like.
The foregoing description of various aspects of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and obviously, many modifications and variations are possible. Such modifications and variations that may be apparent to a person skilled in the art are intended to be included within the scope of the invention as defined by the accompanying claims.
The current application is related to co-owned and co-pending U.S. patent application No. ______ (Attorney Docket No. RSW920050118US1), filed on Aug. 8, 2005, and entitled “Monitoring A Problem Condition In A Communications System”, which is hereby incorporated herein by reference.