Patent Application
20050270973
Embodiments of the invention generally relate to the field of data processing systems. More particularly, the invention relates to a cluster architecture for data processing systems.
In prior art cluster architectures, session communications and internal communications, for example, server-to-server communications, were handled over the same socket connection. Peer-to-peer connections between all the nodes in the cluster additionally resulted in a complicated network configuration that did not scale well in terms of network resources, communication bandwidth and overhead, and lead to limited multicast and/or broadcast communication capabilities.
Embodiments of the invention are generally directed to a cluster architecture, and in particular separate communication facilities for internal cluster communications versus external, client request driven communications. The invention allows for separate and parallel communications between dispatcher nodes and server nodes, and among server nodes, either via a message server or a direct socket connection between servers nodes, even those in different application server instances.
Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figure of the accompanying drawing:
Embodiments of the invention are generally directed to communication facilities and methods among components of a cluster architecture.
An instance typically contains at least one server process, or “application server node”. More commonly, an instance includes a dispatcher and several server nodes. It is also contemplated that more than one dispatcher may reside in a single instance. In
Application server nodes 130, 132 and 134 within application server instance 110 provide the business and/or presentation logic for the applications supported by the system. Each application server node provides a set of core services to the business and/or presentation logic. Likewise, application server nodes 140, 142 and 144 provide support for applications running on application server instance 112.
Each of the application server nodes within a particular instance may be configured with a redundant set of application logic and associated data. In one embodiment, a dispatcher 120 distributes service requests from clients to one or more of server nodes 130, 132 and 134, based, for example, on the load on each of the servers. For example, in one embodiment, a dispatcher implements a round-robin policy of distributing service requests (although various alternate load-balancing techniques may be employed). Application server instances receive requests from one or more clients, for example, via a web client, over a distributed network such as the Internet. In one embodiment, requests from the web client may be transmitted using hypertext transfer protocol (HTTP), HTTPS, SMTP, or SOAP.
In one embodiment of the invention, server nodes 130, 132, 134, 140, 142 and 144 are Java 2 Platform, Enterprise Edition (“J2EE”) server nodes that support Enterprise Java Bean (“EJB”) components and EJB containers (at the business layer) and Servlets and Java Server Pages (“JSP”) (at the presentation layer). A J2EE platform complies with the J2EE Standard. Of course, certain aspects of the embodiment of the invention described herein may be implemented in the context of other software platforms including, by way of example, Microsoft .NET platforms and/or the Advanced Business Application Programming (“ABAP”) platforms developed by SAP AG, the assignee of the this patent application.
Message server 152 is responsible for intra- as well as inter-instance communication. For example, if a server node 130 of instance 110 wishes to send an internal message to instance 112, a message is sent via the message server 152. In one embodiment, each server node and each dispatcher node has a link 180, 182 through which it can communicate with the message server, and through which the message server may send messages notifying of events within the cluster. Message server 152 also supplies a dispatcher 120, 122 with information to facilitate load balancing between various instances in the cluster. The message server 152 also provides notification of events that arise within a cluster, for example, failure or shutdown of an instance or when a service is started or stopped. Because the message server may represent a single point of failure in the cluster, it should support failover to be effectively used in high availability systems. To that end, in one embodiment, the message server has no persistent state such that if the message server fails, it need merely be restarted and then re-register instances in the cluster without performing any state recovery procedures.
In one embodiment, communication and synchronization between each of instances 110 and 112 is enabled via central services instance 114, in particular, by the messaging service provided by message server 152. The message service allows each of the server nodes within each of the instances to communicate with one another via a message passing protocol. For example, messages from one server node may be broadcast to all other server nodes within the cluster via the messaging service. In addition, messages may be addressed directly to specific server nodes within the cluster (e.g., rather than being broadcast to some number or all of the server nodes).
The cluster in
Sockets are the mechanisms that allow the elements of the cluster to communicate, either on the same machine or across a network. Each physical server in the cluster is identified by some address. In a TCP/IP networking environment, the network address is IP address. Apart from the IP address that specifies a machine, each machine has a number of ports, for example, TCP ports, that allows handling multiple socket connections simultaneously.
A program that wants to accept a socket connection with another program creates a socket, binds the socket to a specific address, e.g., an IP address, and port, and then listens for requests on the socket to establish a connection. An application server node creates a server socket and listens on it to accept socket connections from the dispatcher node and other server nodes. In one embodiment, a socket connection between a server node and the message server is initiated by the server node to a server socket on the message server created by the message server.
With reference to
Thus, each server node has at least two socket connections—one with the dispatcher over which to transmit session communications and one with the message server over which to transmit internal communications. A dispatcher maintains two types of socket connections—a first socket connection between the dispatcher and message server for exchanging internal communications, and multiple instances of a second socket connection, duplicated between the dispatcher and each server node to which it is distributing client requests, for exchanging session communications.
In the case of a session communication between a dispatcher and server node, if the socket connection providing such communication breaks, the dispatcher attempts to reinitialize the session communication with the server node. In one embodiment, the server node may send a notification to the message server via its socket connection with the message server for internal communication, the notification providing state information that the server node is unavailable, or down. The dispatcher, before attempting to reinitialize the session communication with the server node, may receive such indication from the message server via its socket connection with the message server, or may poll the message server for the state of the server node. If the dispatcher thereby determines the server node is unavailable, it does not attempt to reinitialize the socket connection for session communications with the server node.
The message server, in one embodiment of the invention, sends a notification or acknowledgement message in response to each message it receives from a server node or a dispatcher. If the message server goes down or otherwise becomes unavailable, a server node that does not receive such an acknowledgement can attempt to retransmit its message to the message server, or wait until it receives an indication that the message server is up and operating again. The message server, upon becoming available, can indicate such in a message, for example, a multicast or broadcast message, sent over the socket connection for internal communications between the message server and each of the server nodes in the cluster.
In one embodiment of the invention, the message server is single threaded, and can become overloaded by messages, creating a bottleneck. To ease message congestion at the message server, a third and different socket connection may be established directly between two server nodes, bypassing the respective socket connections between the server nodes and the message server. For example, server node 130 can establish a socket connection 190 with server node 132 in the same application server instance, for direct intra-instance exchange of internal communications. In addition, a server node can establish a socket connection with a server node in a separate application server instance for direct inter-instance exchange of internal communications. For example, server node 132 in application server instance 110 can establish a socket connection 192 with server node 140 in application server instance 112, for direct exchange of internal communications, thereby bypassing socket connections between server node 132 and message server 152 and between server node 140 and message server 152.
This third socket connection provides an alternative way for exchanging internal communications between server nodes. A server node can initiate opening of this additional socket connection directly to another server node if, for example, data transfer rates over the internal communications socket connection via the message server meet or exceed a threshold. Likewise, a server node can initiate tearing down the direct socket connection to another server node if, for example, the data transfer rate over one or both of the contemporaneous socket connections to the other server node by way of the message server fall below a certain threshold for some minimum amount of time.
The order of internal messages sent between two server nodes is maintained, even in the event that a separate socket connection is contemporaneously established directly between the two server nodes. This can be accomplished through the use of a single output queue per service on a server node, so whether a message is transmitted from one server node to another via socket connections with the message server, or via the direct socket connection between the server nodes, the message arrives in the same order with respect to other messages in the output queue.
Transmitting multicasting messages among server nodes is accomplished via socket connections for internal communications between each server node and the message server. A server node need only send one multicast message to the message server. The message server replicates the message and transmits it to each destination server node over its respective socket connection with the message server.
Elements of embodiments of the present invention may also be provided as a machine-readable medium for storing the machine-executable instructions. The machine-readable medium may include, but is not limited to, flash memory, optical disks, CD-ROMs, DVD ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, propagation media or other type of machine-readable media suitable for storing electronic instructions. For example, embodiments of the invention may be downloaded as a computer program which may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., a modem or network connection).
It should be appreciated that reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, it is emphasized and should be appreciated that two or more references to “an embodiment” or “one embodiment” or “an alternative embodiment” in various portions of this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined as suitable in one or more embodiments of the invention.
Similarly, it should be appreciated that in the foregoing description of embodiments of the invention, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed subject matter requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
