Hybrid method for flushing transaction state in a fault-tolerant clustered database

Abstract

A Registration protocol is used in transaction processing for normal operations. If an error occurs, the system reverts to a Full Broadcast protocol. The Registration Protocol reduces the number of messages that are sent among CPUs in a cluster thereby permitting performance improvements in the system. The Registration Protocol has Begin, DP2 Check, Phase 1 Flush and Phase 2 (lock release) phases just as does the Full Broadcast Protocol, thereby permitting the Full Broadcast protocol to step in at any phase after an error is detected.

Description

FIELD OF THE INVENTION

[0003] The present invention relates generally to improving performance of transaction processing systems, and more particularly to improving the efficiency of distributed transaction protocols so that performance is maintained or improved as the number of CPUs in the transaction processing system increases.

DESCRIPTION OF THE RELATED ART

[0004] Current transaction processing systems involve networks that interconnects clusters of clusters, each cluster being composed of a set of interconnected CPUs. These networks are called cluster interconnects or external fabrics and permit large numbers, perhaps hundreds, of CPUs to cooperate in accordance with a distributed transaction processing protocol. As the number of clusters connected by the cluster interconnect increases, however, performance problems with respect to the protocols used by the entire transaction processing system begin to occur, especially when the size of the transaction is small. In particular, certain protocols used to end a transaction (a transaction flush) are multicasts that affect each CPU whether or not that CPU was involved in the transaction. The transaction flush multicasts thus create a large number of messages over the cluster interconnect, which, for large numbers of small transactions, tends to excessively load the cluster interconnect. In other parts of the protocol, messages between CPUs are inefficient and not effectively aggregated due to the arrival pattern of transaction flushes and the distribution of transaction operations.

[0005] Historically, systems that send large numbers of small messages suffer from two kinds of performance limiting effects: (a) network communications messaging overhead and (b) hotspots in processing when there is much (i) fan-out or (ii)fan-in.

[0006] Network communications overhead comes from the serialization, synchronization, data integrity, routing, error handling and flow control requirements placed on communications systems, and these are standard issues, standardly handled. It is in the per-message handling costs that box-carring becomes a performance win, with a diminishing return for each additional box-carred message which is sent together with others, amortizing that overhead over the group.

[0007] Hotspots in processing occur when there is high traffic, either by protocol design, or by an unexpected load imbalance, which originates from a particular CPU and fans-out to many CPUs in the cluster, or which originates from many CPUs and fans-in to a single CPU in the cluster. Both of these conditions will occur, due to protocol, when coordination or synchronization are required in fault-tolerant systems, by the very nature of those systems.

[0008] Thus, there is a need to improve the message traffic inherent in the transaction processing protocols so that the transaction processing system can scale to larger systems without impeding the performance of the system.

BRIEF SUMMARY OF THE INVENTION

[0009] The present invention is directed towards the above need. A method in accordance with the present invention includes a method of performing a transaction in a processing cluster that has a plurality of CPUs. One of the CPUs hosts a transaction monitor. At least one CPU hosts a resource manager and each of the CPUs includes volatile storage and has access to stable storage. The method includes the steps of beginning a transaction by the transaction monitor and then performing the following steps if no CPU failure occurs: (i) registering one or more resource managers to participate in the transaction, where the CPUs that host the participating resource managers are participating CPUs; (ii) performing work required by the transaction by the one or more participating resource managers, including obtaining any locks needed by the transaction; (iii) upon completion of the transaction work by the participating resource managers, performing a phase 1 operation in which a flush request is broadcast to each participating CPU, where the flush request instructs each participating CPU to send a response indicating that information in volatile storage is flushed to stable storage; (iv) upon completion of the phase 1 operation, requesting a phase 2 operation by broadcasting a lock-release request to each participating CPU, except the CPU hosting the transaction monitor; (v) releasing any locks acquired for the transaction while performing work required by the transaction in response to the phase 2 request; and (vi) after releasing all locks acquired for the transaction, informing the transaction monitor's CPU that phase 2 is completed, and that the transaction's locks can be released there.

[0010] If at a particular step in the above method, a failure of a CPU occurs, then the processing cluster resorts to some or all of the following steps, depending on where the failure occurs: (i) broadcasting a begin transaction message to all CPUs, except the CPU hosting the transaction monitor; (ii) performing work required by the transaction by the one or more resource managers working on behalf of the transaction, including obtaining any locks needed by the transaction; (iii) upon completion of the transaction work by the resource managers working on behalf of the transaction, performing a phase 1 operation in which a flush request is broadcast to each CPU, the flush request instructing each CPU working on behalf of the transaction to send a response indicating that information in volatile storage is flushed to stable storage and instructing CPUs not working on behalf of the transaction to so indicate; (iv) upon completion of the phase I operation, requesting a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; (v) releasing any locks acquired for the transaction while performing work required by the transaction; and (vi) after releasing all locks acquired for the transaction, informing the transaction monitor's CPU that phase 2 is completed, and that the transaction's locks can be released there.

[0011] For example, if a failure occurs during the registration step, then all of the above steps are performed. If the failure occurs during the phase 2 step, then steps (iv) through (vi) are performed.

[0012] One advantage of the present invention is that the performance of a transaction system is improved when the transaction system includes a large number of clusters connected by an external fabric. In particular, the number of clusters that participate in a distributed transaction can be doubled without a significant effect on performance.

[0013] Another advantage of the present invention is that network messages between clusters are made more efficient.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] These and other features, aspects and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:

[0015] FIG. 1 shows a system setting in which the present invention operates;

[0016] FIG. 2 sets forth the various phases of the transaction protocols involved in the present invention and their inter-relationship;

[0017] FIG. 3A sets forth a timing chart for a TMP-Owned Begin Transaction according to the Full protocol;

[0018] FIG. 3B sets forth a timing chart for a DP2 Check-In operation according to the Full Protocol;

[0019] FIG. 3C sets forth a timing chart for a Phase 1 Flush operation according to the Full Protocol;

[0020] FIG. 3D sets forth a timing chart for a TMP-Owned Phase 2 operation according to the Full Protocol;

[0021] FIG. 4A sets forth a timing chart for a TMP-Owned Begin Transaction according to a MMI protocol;

[0022] FIG. 4B sets forth a timing chart for a DP2 Check-In operation according to the MMI protocol;

[0023] FIG. 4C sets forth a timing chart for a Phase 1 Flush operation according to the MMI protocol;

[0024] FIG. 4D sets forth a timing chart for a TMP-Owned Phase 2 operation according to the MMI protocol;

[0025] FIG. 5A sets forth a timing chart for a Application-Initiated Begin Transaction operation; and

[0026] FIG. 5B sets forth a timing chart for an Application-Initiated Transaction Phase 2 operation.

DETAILED DESCRIPTION OF THE INVENTION

[0027] FIG. 1 shows a system setting in which the present invention operates. The system includes one or more clusters 10, 12, 14, 16 of CPUs 18a-f, 20a-f, 22a-f, 24a-f, interconnected by an internal fabric 26, 28, 30, 32. In some embodiments, up to 16 CPUs (CPUs and volatile storage, such as semiconductor memory) are interconnected locally by the internal fabric. Each CPU 18a-f, 20a-f, 22a-f, 24a-f has access to a stable storage system, such as a disk system (not shown), which survives a failure in one of the CPUs. An external fabric 34 interconnects the clusters 10, 12, 14, 16. The external fabric 34 can be a local-area network or a wide-area network.

[0028] FIG. 2 sets forth the various phases of the transaction protocols involved in the present invention and their inter-relationships. These transaction phases are the phases involved in the life of a transaction within a cluster and may be part of a two-phase commit protocol that operates among two or more clusters.

[0029] A transaction is begun either by an Application process (Application-Initiated) 56 or by a TMP Owner 58, 60. A TMP Owner, in the usual case, is a process to which an Application process has exported a transaction. A TMP Owner can begin a transaction according to the Full Broadcast Protocol (Full) 52 or the Registration Protocol (MMI) 54. After a transaction is begun, a resource manager check-in phase 62, 64, 66 (also referred to as a DP Check-In phase) follows. Work required by the transaction is accomplished and upon completion, a Phase 1 Flush 68, 70, 72 is initiated to end the transaction. There are two kinds of Phase 1 Flush operations, one 70 in accordance with the Full protocol and one 72 in accordance with the MMI protocol. Following the Phase 1 flush, there is a Phase 2 operation 74, 76, 78, which releases locks and allows resources to be made available to users, because the transaction is completed. A Phase 2 phase is begun by the BOwner, which is either the TMP-owner or the Application process. The TMP Owner may either perform a Phase 2 flush according to the Full or MMI protocol.

[0030] FIG. 2 also sets forth the overall flow of these various phases. The normal or preferred flow of phases is from an Application- or TMP Owner-begun transaction 56, 58, 60 to a DP Check-In phase 62, 66 according to the MMI protocol, to a Phase 1 Flush 68, 72 with the MMI protocol to the Phase 2 phase 74, 78 by the Application or the TMP Owner in accordance with MMI. If however, a failure (of a CPU) occurs during the performance of the transaction, then flow is permanently transferred to the Full protocol 52, as indicated by the dotted connections.

[0031] A resource manager (DP) Check In phase 64, according to the Full protocol, occurs if there is a failure in the MMI protocol Check In phase 62, 66. A Phase 1 Flush 70 according to the Full protocol 52 occurs if there is an failure in the MMI Phase 1 Flush phase 68, 72 and a Phase 2 operation 76 occurs according to the Full protocol if there is a failure in the MMI Phase 2 operation 74, 78.

[0032] Thus, it can be seen that the Full protocol 52 is resorted to when a failure occurs along the flow of the MMI protocol 50, 54. Operating according to the MMI protocol, except under failure conditions, rather than operating at all times with the Full protocol 52, causes a significant reduction in the messages involved in processing a distributed transaction.

[0033] The phases of the Full Broadcast protocol are first set forth below, followed by phases of the MMI Protocol.

[0034] Begin Transaction—TMP-Owned (Full Broadcast Protocol)

[0035] FIG. 3A sets for the Begin Transaction operation for a TMP-Owned Transaction in accordance with the Full Broadcast Protocol. Shown in FIG. 3A are the key messages that are sent and received in the Begin Transaction operation. The TMP-Owned transaction is started by the TMP-P process in the TMP-P CPU and essentially has two sub-phases. The first phase informs a backup TMP process (TMP-B) that TMP-P is starting a transaction. The second phase informs all of the non-TMP CPUs.

[0036] In the first sub-phase, a TMP process in the primary TMP CPU (TMP-P) makes a ‘Begin’ call 100 to the TMF Library (TmfLib) on the TMP-P CPU. The TmfLib is a part of a system library, in each CPU, that provides TMF (Transaction Monitor Facility) functions to applications, disk (resource manager) processes and TMF-related processes. In response to the call, the TmfLib creates a new Library Transaction Control Block (LibTcb) and searches for an empty entry in the Slot table (SlotTable). A LibTcb is a transaction control block, residing in the CPU global memory, that is managed by the TMF Library facility. A separate SlotTable reflecting each CPU resides in each CPU of a cluster. For example, if there are 16 CPUs there are 16 slot tables in each CPU. Each slot table has a slot for every LibTcb created by an originating CPU, which controls that slot entry in all of the SlotTables. If an empty SlotTable entry is found, the entry is updated with the LibTcb's address, a sequence number for the transaction (from the SlotTable entry number) and an Epoch number of 0. An Epoch number is a sequence number of a flush operation, since flushes can occur multiple times per transaction. The TmfLib next sends a ‘TmpTxBegin’ message 102 to the backup TMP process on the backup TMP CPU (TMP-B).

[0037] When the ‘TmpTxBegin’ message is received by the TMP-B process on the backup TMP CPU, a LibTcb is created for the transaction in the copy of the TMP-P CPU's SlotTable that resides on the TMP-B CPU. Then TMP-B process on the TMP-B CPU then returns a ‘DataAck’ message 104 to the TMP-P process on the TMP-P CPU.

[0038] Returning the ‘DataAck’ message causes the TMP-P process to now inform the other non-TMP CPUs of the transaction, thus beginning the second sub-phase of the ‘Begin’ operation.

[0039] In this second sub-phase, the TmfLib of the TMP-P process broadcasts, via its PIO transport facility, a ‘TmpTxBegin’ message 106 to all of the non-TMP CPUs. The PIO transport facility is a packet I/O facility by which packets can be sent to a single CPU or broadcast to multiple CPUs. When this message is received by each of the non-TMP CPUs, each CPU creates a LibTcb for the transaction in a copy of the TMP-P CPUs SlotTable residing therewith. A ‘DataAck’ message 108 is returned by each non-TMP CPU to the TMP-P process, which triggers a procedure in the TmfLib of the BOwn CPU (the TMP-P CPU, in this case) to update the BOwn CPU's LibTcb state fields and starts the BOwn CPU's state machine for the transaction. The TmfLib in the TMP-P CPU then queues a ‘NetTxBegin GetWork’ for the TMP-P process in the TMP-P CPU confirming that the transaction's initialization is complete.

[0040] In summary, these two sub-phases inform the all of the running CPUs of the transaction. Each running CPU returns an acknowledgment that it has been so informed.

[0041] Resource Manager (DP2) Check In Trans (Full Broadcast Protocol)

[0042] This phase is an instance of a resource manager, such as a disk process, becoming associated with a transaction, by checking in with the Transaction Monitor Facility (TMF) and entering the transaction into the TmfLibrary. Both the primary resource manager process (DATA-P) and its backup process (DATA-B) must be associated with the transaction.

[0043] Referring to FIG. 3B, after a transaction has been started by the TMP-P CPU, if a DP2 data volume must work as a resource on behalf of the transaction, the primary DP2 process must check in with the TMF for every DP2 request message with a new transactional operation (READUPDATE, WRITEREAD, etc.). In the Full Broadcast Protocol, checking in on a transaction consists of entering that transaction into the Tmf Library, which is appropriately called TmfLib.

[0044] The first event in the resource manager Check In process, according to the Full Broadcast Protocol, is then a call to the TmfLib on the primary DP2 CPU (the primary resource manager CPU) with a ‘CheckInTrans’ request 120 that includes the transaction's TcbRef and epoch number. The TcbRef number is a transaction ID, which is only valid for one cluster's usage, the cluster being the cluster at which the transaction was issued, and is only valid until a TMF crash.

[0045] The TmfLib in the primary DP2 CPU checks to see if the TcbRef has an entry in the BOwn CPU's SlotTable in the primary DP2 CPU. If there is no entry, the TmfLib creates a new LibTcb in the SlotTable, if the TcbRef's sequence number and epoch number are valid. An error response is returned to the primary DP2 process, if a LibTcb does not exist and a new one cannot be created.

[0046] Next, a DpCross Link is found or created for the primary DP2 in step 122. A DpCrossLink is an association between a particular transaction and a particular data volume process pair. An error response is returned to the primary DP2 process if a DPCross link cannot be created or the epoch in the DPCross link differs from the epoch number for the transaction. Epoch numbers can differ if there has been a flush (due to a transaction abort) prior to the step of finding or creating the DpCross link.

[0047] The TmfLib in the primary DP2 CPU then sends a ‘DpCheckIn’ message 124 to the backup DP2 CPU, which responds by creating a LibTcb for the transaction and a DPCross link for the backup DP2 CPU, in step 126. The Backup DP2 CPU then sends an ‘FeOk’ message 128 response back to the primary DP2 process on the primary DP2 CPU. An ‘FeOk’ message is a message indicating that the prior request was successful.

[0048] Thus, in this phase, only those resource managers (e.g., data volume CPUs) that are performing work on behalf of the transaction check in with TMF.

[0049] Phase 1 Flush (Full Broadcast Protocol)

[0050] This part of the protocol, shown in FIG. 3C, is complex because the BOwn CPU does not know which resource managers participated in the transaction. All of the CPUs in the system must be sent a request to end the transaction to find out which resource managers actually participated.

[0051] First, the Application requester (the transaction beginner) process calls ‘EndTransaction’ 132. The BOwn CPU (in this case, the CPU on which the Application Requester resides) then broadcasts a ‘FlushCpu’ message 134 for epoch 0 to all running CPUs, including itself for the transaction in question.

[0052] Each CPU having a resource manager that participated in the transaction instructs its participating primary resource manager CPUs, in response to the ‘FlushCpu message,’ to flush its audit buffers to the audit log disk process input buffer, if the audit buffer is not already flushed for the transaction. If the audit is already flushed or when the flush is completed, each participating CPU updates its LibTcb, DpCross links and SlotTable epoch number and sends a ‘CpuFlushed’ message 136 to the BOwn CPU. The BOwn CPU maintains a bit mask for the participating CPUs (i.e., CPUs having participating resource managers) and a bit is set in that mask for each CPU that responds with a ‘CpuFlushed’ message.

[0053] CPUs without a LibTcb for the transaction (because they were never checked in by any participating resource manager) send a ‘NonParticipating’ message 138 to the BOwn CPU. These CPUs do not have their bit set in the bit mask.

[0054] After all of the CPUs have responded to the ‘FlushCpu’ message, the BOwn CPU sends an ‘AllCpusFlushed’ message 140 to the primary TMP (TMP-P) process. The TMP-P process controls the transaction from this point forward. An ‘FeOk’ response 142 is returned by the TMP-P process to the Application Requester.

[0055] Thus, a Phase 1 Flush, according to the Full Broadcast protocol, involves each CPU receiving a ‘FlushCpu’ message and each CPU responding either that it did or did not participate. The result is that a large number of messages are transferred over the cluster interconnect during this operation. As the number of CPUs interconnected in a cluster increases, the number of messages also increases for each transaction, indicating that the Full Broadcast Protocol does not scale well with the addition of CPUs.

[0056] Phase 2 (Full Broadcast Protocol)

[0057] The final phase of the Full protocol is shown in FIG. 3D and also involves all of the CPUs. There are two subphases. The first subphase involves sending a broadcast to all of the CPUs except for the TMP CPUs. The second subphase involves sending a broadcast to the TMP CPUs.

[0058] To start the Phase 2 (lock release) operation for a transaction, the primary TMP process calls its TmfLib with a ‘DoPhase2’ request 150. The TmfLib in the primary TMP CPU sends a ‘Phase2Needed’ message 152 to the BOwn CPU, which in this case is the same CPU.

[0059] The BOwn CPU broadcasts a ‘DoPhase2’ message to all CPUs except the TMP CPUs. Each CPU receiving the ‘DoPhase2’ message 154, then tells its involved resource managers (e.g., DP2 CPUs) to perform a Phase2 operation 156, and then deletes the DpCross links and LibTcb from the involved resource managers.

[0060] A ‘DataAck’ message 158 is returned to the BOwn CPU by each CPU that received the ‘DoPhase2’ message.

[0061] Next, the BOwn CPU broadcasts a ‘DoPhase2’ message 160 to the TMP CPUs. Each TMP CPU then tells its involved resource managers (e.g., DP2 CPUs) to each perform a Phase2 operation 162 and then delete its DpCross links. The backup TMP CPU deletes its own LibTcb entry and a ‘DataAck’ message 164 is returned by the backup TMP CPU, which triggers a procedure that deletes the LibTcb in the primary TMP CPU. Finally, the TmfLib in the primary TMP CPU queues a ‘TxPhase2 GetWork’ for the TMP process, thereby informing the process that the transaction is terminated in the TMF Library.

[0062] In summary, the Full Broadcast protocol involves a begin message to the TMP-B CPU, a begin transaction broadcast to all non-TMP CPUs, a flush broadcast to all CPUs to end the transaction, a broadcast to the non-TMP CPUs, and then a broadcast to the TMP CPUs to release locks.

[0063] The Registration (MMI) Protocol—TMP-Owned

[0064] FIG. 4A illustrates the registration protocol for a TMP-Owned transaction. The protocol starts with the primary TMP Process calling its TmfLib with a ‘Begin’ request 170. The TmfLib creates a new LibTcb and searches for an empty SlotTable entry, as described above. If an empty SlotTable entry is found, the SlotTable is updated with the LibTcb's address, the transaction's sequence number (from the SlotTable entry number) and an epoch number of 0.

[0065] The TmfLib of the primary TMP CPU (TMP-P) sends a ‘TmpTxBegin’ message 172 to the backup TMP CPU (TMP-B) which creates, in response, a LibTcb for the transaction in the copy of the primary TMP CPU's SlotTable that resides in the backup TMP CPU.

[0066] A ‘DataAck’ message 174 is returned by the backup TMP CPU to the primary TMP CPU, triggering a procedure in the TmfLib of the primary TMP CPU that updates the BOwn CPU's (in this case the primary TMP CPU) LibTcb state fields and initiates a state machine in the BOwn CPU for the transaction.

[0067] The TmfLib in the primary TMP CPU queues a ‘NetTxBegin GetWork’ command for the primary TMP process informing the process that the transaction's initialization is complete.

[0068] In summary, only the TMP CPUs are involved in starting a transaction.

[0069] Resource Manager Check in Phase (i.e., DP CheckIn) Phase (Registration Protocol)

[0070] The registration protocol is more complex than the Full Broadcast Protocol in the resource manager check In phase, shown in FIG. 4B, but only the DP2 CPUs are involved.

[0071] The primary resource manager (primary DP2) process calls requests a ‘DpCheckInTrans’ 180 in its TmfLib with the transaction's TcbRef and epoch number. The TmfLib in the primary DP2 CPU checks to see if the TcbRef has an entry in the SlotTable and, if not, creates a new LibTcb, if the TcbRef's sequence number and epoch number are valid. If a LibTcb does not exist and a new one cannot be created, an error response is returned to the primary DP2 process.

[0072] Next, the TmfLib in the primary DP2 CPU sends a ‘Participating’ message 182 to the BOwn CPU for any brother CPUs which are not already registered. The TmfLib in the primary DP2 CPU then sends a ‘May-I-Participate’ message 184 to the BOwn CPU. One thread of the primary DP2 process is held in a wait loop waiting for a response to the message and queuing requests.

[0073] A ‘May-I-Participate’ response message 186 is next sent by the BOwn CPU back to the waiting primary DP2 process. The primary DP2 process is awakened by a special wakeup, LTmf, and a DPCross link is found or created for the primary DP2 process. If the primary DP2 process does not accept the special wakeup, LTmf, message, the transaction's LibTcb is deleted from the BOwn CPU's SlotTable in the primary DP2 CPU, and an error response is returned to the primary DP2 process.

[0074] The DpCross link is found or created for the primary DP2 CPU in step 188 or an error response is returned to the primary DP2 process if one cannot be created or the cross link epoch number and the slot epoch number disagree.

[0075] The TmfLib in the primary DP2 CPU next sends ‘DpCheckIn’ message 190 to the backup DP2 process. When the ‘DpCheckIn’ message is received, a LibTcb is created for the transaction in the BOwn CPU's SlotTable in the backup DP2 CPU and a DpCross link is created for the backup DP2 process.

[0076] An ‘FeOk’ response 194 is returned by the backup DP2 process to the primary DP2 process. If the TmfLib in the backup DP2 CPU was late, the TmfLib in the backup CPU sends a CheckInResult message to the primary CPU informing it of the CheckIn.

[0077] In summary, in the registration protocol only the participating resource managers register to participate in a transaction.

[0078] Phase 1 Flush (Registration Protocol)

[0079] In the Registration Protocol, when the time comes to flush the changes made by a transaction, the list of CPUs that participated is completely known (except during the processing of failures, in which case the system reverts to the Full Broadcast protocol). This permits a more simplified protocol near the end of a transaction, which is the time critical portion of the protocol.

[0080] In the FIG. 4C, the Application requester (the transaction beginner) process makes an ‘EndTransaction’ call 200. The BOwn CPU receives the call and broadcasts a ‘FlushCpu’ message 202 for epoch 0, only to the CPUs that have registered as participating in the transaction including the BOwn CPU.

[0081] The participating CPUs instruct their primary resource managers (i.e., DP2 processes) to flush their audit buffers to the audit-log, disk-process input buffer, if the audit buffer is not already flushed for the transaction being completed. If the audit buffer is already flushed or when the flush is completed, each CPU updates its LibTcB, DpCross links, and SlotTable EpochNum, and sends ‘CpuFlushed’ message 204 to the BOwn CPU.

[0082] After each participating CPU has responded to the transaction flush, the BOwn CPU sends an ‘AllCpusFlushed’ 206 message to the primary TMP CPU.

[0083] When the ‘AllCpusFlushed’ message 206 is received by the primary TMP CPU, the TmfLib queues a ‘Phase1Flush GetWork’ for the primary TMP Process. The primary TMP is now informed of the transaction and returns an ‘FeOk’ response message 208 to the Application Requester process.

[0084] Thus, the message traffic is substantially reduced in this phase compared to the Phase 1 Flush phase in the Full Broadcast protocol. Message traffic is confined to the registered and participating CPUs, the Broadcast Owner and the primary TMP CPU.

[0085] Phase 2 (Lock Release) Operation (Registration Protocol)

[0086] The Phase 2 operation, shown in FIG. 4D, starts with the primary TMP process making a ‘DoPhase2’ call 220 to its TmfLib. The TmfLib in the primary TMP CPU sends a ‘Phase2Needed’ message 222 to the BOwn CPU, which is the same CPU. The BOwn CPU next broadcasts a ‘DoPhase2’ message 224 to all registered and participating CPUs, except the TMP CPUs. When received, each CPU requests that its involved resource managers (DP2s) perform a Phase2 (lock release) 226, and then delete its DpCross links and the LibTcb in the resource manager CPU (i.e., DP2 CPU).

[0087] A ‘DataAck’ message 228 is returned to the BOwn CPU by each recipient CPU of a ‘DoPhase2’ message, which causes the BOwn CPU to broadcast a message ‘DoPhase2’ message 230 to the TMP CPUs. When received, each TMP CPU tells its involved resource managers (DP2 CPUs) to do a Phase2 (lock release) 232 and then deletes its DPCross links. The backup TMP CPU also deletes its LibTcb. A ‘DataAck’ message 234 is returned by the backup TMP CPU which causes the primary TMP CPU to delete its LibTcb. The TmfLib in the primary TMP CPU then queues a ‘TxPhase2 GetWork’ message for the TMP informing the primary TMP CPU that the transaction is completed in the TMF Library.

[0088] Thus, the Phase 2 operation, according to the registration protocol, is performed in two sub-phases. In the first sub-phase, the registered and participating resource managers perform a lock release and in the second phase, the primary and backup TMP CPUs perform a lock release.

[0089] Application-Initiated Begin Transaction

[0090] FIG. 5A sets forth a timing chart for a Application-Initiated Begin Transaction operation. As described above, an Application process on a CPU in the cluster can start a transaction. If the Application process exports a transaction to another node, then the primary TMP (TMP-P) process will initiate the TMP-owned transaction on that node.

[0091] The process of beginning a transaction, for a Transaction Monitor Facility (TMF) Application, involves reliably establishing a transaction in the cluster. Each cluster that is involved in the transaction has a TMF Library and a PIO transport facility. The CPU that sends a packet sent by its PIO transport facility receives a ‘DataAck’ when the CPU or CPUs to which the packet was sent has guaranteed to respond to the contents of the message or if the CPU is down.

[0092] The first event in the operation is the Application making a ‘BeginTransaction’ call 240 to the TmfLib on the CPU of the Application requester. The TmfLib responds to the call by creating a new LibTcb and searching for an empty slot in the SlotTable. If an empty entry is found, the SlotTable entry is updated with the LibTcb's address, the transaction's sequence number (from the SlotTable entry number) and an Epoch Number of 0.

[0093] The BOwn CPU (the CPU on which the Application process is running) then broadcasts, via its PIO facility, a ‘BeginTmpPair’ message 242 to the TMP CPUs. When received, each TMP CPU creates a LibTcb for the transaction in the copy of the BOwn CPU's SlotTable that exists in the TMP CPUs. A ‘DataAck’ message 244 (again via the PIO facility) is returned by the TMP CPUs causing the TmfLib to update the BOwn CPUs LibTcb state fields and the BOwn CPU's state machine to being for the transaction. An ‘FeOK’ response message 246 is returned to the Application requester process, indicating a successful completion of the Begin transaction request.

[0094] Application-Initiated Transaction—Phase 2

[0095] As shown in FIG. 5B, to perform a Phase 2 operation, the primary TMP process makes a ‘DoPhase2’ call 250 into its TmfLib, which sends a ‘Phase2Needed’ message 252 to the BOwn CPU.

[0096] The BOwn CPU then broadcasts a ‘DoPhase2’ message 254 to all registered and participating CPUs, except both of the TMP CPUs and the BOwn CPU. When this message is received, each CPU requests that its involved resource manager processes perform a Phase2 (lock release) operation 256 and delete its DpCross links and the LibTcb in the resource manager CPUs.

[0097] Each CPU receiving the ‘DoPhase2’ message returns a ‘DataAck’ message 258 to the BOwn CPU, which causes BOwn CPU to broadcast a ‘DoPhase2’ message 260 to the TMP CPUs and itself. When received, each TMP CPU requests that its involved resource managers (DP2s) perform a Phase2 (lock release) 262, and the involved resource manager (DP2) CPUs then delete their DPCross links. Each TMP CPU deletes its LibTcb for the transaction being completed and returns a ‘DataAck’ message 264 to the BOwn CPU. This causes the BOwn CPU to delete the LibTcb in the BOwn CPU for the transaction.

[0098] The TmfLib in the primary TMP CPU then informs the TMP process that the transaction is completed in the TMF Library by queuing a ‘TxPhase2 GetWork’ request for the TMP process.

[0099] Although the present invention has been described in considerable detail with reference to certain preferred versions thereof, other versions are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the preferred versions contained herein.

Claims

1. A method of performing a transaction in a processing cluster having a plurality of CPUs, one of the CPUs hosting a transaction monitor, at least one CPU hosting a resource manager, each of the CPUs including volatile storage and having access to stable storage, the method comprising: beginning a transaction by the transaction monitor; and performing the following steps if no CPU failure occurs: registering one or more resource managers to participate in the transaction, the CPUs hosting the participating resource managers being participating CPUs; performing work required by the transaction by the one or more participating resource managers, including obtaining any locks needed by the transaction; upon completion of the transaction work by the participating resource managers, performing a phase 1 operation in which a flush request is broadcast to each participating CPU, the flush request instructing each participating CPU to send a response indicating that information in volatile storage is flushed to stable storage; upon completion of the phase 1 operation, requesting a phase 2 operation by broadcasting a lock-release request to each participating CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction in response to the phase 2 request; and after releasing all locks acquired for the transaction, informing the transaction monitor that phase 2 is completed.

2. A method of performing a transaction as recited in claim 1, wherein the transaction monitor is implemented as a process pair that includes a primary and backup transaction monitor; and further comprising the step of informing the backup transaction monitor of the beginning of the transaction prior to performing the registering step.

3. A method of performing a transaction as recited in claim 1, wherein the transaction monitor is implemented as a process pair that includes a primary and backup transaction monitor; and wherein the primary transaction monitor receives a response that all CPUs are flushed prior to the completion of the phase 1 operation.

4. A method of performing a transaction as recited in claim 1, wherein the transaction monitor is implemented as a process pair that includes a primary and backup transaction monitor; wherein the primary transaction monitor receives a message requesting the transaction monitor process pair to release locks, after all participating CPUs, except the CPUs hosting the transaction monitor process pair, have completed a phase 2 lock release operation; and further comprising the step of performing, in response to the message, a phase 2 lock release operation by the transaction monitor process pair.

5. A method of performing a transaction as recited in claim 1, wherein the resource manager is implemented as a process pair that includes a primary and backup resource manager; and wherein the step of registering one or more resource managers to participate in the transaction includes registering the primary and backup resource manager.

6. A method of performing a transaction as recited in claim 1, wherein the resource manager is implemented as a process pair that includes a primary and backup resource manager; and wherein the step of performing a phase 1 flush operation includes performing a phase 1 flush operation by each of the primary and backup resource managers.

7. A method of performing a transaction as recited in claim 1, wherein the resource manager is implemented as a process pair that includes a primary and backup resource manager; and wherein the step of releasing any locks includes releasing, by each of the primary and backup resource managers, any locks acquired by the primary and backup resource managers.

8. A method of performing a transaction as recited in claim 1, the method further comprising: if a failure occurs during registering, broadcasting a begin transaction message to all CPUs, except the CPU hosting the transaction monitor; performing work required by the transaction by the one or more resource managers working on behalf of the transaction, including obtaining any locks needed by the transaction; upon completion of the transaction work by the resource managers working on behalf of the transaction, performing a phase 1 operation in which a flush request is broadcast to each CPU, the flush request instructing each CPU working behalf of the transaction to send a response indicating that information in volatile storage is flushed to stable storage and instructing CPUs not working on behalf of the transaction to so indicate; upon completion of the phase 1 operation, requesting a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction; and after releasing all locks acquired for the transaction, informing the transaction monitor that phase 2 is completed.

9. A method of performing a transaction as recited in claim 1, the method further comprising: if a failure occurs during a phase 1 operation, performing a phase 1 operation in which a flush request is broadcast to each CPU, the flush request instructing each CPU working behalf of the transaction to send a response indicating that information in volatile storage is flushed to stable storage and instructing CPUs not working on behalf of the transaction to so indicate; upon completion of the phase 1 operation, requesting a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction; and after releasing all locks acquired for the transaction, informing the transaction monitor that the transaction is completed.

10. A method of performing a transaction as recited in claim 1, the method further comprising: if a failure occurs during a phase 2 operation, requesting a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction; and after releasing all locks acquired for the transaction, informing the transaction monitor that the transaction is completed.

11. A method of performing a transaction in a processing cluster having a plurality of CPUs, one of the CPUs hosting a transaction monitor, at least one CPU hosting a resource manager, each of the CPUs including volatile storage and having access to stable storage, the method comprising: beginning a transaction by the transaction monitor; broadcasting a begin transaction message to all CPUs, except the CPU hosting the transaction monitor; performing work required by the transaction by the one or more resource managers working on behalf of the transaction, including obtaining any locks needed by the transaction; upon completion of the transaction work by the resource managers working on behalf of the transaction, performing a phase 1 operation in which a flush request is broadcast to each CPU, the flush request instructing each CPU working behalf of the transaction to send a response indicating that information in volatile storage is flushed to stable storage and instructing CPUs not working on behalf of the transaction to so indicate; upon completion of the phase 1 operation, requesting a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction; and after releasing all locks acquired for the transaction, informing the transaction monitor that phase 2 is completed.

12. A method of performing a transaction in a processing cluster having a plurality of CPUs, one of the CPUs hosting a transaction monitor, at least one CPU hosting an application program and a resource manager, each of the CPUs including volatile storage and having access to stable storage, the method comprising: beginning a transaction by the application program; informing the transaction monitor that a transaction is begun; and performing the following steps if no CPU failure occurs: registering one or more resource managers to participate in the transaction, the CPUs hosting the participating resource managers being participating CPUs; performing work required by the transaction by the one or more participating resource managers, including obtaining any locks needed by the transaction; upon completion of the transaction work by the participating resource managers, performing a phase 1 operation in which a flush request is broadcast to each participating CPU, the flush request instructing each participating CPU to send a response indicating that information in volatile storage is flushed to stable storage; upon completion of the phase 1 operation, requesting a phase 2 operation by broadcasting a lock-release request to each participating CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction in response to the phase 2 request; and after releasing all locks acquired for the transaction, informing the transaction monitor that phase 2 is completed.

13. A method of performing a transaction as recited in claim 12, wherein the transaction monitor is implemented as a process pair that includes a primary and backup transaction monitor; and further comprising the step of informing the backup transaction monitor of the beginning of the transaction prior to performing the registering step.

14. A method of performing a transaction as recited in claim 12, wherein the transaction monitor is implemented as a process pair that includes a primary and backup transaction monitor; and wherein the primary transaction monitor receives a response that all CPUs are flushed prior to the completion of the phase 1 operation.

15. A method of performing a transaction as recited in claim 12, wherein the transaction monitor is implemented as a process pair that includes a primary and backup transaction monitor; wherein the primary transaction monitor receives a message requesting the transaction monitor process pair to release locks, after all participating CPUs, except the CPUs hosting the transaction monitor process pair, have completed a phase 2 lock release operation; and further comprising the step of performing, in response to the message, a phase 2 lock release operation by the transaction monitor process pair.

16. A method of performing a transaction as recited in claim 12, wherein the resource manager is implemented as a process pair that includes a primary and backup resource manager; and wherein the step of registering one or more resource managers to participate in the transaction includes registering the primary and backup resource manager.

17. A method of performing a transaction as recited in claim 12, wherein the resource manager is implemented as a process pair that includes a primary and backup resource manager; and wherein the step of performing a phase 1 flush operation includes performing a phase 1 flush operation by each of the primary and backup resource managers.

18. A method of performing a transaction as recited in claim 12, wherein the resource manager is implemented as a process pair that includes a primary and backup resource manager; and wherein the step of releasing any locks includes releasing, by each of the primary and backup resource managers, any locks acquired by the primary and backup resource managers.

19. A method of performing a transaction as recited in claim 12, the method further comprising: if a failure occurs during registering, broadcasting a begin transaction message to all CPUs, except the CPU hosting the transaction monitor; performing work required by the transaction by the one or more resource managers working on behalf of the transaction, including obtaining any locks needed by the transaction; upon completion of the transaction work by the resource managers working on behalf of the transaction, performing a phase 1 operation in which a flush request is broadcast to each CPU, the flush request instructing each CPU working behalf of the transaction to send a response indicating that information in volatile storage is flushed to stable storage and instructing CPUs not working on behalf of the transaction to so indicate; upon completion of the phase 1 operation, requesting a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction; and after releasing all locks acquired for the transaction, informing the transaction monitor that phase 2 is completed.

20. A method of performing a transaction as recited in claim 12, the method further comprising: if a failure occurs during a phase 1 operation, performing a phase 1 operation in which a flush request is broadcast to each CPU, the flush request instructing each CPU working behalf of the transaction to send a response indicating that information in volatile storage is flushed to stable storage and instructing CPUs not working on behalf of the transaction to so indicate; upon completion of the phase 1 operation, requesting a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction; and after releasing all locks acquired for the transaction, informing the transaction monitor that phase 2 is completed.

21. A method of performing a transaction as recited in claim 12, the method further comprising: if a failure during a phase 2 operation, requesting a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; releasing any locks acquired for the transaction while performing work required by the transaction; and after releasing all locks acquired for the transaction, informing the transaction monitor that phase 2 is completed.

22. A processing cluster comprising: a stable storage system; a plurality of CPUs, each CPU including a volatile memory in which data and programs reside, and configured to access the stable storage system, wherein a designated one of the CPUs hosts a transaction monitor program and at least one CPU hosts a resource manager program, wherein the transaction monitor is configured to begin a transaction, and if no CPU failure occurs: register one or more resource managers to participate in the transaction, the CPUs hosting the participating resource managers being participating CPUs; cause the one or more participating resource managers to perform work required by the transaction, including obtaining any locks needed by the transaction; upon completion of the transaction work by the participating resource managers, cause a phase 1 operation to be performed in which a flush request is broadcast to each participating CPU, the flush request instructing each participating CPU to send a response to the transaction monitor indicating that information in volatile storage is flushed to stable storage; upon completion of the phase 1 operation, cause a phase 2 operation by broadcasting a lock-release request to each participating CPU, except the CPU hosting the transaction monitor, any locks acquired for the transaction while performing work required by the transaction being released in response to the phase 2 request; and receive a message indicating that phase 2 is completed.

23. A method of performing a transaction as recited in claim 22, wherein the transaction monitor is further configured to: if a failure occurs during registering, broadcast a begin transaction message to all CPUs, except the CPU hosting the transaction monitor; cause the one or more resource managers to perform work required by the transaction on behalf of the transaction, including obtaining any locks needed by the transaction; upon completion of the transaction work by the resource managers working on behalf of the transaction, cause a phase 1 operation to be performed in which a flush request is broadcast to each CPU, the flush request instructing each CPU working behalf of the transaction to send a response indicating that information in volatile storage is flushed to stable storage and instructing CPUs not working on behalf of the transaction to so indicate; upon completion of the phase 1 operation, cause a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; any locks acquired for the transaction while performing work required by the transaction being released in response to the release-lock request; and receive a message that phase 2 is completed.

24. A method of performing a transaction as recited in claim 22, wherein the transaction monitor is further configured to: if a failure occurs during a phase 1 operation, cause a phase 1 operation to be performed in which a flush request is broadcast to each CPU, the flush request instructing each CPU working behalf of the transaction to send a response indicating that information in volatile storage is flushed to stable storage and instructing CPUs not working on behalf of the transaction to so indicate; upon completion of the phase 1 operation, causing a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; any locks acquired for the transaction while performing work required by the transaction being released in response to the release-lock request; and receive a message that phase 2 is completed.

25. A method of performing a transaction as recited in claim 22, wherein the transaction monitor is further configured to: if a failure occurs during a phase 2 operation, cause a phase 2 operation in which a release-lock request is broadcast to each CPU, except the CPU hosting the transaction monitor; any locks acquired for the transaction while performing work required by the transaction being released in response to the release-lock request; and receive a message that phase 2 is completed.