Patent Application
20070185872
The present invention relates to locking resources, and more particularly to aggregate locking.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
Multiple processes running on multi-processing systems may access shared resources, such as disk blocks. Some of these shared resources may be accessed by only one process at a time, while others may be accessed concurrently by multiple processes. Consequently, “synchronization mechanisms” have been developed to control access by multiple processes to shared resources. The synchronization mechanism grants locks to processes. Locks grant to holders of the locks the right to access a particular resource in a particular way. Once a lock is granted to a process, the process holds or owns the lock until the lock is relinquished, revoked, or otherwise terminated. Locks are represented by data structures such as semaphores, read/write latches, and condition variables. There are many types of locks. Some types of locks allow shared resources to be shared by many processes concurrently (e.g. shared read lock), while other types of locks prevent any type of lock from being granted on the same resource (exclusive write lock).
The entity responsible for granting locks is referred to herein as a lock manager. In a single node multi-processing system, a lock manager is typically a software component executed and invoked by processes on the node accessing a shared resource.
In contrast to a single node system, a multi-node system consists of network of computing devices or “nodes, each of which may be a multi-processing system. Each of the nodes can access a set of shared resources. Multi-node systems use synchronization mechanisms, referred to as global synchronization mechanisms, to control access to the set of shared resources by nodes in the multi-node system.
A global lock mechanism includes a global-lock manager that is responsible for issuing locks to processes on the multi-node system. In order for a node to access a shared resource, it is granted a “global lock” by a global lock manager. A global lock is a lock that can be granted by a global lock manager on a node in a multi-node system to one or more processes on another node to coordinate access to the shared resources among the processes executing on any node in a multi-node system.
A type of global lock manager is a distributed lock manager, which is comprised of local lock managers that are distributed on the nodes of a multi-node system, with one or more of the local lock managers running on each node in a multi-node system. Each lock manager is responsible for coordinating the global locks for processes on the local lock manager's node. A local lock manager is referred to as the local lock manager with respect to the node on which it resides; the node and a process running on the node are referred to as a local node and local process with respect to the local lock manager and the node.
A local lock manager residing on a node issues global locks to lock managers on the other nodes and to processes running on the same node as the local lock manager. A process needing a global lock on a resource managed by a non-local lock manager requests the global lock from its local lock manager. If a local lock manager already holds a compatible global lock, the local lock manager issues a global lock to the local process. If the local lock manager does not hold a compatible global lock, the local lock manager first obtains one from the non-local lock manager. Once obtained, the local lock manager issues the global lock to the local process.
For convenience of expression, the global locks issued by local lock managers to local processes are referred to herein as local locks. Thus, a local lock manager obtains a global lock from another lock manager and issues compatible local locks to local processes.
Also, for convenience of expression, nodes are described herein as performing actions and as being the object of actions. However, this is just a convenient way of expressing that one or more processes on a node are performing an action or is the object of an action. For example, a lock manager requesting, obtaining, and issuing a global lock or local lock may be described as a node requesting, obtaining, and issuing a global lock or local lock.
Acquiring global locks can be more expensive to acquire than acquiring only local locks. This is because a global lock may entail inter-node communication and interaction between a local lock and a local lock manager on another node.
Such interaction can entail a particularly expensive form of an operation referred to as a ping. A ping occurs when the version of a resource that resides in the cache of one server must be supplied to the cache of a different server. Thus, a ping occurs when, after a node A modifies resource x in its cache, another node B requires resource x.
Cache Fusion
One way of performing a ping is referred as cache fusion. Transferring cache copies of a resource between nodes is performed to speed up locking mechanisms.
Master node 102 receives the request and determines whether any other node holds an incompatible lock on the shared resource. In the simple case where no node holds an incompatible lock on the shared resource, the master node grants the lock directly to the requesting node. If the master node itself holds an incompatible lock on the shared resource, then the master node will eventually grant a lock on the shared resource directly to the requesting node. Otherwise, another node (i.e., a holding node 106) holds an incompatible lock on the shared resource.
The master node sends a message to holding node 106 (step 114) indicating that requesting node 104 requests a lock on the particular shared resource for which holding node 106 holds the lock. Holding node 106 grants the lock and may send a copy of the shared resource directly to requesting node 104 (step 116). In some cases, even a compatible lock held by holding node 106 on the shared resource (e.g., requesting node 104 requesting a shared lock on the resource and holding node 106 holds a shared lock on the resource) will trigger an interconnect message from holding node 106 to requesting node 104 because holding node 106 may hold a dirty, or modified, version of the shared resource.
Lastly, once requesting node 104 receives the shared resource and the lock, requesting node 104 notifies master node 102 (step 118) that requesting node 104 has the lock on the shared resource. Therefore, each request for a lock on a shared resource may cause four inter-node messages to be generated. Techniques are thus needed to reduce the cost of acquiring global locks.
One technique to reduce the cost of acquiring global locks is to use a “mastering technique” that assigns a master node to a subset of shared resources based on patterns of access to shared resources. (A master node for a shared resource governs access by other nodes to the shared resource.) For example, if most of the accesses to a portion of shared data are performed by a particular node, that node is assigned as the master node for that portion of the shared data. This reduces the messaging overhead between nodes because less global locks will have to be acquired since the particular node demanded most of the accesses to the portion of shared data. Future accesses to the portion of shared data will only require the granting of local locks with respect to the particular node. However, mastering does not eliminate the cost of executing more instructions to acquire a global lock.
Another technique to reduce the cost of acquiring global locks is to use coarse-grain locking. In this scheme, locks are acquired at a higher level of granularity, such as a table or file, instead of a finer level of granularity, such as a row or a disk block. When a lock is acquired at the higher level of granularity, it is implicitly granted for levels of shared data at a finer level of granularity. For example, if a global lock is acquired for an entire table, individual global locks for the rows or blocks for the table are implied and do not have to be acquired, avoiding the cost of obtaining a global lock for each row and block.
The advantage of this technique is that it does not depend on the assignment of a master node. A significant disadvantage, however, is that this technique can lead to false contention. Specifically, if a node needs to modify a row in a table that has been locked by another node in a conflicting mode, that node must relinquish the lock on the table although the two nodes may be accessing different rows or even different blocks.
Another technique to reduce the cost of acquiring global locks is to use hierarchical locking. In this scheme, locks are first acquired at a higher level in the hierarchy, such as a table. If a global lock is acquired at a higher level in the hierarchy, global locks are implicitly granted at the lower level of the hierarchy. When another node subsequently needs to access data in the lower level of the hierarchy, such as a row or a block, in a conflicting mode, the first node de-escalates its lock and acquires locks at the lower level in the hierarchy.
The disadvantage of this technique is that the cost of obtaining a global lock is inflated and shifted to the requesting node whose lock request triggers the de-escalation. To honor the request, work is performed to acquire global locks for all the shared data at the lower level of the hierarchy. This work is performed despite the requesting node having requested a lock on only a small portion of the shared data.
As clearly shown, techniques are needed to reduce the cost of acquiring global locks that avoid the pitfalls attendant to techniques described above for reducing the cost of global locks.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
A method and apparatus for managing locks in a database system is provided. A first requester node requests a global lock on a first resource and a master node grants the lock on the first resource as well as a group of resources that includes the first resource. Granting a global lock on the group of resources is performed based on the assumption that the first requester will tend to access other resources in the group. The requester node receives a mapping corresponding to the group of resources that may indicate that a global lock held by another node already exists for a second resource in the group. The requester may grant itself a local lock compatible with the lock on the group of resources without notifying the master node.
A second requester node may request a global lock for a particular resource in the group of resources. The first requester node may then relinquish or down convert the lock on the particular resource and grant the lock on the particular resource to the second requester. The first requester may update the first requester node's mapping to indicate that a different node (i.e., the second requester node) holds the lock on the particular resource. If the first requester node has been modified, or dirtied, the particular resource, the first requester node will also send data corresponding to the modification.
A master 102 node is responsible for granting global locks to other nodes for shared resources in a database system. Initially, a node 204 requests a global lock of a particular type for a resource or a group of resources (step 212). The request may include a request for both a global lock on a particular resource and a global lock on a particular group of resources that includes the resource. The request from node 204 may also be a request for a global lock only on either the particular resource or the particular group of resources.
Master 102 sends node 204 a message granting the request for the global lock, such as a shared lock, on at least a group of resources (step 214). If the request from node 204 is for a particular resource, then master 102 determines dynamically (or may have determined statically) that the particular resource is within a particular group of resources (e.g., set of consecutive data blocks). Master 102 sends a mapping corresponding to the group of resources that indicates a lock and lock type for all the resources in the group. If a local process on node 204 requests a global lock on any resource in the group of resources identified in the mapping, wherein the requested global lock is compatible with the global lock on the group of resources, the local lock manager of node 204 simply grants the local process the lock and updates the mapping without having to request master 102 for the lock.
For example, if the global lock for the group of resources is a shared lock, then node 204 will not have to request a shared lock from master 102 for any resources in the group to grant a local shared lock to a local process. In essence, node 204 receives a lock for a plurality of resources in only one message.
The global lock granted for the group of resources may be the same as or different from the global lock for the particular resource. For instance, if node 204 requests an exclusive lock on the particular resource, then master 102 may grant the exclusive lock on the particular resource and grant a shared lock on the group of resources that includes the particular resource. In that case, the mapping itself may also indicate that there is an exclusive lock on the particular resource granted to node 204.
If node 204 initially requests (i.e., step 1) at least a global lock on a resource and another node in the system holds an incompatible lock on the resource, then master 102 grants a global lock on a group of resources that includes the resource and sends the other node a message indicating that node 204 requests a global lock on the resource. The mapping may then indicate, in addition to the type of lock on the group of resource, that an incompatible global lock is held by another node.
Depending on the how the nodes in the database server network are configured, the other node may not grant the global lock on the resource to node 204 until a pre-determined period of time has elapsed, especially if the other node has an exclusive lock on the resource because exclusive locks are relatively “expensive” to obtain, in terms of inter-node messaging. The other node may also be configured to release its global lock on the resource despite the pre-determined period of time, if a certain number of requests for the resource from master 102 have been received.
The other node eventually either down converts the global lock on the resource to a compatible lock mode or relinquishes the lock entirely and sends node 204 a copy of the latest version of the resource, similar to the process described with respect to
The mapping sent to node 204 may also indicate, in addition to a global lock for the group, that a lock is held by another node in the system on a particular resource in the group.
In this example, mapping 300 indicates that a shared global lock 302 has been granted on the group of resources corresponding to mapping 300. Also, mapping 300 indicates that an exclusive global lock 304 is held on the resource corresponding to block 7 (“resource 7”) by another node. Lastly, mapping 300 indicates that node 204 has granted a shared lock to itself (i.e., local lock manager of node 204 granted the shared lock to a local process relative to node 204).
A mapping, such as mapping 300, may maintain other information pertaining to a resource in the group of resources, such as which node in the system has the exclusive lock, for example, and when the lock was granted. Also, although
If requesting node 404 requires a global lock on resource 7 in mapping 300 and the lock is incompatible with the current lock on resource 7 held by holding node 406, then inter-node messages are sent to master 102 and holding node 406 according to the cache fusion protocol. For example, requesting node 404 sends a message to master 102 (step 412) requesting a global lock on resource 7. Master 102 determines which node holds the exclusive lock on resource 7 and then sends a message to holding node 406 (step 414) informing holding node 406 that requesting node 404 requires an exclusive global lock on resource 7. Holding node 406 down converts its global lock to null mode or relinquishes its global lock altogether and sends a message to requesting node 404 (step 416) granting the lock on resource 7. Holding node 406 also sends a copy of the latest version of resource 7 reflecting any modifications to resource 7 that may have occurred. Requesting node 404 informs master 102 (step 418) that requesting node 404 holds the exclusive global lock on resource 7. Master 102 updates its own table of information indicating that requesting node 404 now holds an exclusive lock on resource 7.
In situations where requesting node 404 requests a global lock for a resource in the group of resources identified by mapping 300 and either no other node holds a global lock or no other node holds an incompatible global lock on the resource, multiple inter-node messages do not need to be generated. However, if requesting node 404 wants an exclusive lock on a resource in the group of resources identified by mapping 300 which holds a region lock in shared mode, node 404 expands this global lock into a fusion lock and submits the escalate request to master 102. Requesting node 404 may indicate the lock expansion in mapping 300.
After holding node 506 obtains a mapping 300 corresponding to a group of resources, another node, such as requesting node 504, may request a global lock on a resource that is included in the group of resources corresponding to mapping 300. For instance, suppose requesting node 504 requests an exclusive global lock on a resource associated with block 3 in mapping 300 (i.e., “resource 3”) and holding node 506 holds the group lock in shared mode. Requesting node 504 sends a request for an exclusive global lock on resource 3 to master 102 (step 512). Master 102 determines that holding node 506 has a shared global lock on the group of resources that includes resource 3 based on, e.g., master 102's own mapping.
There are at least two possible cases at this point: 1) holding node 506 either holds an individual shared global lock on resource 3 (i.e., a global lock just on the resource in addition to a group lock) or 2) holding node 506 does not hold an individual global lock on resource 3. If holding node 506 holds an exclusive lock, then master 102 will have the information.
If holding node 506 holds an individual shared global lock on resource 3, then the cache fusion protocol for releasing and granting locks is followed. Specifically, once holding node 506 is notified by master 102 that requesting node 504 requests an exclusive global lock, holding node 506 either down converts its lock on resource 3 from shared mode to a null mode, or relinquishes the lock on resource 3. Holding node 506 then sends node 504 a copy of resource 3 and grants an exclusive global lock on resource 3 to requesting node 504. Requesting node 504 notifies master 102 that requesting node 504 holds an exclusive global lock on resource 3. Mapping 300 on holding node 506 is updated to indicate that an exclusive global lock on resource 3 is held by another node.
If holding node 506 does not hold an individual global lock on resource 3, then master 102 sends holding node 506 a message (step 514) that another node requires an exclusive lock on resource 3. Holding node 506 notifies master 102 that is does not hold an individual global lock (step 516). Master 102 updates its mapping that corresponds to the group of resources that includes resource 3 or master 102 updates another log that indicates requesting node 504 now holds an exclusive global lock on resource 3. Master 102 grants to requesting node 504 the exclusive lock originally requested (step 518).
There may be circumstances where the lock for an entire group of resources, which corresponds to a particular mapping, needs to be upgraded or downgraded to a different lock (e.g., downgrade from an exclusive lock to shared lock), or where the entire mapping needs to be dissolved.
For example, suppose node 204 in
A more common situation may be the occurrence of one or more other nodes requesting multiple incompatible global locks for resources in the group of resources identified in mapping 300. Node 204 may determine that the group lock needs to be downgraded or dissolved, based on one or more factors, such as how many individual locks on resources in the group of resources are held by other nodes, the likelihood of more conflicting requests in the future, and whether node 204 requires locks on any resources in the group. In such a case of a downgrade of the group lock, node 204 notifies master 102 to downgrade a mapping on master 102 corresponding to mapping 300, e.g., from an exclusive lock to a shared lock, or a shared lock to a null lock.
Node 204 may instead notify master 102 to dissolve the group lock. A situation where a dissolution may be required is where the lock mode is being converted to a lock mode of a different type, such as a read-mostly lock. In order to convert to a lock of a different type, the group lock must first be dissolved and then a new lock type is obtained.
Another circumstance that may require the communication of group locks is the failure of the master node. If master 102 fails, then a new master node is chosen among the nodes in the multi-node system. Each non-master node that maintains a mapping of a group of resources sends their respective mappings to the new master node in order for the new master node to be able to grant locks on individual resources and locks on groups of resources.
Embodiments of the invention discussed thus far have been described in a distributed locking environment. However, embodiments of the invention are not limited to a distributed environment. They may also be implemented in a non-distributed locking environment.
Hardware Overview
Computer system 600 may be coupled via bus 602 to a display 612, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to bus 602 for communicating information and command selections to processor 604. Another type of user input device is cursor control 616, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 604 and for controlling cursor movement on display 612. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
The invention is related to the use of computer system 600 for implementing the techniques described herein. According to one embodiment of the invention, those techniques are performed by computer system 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another machine-readable medium, such as storage device 610. Execution of the sequences of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.
The term “machine-readable medium” as used herein refers to any medium that participates in providing data that causes a machine to operation in a specific fashion. In an embodiment implemented using computer system 600, various machine-readable media are involved, for example, in providing instructions to processor 604 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 610. Volatile media includes dynamic memory, such as main memory 606. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
Common forms of machine-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.
Various forms of machine-readable media may be involved in carrying one or more sequences of one or more instructions to processor 604 for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 600 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 602. Bus 602 carries the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received by main memory 606 may optionally be stored on storage device 610 either before or after execution by processor 604.
Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides a two-way data communication coupling to a network link 620 that is connected to a local network 622. For example, communication interface 618 may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 618 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 620 typically provides data communication through one or more networks to other data devices. For example, network link 620 may provide a connection through local network 622 to a host computer 624 or to data equipment operated by an Internet Service Provider (ISP) 626. ISP 626 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 628. Local network 622 and Internet 628 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 620 and through communication interface 618, which carry the digital data to and from computer system 600, are exemplary forms of carrier waves transporting the information.
Computer system 600 can send messages and receive data, including program code, through the network(s), network link 620 and communication interface 618. In the Internet example, a server 630 might transmit a requested code for an application program through Internet 628, ISP 626, local network 622 and communication interface 618.
The received code may be executed by processor 604 as it is received, and/or stored in storage device 610, or other non-volatile storage for later execution. In this manner, computer system 600 may obtain application code in the form of a carrier wave.
In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
