Patent Application
20080222642
The present invention relates generally to data management and, more specifically, to techniques for managing clusterware-managed resources having dynamic attributes.
A clustered computing system is a collection of interconnected computing elements that provide processing to a set of client applications and, to a large extent, can be viewed as though the computing elements are a single computer. Each of the computing elements is referred to as a node. A node may be a computer interconnected to other computers, or a server blade interconnected to other server blades in a grid. A group of nodes in a clustered computing system that have shared access to storage (e.g., have shared disk access to a set of disk drives or other non-volatile storage) and that are connected via interconnects is referred to herein as a cluster.
A clustered computing system is used to host clustered servers. Resources from multiple nodes in a clustered computing system can be allocated to running a server's software. Each allocation of the resources of a particular node for the server is referred to herein as a server instance, or simply an instance. A database server can be clustered, where the server instances may be collectively referred to as a cluster. Each instance of a database server facilitates access to the same database, in which the integrity of the data is managed by a global lock manager. The collection of server instances, and the resources used by the servers, are typically managed by a “clusterware” software application.
Resources managed by clusterware 102 can have certain attributes that may need to be changed at any point in time, and where values for such attributes may not be known at the time of configuration of the clusterware. If an attribute of an online resource needs to be modified, the system administrator has to stop the resource, modify the particular attribute and start the resource again. Thus, this approach involves stopping the resource and all the other resources that may depend on the resource, which may potentially lead to relatively long periods of outage time for numerous resources. Therefore, this approach negates the objective of a high-availability system to always provide availability of the cluster resources that the clusterware 102 manages.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
Allowing for dynamic resource attributes, i.e., resource attributes that may change dynamically while the resource is in use, provides for dynamic changes to the manner in which such resources are managed. For example, cluster resources having dynamic attributes, when such attributes are dynamically changed, may trigger clusterware that is managing such resources to begin managing the resources differently, i.e., according to a different resource management behavior.
Management of dynamic resource attributes by clusterware involves a new entry point to clusterware agent modules, through which resource-specific user-specified instructions for discovering new values for resource attributes are invoked. For example, a clusterware policy manager may know ahead of time that a particular resource has dynamic attributes or may be notified when a resource's dynamic attribute has changed and, periodically or in response to the notification, the policy manager requests that the agent invoke the particular resource-specific instructions for discovering new values for one or more resource attributes for the particular resource.
Management of dynamic resource attributes by clusterware involves a second new entry point to clusterware agent modules, through which resource-specific user-specified instructions for performing a user-specified action in response to the new attribute value are invoked. Hence, in response to a particular resource attribute being changed to a new value, a user can specify any type of action in response to the attribute value change.
These new clusterware agent entry points enable dynamic resource attributes, discovery of attribute value changes, and responsive user-specified actions. During the majority of this process, the resource does not need to be stopped and remains online.
The techniques described herein allow for greater flexibility in defining and configuring resources, in that a resource can have dynamic attributes whose values can be changed and discovered at any time. Consequently, a system administrator need not be burdened with monitoring and changing attribute values that are changing dynamically at run time, so these techniques help save valuable manpower that may otherwise be needed to constantly monitor clusterware and the applications the clusterware monitors. Furthermore, these techniques provide better fault tolerance and ensure continuous availability for resources that have dynamically changing attributes.
As discussed, clusterware 102 is software that allows clusters of networked computers to operate or be controlled as if they are one, where one function of clusterware 102 is to manage applications 106 (
A non-limiting example of a resource profile for a virtual IP address (resource type “VIP”) named “ora.vip” is as follows:
In the foregoing example, the resource's discoverable dynamic attributes are identified with a preceding ‘$’ symbol. However, other symbols, numbers, or characters could be used for the purpose of distinguishing a resource's dynamic attributes from the resource's static attributes within a resource profile.
Resources' discoverable dynamic attributes can have a time period associated with it, indicating how long that particular attribute value is valid. Once that time expires, the policy manager 202 (
To assist the clusterware 102 in managing resources 214, the clusterware 102 comprises a policy manager 202. For example, policy manager 202 may be implemented as a daemon process running in clusterware 102. However, the policy manager architecture with which embodiments of the invention are implemented may vary from implementation to implementation. Policy manager 202 manages each resource 214a, 214b-214n based on policy attributes 218 specified in a corresponding resource profile 206. Stated otherwise, policy manager 202 makes decisions for the cluster regarding cluster resources, such as what to do when a node crashes. For a non-limiting example, a resource policy attribute 218 associated with a particular type of resource 214a, such as a database server, may specify that at least one instance of the database server must always run in the cluster and that the database server instances are configured to operate in a failover manner in the event of a failure of node on which a server is executing.
Generally, the policy attributes 218 of resource profile 206 provide a source of information for policy manager 202 to use in making decisions about a corresponding resource, in the context of resource management goals of clusterware 102, e.g., a high availability goal of always having cluster resources available for certain numbers or types of applications to execute and operate correctly on the cluster. For example, the policy attributes 218 may characterize one or more rules that the policy manager 202 evaluates in response to discovering a resource attribute change via discovery 210 entry point, in response to which the policy manager 202 can cause performance of a certain action associated with clusterware 102 management of resources 214. Any action associated with the clusterware 102 management of resources 214 is in addition to any user-specified action performed, via attribute changed 212 entry point, in response to discovering a resource attribute change via discovery 210 entry point. The policy manager 202 does not understand the different specific resources 214 in depth, so the policy manager 202 makes decisions based on the policy attributes 218 at a relatively abstract level. For example, the policy manager 202 does not know how to communicate with various applications running on the cluster and therefore using cluster resources, such as a database server, an application server, and a web server.
Hence, clusterware 102 further comprises one or more agent 204. Each agent 204 is specific to a particular type of resource managed by the clusterware 102, i.e., is resource type-specific. Thus, each agent 204 is configured to assist with managing a corresponding particular type of resource. For example, each different type of resource from resources 214a, 214b-214n is associated with a different corresponding agent 204. Each agent 204 generally comprises two different sets of instructions: (a) instructions for clusterware management of resources, generally, and (b) user-specified instructions for management of particular resources, specifically. It is possible that each different type of resource is different and, therefore, has different operational behavior. Thus, each agent 204 includes some resource-specific user-specified code about how to manage the resource, such as how to start the resource, how to stop the resource, and how to check the resource state. These types of actions are usually specific to the type of resource and vary from resource type to resource type.
Consequently, agent 204 comprises programmatic entry points 208, which provide programmatic means (e.g., application programming interfaces, API's) for invoking user-specific code for managing and otherwise affecting the corresponding resource. For example, each of the following actions, (a) start the resource, (b) stop the resource, and (c) check the resource state, are associated with corresponding entry points 208 in agent 204, at which the user-specified instructions associated with the particular action can be invoked to cause performance of the particular action. In this context, the user that specifies the user-specified code for performing various actions associated with a particular resource may be, for non-limiting examples, a developer who developed the particular resource or a system administrator who is familiar with how the particular resource operates and is to be managed. Agent 204 has access to resource profile 206, for accessing the resource-specific attributes 216 associated therewith, for use in performing corresponding actions in managing corresponding resources 214.
As mentioned, each agent 204 manages the corresponding resources through user plugged-in code called as entry points. For example, if an entity decides to start a resource, the entity tells the resource type-specific agent to start the resource, and then the agent invokes the START entry point (which is implemented by users for that specific resource). To develop agents, users simply have to implement their entry points and link with an AgentFramework library provided by clusterware 102.
For example, if a user wants to manage an IP address using clusterware 102, the user develops an agent which manages the IP address using a file similar to the following example, which includes start, stop, monitor, and discovery entry points:
Agents 204 for different resource types are developed by writing the code for the entry points and linking the entry points with a certain clusterware 102 library. Thus, agents 204 contains two parts: clusterware library plus user code (in form of entry points). The policy manager 202 decides when to call these entry points. Thus, whenever an attribute needs to be discovered, policy manager 202 calls the discovery entry point 210 through an appropriate resource type-specific agent 204.
According to an embodiment, agent 204 comprises a discovery 210 entry point. Discovery 210 is an entry point 208 for invoking resource type-specific code which, when invoked, executes to discover the current value for one or more attributes associated with a particular resource type. For a non-limiting example, a user develops a set of instructions which, when executed, cause performance of one or more operations for discovering a new IP address for a virtual IP address resource. Through a programmatic interface, the user inserts or references the set of “discovery” instructions within the agent 204, so that the instructions are invokable at the discovery 210 programmatic entry point of agent 204. Because the nature and behavior of resources vary from resource type to resource type, what operations need to be performed to discover a particular resource's new attribute value will also likely vary from resource type to resource type. Hence, the discovery 210 entry point effectively extends the agent 204 functionality to include the capability to invoke resource type-specific instructions for discovering a change to the value of a resource's attribute.
Dynamic resource attributes may change in multiple scenarios, as follows. With a manual intervention, for example, a system administrator may change an attribute of a resource (e.g., a database's uniqueName attribute). The agent 204 can inform the policy manager 202 about the change through the discovery 210 entry point. With a semi-automatic resource attribute change, for example, at the time of initial configuration of the clusterware 102 certain attributes of a resource may not have values set. The policy manager 202 notices that certain attribute values are not set and starts a process to request the agent 204 to query the values of the particular attributes to determine if the values are now set, and to return such values. With an automatic resource attribute change, for example, an attribute value may be invalidated after a certain time interval. Therefore, the attribute value needs to be queried as a result of the invalidation. For example, the agent 204 may inform the policy manager 202 of the expiration of a lease for a certain attribute and the policy manager 202 in turn requests the agent 204 query the new value of the certain attribute to ensure that the policy manager 202 is aware of the most current attribute information. With scheduled resource attribute discovery, discoverable attributes can have an associated schedule, with which users can specify specific intervals of time when an attribute value needs to be discovered. For example, a command such as or similar to the one that follows could be used to specify an attribute discovery schedule: $fooAttribute={“defaultValue”, “6 AM, 7 PM”}. The foregoing ways in which a dynamic resource attribute may change (i.e., manual, semi-automatic, and automatic) and/or be scheduled for discovery are resource attribute discovery triggers because each could trigger, directly or indirectly, execution of a process for discovering the value of a dynamic attribute of a resource as described herein.
A dynamic resource refers to a resource having one or more attributes that are dynamic in the sense that the value for each of the one or more attributes may change. Such changes may occur at runtime when the resource is in use, or otherwise. The techniques described herein enable a resource to be a dynamic resource. According to an embodiment, syntactic means are provided to identify a particular resource as being a dynamic resource. For example, the name of a dynamic resource may be preceded by a certain character, i.e., a syntactic identifier such as a ‘$’ or ‘&’, in a corresponding resource profile, thereby indicating that at least one dynamic attribute is associated with the particular resource. Based on the techniques described herein, identifying a resource as a dynamic resource effectively identifies the resource as having one or more discoverable attributes, whereby the agent 204 can call the discovery 210 entry point for that specific resource to discover the appropriate value for discoverable attributes, by executing the corresponding user-provided discovery 210 entry point code.
Generally, who or what needs to be notified when a resource attribute changes depends on the nature and type of attribute that changed. For example, in addition to the agent 204, the resource itself may need to know and/or other resources may need to know, such as resources that are dependent on the changed resource. According to an embodiment, agent 204 comprises an attribute changed 212 entry point. Attribute changed 210 is an entry point 208 for invoking resource type-specific code which, when invoked, executes to perform a user-specified action in response to a change to a change in value for one or more attributes associated with a particular resource type. For a non-limiting example, a user develops a set of instructions which, when executed, cause performance of essentially any action in response to the change in resource attribute value. Through a programmatic interface, the user inserts or references the set of “resource attribute changed” instructions within the agent 204, so that the instructions are invokable at the attribute changed 212 programmatic entry point of agent 204. Because the nature and behavior of resources vary from resource type to resource type and the type of action desired in response to a resource attribute change is relatively unlimited, what operations need to be performed in response to a particular resource's changed attribute value will also likely vary from resource type to resource type and possibly from resource attribute to resource attribute. Hence, the attribute changed 212 entry point effectively extends the agent 204 functionality to include the capability to invoke resource type-specific instructions for performing one or more user-defined action in response to discovering a change to the value of a resource's attribute.
Following is example pseudo-code which shows usage of the discovery 210 entry point (i.e., “Discovery”) and the attribute changed 212 (i.e., “ResAttrModified”) entry point, as well as Start, Stop, and Check entry points:
The foregoing example implementation of the discovery 210 entry point utilizes a ‘switch’ command to discover the values for the attributes, ‘Attr1’ and ‘Attr2’, via functions ‘discover_attr1( )’ and ‘discover_attr2( )’, respectively. These functions comprise the resource type-specific user-defined instructions for discovering the values for the respective resource attributes, however is necessary for the particular resource type. A call to the discovery 210 entry point would need to include any input parameters and types that may be specified in the function definition and/or switch command, and the functions would return any return values and types that are specified, if applicable.
The foregoing example implementation of the attribute changed 212 entry point (‘ResAttrModified’) utilizes a ‘switch’ command to perform certain actions in response to a new attribute value, ‘attributeValue’, for a particular attribute, ‘attributeName’. For attribute ‘Attr1’, the responsive action is to send a notification to ‘clientx’ with a payload comprising the ‘attributeName’ to notify the client that the particular named attribute has changed. For attribute ‘Attr2’, the responsive action is to tune the resource, with a payload comprising the ‘attributeName’ and the current corresponding ‘attributeValue’. The ‘send_notification_to_client’ and ‘TuneResource’ functions comprise the resource type-specific user-defined instructions for performing certain actions in response to a new attribute value, based on particular users' desires for the particular resource type. Similarly to the discovery 210 entry point, a call to the ‘ResAttrModified’ functions would need to include any input parameters and types that may be specified in the function definition and/or switch command, and the functions would return any return values and types that are specified, if applicable.
At block 302, cause a call to a first set of one or more instructions which, when executed in response to the call, cause performance of one or more operations for discovering a new value of a particular attribute of a particular resource managed by a clusterware application. For example, policy manager 202 (
Any number of events could trigger causing a call to the discovery 210 (
In some scenarios, a change to an attribute's value may be temporary, whereby the attribute returns to its configured value, for example, upon completion of an operation using the resource, upon inactivating the resource, in response to a change to another interdependent resource and/or resource attribute, and the like. In such a scenario, the new attribute value is not stored persistently in non-volatile memory, rather the new attribute value is maintained in volatile memory for some length of time, and is flushed from this memory at some point in time. However, according to an embodiment, an attribute's new value is persistently stored in non-volatile memory and, therefore, is made relatively “permanent”, i.e., until the new value is overwritten in non-volatile memory by another new value. For example, in response to discovering a change to a resource attribute value via the discovery 210 (
At block 304, one or more resource policies associated with the particular resource are evaluated based on the new value of the particular attribute, and a responsive action is caused to be performed if required by the one or more resource policies. For example, policy manager 202 (
At block 306, cause a call to a second set of one or more instructions which, when executed in response to the call, cause performance of a certain action in response to the new value of the particular attribute of the particular resource managed by the clusterware application, where the new value was discovered via block 302. For example, policy manager 202 of clusterware 102 (
Assume each clusterware-managed resource is configured with a resource profile with the attributes required to define the resource and the resource's high availability (“HA”) requirements. However, the attributes need not be populated at clusterware configuration time. For example, consider a Database Resource, whose profile will consist of a list of attribute-value pairs:
(a) Resource Name,
(b) Resource Type,
(c) Database Unique Name,
(d) Timeout Value.
The attributes (a), (b) and (d) may be available at configuration time, but the value of the attribute Database Unique Name is constantly changing, or dynamic in nature. Thus, the attributes (a), (b) and (d) are populated by the policy manager 202 (
The individual agent 204 (
Computer system 400 may be coupled via bus 402 to a display 412, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 414, including alphanumeric and other keys, is coupled to bus 402 for communicating information and command selections to processor 404. Another type of user input device is cursor control 416, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 404 and for controlling cursor movement on display 412. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
The invention is related to the use of computer system 400 for implementing the techniques described herein. According to one embodiment of the invention, those techniques are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. Such instructions may be read into main memory 406 from another machine-readable medium, such as storage device 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.
The term “machine-readable medium” as used herein refers to any medium that participates in providing data that causes a machine to operation in a specific fashion. In an embodiment implemented using computer system 400, various machine-readable media are involved, for example, in providing instructions to processor 404 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 410. Volatile media includes dynamic memory, such as main memory 406. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 402. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
Common forms of machine-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.
Various forms of machine-readable media may be involved in carrying one or more sequences of one or more instructions to processor 404 for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 400 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by main memory 406 may optionally be stored on storage device 410 either before or after execution by processor 404.
Computer system 400 also includes a communication interface 418 coupled to bus 402. Communication interface 418 provides a two-way data communication coupling to a network link 420 that is connected to a local network 422. For example, communication interface 418 may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 418 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 418 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 420 typically provides data communication through one or more networks to other data devices. For example, network link 420 may provide a connection through local network 422 to a host computer 424 or to data equipment operated by an Internet Service Provider (ISP) 426. ISP 426 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 428. Local network 422 and Internet 428 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 420 and through communication interface 418, which carry the digital data to and from computer system 400, are exemplary forms of carrier waves transporting the information.
Computer system 400 can send messages and receive data, including program code, through the network(s), network link 420 and communication interface 418. In the Internet example, a server 430 might transmit a requested code for an application program through Internet 428, ISP 426, local network 422 and communication interface 418.
The received code may be executed by processor 404 as it is received, and/or stored in storage device 410, or other non-volatile storage for later execution. In this manner, computer system 400 may obtain application code in the form of a carrier wave.
In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
