Patent Grant
7340654
This application is related to U.S. patent application Ser. No. 10/871,286, entitled “Job-Centric Scheduling in a Grid Environment” (Inventors: Bingfeng Lu, Jin Song Zhong, and Jason Lam), and U.S. patent application Ser. No. 10/871,502, entitled “Goal-Oriented Predictive Scheduling in a Grid Environment” (Inventors: David Bigagli, James Pang, and Shutao Yuan), each of which were filed on the same day as the present application and each of which are hereby incorporated by reference.
Disclosed embodiments herein relate generally to computing grid environments, and more particularly to improved systems and methods for monitoring electronic tasks in such environments.
The increasing complexity of electronic tasks (e.g. executable programs such as computational tasks, command execution, and data collection) has increased the demand for resources used in accomplishing such tasks. Resources include hardware that aids in completing electronic tasks, such as servers, clients, mainframe computers, networks, network storage, databases, memory, CPU time, and scientific instruments. Resources may also include software, available network services, and other non-hardware resources.
One response to the increased demand for resources has been the development of networked computing grid systems, which operate to integrate resources from otherwise independent grid participants. Computing grid systems generally include hardware and software infrastructure configured to form a virtual organization comprised of multiple resources in often geographically disperse locations.
Grid systems have become increasingly large and complex, often comprising thousands of machines executing hundreds of thousands of electronic tasks, or “jobs,” on any given day. Managing such systems has become increasingly difficult, particularly identifying and correcting errors or “exception conditions” occurring within such systems. Further, providing appropriate levels of security for grid systems has grown more challenging as grids expand in size and complexity. Thus, manual procedures for managing grid systems are quickly becoming outdated.
Exception condition monitoring has previously been accomplished by monitoring the status of the machines or “hosts” providing the resources within the grid. More particularly, exception condition monitoring has typically involved analyzing attributes of the host such as the host's memory capacity, processing capabilities, and input/output, and evaluating whether the host is operating properly. Such exception condition monitoring can be problematic as it monitors the operation of the host instead of the status of the job being executed on the host. Thus, current exception condition monitoring techniques fail to identify exception conditions associated with jobs and job execution.
Therefore, improved systems and methods for automating functions associated with managing large-scale distributed computing grid systems is desired in which autonomic monitoring is provided to evaluate the execution of jobs on the grid and to correct such execution when exception conditions are detected.
Disclosed herein are autonomic monitoring systems for implementation into computing grid systems and related methods of performing autonomic monitoring in grid systems. In one exemplary embodiment, an autonomic monitoring system for deployment into a computing grid is described. The system monitors jobs being executed within the computing grid to identify exception conditions associated with the jobs. The system includes a configuration module, which selects the jobs to be monitored and defines the exception conditions for the jobs. The system further includes an information collection module, which monitors the jobs having exception conditions. Further provided is an exception module configured to analyze the jobs and identify the existence of exception conditions associated with the jobs.
In another embodiment, an action module is provided with the above-described configuration, information collection, and exception modules. The action module takes actions to correct the exception conditions identified for the jobs, such as closing the host running the job and/or notifying a system administrator of the problem.
In other embodiments, any combination of the above-described modules may be used in achieving autonomic monitoring within the computing grid.
In still further embodiments, related methods for performing autonomic monitoring in computing grid systems are described.
Reference is now made to the following descriptions taken in conjunction with the accompanying drawings.
The present disclosure relates to autonomic monitoring systems and related methods that may be deployed into distributed resource management, or grid, systems.
Grid systems, such as the grid 10, are typically built by configuring each cluster 12 to facilitate resource sharing on the grid. Such configuration may occur by loading grid-enabling software onto the hosts 14 or other devices associated with the clusters 12. In doing so, the operating system services provided by the hosts 14 are extended to create a single system image for the cluster 12. The grid-enabling software of the present disclosure may be in the form of simple object access protocol (SOAP/XML protocol) and may be configured to support operating systems such as Linux®, Windows® and Unix® by deploying software daemons that run on the preexisting operating systems provided for the cluster 12. The grid-enabling software may be broken down into various software modules, which include various processes for facilitating operation of the grid 10.
Referring now to
A batch system 40 is then loaded as a layer on top of the base system 32, and includes a slave batch daemon 42 and a master batch daemon 44. The slave batch daemon 42 includes processes for receiving and processing instructions provided by the master batch daemon 44. The master batch daemon 44 is a level higher than the slave batch daemon 42, and is configured to manage not only the slave batch daemon 42 of the master host 30, but each of the slave batch daemons associated with the plurality of slave hosts 33. Although only three slave hosts 33 are shown, the number of hosts associated with the cluster 12 can vary considerably.
Grid-enabling software is also deployed onto the slave hosts 33 to facilitate resource sharing on the grid 10. In particular, the base system 32 of the grid software is loaded onto the slave hosts 33. As such, the slave hosts 33 are configured with their own load information managers 34, process information managers 36 and remote execution servers 38. Additionally, the slave hosts 33 are provided with their own slave batch daemons 42, which receive instructions from the master batch daemon 44 and facilitate the execution of the jobs (via the remote execution server 38) that are sent to the particular hosts.
Upon installation, the master batch daemon 44 can check the configuration of the cluster 12 and contact each of the slave batch daemons 42 of the slave hosts 33 to initiate host updates to the load information manager 34 of the master host 30. The host updates may be provided by the load information managers 34 of the slave hosts 33 and sent to the load information manager 34 of the master host 30, thereby aggregating resource data for the cluster 12. If a job is submitted to the master host 30, the master host can analyze the resource data for the cluster 12, and dispatch the job to a particular slave host 33 if the load information corresponds to the requirements of the job. Additionally, the slave batch daemons 42 of the slave hosts 33 can check resource usage for the slave hosts 33 and send such information to the master batch daemon 44 of the master host 30.
Referring now to
In some embodiments, a scheduler 56 is associated with the master batch daemon 44 and is configured to facilitate job scheduling within the cluster 12. In particular, the scheduler 56 may process the job submission 54 to evaluate the job's resource requirements, analyze the job's resource requirements against the resources provided by the hosts 14 within the cluster 12, and schedule the job to a compatible host or hosts. As discussed above, the amount and type of resources provided by the individual hosts 14 within the cluster 12 may be determined by the submission of such data from the load information managers 34 associated with the hosts. In other embodiments, the scheduler 56 may assign a job to a host 14 within the cluster 12 according to a scheduling decision made by the job or according to specified goal requirements of the job.
Various plug-in components may be provided as part of the scheduler 56. For example, reservation, parallel, and user-defined components may be provided as plug-in components to generally aid the scheduler 56 in making job-scheduling decisions. Various other plug-in components are contemplated as being provided with the scheduler 56.
Referring to
A job service component 82 is further provided with each cluster 12 to generally provide an interface for inputting jobs into the grid. The job service component 82 comprises software processes configured to receive input from an end-user and create a job based on the end-user's request. The job service component 82 may be deployed onto any machine associated with the grid 10, such as an end-user's PC. The grid 10 may include one or several job service components 82, depending on the particular parameters of the system. In some embodiments, jobs input into the grid 10 will have predetermined resource requirements, e.g., 2 Linux® servers with over 2 gigabytes of memory, and thus the jobs will need to be executed on the grid according to these resource requirements. If a job is to be executed on a particular cluster 12, the job can be sent to the resource manager 52, which can then schedule the job accordingly.
The resource managers 52 may be configured to autonomously respond to errors occurring during job execution on the grid 10. Errors, which are often referred to as “exceptions” or “exception conditions,” are abnormal behaviors associated with objects in the cluster 12. Objects may be hosts within the clusters 12, job queues associated with the hosts, and/or jobs being executed within the clusters. In the past, system administrators have typically been tasked with monitoring and responding to exceptions that occur within the clusters 12. However, according to the principles of the present disclosure, system administrators may delegate authority to the resource managers 52 by specifying the objects to be monitored, defining the exception conditions for each object, and defining the response to each exception condition. Accordingly, the objects effectively become autonomic objects, and thus exception conditions for the objects are automatically detected.
The system 100 includes a configuration module 102, which, in one example, is accessed by a system administrator 104 through a PC 106 connected to the grid 10 and communicatively linked with one of the resource managers 52. The PC 106 may or may not also be configured as one of the hosts 14 of the grid 10. The system administrator 104 may specify the objects to be monitored and define the exception conditions for the objects by inputting such information into the configuration module 102. The configuration module 102 is able to parse the relevant information provided by the system administrator.
The system administrator 104 may define the exception conditions as Boolean expressions and may further break the exception conditions into various object groups, such as job exceptions and host exceptions. However, as will be further described with regard to the following examples, the object groups such as job exceptions and host exceptions may be evaluated in view of job execution.
In some embodiments, job exceptions may be specified at the queue level, i.e. prior to execution on one of the plurality of hosts 14. Queues may be defined at the global or grid level, and thus the queue could be defined for all the hosts 14 associated with the grid 10. However, queues may also be cluster-specific, and therefore, be associated with the hosts 14 of a particular cluster 12. If exceptions are defined for a queue, then jobs belonging to that queue may be subject to autonomic monitoring for the defined exceptions.
In one example, execution of a job may be evaluated by defining exceptions associated with the run time T(j) of a job, such as “Job J”. In evaluating Job J's run time T(j), it may be pertinent to evaluate the Job J's CPU time over run time, which indicates Job J's CPU consumption. Moreover, the minimum expected run time Tmin, the maximum expected job run time Tmax, and the minimum expected CPU consumption Pmin of Job J might also be defined. Accordingly, if Job J runs for too short an amount of time, the job exception can be defined as T(i)<Tmin. Similarly, if Job J runs for too long an amount of time, the job exception can be defined as T(j)>Tmax. Still further, if Job J consumes too little an amount of CPU time, the job exception will be defined as P(j)<Pmin.
Exceptions may be further defined for individual hosts within the clusters 12 by evaluating job execution on the hosts. A host experiencing an exception condition can be detrimental to the grid 10 because jobs running on the host may exit the host abnormally. Such errors can cause significant drain on resources associated with the grid 10. In some embodiments, host exceptions can be configured to close hosts automatically and send a notification to the system administrator 104, such as via email or by page. In one example, a host exception for a particular host (“h”) is dependent on the number of jobs that exit abnormally within the last minute (“E(t)”) at a particular time (“t”). The host exception will be further dependent on the current time (“T(c)”).
The host exception may also be dependent upon parameters defined by the system administrator 104. For example, parameters such as the minimum duration (“D”) that the exceptional situation should exist for host h before the system issues an exception and the maximum number (“Nmax”) of abnormally exited jobs in one minute on host h when the host is operating correctly may also help define a host exception. According to the above parameters, a host exception can be defined as follows:
Ψ(t)|t<Tc, t>Tc−D:E(t)>Nmax.
The parameters D and Nmax can be defined differently for different hosts, thereby allowing the system administrator 104 to tailor the host exception for specific hosts.
An information collection module 108 is communicatively linked with the configuration module 102 to receive the relevant exception condition information defined by the system administrator 104. The information collection module 108 generally monitors the running cluster 12 in real time and collects status information 110 on the autonomic objects, which are those objects specified and defined to have exception conditions. The information collection module 108 may include an information filter 112, which filters out information unrelated to the specified autonomic objects. For example, the information filter 112 may evaluate data according to the exception definitions, and based on the analysis, pass through or filter out such data. Although described as a sub-component of the information collection module 108, the information filter 112 may be a stand-alone component.
The information collection module 108 may evaluate the relevant data by examining the variables associated with each defined exception condition. For instance, an administrator may define the following two exceptions for a job (“Job A”):
The information collection module 108 transmits the relevant data d to an exception module 114, which identifies exceptions that should be corrected. In some embodiments, the exception module 114 evaluates the above-described Boolean expressions associated with the data received from the information collection module 108 to determine whether an exception should be identified and corrected. For example, if the Boolean expressions evaluate to TRUE, then the exception module 108 can mark the exception for correction.
The system 100 further includes an action module 116, which is configured to take actions to handle the exceptions identified by the exception module 114. The action module 116 may be invoked periodically, particularly if there has been an above-average amount of exceptions identified recently. The actions associated with the action module 116 may be configured as IF-THEN rules that include default actions for each defined exception. In some instances, the system administrator 104 may define actions that override the default actions. The actions are typically implemented by user commands or API calls. For example, if a job runs over the time allotted for completion of the job, the action may be configured as follows:
Moreover, if a job uses an insufficient amount of CPU time consumption, the action may be configured as follows:
By way of example, it may be desired that host h be monitored for exception conditions. Referring to
The autonomic objects according to the present disclosure are not necessarily confined to having only one exception condition. Rather, an autonomic object may have multiple exception conditions, which may cause the exception module 114 to identify multiple exceptions for an object. Accordingly, the action module 116 may function to combine actions for the multiple exceptions, or possibly resolve conflicts between the multiple exceptions. In some embodiments, the action module 116 is designed as a plug-in component, which end-users can use to customize the default actions associated with the action module. Moreover, the end-user may design their own action module 116 for use with the system 100.
Referring to
Referring to
While various embodiments of systems for performing autonomic monitoring in a grid environment according to the principles disclosed herein, and related methods of performing autonomic monitoring, have been described above, it should be understood that they have been presented by way of example only, and not limitation. For example, although the hosts have been generally indicated as PCs and servers, various other types of hosts are contemplated as providing resources for the grid 10. Exception notification may be through one or more different techniques, such as by email, wireless paging, web dashboard monitoring, web-page updates, or other notification technique. Thus, the breadth and scope of the invention(s) should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents. Moreover, the above advantages and features are provided in described embodiments, but shall not limit the application of the claims to processes and structures accomplishing any or all of the above advantages.
Additionally, the section headings herein are provided for consistency with the suggestions under 37 CFR 1.77 or otherwise to provide organizational cues. These headings shall not limit or characterize the invention(s) set out in any claims that may issue from this disclosure. Specifically and by way of example, although the headings refer to a “Technical Field,” the claims should not be limited by the language chosen under this heading to describe the so-called technical field. Further, a description of a technology in the “Background” is not to be construed as an admission that technology is prior art to any invention(s) in this disclosure. Neither is the “Brief Summary” to be considered as a characterization of the invention(s) set forth in the claims found herein. Furthermore, any reference in this disclosure to “invention” in the singular should not be used to argue that there is only a single point of novelty claimed in this disclosure. Multiple inventions may be set forth according to the limitations of the multiple claims associated with this disclosure, and the claims accordingly define the invention(s), and their equivalents, that are protected thereby. In all instances, the scope of the claims shall be considered on their own merits in light of the specification, but should not be constrained by the headings set forth herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3648253 | Mullery et al. | Mar 1972 | A |
| 4839798 | Eguchi et al. | Jun 1989 | A |
| 5031089 | Liu | Jul 1991 | A |
| 5428781 | Duault et al. | Jun 1995 | A |
| 5442791 | Wrabetz et al. | Aug 1995 | A |
| 5473773 | Aman | Dec 1995 | A |
| 5504894 | Ferguson et al. | Apr 1996 | A |
| 5522070 | Sumimoto | May 1996 | A |
| 5603029 | Aman et al. | Feb 1997 | A |
| 5621663 | Skagerling | Apr 1997 | A |
| 5655081 | Bonnell et al. | Aug 1997 | A |
| 5675739 | Eilert et al. | Oct 1997 | A |
| 5805785 | Dias et al. | Sep 1998 | A |
| 5812844 | Jones et al. | Sep 1998 | A |
| 5872931 | Chivaluri | Feb 1999 | A |
| 5893905 | Main et al. | Apr 1999 | A |
| 5903757 | Gretz et al. | May 1999 | A |
| 5978829 | Chung et al. | Nov 1999 | A |
| 6195676 | Spix et al. | Feb 2001 | B1 |
| 6230183 | Yocom et al. | May 2001 | B1 |
| 6247041 | Kruger et al. | Jun 2001 | B1 |
| 6263358 | Lee et al. | Jul 2001 | B1 |
| 6282561 | Jones et al. | Aug 2001 | B1 |
| 6334193 | Buzsaki | Dec 2001 | B1 |
| 6353844 | Bitar et al. | Mar 2002 | B1 |
| 6356917 | Dempsey | Mar 2002 | B1 |
| 6393455 | Eilert et al. | May 2002 | B1 |
| 6412026 | Graf | Jun 2002 | B1 |
| 6463454 | Lumelsky et al. | Oct 2002 | B1 |
| 6516350 | Lumelsky et al. | Feb 2003 | B1 |
| 6643614 | Ding | Nov 2003 | B2 |
| 6687731 | Kavak | Feb 2004 | B1 |
| 6694345 | Brelsford et al. | Feb 2004 | B1 |
| 6714960 | Bitar et al. | Mar 2004 | B1 |
| 6728961 | Velasco | Apr 2004 | B1 |
| 7043659 | Klein et al. | May 2006 | B1 |
| 20030126260 | Husain | Jul 2003 | A1 |
| 20030233602 | Lindquist et al. | Dec 2003 | A1 |
| 20040019624 | Sukegawa | Jan 2004 | A1 |
| 20040059966 | Chan et al. | Mar 2004 | A1 |
| 20040064552 | Chong | Apr 2004 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20050283788 A1 | Dec 2005 | US |
