Patent Application
20050268300
The present invention relates in general to task scheduling for computing environments and more particularly to a two-tier distributed task scheduling method and system that separates and performs task management and task execution on separate computing devices and distributes the task execution over multiple computing devices to achieve scalability and reliability in scheduled task execution.
Task scheduling is an important feature in computing environments. Tasks to be executed may include network requests, dynamic link libraries, the scheduled processing of data, and other tasks such as periodic hard drive maintenance and compression, hard drive defragmentation, and the cleanup of old files. Scheduled task systems are designed to automatically execute a predetermined process at a set time, at a set interval, or both. Traditionally, these systems have been implemented as an application that maintains a list of tasks to be executed and the time at which each task is to be executed. Examples of such scheduled task systems include the Task Scheduler in the Microsoft® Windows® XP operating system, or the CRON daemon included with most Unix® systems.
Current scheduled task systems and designed to operate in a computer environment having a single computer. This means that the list of tasks (or scheduling data) is stored locally and on the same computer that executes the tasks. While these systems work well in this single computer environment, these systems have significant drawbacks when applied to environments where reliability and scalability are paramount. One drawback is that these systems are not scalable. For example, if there are many tasks scheduled to be run at or near the same time, the single host computer may quickly become overburdened or low on resources, and as result some of the tasks may not execute correctly. Another drawback is that the systems are not totally reliable in situations where the task absolutely must be executed. For example, if the single computer hosting the scheduling application fails or is turned off during the time a task is scheduled to run, that task will not be executed. This can have serious consequences in a mission-critical environment such as such as a data center or a distributed network of servers where the task must be executed.
Another example of a mission-critical environment is online gaming. Online gaming is an environment where several people compete in a game over a computer network (typically the Internet). Many activities in an online game require a process to be executed outside the context of user activity, on a scheduled, independent basis. By way of example, these activities may include round progression of a tournament to take place at a specified time and reducing a rank of players in statistics if a player has not participated in the game for a long period. The game is an ongoing process, and it is critical that the tasks necessary to keep the game going are executed in a reliable manner for the competition to continue uninterrupted and on schedule. In addition, because the number of players may fluctuate, scalability of the task scheduler becomes an important factor. Accordingly, there exists a need for a task scheduler for a computing environment that provides scalable and absolutely reliable execution of tasks on a scheduled basis.
The invention disclosed herein includes a distributed task scheduling method and system that separates task management from task execution. In particular, task management and task execution are performed on separate computing devices. In addition, task execution is distributed over multiple computing devices. The task management is performed by at least one data broker while the task execution is performed by a plurality of execution hosts.
The distributed task scheduler achieves scalability by having a plurality of execution hosts capable of executing tasks, all under the direction of at least one data broker, such that a task is only executed by a single execution host at a time. Reliability is achieved by having a multiplicity of execution hosts so that if one execution host fails another host can ensure the task is executed. A timeout period is provided for each task, such that an execution host is given the timeout period to execute the task and notify the data broker of the task completion. If the execution host does not or cannot execute the task within the timeout period, the data broker is free to give the task to a different execution host.
The distributed task scheduler uses a data broker and a plurality of execution hosts to execute tasks in a computing environment. A scheduled task list, which includes a list of scheduled tasks to be executed, is stored on the data broker. The data broker dispenses or gives out tasks from the scheduled task to the plurality of execution hosts for execution of the task. The data broker and each of the plurality of execution hosts are located on separate computing devices. The scheduled task list contains tasks and may also contain a task execution timetable representing a scheduled execution time associated with each of the tasks.
The distributed task scheduler includes a two-tier architecture that distributes task management and task execution over a plurality of computing machines. Each tier has at least two separate computing devices. The data management tier having at least one data broker is used to determine whether any scheduled tasks need to be executed at a current time. This is accomplished by examining the scheduled tasks list and determining whether tasks needing to be executed meet certain availability criteria. If an available task is found, the data management tier assigns the scheduled task needing to be run to the task execution tier. Once a task has been given out to the data execution tier, the task is executed on the data execution tier using an execution host. Once the task has been executed the execution host reports to the data broker that the task execution has been completed.
If a task is not found by the data broker, the execution host notes the check-back time that was received from the data broker and then sleeps for some configurable amount of time. A determination then is made as to whether a wake-up time is before the check-back time. If so, then the host checks back with the data broker for a new task at the wake-up time. If the wake-up time is after the check-back time, the host awakens from sleep at the check-back time and makes a new task request to the data broker.
The data broker examines the schedule task list for any available task. A task is defined as available to be executed if the task meets the following criteria: (1) the next execution time for the task is less than or equal to the current time; (2) the task is not marked as completed; and either (3a) the execution host is blank; or (3b) a timeout period has expired without the execution host marking the job as completed. If these criteria are met, then a task is available for execution. Using this approach, multiple execution hosts may be assigned to execute scheduled tasks. If any one of the execution hosts fails unexpectedly, the task will be attempted from another host. By recording the host identifier with the task, the data broker can ensure only one execution host attempts to execute a task at any given time. The data broker should be a transactional data broker, in order to ensure that an update to a given task can only happen by a single thread and a single process on that data broker.
The present invention can be further understood by reference to the following description and attached drawings that illustrate aspects of the invention. Other features and advantages will be apparent from the following detailed description of the invention, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the present invention.
Referring now to the drawings in which like reference numbers represent corresponding parts throughout:
In the following description of the invention, reference is made to the accompanying drawings, which form a part thereof, and in which is shown by way of illustration a specific example whereby the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.
I. Introduction
Current scheduled task systems are designed to operate on a single computer. This can cause problems if the computer fails, because the tasks needing to be executed will not be executed until the computer starts working again. In addition, the single computer scheduled task system can quickly become overworked if a large number of tasks need to be executed at or near the same time.
The distributed task scheduler disclosed herein overcomes these problems and more by separating and distributing the task management and execution over a plurality of computing devices. A two-tier architecture includes at least one execution host and at least one data broker residing on separate computing devices. The execution hosts actually handle the tasks and the data broker manages the task schedule. A single list of tasks to be executed is shared between each of the devices, with special provisions made to ensure tasks are executed as close to the scheduled time as possible, even in the case of catastrophic failure in one or more of the devices. As a result, the distributed task scheduler provides a reliable and scalable system for scheduled task execution.
II. General Overview
The distributed task scheduler facilitates the scalable and reliable execution of tasks in a computing environment.
The two-tier architecture of the distributed task scheduler 100 includes a task management tier 110 and a task execution tier 120. The task management tier 110 includes one or more data broker machines. The data broker machines are shown in
In an exemplary implementation, M=2 and data broker (1) is the master data broker while data broker (2) is a back-up data broker. In this implementation, only one of the data brokers is operational at a time. The master data broker is the only data broker using the scheduled task list (1) and the distributed task manager (1) to give out tasks to the execution hosts. In case of failure of the master data broker, the back-up data broker becomes the master data broker. In order to facilitate a smooth transition in case of failure, the scheduled task list (2) of the back-up data broker is a copy of the scheduled task list (1) of the master data broker.
In alternate implementations, each of the data brokers (1) to (M) is a master data broker has master separate scheduled task lists (1) to (M) that are different from each other. In this case, the respective distributed task managers (1) to (M) dole out tasks to the execution hosts independently of each other. In addition, the execution hosts (discussed below) request tasks from each of the master data brokers (1) to (M) in a random manner or a sequential manner. Other implementations also are possible, such as having multiple pairs of data brokers, where each pair is a master data broker and a back-up data broker, as described above. In this implementation, each data broker pair uses a single scheduled task list to give out tasks to the execution hosts.
The task execution tier 120 includes one or more execution hosts. The execution hosts are shown in
III. Exemplary Operating Environment
The distributed task scheduler 100 is designed to operate in a computing environment and on a computing device, such as the data brokers (1) to (M) and the execution hosts (1) to (N), shown in
The distributed task scheduler is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the distributed task scheduler include, but are not limited to, personal computers, server computers, hand-held, laptop or mobile computer or communications devices such as cell phones and PDA's, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The distributed task scheduler may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The distributed task scheduler may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices. With reference to
Components of the computer 210 may include, but are not limited to, a processing unit 220, a system memory 230, and a system bus 221 that couples various system components including the system memory to the processing unit 220. The system bus 221 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
The computer 210 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by the computer 210 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer 210. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
Note that the term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
The system memory 230 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 231 and random access memory (RAM) 232. A basic input/output system 233 (BIOS), containing the basic routines that help to transfer information between elements within the computer 210, such as during start-up, is typically stored in ROM 231. RAM 232 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 220. By way of example, and not limitation,
The computer 210 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 241 is typically connected to the system bus 221 through a non-removable memory interface such as interface 240, and magnetic disk drive 251 and optical disk drive 255 are typically connected to the system bus 221 by a removable memory interface, such as interface 250.
The drives and their associated computer storage media discussed above and illustrated in
Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, radio receiver, or a television or broadcast video receiver, or the like. These and other input devices are often connected to the processing unit 220 through a user input interface 260 that is coupled to the system bus 221, but may be connected by other interface and bus structures, such as, for example, a parallel port, game port or a universal serial bus (USB). A monitor 291 or other type of display device is also connected to the system bus 221 via an interface, such as a video interface 290. In addition to the monitor, computers may also include other peripheral output devices such as speakers 297 and printer 296, which may be connected through an output peripheral interface 295.
The computer 210 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 280. The remote computer 280 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 210, although only a memory storage device 281 has been illustrated in
When used in a LAN networking environment, the computer 210 is connected to the LAN 271 through a network interface or adapter 270. When used in a WAN networking environment, the computer 210 typically includes a modem 272 or other means for establishing communications over the WAN 273, such as the Internet. The modem 272, which may be internal or external, may be connected to the system bus 221 via the user input interface 260, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 210, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
IV. Operational Overview
The distributed task scheduler 100 includes a two-tier architecture that distributes task management and task execution over a plurality of computing machines.
Once an available task is found, the distributed task scheduler 100 dispenses the scheduled task needing to be run from the task management tier to the task execution tier (box 320). Typically, this is achieved by having the data broker send the task to an execution host. It is important to note that each task is sent to only one execution host. This means that for a given task, only one execution host is responsible for executing that task. As explained in detail below, if that execution host fails to execute the task within a prescribed time period, then the execution host is assumed to have failed and the task is given to a different execution host. However, at all times only one execution host is responsible for executing a given task.
Once a task has been given out to the data execution tier, the task is executed on the data execution tier (box 330). Typically, this involves an execution host to whom the task has been given executing the task. As explained in greater detail below, once the task has been executed the execution host reports to the data broker that the task execution has been completed.
V. Operational Details
The operational details of the distributed task scheduler 100 will now be discussed.
The execution host 400 and the data broker 410 are physically separate computing devices. The execution host 400 is a computing device that handles the requests, reporting, and execution of tasks. The data broker 410 is a computing device that stores the list of tasks to be executed, the time of scheduled execution, and the brokering of tasks to the execution host 400. In addition, the data broker 410 keeps track of which tasks the execution host 400 has been given, and whether a task has been completed.
When the execution host 400 is ready (such as after start-up or waking up from a sleep), a request is made to the data broker 410 for new tasks (box 420). Upon receiving this request, the data broker 410 checks its scheduled task list (not shown) to determine whether there are any new tasks needing to be executed (box 425). If there are no tasks needing to be executed at the current time, then the data broker 410 informs the execution hosts to check back again at a specific amount of time in the future (box 430). This check-back time is based on the task execution timetable kept in the scheduled task list.
If there is a task needing to be executed at the current time, the data broker 410 next determines whether the task is already assigned to a host (box 435). If the task already is assigned, then the data broker 410 determines whether a timeout period for the task has expired (box 440). The timeout period is the time allotted by the data broker 410 in which the task must be executed by the execution host 400. If a task is given to a execution host 400 and the execution host 400 has not reported the task as completed within the timeout period, the data broker 410 assumes the execution host 400 has failed. In this case, the task still needs to be executed and the data broker 410 assigns the task to another execution host for execution. In this way, the distributed task scheduler ensures reliable task execution.
If the timeout period for the task has not expired, the data broker 410 searches its scheduled task list for any other new tasks (box 445). If the timeout period for the task has expired, then the data broker 410 records which execution host 400 was given the task and the time at which the task was given out (box 450). Similarly, if the determination of whether the task is assigned to a host (box 435) is negative, the data broker 410 assigns the task to the execution host 400 and records the assignment and the time. The recordation of the time is important because once the task is given to the execution host 400 and the timeout period begins.
When the execution host 400 receives a new task from the data broker 410, the execution host 400 executes the task (box 455). Next, the execution host 400 reports to the data broker 410 that the task has been completed (box 460). Once a task is completed the execution host 400 can check back with the data broker 410 for an additional task (box 420).
It should be noted that one execution host 400 can execute a plurality of task simultaneously. The number of tasks that can be simultaneously executed is dependent (at least in part) on the current state of computer processor technology. In one tested implementation, each execution host is capable of executing approximately 25 tasks simultaneously. In practice, multiple threads are sent out from the execution host 400 to obtain new tasks. Thus, one execution host 400 may have multiple threads that are all picking up tasks from the data broker 410. It can be seen, therefore, that
Execution Host
More specifically, the operation begins as the execution host 400 becomes available for a new task (box 500). This can occur when the execution host 400 starts up, wakes up from a sleep, or has completed the task at hand. The execution host 400 then notifies the data broker 410 that the host 400 is available for a new task (box 510). It should be noted that each execution host 400 does not have a complete list of tasks. Instead, each execution host 400 (or more correctly, each thread of the execution host 400) obtains one task at a time, completes that task, and then asks for another task by notifying the data broker 410 that the task is completed.
The execution host 400 then determines whether a new task was received from the data broker 410 (box 520). If a task was received, the host 400 executes the received task (box 530). In a particular implementation, tasks are implemented as dynamic link libraries (DLLs) that expose a pre-defined interface. However, a task can be any set of executable instructions. As soon as the host 400 is finished executing the received task, the host 400 reports to the data broker 410 that the task is completed (box 540). The host 400 then ask for a new task (box 510).
Each task has associated with it a timeout period. If the host 400 does not report back to the data broker 410 with the timeout period that the task has been completed, the data broker 410 is free to assign the task to a different host. In other words, an execution host 400 has an exclusive lock or monopoly on the task during the timeout period. Once the timeout period has expired, that monopoly is broken and the task can be given to a different host for execution. If the host 400 fails or some other bad thing happens to the host 400 such that is cannot report back to the data broker 410 that the task was completed within the time specified, then the data broker 410 puts the task back into the pool of tasks and reassigns the task to a different host.
If a task is not received from the data broker 410, the execution host 400 notes the check-back time that was received from the data broker 410 (box 550). The check-back time is the time when a certain task is due to be executed, and thus will become available. The execution host 400 then sleeps for some configurable amount of time (box 560). In a preferred implementation, the sleep time is approximately one minute. This gives the data broker 410 relief so that the execution hosts are not constantly bothering the data broker 410.
A determination then is made as to whether a wake-up time (the time that the host 400 “awakens” from its sleep) is before the check-back time (box 570). If so, then host 400 makes a request to (“checks back with”) the data broker 410 for a new task at the wake-up time (box 580). In other words, even if the check-back time is an hour later, the host 400 will sleep for until the wake-up time and then make a request to the data broker 400, long before the check-back time. If the wake-up time is after the check-back time, the host 400 will awaken from sleep at the check-back time (and before the wake-up time) to make a request to the data broker 410 for a new task (box 590).
Data Broker
In particular, the operation of the data broker 410 begins by receiving input from an execution host (box 600). A determination then is made as to whether the input is an update or a request (box 605). If the input is an update, the data broker 410 mark the task as completed by the execution host providing the update (box 610). Next, the data broker 410 updates any back-up scheduled task lists on back-up data brokers (box 615).
If the input is a request, the data broker examines the schedule task list for any available task (box 620). In general, a task is defined as available to be executed if the task meets the following criteria: (1) the next execution time for the task is less than or equal to the current time; (2) the task is not marked as completed; and either (3a) the execution host is blank; or (3b) a timeout period has expired without the execution host marking the job as completed. If these criteria are met, then a task is available for execution.
In particular, the data broker 410 determines whether an execution time for a task is less than or equal to the current time (box 625). If not, then the data broker 410 notes the execution time, calculates the check-back time, and sends the requesting execution host the check-back time (box 630). As explained above, the execution host 400 uses the check-back time to determine when to make another request to the data broker 410.
If the execution time is less than or equal to the current time, the data broker 410 makes a further determination as to whether the task is assigned (box 635). If the task is assigned, then the data broker 410 makes a determination whether the task has timed-out (box 640). In other words, has the timeout period expired without the execution host 400 reporting back to the data broker 410 that the task has been executed. If the task has not timed-out, then the data broker 410 searches the scheduled task list for another task (box 645).
If either the task is not assigned (box 635) or the task has timed-out (box 640), the data broker 410 marks the task as available for execution and assigns the task to the execution host 400 (box 650). Next, a timeout period is selected (box 655). In one implementation, the timeout period is a globally configured timeout period valid for all tasks. In a preferred implementation, the timeout period is dependent on the duration of the task. For example, a task having a short duration (such as a “ping” to a network server) would have an associated short timeout period, while a task having a longer duration (such as defragmenting a large-capacity hard drive) would have an associated longer timeout period. The assignment of the task to the execution host is recorded by the data broker along with the current time and the timeout period associated with the task (box 660). In a preferred implementation, a unique identifier of the execution host 400 (such as the computer's DNS name, IP address, MAC address, etc.) is used to record the task assignment. The task then is transmitted to the assigned execution host (box 665). Once the execution host 400 has completed execution of the task, it notifies the data broker 410, which marks the task as completed, reschedules the task for the next execution time, or both.
Using this approach, multiple execution hosts may be assigned to execute scheduled tasks. If any one of the execution hosts fails unexpectedly, the task will be attempted from another host. By recording the host identifier with the task, the data broker 410 can ensure only one execution host attempts to execute a task at any given time. If the data broker 410 does not receive a notification of completion from the host 400 before a timeout period expires, the data broker 410 assumes that the host 400 has failed unexpectedly and reassigns the task to a different host.
The data broker 410 preferably is a standard database management system (such as Structured Query Language (SQL)). This means that the data broker 410 can be backed up and can use data replication or similar technology to provide redundancy. In addition, the data broker 410 has a redundancy of the scheduled task list. The key thing is that there is only one master scheduled task list at a time. If the master fails, then a backup can be used and become the master.
The data broker 410 should be a transactional data broker. The data broker 410 should be able to guarantee that an update to a given task can only happen by a single thread, a single process on that data broker 410. The record cannot be overwritten by anyone else. And, if something fails during the update of the process of that task, the data broker 410 should make sure that everything is rolled back to the point at which it was before the transaction began. These types of transactions are called atomic transactions. Atomic is a database term meaning that the transaction is all or nothing. Either everything is successfully updated for nothing is updated.
The transactional data broker should also follow ACID rules. ACID is a database acronym standing for Atomic, Consistent, Isolated and Durable. A consistent transaction is one that leaves data in a consistent state, such that the data does not contradict each other. An isolated transaction is one that cannot be viewed by another transaction before it is committed. In other words, the transaction cannot be viewed in its transitional state. A durable transaction means that changes due to a transaction should be stored in a stable storage and should be recoverable in case of system failure. Any data broker that fits this ACID transactional criteria will work.
In a preferred implementation the data broker is a SQL server that is also an ACID transactional data broker. The scheduled task list of tasks to be executed is stored and owned by the central SQL server. When an execution host engine is ready for a task (i.e. wakes up or has completed the task at hand), the host goes to the SQL server and asks for a task. If the task is available, then the SQL server tracks that it gave this task to a specific execution host. The SQL server should do this in a transactional nature make sure that the exact same task is not given to another host. That is why the SQL server should be a transactional database.
The foregoing description of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description of the invention, but rather by the claims appended hereto.
