Patent Application
20150332195
The disclosed embodiments generally relate to techniques for executing computational workflows on computing clusters. More specifically, the disclosed embodiments relate to a technique for monitoring the performance of periodically scheduled workflows and associated jobs while they are executing on a computing cluster.
Perhaps the most significant development on the Internet in recent years has been the rapid proliferation of online social networks, such as Facebook™ and LinkedIn™. Billions of users are presently accessing such online social networks to connect with friends and acquaintances and to share personal and professional information. However, to operate effectively, these online social networks need to perform a large number of computational operations. For example, an online professional network typically executes computationally intensive algorithms to identify other members of the network that a given member will want to link to.
These computational operations are often performed using periodically scheduled “workflows,” wherein each workflow comprises a collection of interdependent jobs that are scheduled to execute on nodes of a computing cluster. Note that this type of computing cluster can comprise a multi-tenant system, such as Apache Hadoop™. The scheduling process can be somewhat complicated because an intricate dependency chain exists among the jobs that comprise a task, and the scheduler must ensure that all preceding jobs in a dependency graph complete before a given job can execute.
Moreover, these periodically scheduled workflows can encounter performance problems during execution. For example, a node in the computing cluster can have performance problems, and this problematic node can cause a job to be delayed, which can prevent an associated workflow from completing. Therefore, to ensure successful completion of such scheduled workflows, it is necessary to carefully monitor the performance of the workflows and associated jobs to detect performance problems, thereby enabling remedial actions to be performed. For example, a remedial action can involve moving a delayed job from a problematic node to another node in the computing cluster.
Hence, what is needed is a system that facilitates monitoring the performance of periodically scheduled workflows and associated jobs in the computing cluster.
The following description is presented to enable any person skilled in the art to make and use the disclosed embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed embodiments. Thus, the disclosed embodiments are not limited to the embodiments shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.
The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing code and/or data now known or later developed.
The methods and processes described in the detailed description section can be embodied as code and/or data, which can be stored on a non-transitory computer-readable storage medium as described above. When a system reads and executes the code and/or data stored on the non-transitory computer-readable storage medium, the system performs the methods and processes embodied as data structures and code and stored within the non-transitory computer-readable storage medium.
Furthermore, the methods and processes described below can be included in hardware modules. For example, the hardware modules can include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), and other programmable-logic devices now known or later developed. When the hardware modules are activated, the hardware modules perform the methods and processes included within the hardware modules.
The disclosed embodiments provide a system for monitoring the performance of periodically scheduled workflows and associated jobs while they are executing a computing cluster. During operation, the system monitors the total execution time for the workflow, wherein the workflow comprises a set of jobs that execute on nodes of a computing cluster. While monitoring the total execution time for the workflow, the system also monitors execution times for individual jobs in the set of jobs that comprise the workflow. The system also periodically determines an execution-time threshold for the workflow based on prior executions of the workflow. If the monitored execution time for the workflow exceeds the determined execution-time threshold for the workflow, the system sends an alert about the workflow to a user. The system also enables the user to examine the monitored execution time for the workflow and the monitored execution times for the associated jobs. This can potentially help the user to determine a solution to a performance problem for the workflow.
In some embodiments, the system also determines execution-time thresholds for jobs that comprise the workflow based on previous executions of the jobs. Then, if an execution time for a job exceeds the determined execution-time threshold for the job, the system sends an alert about the job to the user.
In some embodiments, the system also enables the user to examine a dependency graph for the workflow to facilitate determining a solution to a performance problem for the workflow. This dependency graph specifies dependencies between jobs in the workflow, wherein a dependency between a first job and a second job indicates that the first job must complete before the second job can begin executing.
In some embodiments, while determining the execution-time threshold, the system first determines a mean value and a standard deviation for the execution time for the workflow based on prior successful executions of the workflow. Next, the system adds the determined standard deviation and a buffer time to the determined mean value to produce the execution-time threshold.
In some embodiments, the system additionally monitors values for one or more internal counters for events associated with the flow, and then enables the user to examine the monitored values for the one or more internal counters.
Before describing details of the operation of the monitoring system, we first describe a computing environment that contains the monitoring system.
More specifically, mobile devices 104 and 108, which are operated by users 102 and 106 respectively, can execute mobile applications that function as portals to an online application, which is hosted on mobile server 110. Note that a mobile device can generally include any type of portable electronic device that can host a mobile application, including a smartphone, a tablet computer, a network-connected music player, a gaming console and possibly a laptop computer system.
Mobile devices 104 and 108 communicate with mobile server 110 through one or more networks (not shown), such as a WiFi® network, a Bluetooth™ network or a cellular data network. Mobile server 110 in turn interacts through proxy 122 and communications bus 124 with a storage system 128, which for example can be associated with an Apache Hadoop™ system. Note that although the illustrated embodiment shows only two mobile devices, in general a large number of mobile devices and associated mobile application instances (possibly thousands or millions) can simultaneously access the online application.
The above-described interactions allow users to generate and update “member profiles,” which are stored in storage system 128. These member profiles include various types of information about each member. For example, if the online social network is an online professional network, such as LinkedIn™, a member profile can include: first and last name fields containing a first name and a last name for a member; a headline field specifying a job title and a company associated with the member; and one or more position fields specifying prior positions held by the member.
The disclosed embodiments also allow users to interact with the online social network through desktop systems. For example, desktop systems 114 and 118, which are operated by users 112 and 116, respectively, can interact with a desktop server 120, and desktop server 120 can interact with storage system 128 through communications bus 124.
Note that communications bus 124, proxy 122 and storage device 128 can be located on one or more servers distributed across a network. Also, mobile server 110, desktop server 120, proxy 122, communications bus 124 and storage device 128 can be hosted in a virtualized cloud-computing system.
The computing environment 100 illustrated in
As illustrated in
During operation of the system illustrated in
Note that a related set of workflows can collectively form a “macro-flow,” which includes a set of interrelated workflows with associated interdependencies. In addition to optimizing the execution of a single workflow, the system can also optimize the execution of a macro-flow associated with multiple interrelated workflows.
Next, the system periodically determines an execution-time threshold for the workflow based on prior executions of the workflow (step 308). The system similarly determines execution-time thresholds for jobs that comprise the workflow based on previous executions of the jobs (step 310).
Returning to
After user 134 receives an alert for a workflow or a job, user 134 may want to examine status information relating to the execution of the workflow. Referring to the flow chart illustrated in
Landing page 600 can also provide an accordion view 602, wherein a specific workflow exec_id=168576 is expanded to display the jobs that comprise the workflow, along with statistics for the jobs. This accordion view 602 is produced when the user clicks on the parent workflow. Similarly, if the user clicks on an individual job, the system can display job history information.
The user can also examine a workflow view 700 for a specific workflow as illustrated in
The foregoing descriptions of disclosed embodiments have been presented only for purposes of illustration and description. They are not intended to be exhaustive or to limit the disclosed embodiments to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the disclosed embodiments. The scope of the disclosed embodiments is defined by the appended claims.
