Scalable real time metrics management

Abstract

Managing performance metrics includes: obtaining a plurality of performance metrics associated with a plurality of sources on a network; aggregating, at a first rate, the plurality of performance metrics associated with the plurality of sources to generate a plurality of first aggregated results; maintaining at least some of the plurality of first aggregated results in one or more memories; aggregating, at a second rate, the plurality of first aggregated results to generate a plurality of second aggregated results, the second rate being a lower rate than the first rate; and maintaining at least some of the plurality of second aggregated results in the one or more memories.

Description

BACKGROUND OF THE INVENTION

Metrics (also referred to as performance metrics) are used by computer systems to quantify the measurement of system performance. Metrics are critical for analyzing systems' operations and providing feedback for improvements.

In modern computer systems, the quantity of metrics can be large. For example, suppose that a single cloud application collects 1000 metrics for analysis every 5 seconds, which means that 720,000 metrics are collected every hour. In a typical high scale environment such as an enterprise data center that supports thousands of applications each executing on multiple servers, the rate can be on the order of billions of metrics per hour.

Currently, most performance monitoring tools save collected metrics to a database, then perform analysis offline. These tools tend to scale poorly because of the high number of input/output (I/O) operations (such as database reads and writes) required for storing and processing a large number of metrics. Further, these tools typically do not support real time analytics due to the latency and processing overhead in storing and processing metrics data in the database.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

FIG. 1 is a functional diagram illustrating a programmed computer system for managing metrics in accordance with some embodiments.

FIG. 2 is a block diagram illustrating an embodiment of a data center that includes a scalable distributed metrics manager.

FIG. 3A is a block diagram illustrating an embodiment of a metrics pipeline in a scalable distributed metrics manager.

FIG. 3B is a diagram illustrating an embodiment of a metric data structure.

FIG. 3C is a diagram illustrating an embodiment of a metrics message.

FIG. 4 is a flowchart illustrating an embodiment of a process for managing metrics.

FIGS. 5A-5B are diagrams illustrating an embodiment of an approach for archiving the aggregated results.

FIGS. 6A-6B are diagrams illustrating another embodiment of an approach for archiving the aggregated results.

FIG. 7 is a flowchart illustrating an embodiment of a process for querying metrics data stored in a database.

FIG. 8 is a diagram illustrating an example of a query to a database comprising multiple time series based database tables.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.

Managing metrics for high scale environments is disclosed. In some embodiments, the metrics are managed and processed in a pipeline comprising multiple stages. A plurality of performance metrics associated with a plurality of sources on a network is obtained. The plurality of performance metrics is aggregated at a first rate to generate a plurality of first aggregated results, and at least some of the plurality of first aggregated results are maintained for a time in one or more memories. The plurality of first aggregated results is aggregated at a second rate to generate a plurality of second aggregated results, the second rate being a lower rate than the first rate. At least some of the plurality of second aggregated results are maintained in the one or more memories. Additional aggregation stages can be used. The aggregated results can be persisted to a persistent storage.

FIG. 1 is a functional diagram illustrating a programmed computer system for managing metrics in accordance with some embodiments. As will be apparent, other computer system architectures and configurations can be used to manage and process metrics. Computer system 100, which includes various subsystems as described below, includes at least one microprocessor subsystem (also referred to as a processor or a central processing unit (CPU)) 102. For example, processor 102 can be implemented by a single-chip processor or by multiple processors. In some embodiments, processor 102 is a general purpose digital processor that controls the operation of the computer system 100. Using instructions retrieved from memory 110, the processor 102 controls the reception and manipulation of input data, and the output and display of data on output devices (e.g., display 118). In some embodiments, processor 102 includes and/or is used to provide server functions described below with respect to server 202, etc. of FIG. 2.

Processor 102 is coupled bi-directionally with memory 110, which can include a first primary storage, typically a random access memory (RAM), and a second primary storage area, typically a read-only memory (ROM). As is well known in the art, primary storage can be used as a general storage area and as scratch-pad memory, and can also be used to store input data and processed data. Primary storage can also store programming instructions and data, in the form of data objects and text objects, in addition to other data and instructions for processes operating on processor 102. Also as is well known in the art, primary storage typically includes basic operating instructions, program code, data, and objects used by the processor 102 to perform its functions (e.g., programmed instructions). For example, memory 110 can include any suitable computer-readable storage media, described below, depending on whether, for example, data access needs to be bi-directional or uni-directional. For example, processor 102 can also directly and very rapidly retrieve and store frequently needed data in a cache memory (not shown).

A removable mass storage device 112 provides additional data storage capacity for the computer system 100, and is coupled either bi-directionally (read/write) or uni-directionally (read only) to processor 102. For example, storage 112 can also include computer-readable media such as magnetic tape, flash memory, PC-CARDS, portable mass storage devices, holographic storage devices, and other storage devices. A fixed mass storage 120 can also, for example, provide additional data storage capacity. The most common example of mass storage 120 is a hard disk drive. Mass storages 112, 120 generally store additional programming instructions, data, and the like that typically are not in active use by the processor 102. It will be appreciated that the information retained within mass storages 112 and 120 can be incorporated, if needed, in standard fashion as part of memory 110 (e.g., RAM) as virtual memory.

In addition to providing processor 102 access to storage subsystems, bus 114 can also be used to provide access to other subsystems and devices. As shown, these can include a display monitor 118, a network interface 116, a keyboard 104, and a pointing device 106, as well as an auxiliary input/output device interface, a sound card, speakers, and other subsystems as needed. For example, the pointing device 106 can be a mouse, stylus, track ball, or tablet, and is useful for interacting with a graphical user interface.

The network interface 116 allows processor 102 to be coupled to another computer, computer network, or telecommunications network using a network connection as shown. For example, through the network interface 116, the processor 102 can receive information (e.g., data objects or program instructions) from another network or output information to another network in the course of performing method/process steps. Information, often represented as a sequence of instructions to be executed on a processor, can be received from and outputted to another network. An interface card or similar device and appropriate software implemented by (e.g., executed/performed on) processor 102 can be used to connect the computer system 100 to an external network and transfer data according to standard protocols. For example, various process embodiments disclosed herein can be executed on processor 102, or can be performed across a network such as the Internet, intranet networks, or local area networks, in conjunction with a remote processor that shares a portion of the processing. Additional mass storage devices (not shown) can also be connected to processor 102 through network interface 116.

An auxiliary I/O device interface (not shown) can be used in conjunction with computer system 100. The auxiliary I/O device interface can include general and customized interfaces that allow the processor 102 to send and, more typically, receive data from other devices such as microphones, touch-sensitive displays, transducer card readers, tape readers, voice or handwriting recognizers, biometrics readers, cameras, portable mass storage devices, and other computers.

In addition, various embodiments disclosed herein further relate to computer storage products with a computer readable medium that includes program code for performing various computer-implemented operations. The computer-readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of computer-readable media include, but are not limited to, all the media mentioned above: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks; and specially configured hardware devices such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and ROM and RAM devices. Examples of program code include both machine code, as produced, for example, by a compiler, or files containing higher level code (e.g., script) that can be executed using an interpreter.

The computer system shown in FIG. 1 is but an example of a computer system suitable for use with the various embodiments disclosed herein. Other computer systems suitable for such use can include additional or fewer subsystems. In addition, bus 114 is illustrative of any interconnection scheme serving to link the subsystems. Other computer architectures having different configurations of subsystems can also be utilized.

FIG. 2 is a block diagram illustrating an embodiment of a data center that includes a scalable distributed metrics manager. In this example, client devices such as 252 connect to a data center 250 via a network 254. A client device can be a laptop computer, a desktop computer, a tablet, a mobile device, a smart phone, a wearable networking device, or any other appropriate computing device. In some embodiments, a web browser and/or a standalone client application is installed at each client, enabling a user to use the client device to access certain applications hosted by data center 250. Network 254 can be the Internet, a private network, a hybrid network, or any other communications network.

In the example shown, a networking layer 255 comprising networking devices such as routers, switches, etc. forwards requests from client devices 252 to a distributed network service platform 204. In this example, distributed network service platform 204 includes a number of servers configured to provide a distributed network service. A physical server (e.g., 202, 204, 206, etc.) has hardware components and software components, and may be implemented using a device such as 100. In this example, hardware (e.g., 208) of the server supports operating system software in which a number of virtual machines (VMs) (e.g., 218, 219, 220, etc.) are configured to execute. A VM is a software implementation of a machine (e.g., a computer) that simulates the way a physical machine executes programs. The part of the server's operating system that manages the VMs is referred to as the hypervisor. The hypervisor interfaces between the physical hardware and the VMs, providing a layer of abstraction to the VMs. Through its management of the VMs' sharing of the physical hardware resources, the hypervisor makes it appear as though each VM were running on its own dedicated hardware. Examples of hypervisors include the VMware Workstation® and Oracle VM VirtualBox®. Although physical servers supporting VM architecture are shown and discussed extensively for purposes of example, physical servers supporting other architectures such as container-based architecture (e.g., Kubernetes®, Docker®, Mesos®), standard operating systems, etc., can also be used and techniques described herein are also applicable. In a container-based architecture, for example, the applications are executed in special containers rather than virtual machines.

In some embodiments, instances of applications are configured to execute within the VMs. Examples of such applications include web applications such as shopping cart, user authentication, credit card authentication, email, file sharing, virtual desktops, voice/video streaming, online collaboration, and many others.

One or more service engines (e.g., 214, 224, etc.) are instantiated on a physical device. In some embodiments, a service engine is implemented as software executing in a virtual machine. The service engine is executed to provide distributed network services for applications executing on the same physical server as the service engine, and/or for applications executing on different physical servers. In some embodiments, the service engine is configured to enable appropriate service components that implement service logic. For example, a load balancer component is executed to provide load balancing logic to distribute traffic load amongst instances of applications executing on the local physical device as well as other physical devices; a firewall component is executed to provide firewall logic to instances of the applications on various devices; a metrics agent component is executed to gather metrics associated with traffic, performance, etc. associated with the instances of the applications, etc. Many other service components may be implemented and enabled as appropriate. When a specific service is desired, a corresponding service component is configured and invoked by the service engine to execute in a VM.

In the example shown, traffic received on a physical port of a server (e.g., a communications interface such as Ethernet port 215) is sent to a virtual switch (e.g., 212). In some embodiments, the virtual switch is configured to use an API provided by the hypervisor to intercept incoming traffic designated for the application(s) in an inline mode, and send the traffic to an appropriate service engine. In inline mode, packets are forwarded on without being replicated. As shown, the virtual switch passes the traffic to a service engine in the distributed network service layer (e.g., the service engine on the same physical device), which transforms the packets if needed and redirects the packets to the appropriate application. The service engine, based on factors such as configured rules and operating conditions, redirects the traffic to an appropriate application executing in a VM on a server. Details of the virtual switch and its operations are outside the scope of the present application.

Controller 290 is configured to control, monitor, program, and/or provision the distributed network services and virtual machines. In particular, the controller includes a metrics manager 292 configured to collect performance metrics and perform analytical operations. The controller can be implemented as software, hardware, firmware, or any combination thereof. In some embodiments, the controller is implemented on a system such as 100. In some cases, the controller is implemented as a single entity logically, but multiple instances of the controller are installed and executed on multiple physical devices to provide high availability and increased capacity. In embodiments implementing multiple controllers, known techniques such as those used in distributed databases are applied to synchronize and maintain coherency of data among the controller instances.

Within data center 250, one or more controllers 290 gather metrics data from various nodes operating in the data center. As used herein, a node refers to a computing element that is a source of metrics information. Examples of nodes include virtual machines, networking devices, service engines, or any other appropriate elements within the data center.

Many different types of metrics can be collected by the controller. For example, since traffic (e.g., connection requests and responses, etc.) to and from an application will pass through a corresponding service engine, metrics relating to the performance of the application and/or the VM executing the application can be directly collected by the corresponding service engine. As another example, to collect metrics relating to client responses, a service engine sends a script to a client browser or client application. The script measures client responses and returns one or more collected metrics back to the service engine. In both cases, the service engine sends the collected metrics to controller 290. Additionally, infrastructure metrics relating to the performance of other components of the service platform (e.g., metrics relating to the networking devices, metrics relating to the performance of the service engines themselves, metrics relating to the host devices such as data storage as well as operating system performance, etc.) can be collected by the controller. Specific examples of the metrics include round trip time, latency, bandwidth, number of connections, etc.

The components and arrangement of distributed network service platform 204 described above are for purposes of illustration only. The technique described herein is applicable to network service platforms having different components and/or arrangements.

FIG. 3A is a block diagram illustrating an embodiment of a metrics pipeline in a scalable distributed metrics manager. Pipeline 300 implements the process for aggregating metrics and can be used to implement scalable metrics manager 292. A pipeline processes one or more specific types of metrics, and multiple pipelines similar to 300 can be configured to process different types of metrics. In this example, pipeline 300 receives metrics from a variety of sources, such as service engines 214, 224, etc., via data streams. Metrics can also be received from other sources such as network devices, an operating system, a virtual switch, etc. (not shown). The performance metrics are continuously collected at various sources (e.g., service engines, network devices, etc.) and sent to the first stage (e.g., stage 302 of FIG. 3A) of the pipeline. The rate at which a metric is generated is arbitrary and can vary for different sources. For example, one service engine can generate metrics at a rate of 1 metric/second, while another service engine can generate metrics at a rate of 2 metrics/second.

Pipeline 300 comprises multiple stages aggregating metrics at different rates. In particular, the first stage aggregates raw metrics, and each successive stage aggregates the outputs from the previous stage at a lower rate (or equivalently, a coarser granularity of time or lower frequency). In the example shown, three stages are used: stage 302 aggregates metrics from their sources every 5 seconds, stage 304 aggregates the results of stage 302 every 5 minutes, and stage 306 aggregates the results of stage 304 aggregated every hour. Different numbers of stages and/or aggregation rates can be used in other embodiments. The metrics pipeline is implemented in memory to allow for fast access and analytical operations. Each stage only needs to maintain a sufficient number of inputs (e.g., metrics or results from the previous stage) in memory to perform aggregation, thus the overall number of metrics to be stored and the total amount of memory required for real time analysis are reasonable and can be implemented for high scale environments such as enterprise data centers where large volumes of metrics are constantly generated. Note that although separate buffers are shown for the output of one stage and the input of the next stage, in some implementations only one set of buffers needs to be maintained. As will be described in greater detail below, each stage performs one or more aggregation functions to generate aggregated results. Further, each of the pipeline stages is optionally connected to a persistent storage 310, such as a database, a file system, or any other appropriate non-volatile storage system, in order to write the aggregated results to the storage and back up the metrics data more permanently. For example, MongoDB is used in some implementations. Further details of the pipeline's operations are explained in connection with FIG. 4 below.

FIG. 3B is a diagram illustrating an embodiment of a metric data structure. In this example, metric 350 is a key-tuple data structure that includes the following fields: {MetricsObjectType, Entity, Node, ObjectID}. Depending on implementation, the values in the fields can be alphanumeric strings, numerical values, or any other appropriate data formats. MetricsObjectType specifies the type of metric being sent. Examples of MetricsObjectType include client metric, front end network metric, backend network metric, application metric, etc. Entity specifies the particular element about which the metric is being reported, such as a particular server or application executing on a virtual machine. Node specifies the particular element that is reporting the metric, such as a particular service engine. An Entity may include multiple objects, and ObjectID specifies the particular object within the entity that is generating the metric, such as a particular Internet Protocol (IP) address, a particular port, a particular Universal Resource Identifier (URI), etc. In the example shown, MetricsObjectType is set to vserver_14_client (which corresponds to a type of metric related to virtual server layer 4 client), Entity is set to vs−1 (which corresponds to a server with the identifier of vs−1), Node is set to se−1 (which corresponds to a service engine with the identifier of se−1), and ObjectID is set to port (which corresponds to the port object within the server). When a metric is stored to the database, each field can be used as an index for lookups. Metrics with different fields can be defined and used. For example, in one implementation, a metric also includes a timestamp field.

In some embodiments, a source can report metrics to the metrics manager without requiring explicit registration with the metrics manager. A source can report metrics to the metrics manager by sending one or more messages having predetermined formats. FIG. 3C is a diagram illustrating an embodiment of a metrics message. The message includes a header that specifies certain characteristics of the metrics being sent (e.g., number of metrics in the message, timestamp of when the message is sent, etc.), and multiple metrics in the message body. In this example, the node batches multiple metrics in a single message and sends the message to the metrics manager.

In some implementations, the metrics manager maintains multiple pipelines to process different types of metrics. Upon receiving a metrics message, the metrics manager parses the message to obtain metrics and places each metric in an appropriate pipeline for processing. In some embodiments, upon detecting that a metric includes a new instance of key-tuples as discussed above, the metrics manager establishes a new in-memory processing unit (e.g., a specific pipeline such as 300 that is configured with its own memory, thread, and/or process for handling metrics associated with the key-tuple), and future metrics messages having the same key-tuple will be processed by this in-memory processing unit. In some embodiments, the metrics manager can establish one or more pipelines that receive as inputs multiple key-tuple in order to generate certain desired results. The configuration of specific pipelines depends on implementation.

FIG. 4 is a flowchart illustrating an embodiment of a process for managing metrics. Process 400 can be implemented by scalable metrics manager 292 operating on a system such as 100.

At 401, metrics associated with a plurality of sources are obtained. As discussed above, the metrics can be sent in messages to the metrics manager.

The obtained metrics are managed in a metrics pipeline as described above in connection with FIG. 3A.

Specifically, at 402, the metrics associated with a plurality of sources are aggregated at a first rate to generate a plurality of first aggregated results.

In some cases, aggregation includes applying one or more transform operations (also referred to as aggregation operations) that transform the received metrics to generate new metrics. For example, suppose that four instances of a particular application periodically generate a set of four metrics reporting the number of connections to each application instance. One or more aggregation functions (F1) can be performed to combine (e.g., add) the four metrics to generate a new aggregated result of the total number of connections, average the four metrics to generate a new aggregated result of average number of connections, determine the minimum and/or maximum number of connections among the four metrics, compute the difference between the maximum number of connections and the minimum number of connections, etc. Many other aggregation/transform functions are possible for various metrics manager implementations. In some cases, the raw metrics are sampled at the first rate to generate the aggregated results. More commonly, a transform operation generates a corresponding aggregated result (also referred to as derived metric) based on the inputs to the transform function. Multiple transform operations can generate a vector of aggregated results. Aggregations can be performed across service engines, across multiple servers in a pool, across multiple entities, across multiple objects, etc. A pipeline can be configured to transform any appropriate metrics into a new result. The specific aggregation functions in a pipeline can be configured by the programmer or administrator according to actual system needs.

The first rate at which aggregation takes place corresponds to the rate at which the aggregation function is performed on the collected data. In the following examples, a constant rate is discussed extensively for purposes of example, but the rate can be a non-constant rate as well (e.g., aggregation happens when the number of metrics collected meets or exceeds a threshold or when some other triggering condition for aggregation is met). Because the aggregation only uses metrics stored in memory, it does not require any database calls and is highly efficient. Further, because the aggregation is done periodically and in a batched fashion (e.g., all first stages of the pipelines perform aggregation every 5 seconds), timers do not need to be maintained per object or per metric. Thus, aggregation can be performed quickly and efficiently.

The received metrics are temporarily maintained in a memory such as a Random Access Memory (RAM). The first aggregated results and/or the received metrics will be rolled up into the next stage periodically. The first stage only needs to maintain a sufficient amount of input metrics in the memory until the first aggregation is performed, after which the metrics used in the aggregation can be removed from the memory in order to save space and make room for new aggregated results and/or metrics. Before being deleted from the memory, the first aggregated results and/or the obtained metrics are optionally output to a persistent storage such as a database. In some embodiments, the aggregation of the performance metrics and the maintenance of the first aggregated results are performed in a single process to reduce the overhead of context switches. It is permissible to implement the aggregation and the maintenance steps in separate processes.

As will be described in greater detail below, analytical operation, event detection and generation, as well as storing the aggregation results and/or the metrics to a persistent storage can be performed.

At 404, the first aggregated results are aggregated at a second rate to generate a plurality of second aggregated results. This is also referred to as a roll-up operation. In this case, the second rate (which can also be constant or non-constant) is on average lower than the first rate, and the aggregation function performed in the second stage is not necessarily the same as the aggregation function performed in the first stage. Referring again to the example shown in FIG. 3A, aggregated results of the first stage (stage 302) are sent to the second stage (stage 304), to be aggregated at a rate of every 5 minutes. Suppose that the first stage generates an aggregated result every 5 seconds, then in every 5 minutes there will be 60 first aggregated results. These 60 first aggregated results from the first stage are aggregated again at the second stage according to the one or more aggregation functions (F2) specified in the second stage to generate one or more second aggregated results.

Similar to the first aggregated results, the second aggregated results are maintained in memory for a time. A sufficient number of the second aggregated results is maintained in the memory for the third stage of aggregation to be performed. The second aggregated results are optionally output to a persistent data store. After the second aggregated results are aggregated in the third stage, those second aggregated results that are used by the third stage for aggregation can be deleted from memory to save space. One or more analytical operations can be performed on the second aggregated results. Event detection and generation can also be performed on the second aggregated results. These operations are preferably implemented as inline operations of the metrics manager.

At 406, the plurality of second aggregated results is aggregated at a third rate to generate a plurality of third aggregated results. Referring again to the example shown in FIG. 3A, the aggregation results of the second stage (stage 304) are sent to the third stage (stage 306), to be aggregated at a rate of every hour. Suppose that the second stage generates an aggregated result every 5 minutes, then in one hour there will be 12 aggregated results. These 12 aggregated results from the second stage are aggregated again at the third stage according to one or more aggregation functions (F3) associated with the third stage.

Although three stages are shown for purposes of illustration, other numbers of stages (e.g., two stages, four, or more stages) can be implemented in various embodiments.

In the above process, at each stage, the metrics manager can invoke one or more corresponding analytical operations (such as anomaly detection) on the aggregated results. For example, the Holt-Winters algorithm used to detect outlier metrics and remove anomalies can be performed at any of the stages. As another example, an aggregated result (e.g., the total number of connections) is compared with a threshold (e.g., a maximum number of 100) to detect if the threshold has been exceeded. Many analytical operations are possible and can be configured by the programmer or administrator according to actual system needs. Preferably, the analytical operation is implemented as an inline function within the metrics manager process. The inline function implements additional steps in the continuous processing of the metrics within the same process and software module. Because the aggregated results are kept in memory rather than streamed to a database and because the analytical operation is inline, the analytical operation does not require any input/output (I/O) operations such as database read/write, inter-process communication, system call, messaging, etc. Thus, the analytical operations are highly efficient compared with existing analytics tools that typically require database access or file system access. The analytical operations can be performed in real time (e.g., at substantially the same time as when the performance metrics are received, or at substantially the same time as when the aggregated results are generated).

In some embodiments, the metrics manager generates events when metrics and/or aggregated results meet certain conditions. Event detection and generation can be implemented as an inline function where certain conditions are tested on a per metric type, per entity, and per node basis. For example, if the network connections metrics of a server sent by a service engine indicate that the connection exceeds a threshold, then an event such as an alarm or log is triggered. Other examples of event triggering conditions include: a metric meeting or exceeding a high watermark level for the first time after the metric has stayed below a low threshold; a metric meeting or falling below a low watermark level after the metric has stayed above the threshold; a metric crossing a predefined threshold; a metric indicating that an anomaly has occurred, etc. Many conditions are possible, and in some embodiments, a set of rules is specified for these conditions, and a rules processing engine compares the values associated with metrics against the rules to detect whether any specified conditions are met.

As discussed above, in some implementations metrics and aggregated results are recorded in a persistent storage such as a database for backup purposes. A retention policy is specified by the administrator to determine the amount of time for which corresponding stored data remains in the database. When the retention period is over, any data that is outside the retention policy period is erased from the database to conserve space.

FIGS. 5A-5B are diagrams illustrating an embodiment of an approach for archiving the aggregated results. In this example, aggregated results are written to the database at the time of aggregation, as shown in FIG. 5A. The retention policy periods for the first stage, the second stage, and the third stage are 2 hours, 2 days, and 1 year, respectively. Thus, after 2 hours, the database records corresponding to the first aggregated results that occurred before the current two hour window are deleted from the database, as shown in FIG. 5B. As can be seen, because the data from different stages is interspersed, deleting records associated with a particular stage can leave “holes” in the database and will slow down the query of the aggregated results, negatively impacting the database's write performance, and ultimately degrading the rate of aggregation.

To overcome the problem illustrated in FIGS. 5A-5B, time series based database tables are used in some embodiments. FIGS. 6A-6B are diagrams illustrating another embodiment of an approach for archiving the aggregated results. In this example, aggregated results from different stages of a pipeline occupy separate tables. Within a stage, multiple tables can be used to store the aggregated results. These tables are referred to as time series based database tables since they each correspond to a different period of aggregated results. Different time series based database tables can be subject to different retention policies. In this example, two tables are used to store the first aggregated results, where each table is configured to store one hour's worth of first aggregated results from the first stage; one table is used to store the second aggregated results, where the table is configured to store one day's worth of second aggregated results from the second stage; and one table is used to store the third aggregated results, where the table is configured to store one year's worth of third aggregated results from the third stage.

The aggregated results are written to the database in a batch in append mode. For example, the first aggregated results can be written to the database every 30 minutes rather than every five seconds. Maintaining a greater amount of aggregated results in memory permits less frequent database writes, which is more efficient. Thus, the rate at which the aggregated results are written can be configured based on tradeoffs of memory required to keep the aggregated results and efficiency of database writes. Further, the aggregated results are written to the database in tables according to the retention period of the corresponding retention policy.

Note that the table size does not need to exactly correspond to the amount of data generated during the retention period but can be on the same order of magnitude. Suppose the retention period for the first stage is two hours. Table 602 is initially filled with first aggregated results obtained during the first hour, and table 604 is initially filled with first aggregated results obtained during the second hour. In this example, at the end of two hours, some of the old aggregated results need to be removed to make room for new aggregated results. Thus, for data in the next two hour window, the entire contents of table 602 is deleted, and table 604 now stores aggregated data for the first hour, and table 602 is used to store aggregated data for the second hour. Because aggregated results are stored in separate tables and deleted separately, holes in the database are avoided.

Because the database uses the time series based database tables used to store aggregated results, when the database is queried, the query will not necessarily be performed on a single table. Thus, in some embodiments, the metric manager provides a query application programming interface (API) that hides the details of the underlying time series based database table and gives the appearance of making query to and receiving results from a single table.

FIG. 7 is a flowchart illustrating an embodiment of a process for querying metrics data stored in a database. Process 700 can be performed by the metrics manager in response to a database query, which can be initiated manually by a user via a user interface tool provided by a performance monitoring application, automatically by the performance monitoring application, etc.

At 702, the database query is analyzed to determine one or more corresponding time series based database tables associated with the database query. Specifically, the time window of the query is compared with the time windows of the time series based database tables.

At 704, it is determined whether the time window being queried spans only a single time series based database table. If so, the database query is performed normally without changes to the query, at 706. If, however, the time window being queried spans multiple time series based database tables, then the particular time series based database tables are determined and the process continues at 708.

At 708, the database query is converted into a union of multiple sub-queries across the determined time series based database tables.

At 710, filters from the database query are applied to the sub-queries such that the database's efficient filtering can be used optimally. The efficiency of filtering is gained as filters are applied on a per table basis before the results are joined together. Thus, the time complexity of filtering becomes K (the max number of rows in any table) instead of N (the number of combined rows across tables), where K<<N.

At 712, the sub-queries are performed on the database.

At 714, the responses to the sub-queries are combined into a single response.

This way, to the generator of the query (e.g., the performance monitoring application), it appears as if the query were performed on a single table.

FIG. 8 is a diagram illustrating an example of a query to a database comprising multiple time series based database tables. In FIG. 8, a plurality of database tables is used to store aggregated metrics from various stages of the pipeline. In particular, tables 802 and 804 are shown to store the first hour of the first aggregated results and the second hour of the first aggregated results, respectively. As shown, table 802 stores metrics gathered between 21:00:00-22:59:55 and table 804 stores metrics gathered between 22:00:00-22:59:55. Metrics in both tables are gathered in 5-second increments.

Suppose the following database query is made to query the database:

SELECT se_stats_table.metric_timestamp AS se_stats_table_metric_timestamp,

se_stats_table.avg_cpu_usage AS se_stats_table_avg_cpu_usage, entity_table.entity_id AS entity_id

FROM se_stats_table JOIN entity_table ON entity_table.entity_key=se_stats_table.entity_key

WHERE se_stats_table.metric_timestamp>=‘2015-03-19T21:03:25’ AND

se_stats_table.metric_timestamp<=‘2015-03-19T22:03:20’ AND se_stats_table.metric_period=‘5SECOND’ AND entity_table.entity_id=‘se−1’

Referring to FIG. 7, at 702, the database query is analyzed and it is determined that there are two time series based database tables (802 and 804) that correspond to the database query.

At 708 and 710, the database query is converted into a union of two sub-queries, and filters are applied to the sub-queries. In this example, the sub-queries correspond to their respective database tables. The sub-query that spans the time window of ‘2015-03-19T21:03:25’ to ‘2015-03-19T21:59:55’ is:

SELECT se_stats_table_1hour_396333.metric_timestamp AS

se_stats_table_1hour_396333_metric_timestamp, se_stats_table_1hour_396333.avg_cpu_usage

AS se_stats_table_1hour_396333_avg_cpu_usage, entity_table.entity_id AS entity_id

FROM se_stats_table_1hour_396333 JOIN entity_table ON entity_table.entity_key=

se_stats_table_1hour_396333.entity_key

WHERE se_stats_table_1hour_396333.metric_timestamp>=‘2015-03-19T21:03:25’ AND

se_stats_table_1hour_396333.metric_timestamp ‘2015-03-19T21:59:55’ AND

se_stats_table_1hour_396333.metric_period=‘5SECOND’ AND entity_table.entity_id=‘se−1’

The sub-query that spans the time window of 2015-03-19T22:00:00 to ‘2015-03-19T22:03:20’ is:

SELECT se_stats_table_1hour 396334.metric_timestamp AS

se_stats_table_1hour_396334_metric_timestamp, se_stats_table_1hour_396334.avg_cpu_usage

AS se_stats_table_1hour_396334 avg_cpu_usage, entity_table.entity_id AS entity_id

FROM se_stats_table_1hour_396334 JOIN entity_table ON entity_table.entity_key=

se_stats_table_1hour_396334.entity_key

WHERE se_stats_table_1hour_396334.metric_timestamp>=‘2015-03-19T22:00:00’ AND

se_stats_table_1hour_396334.metric_timestamp ‘2015-03-19T22:03:20’ AND

se_stats_table_1hour_396334.metric_period=‘5SECOND’ AND entity_table.entity_id=‘se−1’

The union of the sub-queries with filters is:

SELECT anon 1.se_stats_table_1hour_396333 metric_timestamp AS metric_timestamp,

anon_1.se_stats_table_1hour_396333 avg_cpu_usage AS avg_cpu_usage, anon_1.entity_id AS

entity_id

FROM (SELECT se_stats_table_1hour_396333.metric_timestamp AS

se_stats_table_1hour_396333 metric_timestamp, se_stats_table_1hour_396333.avg_cpu_usage

AS se_stats_table_1hour_396333 avg_cpu_usage, entity_table.entity_id AS entity_id

FROM se_stats_table_1hour_396333 JOIN entity_table ON entity_table.entity_key=se_stats_table_1hour_396333.entity_key

WHERE se_stats_table_1hour_396333.metric_timestamp>=‘2015-03-19T21:03:25’ AND

se_stats_table_1hour_396333.metric_timestamp<=‘2015-03-19T21:59:55’ AND

se_stats_table_1hour_396333.metric_period=‘5SECOND’ AND entity_table.entity_id=‘se−1’

UNION ALL SELECT se_stats_table_1hour_396334.metric_timestamp AS

se_stats_table_1hour_396334 metric_timestamp, se_stats_table_1hour_396334.avg_cpu_usage

AS se_stats_table_1hour_396334 avg_cpu_usage, entity_table.entity_id AS entity_id

FROM se_stats_table_1hour_396334 JOIN entity_table ON entity_table.entity_key=se_stats_table_1hour_396334.entity_key

WHERE se_stats_table_1hour_396334.metric_timestamp>=‘2015-03-19T22:00:00’ AND

se_stats_table_1hour_396334.metric_timestamp ‘2015-03-19T22:03:20’ AND

se_stats_table_1hour_396334.metric_period=‘5SECOND’ AND entity_table.entity_id=‘se−1’)

AS anon_1

ORDER BY anon_1.se_stats_table_1hour_396333 metric_timestamp

LIMIT 720

The rearrangement of the query across multiple time series tables shown above does not compromise the performance of read operations to the database, and facilitates efficient write operations to the database by the metrics manager.

Managing performance metrics has been disclosed. By processing the metrics in a pipeline in memory, the technique described above significantly reduces the amount of I/O operations and latency associated with processing the metrics, and allows for real time analytics.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A method of producing aggregated performance metric sets associated with a set of elements in a network, the method comprising: iteratively; (i) storing a plurality of performance metric sets, associated with the set of network elements, in a memory until a first threshold amount of storage is reached; and(ii) upon reaching the first threshold amount, performing a first aggregation operation on the performance metric sets stored in a particular iteration to generate an aggregate performance metric set that is at a first aggregation level;storing a plurality of aggregate performance metric sets generated by a plurality of iterations at the first aggregation level in the memory until a second threshold amount of storage is reached; andupon reaching the second threshold amount, performing a second aggregation operation on the plurality of aggregate performance metric sets at the first aggregation level to generate an aggregate performance metric set at a second aggregation level,said first and second aggregation operations performed to reduce an amount of memory consumed while producing aggregated performance metric sets for a set of analytical tools to analyze.

2. The method of claim 1 further comprising: iteratively; storing a plurality of aggregate performance metric sets generated by a plurality of second aggregation operations at the second aggregation level in the memory until a third threshold amount of storage is reached; andupon reaching the third threshold amount, performing a third aggregation operation on the plurality of aggregate performance metric sets at the second aggregation level to generate an aggregate performance metric set at a third aggregation level.

3. The method of claim 2 further comprising outputting at least one of (1) the plurality of aggregate performance metric sets at the second aggregation level and (2) the aggregate performance metric set at the third aggregation level to a persistent storage.

4. The method of claim 1 further comprising performing a first set of analytical operations based at least in part on the plurality of aggregate performance metric sets at the first aggregation level to detect anomalies in the plurality of aggregate performance metric sets at the first aggregation level.

5. The method of claim 1, wherein the first aggregation operation comprises one or more of: averaging values associated with a first performance metric in the plurality of performance metric sets;determining a maximum of a set of values associated with a second performance metric in the plurality of performance metric sets;determining a minimum of a set of values associated with a third performance metric in the plurality of performance metric sets; andcalculating a difference between a maximum and minimum value associated with a fourth performance metric in the plurality of performance metric sets.

6. The method of claim 1, wherein the set of network elements comprises a plurality of compute nodes executing on a plurality of host computers and the plurality of performance metric sets is received from a plurality of service engines executing on the plurality of host computers to provide a distributed network service to the plurality of compute nodes.

7. The method of claim 1 further comprising: performing a first set of analytical operations based at least in part on the plurality of aggregate performance metric sets at the first aggregation level to detect anomalies in the plurality of aggregate performance metric sets at the first aggregation level; andperforming a second set of analytical operations based at least in part on the aggregate performance metric set at the second aggregation level to detect anomalies in the aggregate performance metric set at the second aggregation level.

8. The method of claim 1, wherein the plurality of aggregate performance metric sets at the first aggregation level are removed from the memory after the plurality of aggregate performance metric sets defined at the first aggregation level are aggregated into the aggregate performance metric set at the second aggregation level.

9. The method of claim 1 further comprising performing event detection based on one or more of: the plurality of performance metric sets, the generated aggregate performance metric sets at the first aggregation level, and the aggregate performance metric set at the second aggregation level.

10. The method of claim 1 further comprising, after the first aggregation operation on the performance metric sets stored in the particular iteration, removing the plurality of performance metric sets stored in the particular iteration from the memory.

11. The method of claim 1, wherein at least one of (1) the plurality of aggregate performance metric sets at the first aggregation level and (2) the aggregate performance metric set at the second aggregation level are persisted to a persistent storage.

12. The method of claim 1, wherein: the plurality of aggregate performance metric sets at the first aggregation level are persisted to one or more first time-series-based database tables in a persistent storage;a plurality of generated aggregate performance metric sets at the second aggregation level are persisted to one or more second time-series-based database tables in the persistent storage;the one or more first time-series-based database tables are separate from the one or more second time-series-based database tables;data stored in the one or more first time-series-based database tables is subject to a first retention period; anddata stored in the one or more second time-series-based database tables is subject to a second retention period that is different from the first retention period.

13. The method of claim 12 further comprising: analyzing a database query to determine whether there are a plurality of corresponding time-series-based database tables associated with the database query;in the event that there are a plurality of corresponding time-series-based database tables associated with the database query, converting the database query into a union of a plurality of sub-queries across the plurality of corresponding time-series-based database tables associated with the database query;applying any filters from the database query to the sub-queries;performing the plurality of sub-queries; andcombining responses to the plurality of sub-queries into one response to the database query.

14. A system, comprising: a set of one or more processors;one or more memories coupled to the one or more processors; anda non-transitory machine readable medium storing a program for execution by the set of processors, the program for producing aggregated performance metric sets associated with a set of elements in a network, the program comprising sets of instructions for: iteratively; (i) storing a plurality of performance metric sets, associated with the set of network elements, in a memory of the one or more memories until a first threshold amount of storage is reached; and(ii) upon reaching the first threshold amount, performing a first aggregation operation on the performance metric sets stored in a particular iteration to generate an aggregate performance metric set that is at a first aggregation level;storing a plurality of aggregate performance metric sets generated by a plurality of iterations at the first aggregation level in the memory until a second threshold amount of storage is reached; andupon reaching the second threshold amount, performing a second aggregation operation on the plurality of aggregate performance metric sets at the first aggregation level to generate an aggregate performance metric set at a second aggregation level,said first and second aggregation operations performed to reduce an amount of memory consumed while producing aggregated performance metric sets for a set of analytical tools to analyze.

15. The system of claim 14, wherein the program further comprises a set of instructions for: iteratively storing a plurality of aggregate performance metric sets generated by a plurality of second aggregation operations at the second aggregation level in the memory until a third threshold amount of storage is reached; andupon reaching the third threshold amount, performing a third aggregation operation on the plurality of aggregate performance metric sets at the second aggregation level to generate an aggregate performance metric set at a third aggregation level.

16. The system of claim 14, wherein the program further comprises a set of instructions for outputting at least one of (1) the plurality of aggregate performance metric sets at the first aggregation level, and the aggregate performance metric set at the second aggregation level are persisted to a persistent storage.

17. The system of claim 14, wherein the first aggregation operation comprises one or more of: averaging values associated with a first performance metric in the plurality of performance metric sets;determining a maximum of a set of values associated with a second performance metric in the plurality of performance metric sets;determining a minimum of a set of values associated with a third performance metric in the plurality of performance metric sets; andcalculating a difference between a maximum and minimum value associated with a fourth performance metric in the plurality of performance metric sets.

18. The system of claim 14, wherein the plurality of aggregate performance metric sets at the first aggregation level are removed from the memory after the plurality of aggregate performance metric sets at the first aggregation level are aggregated into the aggregate performance metric set at the second aggregation level.

19. A non-transitory computer readable medium storing a program for execution by the set of processors, the program for producing aggregated performance metrics associated with a set of elements in a network, the program comprising sets of instructions for: iteratively; (i) storing a plurality of performance metric sets, associated with the set of network elements, in a memory until a first threshold amount of storage is reached; and(ii) upon reaching the first threshold amount, performing a first aggregation operation on the performance metric sets stored in a particular iteration to generate an aggregate performance metric set that is at a first aggregation level;storing a plurality of aggregate performance metric sets generated by a plurality of iterations at the first aggregation level in the memory until a second threshold amount of storage is reached; andupon reaching the second threshold amount, performing a second aggregation operation on the plurality of aggregate performance metric sets at the first aggregation level to generate an aggregate performance metric set at a second aggregation level,said first and second aggregation operations performed to reduce an amount of memory consumed while producing aggregated performance metric sets for a set of analytical tools to analyze.

20. The non-transitory machine readable medium of claim 19, wherein the set of network elements comprises a plurality of compute nodes executing on a plurality of host computers and the plurality of performance metrics is received from a plurality of service engines executing on the plurality of host computers to provide a distributed network service to the plurality of compute nodes.