Patent Application
20050262237
This invention pertains to network, server, and service monitoring; more specifically, it pertains to dynamic identification, tracking, and investigation of service performance and availability incidents based on monitoring of application network communications. The service may be provided by a single device, a network of devices, applications running on a device or network, etc.
Almost from the earliest days of computing, users have been attaching devices together to form networks. Several types of networks include local area networks (LANs), metropolitan area networks (MANs) and wide area networks (WANs). One particular example of a WAN is the Internet, which connects millions of computers around the world.
Networks provide users with the capacity of dedicating particular computers to specific tasks and sharing resources such as a printer, applications and memory among multiple machines and users. A computer that provides functionality to other computers on a network is commonly referred to as a server. Communication among computers and devices on a network is typically referred to as traffic.
Of course, the networking of computers adds a level of complexity that is not present with a single machine, standing alone. A problem in one area of a network, whether with a particular computer or with the communication media that connects the various computers and devices, can cause problems for all the computers and devices that make up the network. For example a file server, a computer that provides disk resources to other machines, may prevent the other machines from accessing or storing critical data; it thus prevents machines that depend upon the disk resources from performing their tasks.
Network and MIS managers are motivated to keep business-critical applications running smoothly across the networks separating servers from end-users. They would like to be able to monitor response time behavior experienced by the users, and to clearly identify potential network and server bottlenecks as quickly as possible. They would also like the management/maintenance of the monitoring system to have a low man-hour cost due to the critical shortage of human expertise. It is desired that the information be consistently reliable, with few false positives (else the alarms will be ignored) and few false negatives (else problems will not be noticed quickly).
Existing response-time monitoring solutions fall into one of three main categories: those requiring a client-site agent (an agent located near the client, on the same site as the client); subscription service; and solutions for specialized applications only. These existing solutions are briefly described below.
There are several existing response-time monitoring tools (e.g., NetIQ's Pegasus and Compuware's Ecoscope) that require a hardware and/or software agent be installed near each client site from which end-to-end or total response times are to be computed. The main problem with this approach is that it can be difficult or impossible to get the agents installed and keep them operating. For a global network, the number of agents can be significant; installation can be slow and maintenance painful. For an eCommerce site, installation of the agents is not practical; requesting potential customers to install software on their computers probably would not meet with much success. A secondary issue with this approach is that each of the client-site agents must upload their measurements to a centralized management platform; this adds unnecessary traffic on what may be expensive wide-area links. A third issue with this approach is that it is difficult to accurately separate the network from server delay contributions.
To overcome the issue with numerous agent installs, some companies (e.g., KeyNotes and Mercury Interactive) offer a subscription service whereby one may use their preinstalled agents for response-time monitoring. There are two main problems with this approach. One is that the agents are not monitoring “real” client traffic but are artificially generating a handful of “defined” transactions. The other is that the monitoring does not generally cover the full range of client sites—the monitoring is limited to where the service provider has installed agents.
A third approach used by a few companies is to provide a monitoring solution via a server-site agent (an agent located near the server, on the same site as the server), rather than a client-site agent. The shortcoming with some of these tools is that they either support only a single application (e.g., SAP/R3 or web), or that they are using generated Internet control message protocol (ICMP) packets rather than the actual client application packets to estimate network response times, or that they assume a constant network response time throughout the life of a TCP session. The ICMP packets may be treated very different than the actual client application packets because of their protocol (separate management queue and/or QoS policy), their size (serialization and/or scheduling discipline), and their timing (not sent at same time as the application packets). Network response times typically vary considerably throughout a TCP session. Other of these tools, such as the NetQoS(™) SuperAgent(™) service monitor, does not have these shortcomings.
A common monitoring technique is to dedicate a particular device, such as a probe or server, to passively monitor the service (provided by a network, system, and/or application) in order to identify troublesome traffic. However, this method does not distinguish whether a particular busy period represents a normal or abnormal deviation. For example, at the start of a business day it may be common for many users to simultaneously log in to their machines and access a given application, generating a spike in network traffic. Further, during a holiday period, a business network may normally have very little or no traffic.
Another common monitoring technique is the use of active agents to periodically test (or probe) the network, including computers and devices connected to the network and any particular services those computers and devices provide. If such an agent is scheduled to run every fifteen (15) minutes, then this implies that on average it will detect a sustained outage after seven and one half (7.5) minutes have elapsed. Intermittent, brief outages may very well go undetected. More frequent probing allows the agent to detect sustained outages more quickly and increases the probability the agent will detect intermittent issues; but more frequent probing places an additional, and sometimes unacceptable, load on the environment.
Developers continue to improve methods and systems for testing networks, servers and services for availability and performance. Among what is needed is a reliable method and system for monitoring networks, servers and services for availability and performance that provides sufficiently accurate information while avoiding excessive load on the networks, servers and services. Another issue, however, is the complexity of interpreting the rich dense data that arises from the monitoring. Among what is needed is intelligent automation that identifies issues and probably causes.
Embodiments are directed to providing a system and method of monitoring a data network and its services that incorporates both passive and active approaches and thereby benefits from the advantages of both approaches while avoiding the drawbacks of either. In a manner suitable for LANs, Manes and WANs, a Service Monitor provides server-side monitoring of a computing environment. The method includes monitoring application network transactions and behaviors for a computing environment including one or more client subnets accessing a service provided by one or more servers; decomposing the monitored transactions into network, server and application delay components; using the original and decomposed delay components to identify application(s), server(s) and/or client subnet(s) associated with a response-time issue; and implementing an active investigation on the applications and/or servers and/or client subnets. Additionally, the method includes monitoring application network transactions for a computing environment including one or more client subnets accessing a service provided by one or more servers; deriving non-delay quality metrics (e.g., loss rates, good put) from the monitored transactions; using these quality metrics to identify application(s), server(s) and/or client subnet(s) associated with a quality issue; and implementing an active investigation on the applications and/or servers and/or network devices and/or client subnets. The active investigation includes gathering statistical data to assist root cause analysis without causing an interruption of service monitoring.
The invention provides a method of monitoring a data network and its services that incorporates both passive and active approaches and thereby benefits from the advantages of both approaches while avoiding the drawbacks of either. In a manner suitable for LANs, Manes and WANs, a Service Monitor collects information related to service traffic on a target network. The information is correlated to specific devices on the network and specific services provided by the devices. The correlated information is employed to construct a profile of the network's traffic as the traffic relates to devices and services. The profile is used to monitor the network for periods of either less than or more than typical amounts of traffic corresponding to the devices and services. If such a period is detected, then intelligent agents investigate to determine whether or not a problem exists.
In addition, parameters are defined for “exclusion periods,” i.e. particular times that information is not collected. For example, during a Monday holiday, a business network might typically be expected to show less than the common data traffic for a service(s). Similarly during server maintenance windows, server traffic would be atypical. By excluding this data from the generation of a profile of typical Monday business days, a more accurate profile is generated.
In one embodiment, the method includes analyzing the decomposed components and derived metrics to identify anomalies, reduce alarms, perform an active investigation, and further isolate an identified problem. The decomposing can be based on response size. If the element with an identified problem is a server, the statistical data can include server statistics, and if the element with an identified problem is a client subnet, the statistical data can include network statistics.
The active investigation can include either a continuous mode or a snapshot mode. A snapshot mode can be operational only when triggered by an event, the snapshot mode providing a snapshot of performance around a predetermined period of time, such as about five to 15 minutes from the beginning of an event. The snapshot does not have to include context or historical information. The continuous mode can poll a source of network or server or service information continuously to provide a performance history and store and report performance data in a database for storing the event detection data concerning anomalies in the computer environment. Also, the continuous mode can store and report performance data in a dedicated database for active investigations.
In another embodiment, the monitoring is server-side monitoring that includes event detection capable of identifying sudden, gradual, and/or periodic anomalies in the service via auto-thresholding according to one or more baselines. The baselines can include one or more of baselines based on a past week, based on a same day of week over three months, based on a same day of week and similar day of month over six months, based on an hourly calculation, based on work days, or based on user-configured time periods. The baselines may use time filters to exclude “atypical” time periods—such as maintenance windows. The baselines may use other criteria to exclude “atypical” time periods, such as time intervals containing a very low number of measurements. The auto-thresholding can calculate a single threshold from a weighted average of each baseline calculation, or the server-side monitoring can include checking data against each baseline threshold individually and record any baseline violated, each violation indicative of a different problem.
A violation can be of a 6-month baseline threshold but not a 7-day baseline threshold, which indicates a gradual increase condition, in which case the active investigation includes inspecting time-series event data.
Another embodiment is directed to a service monitoring system configured to monitor application network transactions and behaviors for the computing environment. The system includes an event detection module capable of operating independent of client site monitors, the event detection module configured to decompose the monitored transactions and behaviors into at least network, server and application delay components and to use the original and decomposed delay components along with other derived quality metrics to identify one or more of the services, servers, networks and client subnets as being associated with a response-time or other quality issue. The system further includes one or more active investigation modules coupled to the event detection modules, the active investigation modules configured to investigate the one or more services, servers and client subnets according to criteria determined by the event detection module, the active investigation module configured to gather statistical data to assist root cause analysis independent of a service monitoring interruption. The system can include a data store coupled to the service monitor, the data store configured to hold one or more of historic data, sensitivity data, threshold data, server settings, investigation settings, incident data, current configuration data and metrics collected by the service monitor.
In one embodiment, the system event detection component interacts with a second monitoring system disposed in a network performance agent, the network performance agent disposed near one or more clients or servers. The event detection component can act on data from multiple service monitors distributed across the globe. Active investigations are launched from the appropriate service monitors to collect relevant information pertaining to the service degradation.
These and other advantages of the invention, as well as additional inventive features, will be apparent from the description of the invention provided herein.
This summary is not intended as a comprehensive description of the claimed subject matter but, rather is intended to provide a short overview of some of the matter's functionality. Other systems, methods, features and advantages of the invention will be or will become apparent to one with skill in the art upon examination of the following FIGUREs and detailed description. It is intended that all such additional systems, methods, features and advantages be included within this description, be within the scope of the invention, and be protected by the accompanying claims.
For a more complete understanding of the present invention, and the advantages thereof, reference is now made to the following brief descriptions taken in conjunction with the accompanying FIGUREs, in which like reference numerals indicate like features.
a and 13b are flow diagrams illustrating an Examine Issues process flowing from
Although described with particular reference to a computing environment that includes personal computers (PCs), a wide area network (WAN) and the Internet, the claimed subject matter can be implemented in any information technology (IT) system in which it is necessary or desirable to monitor performance of a network and individual system, computers and devices on the network. Those with skill in the computing arts will recognize that the disclosed embodiments have relevance to a wide variety of computing environments in addition to those specific examples described below. In addition, the methods of the disclosed invention can be implemented in software, hardware, or a combination of software and hardware. The hardware portion can be implemented using specialized logic; the software portion can be stored in a memory and executed by a suitable instruction execution system such as a microprocessor, PC or mainframe.
All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.
In the context of this document, a “memory,” “recording medium” and “data store” can be any means that contains, stores, communicates, propagates, or transports the program and/or data for use by or in conjunction with an instruction execution system, apparatus or device. Memory, recording medium and data store can be, but are not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device. Memory, recording medium and data store also includes, but is not limited to, for example the following: a portable computer diskette, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), and a portable compact disk read-only memory or another suitable medium upon which a program and/or data may be stored.
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments wherein tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote computer storage media including memory storage devices.
With reference to
The computer 10 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by the computer 10 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer 10. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
The system memory 30 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 31 and random access memory (RAM) 32. A basic input/output system 33 (BIOS), containing the basic routines that help to transfer information between elements within computer 10, such as during start-up, is typically stored in ROM 31. RAM 32 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 20. By way of example, and not limitation,
The computer 10 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media, discussed above and illustrated in
The computer 10 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 80. The remote computer 80 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 10, although only a memory storage device 81 has been illustrated in
When used in a LAN networking environment, the computer 10 is connected to the WAN 127 through a network interface or adapter 70. When used in a WAN networking environment, the computer 10 typically includes a modem 72 or other means for establishing communications over the WAN 73, such as the Internet. The modem 72, which may be internal or external, may be connected to the system bus 21 via the user input interface 60 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 10, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
In the description that follows, the invention will be described with reference to acts and symbolic representations of operations that are performed by one or more computers, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processing unit of the computer of electrical signals representing data in a structured form. This manipulation transforms the data or maintains it at locations in the memory system of the computer, which reconfigures or otherwise alters the operation of the computer in a manner well understood by those skilled in the art. The data structures where data is maintained are physical locations of the memory that have particular properties defined by the format of the data. However, while the invention is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that several of the acts and operation described hereinafter may also be implemented in hardware.
Referring now to
Each of Service Monitors 125(1,2,3) can be configured to implement all or some of the claimed subject matter and can be executed on one or more servers coupled to WAN 127, such as file server 121. The data provided by each of the Service Monitors is analyzed as a whole, such that each Service Monitor may provide additional insight and information into the source of the issue. Service Monitors 125 (1,2,3) could also be implemented on other computing systems, such as computing system client 101, on a dedicated application server such as application server 111, or on routers 117(1,2). Service Monitors 125(1,2,3) are explained in more detail below. Data store 113 can store an exemplary shared application 115. One example of a commonly shared application is a database management system (DBMS). One with skill in the computing arts should be familiar with applications and types of applications that are commonly implemented as shared applications.
Server 121 can be connected to the Internet or another LAN/WAN via any suitable communication medium such as, but not limited to, a dial-up telephone line, a digital subscriber line (DSL) or some type of wireless connection. Thus, file server 121 can be configured to provide a gateway, or access point to one or more computer networks, including the Internet.
Referring now to
As shown in
As described below, the computing environment 100 illustrates Service Monitors 125(1,2,3) that provide monitoring processes that report service behavior based on both active and passive monitoring and investigations. Advantageously, the Service Monitors operate either independent of agents at client sites or with agents at client sites. The Service Monitors may be placed anywhere along the network path, but the optimal (maximum benefit for the cost) locations are usually at the data centers. As described below, embodiments are directed to processes that operate within Service Monitors 125 to provide monitoring, which can include active or passive monitoring and can include application performance monitoring and service availability monitoring. More particularly, some embodiments are directed to determining appropriate active investigations based on passive observations. In one embodiment, Service Monitors actively investigate only when conclusions based on passive observations indicate that an active investigation is appropriate due to performance degradation. In another embodiment, a method is described that determines service availability according to a traffic determination attributable to a service.
Low Overhead Service Availability Monitoring
Examples of devices that might be the target of step 203 are computing system 10, file server 121, print servers, and connections to the Internet. Once a particular device is selected for monitoring, control proceeds to a “Check Services” step 205 during which process 200 monitors the services associated with the particular device selected in step 203. Check Services step 205 is described in more detail below in conjunction with
Following step 205, control proceeds to a “Was Any Service Detected?” step 207 during which process 200 determines whether or not any of the services associated with the particular device selected for monitoring in step 203 has been determined to be available during Check Services step 205. The theory is that, if a service is available, then the monitored device must also be available. If one service has been determined to be available, then control proceeds to a “Device Is Up” state 213. In one embodiment, if so configured, the state of the device can be stored in Current Metrics 171 of data store 123 (
If, in step 207, process 200 determines that no service associated with the selected device is available, then control proceeds to a “Probe Device” step 209 during which process 200 attempts to establish a connection or otherwise communicate with the targeted device. The transition from step 207 to step 209 represents a transition from passive component 151 to active component 153 in that a passively-detected condition indicates that affirmative action needs to be initiated to determine the state of the particular targeted device.
The particular method used to establish this connection depends upon the type of device. For example, if the targeted device is computing system 139, then an ICMP ping command may be sent to computing system 139 using an Internet protocol (IP) address associated with computing system 139 to determine whether or not computing system 139 is on-line or off-line. The device could also be a router.
Control proceeds from step 209 to a “Device Response?” step 211 during which process 200 determines whether or not the communication attempted in step 209 was successful. If the communication, whether a ping command or some other communication, was successful, then control proceeds to “Device Is Up” state 213 and metrics can be recorded if desired. If the attempted communication was not successful, then control proceeds to a “Device Is Down” state 215. If metrics are recorded, information gathered during steps 207, 209 and 211 corresponding to the current state, as indicated by one of states 213 and 215, and observed activity corresponding to the targeted device is stored in Current Metrics 171 of data store 123. Control then proceeds to “More Devices?” step 219 during which, process 200 determines whether or not each device listed in Current Configuration 169 has been monitored by process 200.
If there are unexamined devices listed in Current Configuration 169 that have not yet been processed in the current iteration of process 200, then control returns to Check Device Availability step 203, the next device in Current Configuration 169 is selected as the target and processing continues as described above. If, in step 219, process 200 determines there are no more devices to be monitored, then control proceeds to a “Sleep” step 221 during which a predefined interval of time is allowed to pass. Following the predefined interval of time, control then returns to Start Availability Check step 201 and processing continues as before starting from the top of the device list of Current Configuration 169. In other words, periodically, based upon the length of the predefined interval, process 200 monitors each device and service listed in Current Configuration 169.
It should be noted that process 200 does not include an “End” step in which processing is complete because, once initiated, process 200 continues to periodically analyze the devices and services of computing environment 100 shown in
One example of a service that might be the target of step 233 could include services provided by a router, a server, a switch and the like and the service can include an application, the operability of a URL, routing services and the like. Once a particular service is selected for monitoring, control proceeds to a “Has Valid Traffic Been Seen for the Service?” step 235 during which process 200 analyzes the targeted service and determines whether or not there has been recent traffic corresponding to that service. Note that traffic for all configured services is passively monitored continuously; step 235 refers to the analysis of the monitoring for the selected service.
If service is detected, then control proceeds to a “Service Is Up” state 241. At this time, if so configured, metrics can be recorded and results of process' 200 observations can be stored in Current Metrics 171 of data store 123 (
If, in step 235, process 200 does not observe traffic that can be associated with the targeted service, then control proceeds to a “Can Use of Service Be Acquired?” step 237 during which process 200 requests performance of a task associated with the targeted service. The transition from step 235 to step 237 represents a transition from passive component 151 to active component 153.
The particular task requested depends, upon the type of service. For example, if the targeted service relates to network connectivity, then a “trace route” command can be sent to determine if the destination is reachable from the source. As another example, if the targeted service is a web application transaction, then an appropriate HTTP command(s) can be sent to the server to determine whether or not that transaction is available.
In step 237 process 200 determines whether or not the service requested was successfully completed. If so, then control proceeds to “Service Is Up” state 241. If the requested task is not completed, then control proceeds to a “Service Is Down” step 243.
If configured, metrics can be recorded related to information gathered during steps 235, 237 and 239 corresponding to the current state, as indicated by one of states 241 and 243, and observed activity of the targeted service is stored in Current Metrics 171 of data store 123. Control then proceeds to an “Another Service?” step 247 during which process 200 determines whether or not each service listed in Current Configuration 169 that corresponds to the targeted device has been monitored by process 200. As explained above in conjunction with
If there are additional services corresponding to the targeted device listed in Current Configuration 169 that have not yet been examined in the current iteration of process 200, then control returns to Check Next Service step 233 and processing continues as described above with the next unexamined service as the target of process 200. If, in step 247, process 200 determines there are no more service to be monitored, then control proceeds to an “End Service Check” step 249 in which processing associated with step 205 is complete. Control then returns to Was Any Service Detected? step 207 (
Referring now to
Augmenting Passive Probes with Active Investigations
Referring now to
According to an embodiment, performance agents can be situated near server farms, such as within Service Monitor 125(2) near server farm 109 shown in
Investigation console component 600 can be implemented within a server, such as file server 121, operable as Web Server 610. Server 610 is configured to implement Investigator Web Interface 620 and Event Handler Web Service 630. Investigator Web Interface 620 is operable to provide security for operating command line tools 640. Command line tools can include ping, trace route, TCP echo, TCP trace route, performance agent query and Simple Network Management Protocol (SNMP) query. Event Handler Web Service 630 can be implemented as an alarm handler web service that accepts alarms from agents. The alarms are logged in Investigator database 650. If an alarm occurs, a signal to expert system 660 takes place. Investigation console component 600 can be coupled to a plurality of performance agents. For example, Service Management Console 131 can include an investigation console component, and each of Service Monitors 125 can include a performance agent that includes a module or the like to integrate with the investigation console component.
In one embodiment, the module provides an active component coupled to an otherwise passive performance agent. The active component gathers additional specific statistics based on results of an event correlation engine. In operation, if the passive component determines that an issue is present with a server, active component gathers additional server statistics. Likewise, if an issue is discovered in a subnet, active component gathers additional network statistics. Thus, any response-time issues in a network are isolated using additional data. The additional data can be collected via one or more modes, including a snapshot mode and a continuous collection mode.
Investigation console component 600 receives the additional data generated by the active component and operates on the received data if available. Investigation console component 600, in an embodiment, is operable whether or not some or any additional data is received from active component.
The console 600 and network performance agents, in one embodiment, include event detection algorithms that are capable of identifying sudden, gradual, and periodic anomalies. For example, an Auto-Thresholding method, described in further detail below, can be configured to generate a separate threshold for each of three or more baselines. One baseline can be based on the past week, one can be based on the same day of week over the past three months, and one can be based on the same day of week similar day of month over the past six months. These baselines are exemplary, and one of ordinary skill in the art will appreciate with the benefit of this disclosure that system requirements can dictate alternate baselining techniques such as hourly thresholds or baselines using workdays only.
The baselines are computed using related historical data that can be weighted according to different means. For example, a network delay metric for a specific service A from a specific site B to a specific server C might be compared against thresholds computed from historical data of the network delays experience by service A for communication between site B and server C located at data farm D. Also, a network delay metric for service A from a specific site B to a specific server C might be compared against thresholds computed form historical data of the network delays experienced by service A for communication between site B and all servers C1-CN that host service A at data farm D, where the measurements from the different servers could be weighted equally or according to their amount of service-related traffic or according to some other means.
The event detection can be triggered a single transaction or behavior, or it can be triggered by a function of the related transactions or behaviors. For example, a single Purchase Order transaction response time exceeding a threshold could trigger an incident; similarly, the average of the Purchase Order transaction response times in a 5 min interval exceeding a threshold could trigger an incident. The function can be arbitrary and include different forms of weighting to aggregate the related measurements. The weighting can be based for example on the type of service, the user, the server, and the underlying measurement type.
An Auto-Thresholding method according to an embodiment reports a single threshold from the weighted average of the three baseline thresholds, where each baseline may itself be a weighting of related measurements as explained above. Performance agent 670 can be configured to instead check data against the individual baseline thresholds and record which baseline(s) was violated.
A violation of the 6-month threshold but not the 7-day threshold could indicate a gradual increase condition; the hypothesis could then be confirmed by inspecting time-series event data. Similarly a violation of the 7-day threshold but not the six-month threshold could indicate either a periodicity or a recent jump.
In one embodiment, a network performance agent 670 with an active investigation component has two modes, snapshot and continuous.
The snapshot mode exhibits activity only when triggered by an event. More specifically, in snapshot mode, the active investigation component only provides a snapshot of performance around the time of an event. For example, in some networks an appropriate period of time can be about five to 15 minutes from the beginning of an event without any context or historical information. A snapshot mode can be beneficial to those clients that are collecting network and systems data using other tools in addition to a network performance agent in accordance with embodiments herein. For example, such clients, by using additional tools would have to implement double-polling systems if the snapshot mode were not used. Rather than a double-poll system, such clients can refer to their other tools to provide context.
The continuous mode for the active investigation component polls server and/or network information continuously to provide a performance history. According to this mode, performance data can be stored and reported from a network performance agent database, in which case the Event Detection component 680 should also note anomalies in this data. Alternatively the performance data may be stored and handled separately by the Active Investigation component. The continuous mode allows for the reporting not only of instantaneous values but also of whether those values are atypical thereby providing improved automated root cause analysis.
Referring now to
Active investigator 620 can be coupled to a host of active investigator web services, which can include ping, trace route, TCP Echo, TCP trace route, agent query, SNMP query, and router query.
Control proceeds from step 303 to an “Examine Next Metric” step 305 during which process 300 takes the first unexamined metric from Current Metrics file 171 for examination. Control then proceeds to a “Does Metric Cross Threshold in Specified Direction?” step 307 during which the metric selected in step 305, or “targeted metric,” is compared to a threshold set for that particular metric. Thresholds are stored in and retrieved from Threshold Values file 161 (
If in step 307 the targeted metric does not exceed the corresponding threshold value, then control proceeds to a “Metric Sufficiently Deviate from Normal Behavior?” step 309 during which the targeted metric is subjected to a normality test by being compared to associated information in Historic Data file 157. Historic data file 157 contains information corresponding to historic levels for the targeted metric. In other words, the target metric is checked to see whether or not its current value is in line with previously encountered values, or baselines. If the targeted metric's value sufficiently differs from historic values, then control proceeds to Transition Point A. Otherwise, control proceeds to a “Metric Tracked?” step 311 during which process 300 determines whether or not the targeted metric is one that has been designated as a “tracked” metric, i.e. a metric saved regardless of whether it exceeds a threshold in step 307 or differs sufficiently form normal in step 309. If the targeted metric, is a tracked metric, then control proceeds to a Transition Point B, which leads to the portion of process 300 explained in detail below in conjunction with
If in step 311 the targeted metric is determined not to be a tracked metric, then control proceeds to an “More Metrics?” step 313 during which process 300 determines whether or not there are additional, unexamined metrics in Current Metrics file 171. In addition, metrics that have exceeded a threshold or a normality test, diverted for further processing via Transition Point A, and tracked metrics, diverted for further processing via Transition Point B, are reintroduced to More Metrics? step 313 via a transition Point C.
If there are no more additional metrics to be examined, then control proceeds to a “Store Incident Changes to Database” step 317 during which the current metrics, including tracked metrics, metrics that crossed one or more thresholds in step 307 and metrics that failed a normality step in step 309, are stored in a Investigator database 650 so that the data is available for further processing during an Examine Incidents process 351, described in detail below in conjunction with
Following More Metric Step 313, control returns to Examine Next Metric step 305 and processing continues as described above with the next, unexamined metric designated as the targeted metric.
If process 300 determines in step 313 that there are no more metrics to be processed, then control proceeds to a “Store Incident Changes to Database” step 317 during which all data stored in the temporary file during iterations through step 313 are saved to an Investigator Database 650. In one embodiment, database 123 is implemented as an Investigator database 650, and control updates Incident Data file 167 (
If, in step 321, process 300 determines there is no corresponding open incident, then control proceeds to a “Create Incident” step 323 during which a new incident entry is created in Incident Data 167. Control then proceeds to a “New Issue?” step 325 during which process 300 determines whether or not the targeted metric represents a new issue or one that is already being tracked. Of course, if step 325 is entered via step 323, the targeted metric represents a new issue because the incident is new. Control can also proceed to step 325 if process 300 determines in step 321 that the targeted metric corresponds to a previously opened incident. In this case, there might be a previously opened issue that corresponds to the targeted metric.
If process 300 determines that the target metric does not correspond to a previously opened issue, then control proceeds from step 325 to an “Add New Issue” step 327 during which an additional issue entry is added to the corresponding incident entry in Incident Data 167. Control proceeds to an “Update Issue Within Incident” step 329 if process 300 determines in step 325 that the targeted metric is not a new issue. Further, control can proceed to step 329 directly from step 311 (
Control proceeds from step 327 or 329 to a “Configured To Investigate?” step 331 during which process 300 determines whether or not the tracked metric corresponds to a device, service or metric type that process 300 is configured to investigate. If so, control proceeds to an “Issue Severe?” step 333 during which process 300 determines whether or not the current issue is sufficiently severe or important to trigger an active investigation. If the current issue is severe enough to initiate an investigation, then control proceeds to an “Investigate” step 335. Investigate step 335 includes investigating based on metric type, device and service. In an embodiment, active investigations are launched automatically to collect more data based on the state and type of issue within the incident. If the current issue is not severe enough to investigate or upon completion of the configured investigation, then control proceeds to a “User Notification Required?” step 337 during which process 300 determines whether or not computing environment 100 shown in
If process 300 determines, in step 331, that system 100 is not configured to investigate the current issue or, in step 333, that the issue is not severe enough to trigger an investigation, then control proceeds to User Notification Required step 337. Information regarding whether or not a particular issue corresponds to a service or device that is configured for an investigation is stored in Server Settings 163. Information regarding whether or not notification is required is stored in Current Configuration 169. Information regarding whether or not a particular issue is severe enough to trigger an investigation is stored in Investigation Settings 165.
If, in step 337, process 300 determines that notification is required by the particular issue, then control proceeds to an “Issue Severe?” step 339 during which process 300 determines whether or not the current issue is severe enough to trigger a notification. If so, then control proceeds to a “Notify Users” step 341 during which relevant messages corresponding to the current issue are transmitted (for example, by email or pager) to appropriate users. Finally, following step 341, control proceeds to a Transition Point C which returns control to Another Metric? step 313 (
Process 350 begins in a “Start Examine Incidents” step 351 and control proceeds immediately to an “Import Collector Files” step 353 during which process 350 retrieves collector files stored in Current Metrics directory 171. Agents on each computing device coupled to system 100 collect metrics corresponding to processes, services and devices and transmit those metrics to server 121. Control then proceeds to a “Save Copy” step 355 during which process 300 saves a copy of the collector files for archival purposes.
Control then proceeds to a “Process and Delete Files” step 357 during which process 350 combines all the collector files into a single, summary file and then deletes the collector files. Control then proceeds to a “Transform Data” step 359 during which the summary file is processed. Control then proceeds to an “Add Data” step 361 during which process 350 adds appropriate transformed data.
Once data in the summary file has been processed in step 357 and any additional data added in step 359, the summary file is saved to a data cache 363 and control proceeds to a “Wait For Files” step 365 during which process 350 waits for more collector files to be generated. Once new files have been generated, control returns to step 357 and processing continues as described above. It should be noted that there is no “End” step in process 350 because once initiated, process 350 continues to run until system 100 is brought down or process 350 is expressly halted by a system administrator.
From step 383, control proceeds to a “Correlate Events” step 389 during which any events labeled “Bad” or “Missing” are incorporated into new incidents. Process 380 then proceeds to a “Conduct Investigation” step 391 during which process 380 determines what steps and devices are involved with an attempt to discover the source of the incident. Information concerning the particular actions and targeted devices is stored in Investigation Settings file 165 (
Control proceeds from step 391 to a “Check Availability” step 393 during which time the actions on the devices are executed, if possible (see
Once a targeted device has been tested for availability, control proceeds to an “Update Incidents” step 395 during which Incident Data file 167 is updated to reflect both new information on existing incidents and any new incidents created. Thus, the next iteration of process 380, Open Incident List 387 contains current information. Finally, control proceeds to a Send Notification” step 397 during which appropriate users are notified of new and closed incidents. Control then proceeds to an “End Investigate” step 398 indicative of the completion of process 380.
Control proceeds to “All Issues Closed?” step 409 wherein process 400 determines whether issues are closed. If so, control proceeds to “Close Incident” step 411, followed immediately by “Notify Users” step 413 wherein users are notified that the incident has been closed if the system is so configured. Following the notification of users, control passes to query step “More Incidents?” 415, wherein process 400 determines whether or not there are any more incidents to be examined. [0117] If, in step 409 all issues are not closed, process 400 proceeds to “More Incidents?” query step 415. If more incidents are present to be examined, control returns to step 405 Examine Next Incident. If all issues are closed for a given incident and no further incidents are present, control proceeds to “Store Changes” step 417 wherein any incident changes are stored to a database, such as data store 123. Control proceeds to “Sleep” step 419, wherein process 400 waits for a predetermined period of time before returning to examining incidents at step 401 to perform the process again.
If the examination of an issue reveals that recent availability or performance measurements have taken place in query step Recent Measurement? 425, control passes to “Good State?” query step 431 wherein process 407 determines whether or not the issue is in a good state. If the issue is in a good state, control passes to Wait Enough? query step 429, described above, or passes to More Issues query step 435, also described above.
If there are no more issues that require attention, control is passed to End Examine Issues step 437.
| Number | Date | Country | |
|---|---|---|---|
| 60563535 | Apr 2004 | US |
