Patent Application
20210397538
An application, e.g., an enterprise software-as-a-service application, can suffer a variety of faults which users of the application can experience as slow or unresponsive service. For example, users of an application can experience slow or unresponsive service when one or more resources that underly the application fail or become overloaded.
Application support personnel can diagnose a problem in an application by employing guesswork or manually examining the logs of the underlying resources of the application one by one until a likely cause of the problem is found. Such intuitive manual trial and error methods for diagnosing a problem can consume time, cost money, and annoy users of an application.
In general, in one aspect, the invention relates to a diagnostic service for identifying a likely cause of a problem in an application. The diagnostic service can include: a classifier that selects the likely cause from among a set of predetermined faults that may occur when running the application by associating a set of real-time features sampled from the application and that pertain to the problem to each predetermined fault; and a learning service that trains the classifier to associate the real-time features to the predetermined faults using a set of training features obtained by injecting each predetermined fault into the application and sampling a respective subset of the training features during each predetermined fault injected and by labeling each respective subset sampled with the respective predetermined fault.
In general, in another aspect, the invention relates to a method for identifying a likely cause of a problem in an application. The method can include: injecting a series of a set of predetermined faults into the application; sampling a subset of a set of training features from the application during each predetermined fault injected into the application and labeling each subset with an identifier of the corresponding predetermined fault; and training a classifier to identify the likely cause by associating a set of real-time features sampled from the application that pertain to the problem to one or more of the predetermined faults in response to the training features.
Other aspects of the invention will be apparent from the following description and the appended claims.
Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements.
Reference will now be made in detail to the various embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Like elements in the various figures are denoted by like reference numerals for consistency. While described in conjunction with these embodiments, it will be understood that they are not intended to limit the disclosure to these embodiments. On the contrary, the disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the disclosure as defined by the appended claims. Furthermore, in the following detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be understood that the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.
The diagnostic service 100 includes a classifier 120 that selects the likely cause 124 from among a set of predetermined faults p1-pN that may occur when running the application 110. The predetermined faults p1-pN can pertain to resources underlying the application 110. Examples of resources that can underly the application 110 include processors, servers, communication mechanisms, storage mechanisms, etc. The resources that underly the application 110 can include any number and arrangement of real resources, e.g., storage drives, network adapters, and virtual resources, e.g., cloud storage, virtual machines.
The classifier 120 associates a set of real-time features 114 sampled from the application 110 and that pertain to the problem 112 to each predetermined fault p1-pN. In one or more embodiments, the classifier 120 includes a set of learned logic, e.g., a neural network, that maps the values of the real-time features 114 as inputs to a respective correlation value for each of the predetermined faults p1-pN as outputs.
In one or more embodiments, the real-time features 114 are sampled from logs normally maintained for the application 110, e.g., usage logs, event logs, error logs, statistical logs, etc. For example, the real-time features 114 can be sampled from respective logs pertaining to processors, servers, communication mechanisms, storage mechanisms, etc., both real and virtual, that underly the application 110.
In one or more embodiments, the real-time features 114 correspond to a time window associated with problem 112. For example, timestamps recorded in the resource logs that underly the application 110 can be used to sample the features that correspond to the time of the problem 112.
The diagnostic service 100 includes a learning service 122 that uses a set of training features 126 labeled with the predetermined faults p1-pN to train the classifier 120. The training features 126 are obtained by injecting each predetermined fault p1-pN into the application 110 and sampling a respective subset of the training features 126 during each predetermined fault p1-pN injected and by labeling each respective subset of the training features 126 sampled with the respective predetermined fault p1-pN.
In one or more embodiments, the training features 126 are sampled from logs normally maintained for the application 110, e.g., event logs, error logs, statistical logs, etc., during a time window that corresponds to the injections of the respective predetermined fault p1-pN. For example, the training features 126 can be sampled from logs pertaining to processors, servers, communication mechanisms, storage mechanisms, etc., both real and virtual, that underly the application 110.
In one or more embodiments, the learning service 122 trains the classifier 120 using supervised learning from the labeled data of the training features 126. Examples of supervised learning in various embodiments include Random Forest, XGBoost, and deep learning algorithms involving Convolutional Neural Networks (CNN). For example, the learning service 122 can derive the parameters of a neural network by learning from the training features 126.
In one or more embodiments, the likely cause 124 is a list of one or more of the predetermined faults p1-pN determined by the classifier 120 to have the highest correlations to the real-time features 114. The list of the likely cause 124 can include a confidence indicator pertaining to each identified one of the predetermined faults p1-pN.
The application 110 in this example employs a set of resources R1-Rn while servicing the users 300-303. The resources R1-Rn can include any number and arrangement of, e.g., processor resources, server resources, communication resources, storage resources, etc., with any arrangement of real and virtual resources.
Examples of the predetermined faults p1-pN associated with the resources R1-Rn include a disk filling up, a network partition down, a host down, a storage volume unmounted, a resource bottleneck, a load spike, etc., which are commonly occurring faults in an enterprise application. In the examples below, the resource R1 is a storage resource and the fault p1 is “disk filling up”, the resource R2 is a network resource and the fault p2 is “network partition”, and the resource Rn is a host resource and the fault pN is “host down”.
The user 216 of the diagnostic service 100 obtains a report of the problem 112 via, e.g., one or more messages on the public network 310, and then uses the diagnostic service 100 or other software support service to sample the real-time features 114 via the public network 310. The diagnostic service 100, in turn, presents the likely cause 124 rendered by the classifier 120 in response to the real-time features 114 to the user 216. In some embodiments, the user 216 can access the diagnostic service 100 via the public network 310.
An example of the likely cause 124 presented to the user 216 is a list that includes “disk filling up, 75%”, “volume unmounted, 20%”, load spike, 4%”, where the percentages indicate confidence levels with higher numbers indicating higher confidence based on the training of the classifier 120. The user 216 can provide feedback if, e.g., after investigation, the problem 112 turns out to have been caused by the load spike or some new fault not included in the predetermined faults p1-pN. The user 216 can sample new features from the logs of the resources R1-Rn or from the logs of a new resource so that the training of the classifier 120 can be updated accordingly.
The real-time features 114 are obtained by sampling S sets of F features for each of the resources R1-Rn from one or more respective logs maintained for the resources R1-Rn. The F features sampled can be real numbers extracted from any of the metrices, events, indicators, etc., normally maintained for the resource R1. The S sets for R1-R3 are sampled for a time window TPROBLEM encompassing an occurrence of the problem 112.
The simulated user accesses 510 depend on the nature of the application 110. For example, if the application 110 is a payroll application, the simulated user accesses 510 can include simulated payroll data inputs, outputs, reports, etc., for a large number of simulated users of the payroll application.
The training features 126 that correspond to the injection of the fault p1 are obtained by sampling S sets of F features of each resource R1-Rn from one or more logs maintained for the resources R1-Rn for the time window T1SIMULATE that encompasses the injection of the fault p1. The features sampled for the time window T1SIMULATE can be real numbers extracted from any of the metrices, events, indicators, etc., normally maintained for the resources R1-Rn. Each of the S sets sampled for each R1-Rn are labeled p1 for inclusion in the training features 126.
The training features 126 that correspond to the injection of the fault p2 are obtained by sampling S sets of F features of each resource R1-Rn from one or more logs maintained for the resources R1-Rn for the time window T2SIMULATE that encompasses the injection of the fault p2 and labeling each of the S sets sampled for each R1-Rn with p2 for inclusion in the training features 126.
The training features 126 that correspond to the injection of the fault pN are obtained by sampling S sets of F features of each resource R1-Rn from one or more logs maintained for the resources R1-Rn for the time window TNSIMULATE that encompasses the injection of the fault pN and labeling each of the S sets sampled for each R1-Rn with PN for inclusion in the training features 126.
In one or more embodiments, the feature pre-processor 820 refines S sets of F features of raw labeled data by determining a missing value (∃s,f∥Xp(s,f)∉∧(0≤s<S)∧(0≤f<F) in the column vector Xp(*,f) where f specifies the feature using linear interpolation. For example, if Xp(3,2) is missing for feature 2, the tuples
a,Xp(a,2)
and
b,Xp(b,2)
can be used to create a line by finding max(a) such that ∃a∥(0≤a<3)∧(Xp(a,2)∈
) and min(b) such that ∃b∥(3<b<S)∧(Xp(b,2)∈
). The values along that line are estimates for the samples between those tuples. The estimated value for sample 3 is assigned to Xp(3,2). If an a or b does not exist, the value from the nearest samples is assigned. If all values in a column vector Xp(*,f) are missing for feature f, zeros are assigned to Xp(*,f).
In one or more embodiments, the feature pre-processor 820 refines S sets of F features of raw labeled data by aggregating multiple instances of a resource type into one feature. The function Z(Xp(s,*))m,z returns a vector of values from sample s for features with metric m and resource type z. The values Z(Xp(s,*))m,z∈ are combined by applying the minimum, maximum, and mean to Z(Xp(s,*))m,z. The result from each statistical operation Z(Xp(s,*))m,z∈
1×F→(Z(Xp(s,*))m,z).stat becomes a new feature in Xp, while the old features with the metric m and resource type z are removed. The above steps are repeated for all s,m,z. After aggregation, the dimensions of the feature matrix Xp changes to SxF{circumflex over ( )}. The new features are impervious to resource scaling and preserve the format of the feature vector for any number of instances added or removed.
In one or more embodiments, the feature pre-processor 820 refines S sets of F features of raw labeled data by multi-grain scanning to create new features based on historical data. A window of W×F size captures W samples for F features. The window steps through the samples in the feature matrix Xp∈S×F at a granularity L. For example, the window starts at sample S−1 and moves to sample (S−1)−L for the first step and then to sample (S−1)−2L for the second step. In general, the window moves to sample (S−1)−tL for the tth step. Each step takes W samples within the window and creates a feature vector np∈
1×(w×F) of length W×F in the new feature matrix Np. Multi-grain scanning flattens the window to increase the number of features by W times the original number of features. The scanning process continues until the window reaches the last real valued sample in Np. Any non-real valued samples within the window are assigned zero. Once the multi-grain scanning process completes, the new feature matrix is Np∈
S×(W×F) and replaces the original feature matrix Xp.
In one or more embodiments, the feature pre-processor 820 refines S sets of F features of raw labeled data by Z-score normalization in which the magnitude and the mean for the values of a feature are normalized using z-score. Z-score normalization reduces the error in model training and inference by transforming the values for a feature to the number of standard deviations from the mean. Z-score normalization is applied to each feature f such that Xp(*,f)∈S×1→(Xp(*,f)).zscore. The offline version computes the z-score using the mean and standard deviation of the collected data. The online version computes the z-score by tracking the running mean and the estimated running standard deviation to have similar reduction in model error.
The local classifiers 920-1 through 920-n determine a respective likely cause 924-1 through 924-n of the problem 112 in response to the respective sets of real-time features 914-1 through 914-n, and a voting step 930 selects the likely cause 124 from among the likely causes 924-1 through 924-n. For example, if n=10 and 7 of 10 of the likely causes 924-1 through 924-n indicate fault p3, then the voting step 930 selects fault p3 as the likely cause 124.
The diagnostic service 100 can determine the likely cause 124 based on any subset of the local classifiers 920-1 through 920-n. For example, if n=10 and, at the time of the problem 112, feature samples are available from R1, R3, R4, R7, and R9 only, then the voting step 930 selects the likely cause 124 from among the likely causes 924-1, 924-3, 924-4, 924-7, and 924-9 only. Such a condition might occur, for example, when the resources R1-Rn are services offered in a cloud environment, e.g., compute service, storage system, network load balancing service, database services, caching services, messaging services, etc. Different feature data can be collected from each of these different services.
At step 1110, a series of a set of predetermined faults are injected into the application. The predetermined faults can be preselected from expert knowledge, histories of problems, etc., as faults that may occur in the application and cause the problem experienced by users of the application. The predetermined faults can pertain to faults that may occur in a variety of resources upon which the application runs.
At step 1120, a subset of a set of training features are sampled from the application during each predetermined fault injected into the application and each subset is labeled with an identifier of the corresponding predetermined fault. There can be any number of samples of the features in each subset and there can be any number of features.
At step 1130, a classifier is trained to identify the likely cause by associating a set of real-time features sampled from the application that pertain to the problem to one or more of the predetermined faults in response to the training features. The training can be supervised learning from the labeled training features. The likely cause can identify one or more of the predetermined faults as a likely cause of the problem along with a confidence indicator for the likeliness of the identified predetermined fault.
While the foregoing disclosure sets forth various embodiments using specific diagrams, flowcharts, and examples, each diagram component, flowchart step, operation, and/or component described and/or illustrated herein may be implemented, individually and/or collectively, using a range of processes and components.
The process parameters and sequence of steps described and/or illustrated herein are given by way of example only. For example, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various example methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments may be devised which do not depart from the scope of the invention as disclosed herein.
