Patent Application
20150127595
1. Field of the Disclosure
The disclosure relates to modeling and presenting information regarding whether a target system is in an anomalous state based on the accuracy of predictions made by a predictive model.
2. Description of the Related Arts
Predictive analytics refers to a variety of techniques for modeling and data mining current and past data sets to make predictions. Predictive analytics allows for the generation of predictive models by identifying patterns in the data sets. Generally, the predictive models establish relationships or correlations between various data fields in the data sets. Using the predictive models, a user can predict the outcome or characteristics of a transaction or event based on available data. For example, predictive models for credit scoring in financial services factor in a customer's credit history and data to predict the likeliness that the customer will default on a loan.
Commercially available products for predictive analytics include products from IBM SSPS, KXEN, FICO, TIBCO, Portrait, Angoss, and Predixion Software, just to name a few. These software products use one or more statistical techniques such as regression models, discrete choice models, time series models and other machine learning techniques to generate useful predictive models. These software products generate different predictive models having different accuracies and characteristics depending on, among others, the amount of training data and available resources.
With a perfect predictive model, all patterns and sequences should be predicted. Such predictive model will always make an accurate prediction; and hence, no anomaly in prediction will ever arise. In practice, however, predictive models are imperfect and the data is not always predictable. Hence, the prediction made using a predictive model will often deviate from the actual value being predicted. Some of such deviation may be indicative of critical events or errors that may pose a significant risk or advantage to a user of the predictive model.
Embodiments relate to detecting an anomaly in a target system by making a prediction based on input data received from the target system and determining accuracy of the prediction. The prediction is generated by executing one or more predictive algorithms using the received input data. A current accuracy score representing accuracy of the prediction is generated. An anomaly score representing likelihood that the target system is in an anomalous state is generated based on the current accuracy score by referencing an anomaly model. The anomaly model represents anticipated range or distribution of accuracy of predictions made by the predictive model.
In one embodiment, the prediction is compared with an actual value corresponding to the prediction to generate the current accuracy score.
In one embodiment, the anomaly model is generated by analyzing a plurality of prior accuracy scores generated prior to generating the current accuracy score. The prior accuracy scores are generated by executing the predictive algorithm based on training data or prior input data provided to the predictive algorithm and comparing the plurality of predictions against a plurality of corresponding actual values.
In one embodiment, the accuracy score takes one of a plurality of discrete values. The likelihood is determined by computing a difference in cumulative distribution function (CDF) values at an upper end and a lower end of one of the plurality of discrete values.
In one embodiment, the likelihood is determined by computing a running average of the current accuracy score and prior accuracy scores preceding the current accuracy score, and determining the anomaly score by identifying an output value of the anomaly model corresponding to the running average.
In one embodiment, a number of the prior accuracy scores for computing the running average is dynamically changed based on predictability of the input data.
In one embodiment, the accuracy score is aggregated with one or more prior accuracy scores generated using the input data at time steps prior to a current time step for computing the current accuracy score.
In one embodiment, user input indicating a time period represented by the aggregated accuracy score is received.
In one embodiment, a time period represented by the aggregated accuracy score is increased or decreased when another user input is received.
In one embodiment, the predictive algorithm generates the prediction using a hierarchical temporal memory (HTM) algorithm or cortical learning algorithm.
In one embodiment, a plurality of predictions and a corresponding plurality of current accuracy scores are generated based on the same input data and associated with different parameters of the target system. The likelihood that the target system is in the anomalous state is determined based on a combined accuracy score that combines the plurality of current accuracy scores.
In one embodiment, the likelihood that the target system is in the anomalous state is determined based on change of correlation of at least two of the plurality of current accuracy scores.
The teachings of the embodiments of the present disclosure can be readily understood by considering the following detailed description in conjunction with the accompanying drawings.
Figure (FIG.) 1 is a conceptual diagram illustrating relationships between a target system, a predictive model and an anomaly model, according to one embodiment.
In the following description of embodiments, numerous specific details are set forth in order to provide more thorough understanding. However, note that the embodiments may be practiced without one or more of these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.
A preferred embodiment is now described with reference to the figures where like reference numbers indicate identical or functionally similar elements. Also in the figures, the left most digits of each reference number corresponds to the figure in which the reference number is first used.
Embodiments relate to determining likelihood of anomaly in a target system based on the accuracy of the predictions. A predictive model makes predictions based at least on temporally varying input data from the target system. The accuracy of the predictions over time is determined by comparing actual values against predictions for these actual values. The accuracy of the predictions is analyzed to generate an anomaly model which indicates the historical distribution in the accuracy of predictions made by the predictive model. When the accuracy of subsequent predictions deviates from the distribution as anticipated by the anomaly model, the target system is likely in an anomalous state.
The predictive model described herein refers to software, hardware, firmware or a combination thereof for processing input data from a target system to predict future values associated with the target system. The predictive model may include, for example, a threshold model, a Gaussian model, a regression model, a hierarchical temporal memory (HTM) algorithm or a cortical learning algorithm (available from Numenta, Inc. of Menlo Park, Calif.) and other machine learning algorithms.
The anomaly model described herein refers to a mathematical or conceptual model representing an anticipated range, or distribution of accuracy of predictions made by a predictive system. The anomaly model may be generated by processing prior predictions and corresponding accuracy values or modeling the predictive system and/or the target system. In some embodiments, the anomaly model is embodied in the form of a distribution function.
The anomalous state described herein refers to a state that deviates from an ordinary operating state of a target system. The anomalous state may be the result of change in the target system itself or the environment of the target system.
The target system described herein refers to a system that is the subject of observation by the predictive system for making predictions. The target system generates input data that can be fed to the predictive system. The target system may include, but not limited to, a computer network system, a mechanical system, a financial transaction system, a health maintenance system, and a transport system (e.g., airline reservation system).
Figure (FIG.) 1 is a conceptual diagram illustrating relationships between a target system 102, a predictive model 110 and an anomaly model 120, according to one embodiment. The predictive model 110 does not always produce accurate predictions based on input data 104 from the target system 102. Due to the imperfection of the predictive model 110, inherent characteristics of the target system 102 (e.g., unobservable variables), the predictions may sometimes be accurate while at other times they may be inaccurate. The predictive model 110 compares its prediction against an actual value 108 to determine the accuracy of the prediction and generates the accuracy score 114 representing the accuracy or inaccuracy of its prediction.
The input data 104 may represent, among other information, resource usage data for computers, images, videos, audio signals, sensor signals, data related to network traffic, financial transaction data, communication signals (e.g., emails, text messages and instant messages), documents, insurance records, biometric information, parameters for manufacturing process (e.g., semiconductor fabrication parameters), inventory patterns, energy or power usage patterns, data representing genes, results of scientific experiments, parameters associated with operation of a machine (e.g., vehicle operation) and medical treatment data.
The prediction generated by the prediction model may be indicative of, but not limited to, whether data packets traversing a network will cause network issues (e.g., congestion), weather forecast, a person's behavior, gene expression and protein interactions, the amount of inventory, energy usage in a building or facility, web analytics (e.g., predicting which link or advertisement that users are likely to click), results of pending experiments, illness that a person is likely to experience, whether a user is likely to find contents interesting, result of election, whether a loan will be paid back, occurrence of adverse events, and reaction to medical treatments. The generated prediction may be (i) binary (e.g., 0 or 1), (ii) take one value from a range of values or (iii) indicate a range of values among many possible ranges of values.
An actual value 108 herein refers to information that can be used to determine whether a prediction made by the predictive model 110 was accurate. Although
When the target system 102 is in a non-anomalous state, characteristics (e.g., average, trend or distribution) of the accuracy score 114 over a period of time can anticipate a certain behavior, even though it would be difficult to predict which individual predictions would be accurate or inaccurate. For example, if an average of accuracy scores is 0.5, half of the predictions will be accurate and the other half will be inaccurate. Such anticipated behavior of the accuracy score 114 may be formulated into the anomaly model 120. Models that can be used as the anomaly model 120 include a distribution model. The distribution model can be formatted in terms of probability distribution or other forms of data (e.g., histogram) suitable for deriving the probability distribution. In one or more embodiments, the anomaly model 120 receives the accuracy score 114, identifies a value corresponding to a certain accuracy score or a ranges of accuracy scores, and outputs the identified value as the anomaly parameter 124.
After the anomaly model 120 is generated, the anomaly model 120 may be referenced to determine whether the target system 102 is in an anomalous state. That is, if the accuracy scores 114 over a time period remain within a range or distribution as anticipated by the anomaly model 120, the target system is likely to be in a non-anomalous state. Conversely, if the accuracy score 114 over a time period does deviates from the range or distribution as anticipated by the anomaly model 120, the target system 102 is likely to be in an anomalous state. The likelihood that the target system 102 is in an anomalous state or a non-anomalous state can be represented by the anomaly parameter 124. The anomaly model 120 generates the anomaly parameter 124 based on the accuracy scores 114 over a number of predictions or time that is statistically meaningful.
The anomaly detector 200 may include, among other components, a processor 212, a data interface 224, a data interface 224, a display interface 228, a network interface 230, a memory 232 and a bus 260 connecting these components. One or more software components in the memory 232 may also be embodied as a separate hardware or firmware component in the anomaly detector 200. The anomaly detector 200 may include components not illustrated in
The processor 212 reads and executes instructions from the memory 232. The processor 212 may be a central processing unit (CPU) and may manage the operation of various components in the anomaly detector 200.
The data interface 224 is hardware, software, firmware or a combination thereof for receiving the input data 104. The data interface 224 may be embodied as a networking component to receive the input data over a network from another computing device. Alternatively, the data interface 224 may be a sensor interface that is connected to one or more sensors that generate the input data 104. In some embodiment, the data interface 224 may convert analog signals from sensors into digital signals.
The display interface 228 is hardware, software, firmware or a combination thereof for generating display data to be displayed on a display device. The display interface 228 may be embodied as a video graphics card. In one embodiment, the display interface 228 enables a user of the anomaly detector to views graphical user interface screens associated with the anomaly detection.
The network interface 230 is hardware, software, firmware or a combination thereof for sending data associated with anomaly detection to another device. The network interface 230 may enable the anomaly detector 200 to service multiple devices with anomaly data.
The memory 232 is a non-transitory computer readable storage medium that stores software components including among others, data preprocessing module 236, predictive algorithm module 242, anomaly processor 246, statistics module 250, user interface (UI) generator 252, and application 256. The memory 232 may store other software components not illustrate in
The data preprocessing module 236 receives the input data 104 via the data interface 224 and processes the input data 104 for feeding into the predictive algorithm module 242. The preprocessing may include, among others, converting data formats, filtering data, aggregating data, and adding or removing certain data. With respect to converting the data format, certain predictive algorithms can perform input data only in certain formats, and hence, the data preprocessing module 236 converts data in one format to another format compatible with the predictive algorithm module 242. For example, an HTM algorithm or a cortical learning algorithm (available from Numenta, Inc. of Menlo Park, Calif.) uses data to be in sparse distributed representation. Hence, the data preprocessing module 236 converts data in other formats to data in sparse distributed representation. Further, some input data needs to be aggregated before being fed into the predictive algorithm module 242 for enhanced prediction at the predictive algorithm module 242. The data preprocessing module 242 may perform such aggregation of the input data.
The predictive algorithm module 242 receives preprocessed input data from the data preprocessing module 236 to generate predictions and their corresponding accuracy scores. Various types of predictive algorithms may be deployed in the predictive algorithm module 242. The accuracy scores 114 indicate how accurate the predictions made by the predictive algorithm module 242 are relative to the actual values 108 corresponding to the predictions. The accuracy scores 114 may simply indicate that the prediction was correct (e.g., value of 1) or incorrect (e.g., value of 0). Alternatively, the accuracy scores 114 may take a value within a range (e.g., values between 0 and 1) or take one of a number of discrete values (e.g., 0, 0.25, 0.5, 0.75 and 1).
The anomaly processor 246 may (i) generate an anomaly model 120, and (ii) use the anomaly model 120 to process the anomaly parameter 124 based on the anomaly model 120. To generate the anomaly model 120, the anomaly processor 246 may process a number of accuracy scores (e.g., 1,000 accuracy scores) that were generated while the target system 102 was in a non-anomalous state or process a number of recent accuracy scores. By using the accuracy scores associated with the recent history, the anomaly processor 246 can generate an anomaly model that anticipates the behavior of accuracy scores. Specifically, the anomaly processor 246 uses the accuracy scores associated with the non-anomalous state or recent history to generate a distribution model that functions as the accuracy model 120. For this purpose, the anomaly processor 246 may generate a histogram of the accuracy scores associated with the non-anomalous state or recent anomaly scores, and then from the histogram, derive a normal distribution curve for the non-anomalous state of the target system 102. In other embodiments, a distribution model can be derived directly from a series of time averaged accuracy scores without using a histogram.
In some embodiments, the anomaly processor 246 may use an anomaly model that evolves over time. For various reasons, the target system 102 may become more or less predictable over time. Similarly, the predictive model 110 may also become more or less accurate over time. For example, the predictive model 110 may perform on-line learning, and produce more accurate predictions as the predictive model 110 is exposed to more input data 104. Due to such changes, the accuracy scores 114 may gradually drift to produce high or lower scores even though the target system 102 is in a non-anomalous state. To account for such natural changes in the accuracy score, the anomaly model 120 may be refreshed continuously, periodically, or modified based on accuracy scores that were determined not to be associated with an anomalous state of the target system 102.
After the anomaly model 120 is generated, the anomaly processor 246 may determine the likelihood that the target system 102 is in an anomalous state by determining whether one or more recently received accuracy scores fluctuate within a range or the accuracy scores are distributed in a manner that was anticipated by the anomaly model 120. The likelihood of the anomalous state is represented by the anomaly parameter 124. Because individual accuracy scores for each prediction may vary abruptly, the anomaly processor 246 may use filtered versions of the accuracy scores to generate the anomaly parameter 124. For example, the anomaly model 120 may compute running average of the accuracy scores to determine the anomaly parameter 124.
In one or more embodiments, the time span associated with the running average or the number of accuracy scores for computing the running average may be dynamically adjusted based on the predictability of the input data 104. If the input data 104 is less predictable, the running average may be set longer to detect an anomalous change. On the other hand, the running average can be shortened if the target system has become more predictable. If the parameter being predicted becomes highly predictable, then a shorter running average can be used to detect an anomalous change in the target system.
The statistics module 250 receives and stores the anomaly parameter 124 for aggregating as requested by the application 256. A large number of anomaly parameter values without statistical processing may not be decipherable or meaningful to a human user. The statistics module 250 aggregates the anomaly parameter values over certain time periods of time to generate aggregated values that are easier for a user to decipher and understand. For example, anomaly parameters are generated every short period of time (e.g., seconds) into an aggregated parameter covering a longer period of time (e.g., minutes, hours, or days), as described below in detail with reference to
Various functions may be used to aggregate anomaly parameters. The aggregated anomaly parameter for a longer time period (e.g., a day) may show the highest anomaly parameter aggregated over a shorter time period (e.g., an hour). Alternatively, the aggregated anomaly parameters over a longer time period may be the sum of a number of highest anomaly parameters over a shorter time period and exceeding a certain threshold value. In other embodiments, the aggregated anomaly parameter over a longer time period may be the average of anomaly parameters over a shorter time period.
The UI generator 252 receives the aggregated anomaly parameter values from the statistics module 250 and generates graphical user elements such as charts or listings for presentation to the user. Example charts are described below in detail with reference to
The application 256 is a software component that provides various services to users, including sending graphical user interface screens displaying raw anomaly parameter, aggregated anomaly parameter, and/or alerts associated with anomaly parameters to the users. The application 256 may also perform other functions such as deploying and managing instances of programs onto a cloud computing cluster.
The application 256 also receives user input from a user interface device (e.g., keyboard, mouse or touch screen) connected to the anomaly detector 200 or from another device via the network interface 230. Such user input may instruct the application 256 to generate anomaly parameters with different time aggregations or show anomaly parameters based on different parameters.
Then the input data is preprocessed 316 for processing by the predictive algorithm module 242. As described above with reference to
The predictive algorithm module 242 may be trained using training data or configured with parameters to perform predictions on the input data. Based on the training or configuration of the predictive algorithm module 242, the predictive algorithm module 242 is provided with the preprocessed input data to perform 320 prediction.
The prediction made by the predictive algorithm module 242 is then compared with a corresponding actual value 108 to generate 326 the accuracy score 114. The accuracy score 114 may then be provided to the anomaly processor 246 to determine 330 the anomaly parameter. The accuracy score 114 may be averaged or processed otherwise using, for example, prior accuracy scores, before being provided to the anomaly processor 246.
The statistics module 250 then processes 336 the current anomaly parameters with previously generated anomaly parameters for presenting to a user according to a command from the application 256. The processing may include, but is not limited to, aggregating, averaging and filtering the anomaly parameters.
Based on the processed anomaly parameters, graphical user interface (GUI) elements (e.g., charts or listings) are generated 340. The generated GUI elements facilitate users to decipher or understand the current state of the target system. The generated GUI elements can be used for display on a display device connected to the display interface 228 of the anomaly detector 200. Alternatively, the generated GUI elements can be sent to other devices over the network interface 230 for display on these devices.
Various changes can be made to the processes and the sequence as illustrated in
An example of generating anomaly parameters using a HTM or cortical learning algorithm and a distribution model as the anomaly model is described herein. The HTM or cortical learning algorithm is described, for example, in U.S. patent application Ser. No. 13/046,464 entitled “Temporal Memory Using Sparse Distributed Representation,” filed on Mar. 11, 2011, which is incorporated by reference herein in its entirety.
An HTM or a cortical learning algorithm uses logical constructs called cells and columns where each column includes a subset of these cells. A column in the HTM or the cortical learning algorithm refers to a collection of cells. A cell stores temporal relationship between its activation states and activation states of other cells connected to the cell. A subset of columns may be selected by a sparse vector that is derived from input data. The selection of certain columns causes one or more cells in the selected columns to become active. The relationships of cells in a column and cells in other columns are established during training After training, a column may become predictively active before a sparse vector with an element for activating the column is received.
In one or more embodiments, the number of columns in a current time step and the number of columns predicted to become active in a previous time step can be used to compute the accuracy score. For example, the accuracy score can be computed by the following equation:
PACT/TACT equation (1)
where PACT represents active columns in a current time step that were previously predicted to become active in a current time step, and TACT represents the number of total active columns. The accuracy score will have a value between 0 and 1 where 0 indicates that none of the columns predicted to be active at the current time step is currently active and 1 indicates that all of the columns predicted to be active at the current time step is currently active. The minimum increment of the accuracy score is 1/TACT.
When a distribution model is used as the anomaly model, the anomaly processor 246 may classify an accuracy score (either raw or filtered) to one of these discrete ranges. Taking an example which includes 2048 columns and each sparse vector indicating 40 of such columns as being active, the minimum increment of the accuracy score is 1/40=0.025. The current accuracy score or the running average of the accuracy scores may be binned to one of 40 discrete ranges of [0−0.025], [0.025−0.05], [0.05−0.075] . . . [0.975−1].
The anomaly parameter can then be computed using cumulative distribution function (CDF) for both end values of the classified range. Taking the example where the accuracy score is between 0.05 and 0.075, the anomaly parameter can be computed simply as the difference between CDF (0.075) and CDF (0.05). That is, the anomaly parameter equals CDF (0.075)−CDF (0.05). Assuming that the anomaly parameter is 0.25, this represents that the likelihood of the target system being in a non-anomalous state is 0.25 (25%) based on the current accuracy score.
Although the above example was described with reference to an HTM algorithm or a cortical learning algorithm, similar methods may be used for various other predictive algorithms to generate the anomaly parameters.
A large number of anomaly parameters may be generated as a result of performing predictions on input data 104. When an extensive number of anomaly parameters are presented to users in their raw form, it would be difficult for the users to understand overall trend or changes of the anomaly parameters over time. To facilitate the users to understand the relative significance of the anomaly parameters and trend, the anomaly parameters may be aggregated and presented to users in a graphical user interface such as a bar chart.
In one or more embodiments, an aggregated anomaly parameter over a period as represented by an individual bar can be expanded to reveal a detailed breakdown of multiple aggregated anomaly parameters over a shorter period. For example, when a hashed bar representing the day of Jun. 1, 2014 is selected in
By enabling the user to view aggregated versions of the anomaly parameters over a desired time period, the user can quickly perceive the needed information. The bar charts or other similar graphical user interfaces enables the users, even without detailed technical understanding of the underlying target system 102 or associated models (e.g., predictive model 110 and anomaly model 120), to easily perceive trends and occurrences of anomalous states in the target system 102.
In conjunction with the above embodiment or as an alternative embodiment, a graphical user interface may show a sorted list of a number of highest anomaly parameters over a certain time period. The sorted list may be expanded to show the highest anomaly parameters over a shorter time period or zoomed out to show the highest anomaly parameters over a longer time period.
Based on the indicated time unit of interest and/or specified time of interest. The statistics module 250 retrieves 616 anomaly parameters of the time unit of interest to generate corresponding aggregated anomaly parameters. The aggregated anomaly parameters are then sent to the UI generator 252 or the application 256 for subsequent actions.
It is then determined 626 if another user input indicating a different time unit or time of interest is received from the user. If another user input is received, the process returns to retrieving 616 anomaly parameters within the indicated time of interest and performs subsequent processes. If such user input is not received, the process terminates.
In the example of
Various functions may be used to combine the anomaly parameters. The function may take the following, for example, as the combined anomaly parameter: (i) the highest value of the three anomaly parameters, (ii) the sum of the all three anomaly parameters and (iii) the average of the three anomaly parameters. Alternatively, a correlation function may be used for these anomaly parameters to detect if there is any deviation from the correlation of different anomaly parameters. For example, if anomaly parameters A and B tend to increase or decrease together under a non-anomalous state, the change in such behavior of anomaly parameters is likely to indicate that the target system is in an anomalous state.
Although the above embodiments were described primarily with respect to determining anomalous states in a target system, the same principle may be used to detect other predetermined states of a target system.
In one or more embodiments, multiple predictive models may be used to generate multiple series of predictions. Each predictive model generates a series of predictions. Corresponding predictions (e.g., predictions for the same time step) are then combined to generate a combined prediction in a manner similar to multi-parameter anomaly detection, as described above with reference to
In one or more embodiments, graphical user elements may be generated at a user device remote from the anomaly detector 200. The user device may be, for example, handheld devices such as smartphones. Instead of generating the graphical user elements at the UI generator 252, the user may receive raw anomaly parameters or partially aggregated anomaly parameters from the anomaly detector 200 and then process these raw or partially aggregated anomaly parameters to generate the graphical user elements.
Upon reading this disclosure, those of skill in the art will appreciate still additional alternative designs for processing nodes. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the embodiments are not limited to the precise construction and components disclosed herein and that various modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope of the present disclosure.
The application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 61/899,043 filed on Nov. 1, 2013, which is incorporated by reference herein in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 61899043 | Nov 2013 | US |
