Patent Grant
11928760
Each of the following applications are hereby incorporated by reference: application Ser. No. 15/266,979 filed on Sep. 15, 2016; application No. 62/301,590 filed on Feb. 29, 2016; application No. 62/301,585 filed on Feb. 29, 2016. The Applicant hereby rescinds any disclaimer of claim scope in the parent application(s) or the prosecution history thereof and advises the USPTO that the claims in this application may be broader than any claim in the parent application(s).
This application is related to U.S. application Ser. No. 15/140,358, entitled “SCALABLE TRI-POINT ARBITRATION AND CLUSTERING”; U.S. application Ser. No. 15/057,065, entitled “SYSTEM FOR DETECTING AND CHARACTERIZING SEASONS”; U.S. application Ser. No. 15/057,060, entitled “SUPERVISED METHOD FOR CLASSIFYING SEASONAL PATTERNS”; and U.S. application Ser. No. 15/057,062, entitled “UNSUPERVISED METHOD FOR CLASSIFYING SEASONAL PATTERNS”, the entire contents for each of which are incorporated by reference herein as if set forth in their entirety.
The present disclosure relates to computer-implemented techniques for generating forecasts. In particular, the present disclosure relates to detecting and accommodating state changes in a time-series signal when generating projections of future values.
Organizations, data analysts, and other entities are often interested in forecasting future values for a time-series signal. In the context of capacity planning, for example, a forecast may be used to determine how many hardware and/or software resources to deploy to keep up with demand. An inaccurate forecast may result in poor capacity planning decisions, leading to an inefficient allocation of resources. For instance, a forecast that underestimates future demand may lead to insufficient hardware and/or software resources being deployed to handle incoming requests. As a result, the deployed resources may be over-utilized, increasing the time spent on processing each request and causing performance degradation. On the other hand, a forecast that overestimates future demand may result in too many resources being deployed. In this case, the deployed resources may be underutilized, which increases costs and inefficiencies associated with maintaining a datacenter environment.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
The embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings. It should be noted that references to “an” or “one” embodiment in this disclosure are not necessarily to the same embodiment, and they mean at least one. In the drawings:
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in a different embodiment. In some examples, well-known structures and devices are described with reference to a block diagram form in order to avoid unnecessarily obscuring the present invention.
A time-series signal may exhibit various behaviors such as seasonal variations in peaks and lows, trends, and/or states. A failure to account for such characteristics may result in unreliable forecasts and, as previously indicated, poor planning decisions. For instance, a middleware administrator in charge of a web-service based application may be responsible for ensuring that there are enough hardware and/or software resources during peak times to satisfy demand. The administrator may plot a trend line using a linear regression model to predict whether current hardware is sufficient for peak months. However, linear regression does not account for seasonal fluctuations in the time-series. In the event that online traffic is greatly reduced in the late evening hours, the linear regression model may underestimate future peak values or overestimate future trough values, both of which lead to a wasteful use of computational resources (including computer hardware, software, storage, and processor resources, and any services or other resources built on top of those resources). Other seasonal factors, such as increased volume around holidays or sales event, as well as non-seasonal factors, such as changes in the state of a signal due to external factors, may also cause the linear regression model to generate inaccurate forecasts.
In addition, linear regression models generally do not account for potential state changes in the time series data when formulating predictions about future values. A large step down in the time-series caused by a state change may lead the linear regression model to significantly overestimate future values while a step up in the time-series may lead the linear regression model to significantly underestimate future values. Thus, in the context of capacity planning operations, a failure to accommodate step changes into the forecast may result in inefficient use and deployment of computational resources.
Rather than relying on linear regression, an administrator may instead use a Holt-Winters forecasting model to account for seasonality in the time-series. The Holt-Winters forecasting model relies on a triple exponential smoothing function to model levels, trends, and seasonality within the time-series. A “season” in this context refers to a period of time before an exhibited behavior begins to repeat itself. The additive seasonal model is given by the following formulas:
Lt=α(Xt−St−p)+(1−α)(Lt−1+Tt−1) (1)
Tt=γ(Lt−Lt−1)+(1-γ)Tt−1 (2)
St=δ(Xt−Lt)+(1−δ)St−p (3)
where Xt, Lt, Tt, and St denote the observed level, local mean level, trend, and seasonal index at time t, respectively. Parameters α, γ, δ denote smoothing parameters for updating the mean level, trend, and seasonal index, respectively, and p denotes the duration of the seasonal pattern. The forecast is given as follows:
Ft+k=Lt+k Tt+St+k-p (4)
where Ft+k denotes the forecast at future time t+k.
The additive seasonal model is typically applied when seasonal fluctuations are independent of the overall level of the time-series data. An alternative, referred to as the multiplicative model, is often applied if the size of seasonal fluctuations vary based on the overall level of the time series data. The multiplicative model is given by the following formulas:
Lt=α(Xt/St−p)+(1−α)(Lt−1+Tt−1) (5)
Tt=γ(Lt−Lt−1)+(1−γ)Tt−1 (6)
St=δ(Xt/Lt)+(1−δ)St−p (7)
where, as before, Xt, Lt, Tt, and St denote the observed level, local mean level, trend, and seasonal index at time t, respectively. The forecast is then given by the following formula:
Ft+k=(Lt+k Tt)St+k-p (8)
While the Holt-Winter additive and multiplicative models take into account seasonal indices to generate the forecast, these models can be brittle in the manner in which forecasts are generated. If a user has not carefully defined the context of an input set of data to accommodate the eccentricities of an environment in which time-series data are captured, then the Holt-Winters model may generate inaccurate results. For instance, the above forecasting models are not tailored to determine if a seasonal pattern or other characteristic of a time-series has gone through a change in state. If the user has not explicitly defined how the state change should be handled, then multiple states may affect the projections output by the forecasting model, leading to overestimates and/or underestimates of future resource usage. A user may attempt to manually filter the time-series data that are used to train the model in an attempt to mitigate the impact of a time-series having multiple states; however, the user may not always be aware and/or able to detect changes in the state of a time-series signal. Also, manually inputting this data each time a state change is detected may be difficult and cumbersome, especially in systems where state changes are frequent.
Systems and methods are described through which state changes within a time-series signal may be automatically detected and accommodated in a forecast or other analytical model. A “state change” in this context refers to a change in the “normal” characteristics of the signal. For instance, time-series data before a particular point in time (a “change time”) may exhibit a first pattern or set of one or more characteristics that are determined to be normal. After the change time, the time-series data may exhibit a second pattern or second set of characteristics that were not normal before the change time. In other words, a state change results in a “new normal” for the time-series signal. The system may determine whether a change is normal or not by analyzing the patterns within the signal, as described further herein.
In one or more embodiments, a process for automatically detecting and accommodating state changes comprises generating, within volatile and/or non-volatile storage, a representation of a time-series signal. The representation may be generated in such a way as to approximate the behavior of the time-series signal across one or more seasonal periods. Once generated, the process identifies a set of one or more state changes within the representation of the time-series signal. The process then selects, based at least in part on at least one state change in the set of one or more state changes, a subset of values from the sequence of values to train a forecasting or other analytical model and trains the analytical model using the selected subset of values. The subset of values may exclude one, two, or more values within a state that occurs previous to a state change to allow the model to generate a steady-state forecast as if there were no state changes in the time-series signal and no state changes in the forecast. The process generates, within volatile and/or non-volatile storage, forecast data and/or some other analytical output using the trained analytical model.
Automatic detection and accommodation of state changes into forecasting models allows for more accurate forecasts to be generated without the visual guesswork of human analysts. For instance, in a cloud or datacenter environment, updates to software and/or hardware resources deployed within the environment may result in a new normal for the resources. By automatically accommodating these state changes, the forecasting model may more rapidly adjust to the new normal of the environment and generate more accurate projections about future resource patterns. The increased accuracy in forecasts may lead to greater efficiency in resource deployment and utilization.
A time series signal comprises a sequence of values that are captured over time. The source of the time series data and the type of information that is captured may vary from implementation to implementation. For example, a time series may be collected from one or more software and/or hardware resources and capture various performance attributes of the resources from which the data was collected. As another example, a time series may be collected using one or more sensors that measure physical properties, such as temperature, pressure, motion, traffic flow, or other attributes of an object or environment.
Hosts 110a to 110n represent a set of one or more network hosts and generally comprise targets 112a to 112i and agents 114a to 114j. A “target” in this context refers to a resource that serves as a source of time series data. For example, a target may be a software deployment such as a database server instance, middleware instance, or some other software resource executing on a network host. In addition or alternatively, a target may be a hardware resource, an environmental characteristic, or some other physical resource for which metrics may be measured and tracked.
Agents 114a to 114j comprise hardware and/or software logic for capturing time-series measurements from a corresponding target (or set of targets) and sending these metrics to data collector 120. In one or more embodiments, an agent includes a process, such as a service or daemon, that executes on a corresponding host machine and monitors one or more software and/or hardware resources that have been deployed. In addition or alternatively, an agent may include one or more hardware sensors, such as microelectromechanical (MEM) accelerometers, thermometers, pressure sensors, etc., that capture time-series measurements of a physical environment and/or resource. Although only one agent and target is illustrated per host in
Data collector 120 includes logic for aggregating data captured by agents 114a to 114j into a set of one or more time-series. Data collector 120 may store the time series data in data repository 140 and/or provide the time-series data to analytic services 130. In one or more embodiments, data collector 120 receives data from agents 114a to 114j over one or more data communication networks, such as the Internet. Example communication protocols that may be used to transport data between the components illustrated within system 100 may include, without limitation, the hypertext transfer protocol (HTTP), simple network management protocol (SNMP), and other communication protocols of the internet protocol (IP) suite.
Analytic services 130 include a set of analytics that may be invoked to process time-series data. Analytic services 130 may be executed by one or more of hosts 110a to 110n or by one or more separate hosts, such as a server appliance. Analytic services 130 comprises seasonality analytic 132, state change analytic 134, forecasting analytic 136, and capacity planning analytic 138. Each logic unit implements a function or set of functions for processing the collected time series data and generating forecast data.
Seasonality analytic 132 includes logic for detecting and classifying seasonal behaviors within an input time-series signal. In one or more embodiments, seasonality analytic 132 may generate seasonal patterns that approximate detected seasonal behavior within a time-series and/or classify seasonal patterns/behaviors according to techniques described in U.S. application Ser. No. 15/140,358, entitled “SCALABLE TRI-POINT ARBITRATION AND CLUSTERING”; U.S. application Ser. No. 15/057,065, entitled “SYSTEM FOR DETECTING AND CHARACTERIZING SEASONS”; U.S. application Ser. No. 15/057,060, entitled “SUPERVISED METHOD FOR CLASSIFYING SEASONAL PATTERNS”; and/or U.S. application Ser. No. 15/057,062, entitled “UNSUPERVISED METHOD FOR CLASSIFYING SEASONAL PATTERNS”, the entire contents for each of which were previously incorporated by reference herein as if set forth in their entirety. For instance, seasonality analytic 132 may identify and classify sparse highs, dense highs, sparse lows, and/or dense lows that are seasonal within an input set of time-series data.
State change analytic 134 includes logic for detecting and accommodating state changes within the input time-series signal. In one or more embodiments, state change analytic 134 selects a set of training data used by forecasting analytic 136 to train a forecasting model in a manner that accommodates state changes, if any, that exist within a time-series signal. Example logic for detecting and accommodating state changes is described in further detail in the sections below.
Forecasting analytic 136 includes logic for training forecasting models and generating forecasts based on the trained forecasting models. In one or more embodiments, the forecasts that are generated account for seasonal patterns, if any, that are detected by seasonality analytic 132 and state changes that are detected, if any, by state change analytic 134. The forecasting model that is implemented by forecasting analytic 136 may vary depending on the particular implementation. In one or more embodiments, forecasting analytic 136 may implement a forecasting model such as described in U.S. Appln. No. 62/301,590, entitled “SEASONAL AWARE METHOD FOR FORECASTING AND CAPACITY PLANNING”, which was previously incorporated by reference herein as if set forth in their entirety. The forecasting models are trained based on the detected seasonal patterns in an input time-series signal and used to project future values for the time-series signal. In other embodiments, the techniques described herein may be applied to other forecasting models such as the Holt-Winters models described above to generated forecasts. State change analytic 134 may thus accommodate state changes for a variety of different forecasting models.
Capacity planning analytic 138 includes logic for performing and/or recommending capacity planning operations. For example, capacity planning analytic 138 may automatically deploy additional software resources on one or more hosts to satisfy forecasted demands on system 100. As another example, capacity planning analytic 138 may generate and display a recommendation to acquire additional hardware resources to satisfy a forecasted increase in demand. In yet another example, capacity planning analytic 138 may automatically bring down deployed resources during forecasted low seasons to conserve energy/resources. Thus, capacity planning operations may leverage the generated forecasts to increase the efficiency at which resources are deployed within a datacenter or cloud environment.
Data repository 140 includes volatile and/or non-volatile storage for storing data that are generated and/or used by analytic services 130. Example data that may be stored may include, without limitation, time-series data collected by data collector 130, seasonal pattern classifications generated by seasonality analytic 132, state change identification data generated by state change analytic 134, forecast data generated by forecasting analytic, and capacity planning actions/recommendations generated by capacity planning analytic 138. Data repository 140 may reside on a different host machine, such as a storage server that is physically separate from analytic services 130, or may be allocated from volatile or non-volatile storage on the same host machine.
Clients 150a to 150k represent one or more clients that may access analytic services 130 to generate forecasts and/or perform capacity planning operations. A “client” in this context may be a human user, such as an administrator, a client program, or some other application instance. A client may execute locally on the same host as analytic services 130 or may execute on a different machine. If executing on a different machine, the client may communicate with analytic services 130 via one or more data communication protocols according to a client-server model, such as by submitting HTTP requests invoking one or more of the services and receiving HTTP responses comprising results generated by one or more of the services. Analytic services 130 may provide clients 150a to 150k with an interface through which one or more of the provided services may be invoked. Example interfaces may comprise, without limitation, a graphical user interface (GUI), an application programming interface (API), a command-line interface (CLI) or some other interface that allows a user to interact with and invoke one or more of the provided services.
State change analytic 134 receives, as input, a time-series signal. In response state change analytic 134 processes the time-series signal to identify and extract state changes, if any, that exist within the signal. A “state change” in this context refers to a change in the “normal” characteristics of the signal. For example, a particular seasonal behavior/pattern may be detected within a first sub-period of the time-series signal until a particular point in time. After the point in time, the seasonal behavior/pattern may shift such that the new seasonal behavior of the signal differs from the seasonal behavior that was detected in the first sub-period. There are a variety of potential causes of a state change, which may vary depending on the particular implementation. In the context of monitoring the performance of target deployments, for instance, a patch or other update of a target resource may cause a recurring shape, amplitude, phase, and/or other characteristic of a time-series signal to change. In another example, a change in a batch job schedule performed on a target resource may cause a shift in the high and/or sparse high seasonal pattern from the beginning of the week to the end of a given week. If the state change is not accommodated, then the prior state may have a significant impact on the forecasted values (or other analytic output) for the time-series signal.
Referring to
With reference to
At 320, the process generates, within volatile or non-volatile storage, a representation of the time-series signal. The representation may correspond to a simplified, summarized, compressed, and/or smoothed version of the time-series signal. In one or more embodiments, the representation of the time-series signal comprises a set of linear piecewise approximations. In other embodiments, the representation may comprise nonlinear approximations and/or seasonal pattern classifications. Techniques for generating time-series representations are given in further detail below.
At 330, the process determines whether the quality of the time-series representation satisfies a threshold. The quality may be determined, based at least in part, on how well the representation summarizes the data points in the time-series signal. To make this determination, a statistical analysis may be performed to compare the time-series signal with the representation. For example, the coefficient of determination, denoted R2, may be computed where the coefficient of determination indicates a variance between the representation and the sequence of values in the time-series. In order to compute the coefficient of determination, a set of residuals between the values in the time-series signal and the representation of the time-series signal may be determined, where a residual in this context represents a difference between a value in the time-series representation (an “abstracted” or “representative” value) and a corresponding value in the time-series signal. The coefficient of determination may then be computed from the squared coefficient of multiple correlation as follows:
where R2 is the coefficient of determination having a value between 0 and 1, Di is the time-series value at the ith position in the set of time-series data D, Ai is the value of the time-series representation at the ith position, and
is the sum of the squared residuals. An R2 value that is less than a threshold, such as 0.6, indicates that the representation does not account for a relatively large amount of variance in the time-series signal. This may occur, for example, when the data in the time-series cover a large range and is irregular. In this scenario, there may not be any definable states within the time series, and the entire sequence of time-series data may be selected to train the analytical model.
If the threshold is satisfied, then the process continues to 340, where a set of one or more changes within the abstracted time-series are detected. In response to detecting the set of one or more changes, the process may generate, with volatile and/or non-volatile storage, a set of change identification data. In one or more embodiments, the change identification includes a set of change times that identifies each detected state change within the time-series signal. For example, the change identification data may comprise an array of timestamps, with each timestamp identifying and corresponding to a change time of a respective change that was detected. A time within the array may be identified using a logical timestamp, such as a system change number (SCN), that defines the time in relation to other events in the system. In other embodiments, the time may be an actual time measured by a clock, such as the date, hour, minute, etc.
At 350, the process determines whether the extracted changes are normal. Techniques for detecting and classifying changes are described in further detail below.
If the changes are determined to be abnormal, then the process continues to 360 to accommodate the changes in the forecast and/or other analytical output. At 360, the process selects a subset of values from the input set of time series D in order to train the analytic model. In one or more embodiments, the selected subset of data comprises values within D that occur after the most recent abnormal state change that was detected. In order to identify the most recent abnormal state change, the process may read the change identification data to extract the timestamp corresponding to the latest change time. Timestamps corresponding to values within the set of time series data D may be compared to the extracted timestamp to determine which values occur after the corresponding change time. These values may be selected and returned to a forecasting or other analytic, which uses the subset to train an analytical model. Values that occur before the change time are ignored by the analytic and are not used to train the model. In the context of forecasting, values that occur before the change time do not affect the forecast to prevent previous states that are no longer relevant from influencing the forecasting model and causing inaccurate forecasts. Examples of how ignoring these values may lead to more accurate forecasts are given in further detail below.
If the summarized representation quality does not satisfy a threshold or the changes are determined to be normal, then the process continues to 370. At 370, the process selects and returns the set of time-series data D. The circumstances in which changes are determined to be normal may vary depending on the particular implementation. In one or more embodiments, the changes may be determined to be normal when state changes periodically recur. As an example, there may be two different types of seasonal patterns detected on a Monday, with a first pattern having a much higher peak value than the second type of Monday. The time-series signal may periodically alternate between these two types of Mondays, such as every other week, every third week of the month, or according to some other pattern. As the state changes periodically recur, they may be classified as normal. In this case, both states contribute to the forecast/analytic output, and values from the time-series signal that happen both before and after the change time are both included in the training set used to train the analytical model.
Detecting and analyzing state changes directly from the raw time-series data may be difficult since there are often fluctuations in the data from one seasonal period to another. In order to facilitate state change detection and analysis, a time-series representation that models the behaviors within a time-series signal may be generated. For instance, a time-series representation may be generated as a set of one or more data objects that include a set of abstract values representing the behavior of a time-series signal over one or more seasonal periods. The abstract values may derived based on a classification, regression, and or minimum description length (MDL) based analysis as described further below. The abstract values may also be compressed to reduce data access times and storage overhead within a computing system when detecting and analyzing state changes.
A classification-based representation may comprise a set of classifiers for different sub-periods within a time-series signal. For instance, a time-series signal may be chunked into different sub-periods corresponding to one hour intervals or some other sub-period. Each interval may then be assigned a classification based on a seasonal analysis of the signal such as described in U.S. application Ser. No. 15/057,065, entitled “SYSTEM FOR DETECTING AND CHARACTERIZING SEASONS”; U.S. application Ser. No. 15/057,060, entitled “SUPERVISED METHOD FOR CLASSIFYING SEASONAL PATTERNS”; and/or U.S. application Ser. No. 15/057,062, entitled “UNSUPERVISED METHOD FOR CLASSIFYING SEASONAL PATTERNS”, which were previously incorporated by reference. As an example, a given instance of a Monday from 8-9 a.m. may be classified as a sparse high, dense high, dense low, sparse low, or undetermined based on the analysis. In order to compress the representation, adjacent hours that are classified the same way may be merged together. Thus, if four adjacent sub-periods from 8 a.m. to 12 p.m. on Monday are classified as sparse high, then these sub-periods may be merged together and classified as sparse high to compress the summary data. The classifications for different seasonal periods may then be compared to detect state changes.
A regression-based simplified representation may comprise a set of one or more linear and/or nonlinear approximations for sequences (including sub-sequences) of values within the time-series signal. With a linear approximation, for instance, a linear regression model may be applied to different segments of a time series to fit a line to the sequence of values included in each separate segment. As a result, a set of linear approximations, referred to herein as a linear piecewise approximation, is generated for the time-series signal, with each separate “piece” or line within the set of linear approximations corresponding to a different segment within the time-series. In addition to generating linear approximations, regression models may also be used to generate nonlinear approximations, such as polynomial fits. Thus, the manner in which the representation is generated may vary depending on the particular implementation.
An MDL-based analysis may be used to generate an representation of the time-series signal, in accordance with one or more embodiments. The principle of MDL is that given a set of time-series data D, an summarization model is chosen in a manner that is able to compress the abstracted values by the greatest amount. For instance, within a given week or other seasonal period of time-series data, there may be many different ways in which linear piecewise approximations may be generated. A single constant value may be able to achieve the highest level of compression for the abstracted values. However, the single constant value may not serve as an accurate description of the behavior during that week. As a result, the residual values or functions that are stored to derive the original time-series signal from the representation may be relatively large in this case. The constant value may be changed to a best fit line having a slope, which may represent a better description of the data. In this case, the representation may not be able to be compressed as much as a single constant value; however, the representation may represent a more accurate description of the time-series signal and the residual values may be more highly compressed. The line may be broken up into smaller and smaller segments until the model is identified that a) satisfies a threshold level of quality and/or b) achieves the highest level of compression between the representation and the residuals. In some instances, as previously indicated, a representation may be thrown out if the quality is below a threshold. For instance, if the R2 indicates that the representation describes less than a threshold percentage (e.g., 70% or some other threshold) of the sequence of values summarized by the representation, then the representative values may be ignored.
Referring to
At 420, the process generates a linear approximation of the selected segment. As previously indicated, the linear approximation may be computed using a linear regression model in order to fit a line to the values in the selected segment. Depending on the data set, the line may overlap one or more data points from the time-series within the segment or it may not overlap any data points in the segment.
At 430, the process determines whether to stop or continue segmenting the time-series signal based on the linear approximation that was generated for the selected segment. The determination may be made based on a set of segmentation rules or criteria that define the circumstances under which segmentation is continued. For instance, the segmentation rules may implement an MDL-based approach where the sequences of values are segmented until a set of linear approximations that maximize compression while satisfying a threshold level of quality is determined. On the other hand, segmentation may stop if the segments become too small (e.g., they contain less than a threshold number of values). If the process determines that segmentation should continue, the process continues to 440. Otherwise, the process proceeds to 450.
At 440, the process divides the selected segment into two or more sub-segments. The break point(s) between the segments may vary depending on the particular implementation. In one or more embodiments, the segment may be broken in half. However, in other embodiments, the break points may be determined based on an analysis of the slopes/trends of the sequence of values that belong to the segments. By analyzing slopes or trends, a more accurate linear approximation may be derived. For instance, if breaking a segment in two, a first portion of the segment, which may be more or less than half the segment, may generally trend downward. The second portion of the segment may then slope upward. The break point may be selected in between these two portions of the segment, which allows a better fit to be derived through a linear regression model. Once the segment has been split into two or more sub-segments, the process returns to 430, and linear approximations are then generated for the two sub-segments. The process then continues for each of these sub-segments to determine whether to further segment the values within the time-series signal.
At 450, the process returns a linear piecewise approximation comprising multiple pieces/lines, where each piece/line corresponds to and approximates a different segment (or sub-segment) of the time-series signal. For example a first line may approximate values from a first portion of a seasonal period (e.g., Monday 9 a.m. to Tuesday 9 p.m. or any other sub-period), a second line may approximate values from a second portion of the seasonal period (e.g., Tuesday 9 p.m. to Wednesday 10 a.m.), etc.
Once generated, the time-series representation is analyzed to detect and process state changes within the time-series signal. With classification-based representation, the classifications may be compared across different seasonal periods. For example, Friday from 9 p.m. to 11 p.m. may be classified as a sparse high across multiple seasonal periods due to recurring maintenance and batch jobs being performed within a datacenter environment. If the recurring maintenance time is changed to Saturday from 8 p.m. to 10 p.m., a corresponding change in classification may be observed. The new normal for Friday from 9 p.m. to 11 p.m., previously classified as a sparse high, may suddenly change to low while Saturday from 8 p.m. to 10 p.m. becomes a sparse high. Thus, classifications across different seasonal periods may be compared and analyzed to detect if and when state changes have occurred.
State changes may also occur between sub-periods that have been classified the same way. For example, a dense high may undergo a step up or a step down in the normal seasonal pattern due to a change in circumstances. In the context of resource usage, this may be caused by a growth or decline in the number of users of an application or other resource. In such a scenario, the linear approximations across different seasonal periods may be compared and analyzed to detect if a state change has occurred.
At 520, the process selects a seasonal period or sub-period from the time-series data. The seasonal period that is selected may be based on the seasonal pattern and classifications that are detected within the automatic time-series. For example, the techniques described in U.S. application Ser. No. 15/140,358, entitled “SCALABLE TRI-POINT ARBITRATION AND CLUSTERING”; U.S. application Ser. No. 15/057,065, entitled “SYSTEM FOR DETECTING AND CHARACTERIZING SEASONS”; U.S. application Ser. No. 15/057,060, entitled “SUPERVISED METHOD FOR CLASSIFYING SEASONAL PATTERNS”; and/or U.S. application Ser. No. 15/057,062, entitled “UNSUPERVISED METHOD FOR CLASSIFYING SEASONAL PATTERNS”, previously incorporated by reference, may be used to automatically detect and classify seasonal patterns within a time-series signal. For instance, if weekly patterns are detected, then a seasonal period of a week may be selected for analysis. Sub-periods of the week may also be selected based on the characteristics of the patterns that are detected within the week. If dense highs are detected during a certain sub-period of the week, for example, than this sub-period may be selected for analysis.
At 530, the process compares one or more linear approximations and/or classifications for the selected seasonal period (or sub-period) with linear approximations across one or more other seasonal samples within the time-series signal. The analysis compares the sub-periods to detect whether observable change in state may be detected. As an example, the first four sub-periods within a representation may indicate that these sub-periods were classified as sparse highs. The next four sub-periods may then indicate that these sub-periods are classified as low. As another example, the magnitude and/or slope of a sequence of linear approximations for the selected sub-period/segment may significantly and consistently step up or step down after a particular point in time. If the magnitude and/or slope does not change by a threshold amount and/or in more than a threshold number of samples, then the differences may be ignored. In other words, differences in the approximations that are not deemed statistically significant may be ignored and are not classified as changes. On the other hand changes that are statistically significant are classified as changes and undergo further analysis. This step may be repeated for multiple sub-periods within a selected seasonal period to detect whether a state change has occurred for different segments of the week or other seasonal period.
At 540, the process determines whether a state change was detected. As previously mentioned, a state change may be detected if the linear approximations for the sub-period differ by a statistically significant amount between different seasonal samples and/or if the sub-period is classified differently between different samples within the time-series signal. If changes are detected, then the process continues to 550. Otherwise, the process ends and no changes are returned.
If at least one abnormal change has been detected, then at 550, a set of one or more change times for one or more corresponding state changes are extracted and stored. In one or more embodiments, the change times may identify the seasonal sample in which the change occurred. For example, if the state changes are detected in fifth and twelfth seasonal samples for a given set of time-series data, then change identification data may be generated that references these samples. In addition or alternatively, the change time may identify an absolute time, such as a date and hour, at which the change was detected. For instance, if a change was detected on Friday, June 24 at 10 a.m., then a timestamp identifying this time may be generated and stored with the change identification data.
At 560, the process returns change identification data comprising the extracted change times. The change times may be used to select the set of time-series data used to train a forecasting and/or other analytical model as described in further detail below.
Once detected, the state changes are further analyzed to determine whether they are normal or abnormal. Changes that are normal do not cause a “new” normal in the state of the time-series signal. As an example, two types of Monday may recur within a time-series signal—one with a relatively high resource usage and a second with a relatively low resource usage. Analyzing the representation, it may appear that the time-series signal periodically changes from one type of Monday to the other type of Monday. These changes may thus be classified as normal so that both types of Mondays may be used to generate the forecast and/or other analytical output.
With reference to
At 620, the process determines the dimensionality per seasonal period or sub-period of the states. In the context of weekly seasonal periods, for instance, this attribute may be determined as the average number of states per week. To compute the value, the total number of states may be divided by the number of samples of the seasonal period that were analyzed. If four states were detected over eight samples, then this attribute may be assigned a value of “0.5”. Similarly, if eight states were detected over the eight samples, then the attribute may be assigned a value of “1”. However, any other formula that is a function of the dimensionality and the seasonal period may be used to compute a value at this step and may vary depending on the particular implementation.
At 630, the process determines, based on the dimensionality attribute and/or the dimensionality per seasonal period attribute whether to classify the changes as normal. In one or more embodiments, a set of one or more thresholds are used to determine how to classify the changes. For instance, if there is fewer than one state per week (dimensionality/duration) or only two states (dimensionality=2), then it may be inferred that changes are not normal. The thresholds at this step may vary depending on the particular implementation. In other embodiments, it may be determined whether the state changes recur at periodic intervals. If the state changes recur, then the changes may be determined to be normal at this step. On the other hand, if the changes do not recur, then the changes may be classified as abnormal.
Based on the determination at 630, the process proceeds to either 640, where the changes are classified as normal, or 650, where the changes are classified as abnormal. In one or more embodiments, the change times are extracted, as described above, for abnormal changes that are detected within the time-series signal. Change time extraction may be skipped for changes, if any, that have been classified as normal.
Based on the state change identification and classification, a set of training data is selected to train a forecasting model according to one or more embodiments. In the event that an abnormal change was detected, then a subset of the time-series data may be used to train the forecasting model. For instance, time-series data that occur before a change time corresponding to an abnormal change may be excluded from the training set while time-series data occurring after the change time may be included. Excluding the data before the change time prevents previous states that are no longer relevant from affecting the forecast. As a result, forecast accuracy may be improved.
The change time that is used to select the set of training data (herein referred to as a “tipping point”) may either be automatically selected by the analytic or selected via external user input, depending on the particular implementation. For example, if automatically selected, the analytic may use the most recent detected abnormal state change as the target by which the set of training data is selected. Thus, time-series data that occur before the most recent change time is excluded from the training set of data while time-series data after the change time is included. In other embodiments, abnormal state changes that occurs before the most recent abnormal state change may be selected, either automatically or by a user, as the tipping point. If a previous state change is used, then data after the corresponding change time, including data that occur between the tipping point and the most recent abnormal state change, are used to train the forecasting model, and data before the change time corresponding to the tipping point are excluded. Thus, the change time that is selected may vary from implementation to implementation.
With reference to
At 720, the process selects time-series data that are more recent than the identified abnormal state change. In order to select the data, the process may compare the timestamp or other change time identifier against timestamps for the time-series data. Once the timestamp is matched to the timestamp of a sample data point in the time-series signal, data points in the time-series signal may be truncated. The remaining data points are included in a training set of data for the forecasting model.
At 730, the process trains a forecasting model using the selected time-series data. The manner in which the model is trained may vary depending on the particular forecasting model that is used. Example forecasting models are described in described in U.S. Appln. No. 62/301,590, entitled “SEASONAL AWARE METHOD FOR FORECASTING AND CAPACITY PLANNING”, which was previously incorporated by reference. However, other forecasting models may also be trained at this step.
At 740, the process generates a forecast using the trained forecasting model. In the context of software and hardware, metrics, for example, the trained forecasting model may project future values for these metrics, such as predictions about future usage rates, based on the seasonal usage patterns detected after the most recent state changes. These projections may be useful in a variety of applications, including, but not limited to, anomaly detection and capacity and planning operations.
In one or more embodiments, the forecasting model is trained to forecast a steady-state of the time-series signal despite previously detected state changes. As previously described, the forecasting model is trained as if there were a single state in the time-series data used to train the model. This may be done by excluding, from the training set, values that have occurred in one or more states that existed prior to the most recent state change. Excluding values in this context may mean omitting them from the training set or, in other embodiments, adjusting the values as if there were no state changes. As an example, if a step up is detected, the values before the step up changes may be adjusted upward based on the size of the step before being included in the training set of data. In other words, the original value is excluded, and a new, adjusted value is added to the training set. Conversely, with a step down, the values within the time-series may be adjusted downward, and the new, adjusted value added to the training set of data. This allows the forecasting model to treat the training set of data as if they belong to a steady-state signal. The forecast may then project values as if there were no more state changes in the time-series signal.
Chart 820 depicts a forecast when the entire set of time-series data, including values that occur before change time 812, are used to generate the forecast. As can be seen, the slope of the forecast changes direction due to the values that occur before the state change. As a result, the projections may be inaccurate when the previous state no longer represents the normal behavior of the monitored resources.
If multiple state changes are detected, then change identification data may be displayed concurrently with the visualization of the time-series signal and/or forecast. For instance, a visual indicator may identify each time on the chart where an abnormal state change was automatically identified. The user may select the change time to adjust the forecast. Referring again to
In one or more embodiments, a slider may be displayed with a forecast to allow a user to adjust the tipping point. For example, the slider may be initially set at the most recent abnormal state change that was detected. The user may move the slider back in time along the chart to include additional time-series data in the training set of data or forward in time to reduce the amount of time-series data that are included in the training set of data. The forecast may be updated in real-time as the user is adjusting the slider so that the user is able to view how different training sets affect the forecast.
The forecasts that are generated may be used to recommend and/or automatically perform capacity planning operations in accordance with one or more embodiments. Example capacity planning operations may include without limitation, deploying additional resources in anticipation of a forecast of increased demand, shutting down or otherwise bringing down resources in anticipation of a forecast of reduced demand, ordering additional resources to be deployed within an environment, load balancing of resources across multiple hosts, and/or consolidating resources from multiple targets into a single target.
In addition or as an alternative to capacity planning the forecasts may be integrated into other applications, such as anomaly detection and correlation prediction. For instance, an anomaly detection application may predict correlation between two or more time-series signals and raise a flag or other notification to alert a user if observed resource patterns significantly differ from projected behavior. In the event that a state change is detected in one or more of the time-series signals being correlated or in a correlation-time series that is built using the time-series signals, a correlation prediction model may be trained using the data after the most recent state change and excluding time-series data from one or more of the signals before the state change. This allows a steady-state correlation to be projected as if there were no state changes in the signal correlation signal.
According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.
For example,
Computer system 900 also includes a main memory 906, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 902 for storing information and instructions to be executed by processor 904. Main memory 906 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 904. Such instructions, when stored in non-transitory storage media accessible to processor 904, render computer system 900 into a special-purpose machine that is customized to perform the operations specified in the instructions.
Computer system 900 further includes a read only memory (ROM) 908 or other static storage device coupled to bus 902 for storing static information and instructions for processor 904. A storage device 910, such as a magnetic disk or optical disk, is provided and coupled to bus 902 for storing information and instructions.
Computer system 900 may be coupled via bus 902 to a display 912, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 914, including alphanumeric and other keys, is coupled to bus 902 for communicating information and command selections to processor 904. Another type of user input device is cursor control 916, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 904 and for controlling cursor movement on display 912. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
Computer system 900 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 900 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 900 in response to processor 904 executing one or more sequences of one or more instructions contained in main memory 906. Such instructions may be read into main memory 906 from another storage medium, such as storage device 910. Execution of the sequences of instructions contained in main memory 906 causes processor 904 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 910. Volatile media includes dynamic memory, such as main memory 906. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
Storage media are distinct from but may be used in conjunction with transmission media. Transmission media participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 902. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 904 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 900 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 902. Bus 902 carries the data to main memory 906, from which processor 904 retrieves and executes the instructions. The instructions received by main memory 906 may optionally be stored on storage device 910 either before or after execution by processor 904.
Computer system 900 also includes a communication interface 918 coupled to bus 902. Communication interface 918 provides a two-way data communication coupling to a network link 920 that is connected to a local network 922. For example, communication interface 918 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 918 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 918 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 920 typically provides data communication through one or more networks to other data devices. For example, network link 920 may provide a connection through local network 922 to a host computer 924 or to data equipment operated by an Internet Service Provider (ISP) 926. ISP 926 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 928. Local network 922 and Internet 928 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 920 and through communication interface 918, which carry the digital data to and from computer system 900, are example forms of transmission media.
Computer system 900 can send messages and receive data, including program code, through the network(s), network link 920 and communication interface 918. In the Internet example, a server 930 might transmit a requested code for an application program through Internet 928, ISP 926, local network 922 and communication interface 918.
The received code may be executed by processor 904 as it is received, and/or stored in storage device 910, or other non-volatile storage for later execution.
In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.
Embodiments are directed to a system with one or more devices that include a hardware processor and that are configured to perform any of the operations described herein and/or recited in any of the claims below.
In an embodiment, a non-transitory computer readable storage medium comprises instructions which, when executed by one or more hardware processors, causes performance of any of the operations described herein and/or recited in any of the claims.
Any combination of the features and functionalities described herein may be used in accordance with one or more embodiments. In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5448476 | Kurokawa et al. | Sep 1995 | A |
| 6298063 | Coile et al. | Oct 2001 | B1 |
| 6438592 | Killian | Aug 2002 | B1 |
| 6597777 | Ho | Jul 2003 | B1 |
| 6643613 | McGee et al. | Nov 2003 | B2 |
| 6928398 | Fang et al. | Aug 2005 | B1 |
| 6996599 | Anders et al. | Feb 2006 | B1 |
| 7343375 | Dulac | Mar 2008 | B1 |
| 7529991 | Ide et al. | May 2009 | B2 |
| 7672814 | Raanan et al. | Mar 2010 | B1 |
| 7739143 | Dwarakanath et al. | Jun 2010 | B1 |
| 7739284 | Aggarwal et al. | Jun 2010 | B2 |
| 7783510 | Gilgur et al. | Aug 2010 | B1 |
| 7987106 | Aykin | Jul 2011 | B1 |
| 8014983 | Crowe et al. | Sep 2011 | B2 |
| 8200454 | Dorneich et al. | Jun 2012 | B2 |
| 8229876 | Roychowdhury | Jul 2012 | B2 |
| 8234236 | Beaty et al. | Jul 2012 | B2 |
| 8363961 | Avidan et al. | Jan 2013 | B1 |
| 8576964 | Taniguchi et al. | Nov 2013 | B2 |
| 8583649 | Ailon et al. | Nov 2013 | B2 |
| 8635328 | Corley et al. | Jan 2014 | B2 |
| 8650299 | Huang et al. | Feb 2014 | B1 |
| 8676964 | Gopalan et al. | Mar 2014 | B2 |
| 8694969 | Bernardini et al. | Apr 2014 | B2 |
| 8776066 | Krishnamurthy et al. | Jul 2014 | B2 |
| 8880525 | Galle et al. | Nov 2014 | B2 |
| 8930757 | Nakagawa | Jan 2015 | B2 |
| 8949677 | Brundage et al. | Feb 2015 | B1 |
| 9002774 | Karlsson | Apr 2015 | B2 |
| 9053171 | Ailon et al. | Jun 2015 | B2 |
| 9081857 | Huet et al. | Jul 2015 | B1 |
| 9141914 | Viswanathan et al. | Sep 2015 | B2 |
| 9147167 | Urmanov et al. | Sep 2015 | B2 |
| 9195563 | Scarpelli | Nov 2015 | B2 |
| 9218232 | Khalastchi et al. | Dec 2015 | B2 |
| 9292408 | Bernstein et al. | Mar 2016 | B2 |
| 9323599 | Iyer et al. | Apr 2016 | B1 |
| 9323837 | Zhao et al. | Apr 2016 | B2 |
| 9330119 | Chan et al. | May 2016 | B2 |
| 9355357 | Hao et al. | May 2016 | B2 |
| 9367382 | Yabuki | Jun 2016 | B2 |
| 9389946 | Higuchi | Jul 2016 | B2 |
| 9471778 | Seo et al. | Oct 2016 | B1 |
| 9495220 | Talyansky | Nov 2016 | B2 |
| 9495395 | Chan et al. | Nov 2016 | B2 |
| 9507718 | Rash et al. | Nov 2016 | B2 |
| 9514213 | Wood et al. | Dec 2016 | B2 |
| 9529630 | Fakhouri et al. | Dec 2016 | B1 |
| 9658910 | Poola et al. | May 2017 | B2 |
| 9658916 | Yoshinaga et al. | May 2017 | B2 |
| 9692662 | Chan et al. | Jun 2017 | B2 |
| 9710493 | Wang et al. | Jul 2017 | B2 |
| 9727533 | Thibaux | Aug 2017 | B2 |
| 9740402 | Manoharan et al. | Aug 2017 | B2 |
| 9779361 | Jones et al. | Oct 2017 | B2 |
| 9811394 | Kogias et al. | Nov 2017 | B1 |
| 9961571 | Yang et al. | May 2018 | B2 |
| 10073906 | Lu et al. | Sep 2018 | B2 |
| 10210036 | Iyer et al. | Feb 2019 | B2 |
| 10331802 | Garvey et al. | Jun 2019 | B2 |
| 10692255 | Garvey et al. | Jun 2020 | B2 |
| 10699211 | Garvey et al. | Jun 2020 | B2 |
| 10885461 | Garvey et al. | Jan 2021 | B2 |
| 10915830 | Garvey et al. | Feb 2021 | B2 |
| 11146463 | Tedaldi et al. | Oct 2021 | B2 |
| 11295217 | Herzog | Apr 2022 | B2 |
| 20020019860 | Lee et al. | Feb 2002 | A1 |
| 20020092004 | Lee et al. | Jul 2002 | A1 |
| 20020183972 | Enck et al. | Dec 2002 | A1 |
| 20020188650 | Sun et al. | Dec 2002 | A1 |
| 20030149603 | Ferguson et al. | Aug 2003 | A1 |
| 20030154044 | Lundstedt et al. | Aug 2003 | A1 |
| 20030224344 | Shamir et al. | Dec 2003 | A1 |
| 20050119982 | Ito et al. | Jun 2005 | A1 |
| 20050132030 | Hopen et al. | Jun 2005 | A1 |
| 20050159927 | Cruz et al. | Jul 2005 | A1 |
| 20050193281 | Ide et al. | Sep 2005 | A1 |
| 20060065717 | Hurwitz et al. | Mar 2006 | A1 |
| 20060087962 | Golia et al. | Apr 2006 | A1 |
| 20060106743 | Horvitz | May 2006 | A1 |
| 20060106797 | Srinivasa et al. | May 2006 | A1 |
| 20060161485 | Meldahl | Jul 2006 | A1 |
| 20060212593 | Patrick et al. | Sep 2006 | A1 |
| 20060287848 | Li et al. | Dec 2006 | A1 |
| 20070011281 | Jhoney et al. | Jan 2007 | A1 |
| 20070150329 | Brook et al. | Jun 2007 | A1 |
| 20070179836 | Juang et al. | Aug 2007 | A1 |
| 20080221974 | Gilgur et al. | Sep 2008 | A1 |
| 20080288089 | Pettus et al. | Nov 2008 | A1 |
| 20090024427 | Shan | Jan 2009 | A1 |
| 20090030752 | Senturk-Doganaksoy et al. | Jan 2009 | A1 |
| 20090198559 | Wang et al. | Aug 2009 | A1 |
| 20100027552 | Hill | Feb 2010 | A1 |
| 20100036857 | Marvasti et al. | Feb 2010 | A1 |
| 20100050023 | Scarpelli et al. | Feb 2010 | A1 |
| 20100082132 | Marruchella et al. | Apr 2010 | A1 |
| 20100082697 | Gupta et al. | Apr 2010 | A1 |
| 20100185499 | Dwarakanath et al. | Jul 2010 | A1 |
| 20100257026 | Brocklebank | Oct 2010 | A1 |
| 20100324869 | Cherkasova et al. | Dec 2010 | A1 |
| 20110022879 | Chavda et al. | Jan 2011 | A1 |
| 20110040575 | Wright et al. | Feb 2011 | A1 |
| 20110125894 | Anderson et al. | May 2011 | A1 |
| 20110126197 | Larsen et al. | May 2011 | A1 |
| 20110126275 | Anderson et al. | May 2011 | A1 |
| 20110213788 | Zhao et al. | Sep 2011 | A1 |
| 20110265164 | Lucovsky et al. | Oct 2011 | A1 |
| 20120005359 | Seago et al. | Jan 2012 | A1 |
| 20120051369 | Bryan et al. | Mar 2012 | A1 |
| 20120066389 | Hegde et al. | Mar 2012 | A1 |
| 20120110462 | Eswaran et al. | May 2012 | A1 |
| 20120110583 | Balko et al. | May 2012 | A1 |
| 20120203823 | Manglik et al. | Aug 2012 | A1 |
| 20120240072 | Altamura et al. | Sep 2012 | A1 |
| 20120265881 | Chen et al. | Oct 2012 | A1 |
| 20120278477 | Terrell et al. | Nov 2012 | A1 |
| 20120278663 | Hasegawa | Nov 2012 | A1 |
| 20120296696 | Cantor et al. | Nov 2012 | A1 |
| 20120323988 | Barzel et al. | Dec 2012 | A1 |
| 20130013535 | Punera et al. | Jan 2013 | A1 |
| 20130024173 | Brzezicki et al. | Jan 2013 | A1 |
| 20130103616 | Hao et al. | Apr 2013 | A1 |
| 20130151179 | Gray | Jun 2013 | A1 |
| 20130226320 | Berg-Sonne et al. | Aug 2013 | A1 |
| 20130282891 | Davis et al. | Oct 2013 | A1 |
| 20130326202 | Rosenthal et al. | Dec 2013 | A1 |
| 20130329981 | Hiroike | Dec 2013 | A1 |
| 20140058572 | Stein et al. | Feb 2014 | A1 |
| 20140095422 | Solomon et al. | Apr 2014 | A1 |
| 20140101119 | Li et al. | Apr 2014 | A1 |
| 20140101300 | Rosensweig et al. | Apr 2014 | A1 |
| 20140143401 | Carlen et al. | May 2014 | A1 |
| 20140156557 | Zeng et al. | Jun 2014 | A1 |
| 20140215470 | Iniguez | Jul 2014 | A1 |
| 20140310235 | Chan et al. | Oct 2014 | A1 |
| 20140310714 | Chan | Oct 2014 | A1 |
| 20140325649 | Zhang | Oct 2014 | A1 |
| 20140379717 | Urmanov et al. | Dec 2014 | A1 |
| 20150032775 | Yang et al. | Jan 2015 | A1 |
| 20150033084 | Sasturkar et al. | Jan 2015 | A1 |
| 20150040142 | Cheetancheri et al. | Feb 2015 | A1 |
| 20150046123 | Kato | Feb 2015 | A1 |
| 20150046920 | Allen | Feb 2015 | A1 |
| 20150065121 | Gupta et al. | Mar 2015 | A1 |
| 20150180734 | Maes et al. | Jun 2015 | A1 |
| 20150205691 | Seto | Jul 2015 | A1 |
| 20150234869 | Chan et al. | Aug 2015 | A1 |
| 20150242243 | Balakrishnan et al. | Aug 2015 | A1 |
| 20150244597 | Maes et al. | Aug 2015 | A1 |
| 20150248446 | Nordstrom et al. | Sep 2015 | A1 |
| 20150251074 | Ahmed et al. | Sep 2015 | A1 |
| 20150254330 | Chan et al. | Sep 2015 | A1 |
| 20150296030 | Maes et al. | Oct 2015 | A1 |
| 20150302318 | Chen et al. | Oct 2015 | A1 |
| 20150309840 | Ding et al. | Oct 2015 | A1 |
| 20150312274 | Bishop et al. | Oct 2015 | A1 |
| 20150317589 | Anderson et al. | Nov 2015 | A1 |
| 20150371144 | Engle | Dec 2015 | A1 |
| 20150377938 | Bansal et al. | Dec 2015 | A1 |
| 20160042289 | Poola et al. | Feb 2016 | A1 |
| 20160062950 | Brodersen | Mar 2016 | A1 |
| 20160071517 | Beaver et al. | Mar 2016 | A1 |
| 20160092516 | Poola et al. | Mar 2016 | A1 |
| 20160105327 | Cremonesi et al. | Apr 2016 | A9 |
| 20160139964 | Chen et al. | May 2016 | A1 |
| 20160171037 | Mathur et al. | Jun 2016 | A1 |
| 20160253381 | Kim et al. | Sep 2016 | A1 |
| 20160283533 | Urmanov et al. | Sep 2016 | A1 |
| 20160292611 | Boe et al. | Oct 2016 | A1 |
| 20160294773 | Yu et al. | Oct 2016 | A1 |
| 20160299938 | Malhotra et al. | Oct 2016 | A1 |
| 20160299961 | Olsen | Oct 2016 | A1 |
| 20160321588 | Das et al. | Nov 2016 | A1 |
| 20160342909 | Chu et al. | Nov 2016 | A1 |
| 20160357674 | Waldspurger et al. | Dec 2016 | A1 |
| 20160378809 | Chen et al. | Dec 2016 | A1 |
| 20170061297 | Joshi et al. | Mar 2017 | A1 |
| 20170061321 | Maiya et al. | Mar 2017 | A1 |
| 20170249563 | Garvey et al. | Aug 2017 | A1 |
| 20170249564 | Garvey et al. | Aug 2017 | A1 |
| 20170249648 | Garvey et al. | Aug 2017 | A1 |
| 20170249649 | Garvey et al. | Aug 2017 | A1 |
| 20170249763 | Garvey et al. | Aug 2017 | A1 |
| 20170262223 | Dalmatov et al. | Sep 2017 | A1 |
| 20170284903 | Anderson et al. | Oct 2017 | A1 |
| 20170329660 | Salunke et al. | Nov 2017 | A1 |
| 20170351563 | Miki et al. | Dec 2017 | A1 |
| 20170364851 | Maheshwari et al. | Dec 2017 | A1 |
| 20180026907 | Miller et al. | Jan 2018 | A1 |
| 20180039555 | Salunke et al. | Feb 2018 | A1 |
| 20180052804 | Mikami et al. | Feb 2018 | A1 |
| 20180053207 | Modani et al. | Feb 2018 | A1 |
| 20180059628 | Yoshida | Mar 2018 | A1 |
| 20180081629 | Kuhhirte et al. | Mar 2018 | A1 |
| 20180219889 | Oliner et al. | Aug 2018 | A1 |
| 20180268152 | Cuomo et al. | Sep 2018 | A1 |
| 20180321989 | Shetty et al. | Nov 2018 | A1 |
| 20180324199 | Crotinger et al. | Nov 2018 | A1 |
| 20180330433 | Frenzel et al. | Nov 2018 | A1 |
| 20180349797 | Garvey et al. | Dec 2018 | A1 |
| 20180351786 | Pope et al. | Dec 2018 | A1 |
| 20190042982 | Qu et al. | Feb 2019 | A1 |
| 20190065275 | Wong et al. | Feb 2019 | A1 |
| 20190121808 | Chadha et al. | Apr 2019 | A1 |
| 20190305876 | Sundaresan et al. | Oct 2019 | A1 |
| 20190379589 | Ryan et al. | Dec 2019 | A1 |
| 20190392098 | Mehta et al. | Dec 2019 | A1 |
| 20200034745 | Nagpal et al. | Jan 2020 | A1 |
| 20200125988 | Garvey et al. | Apr 2020 | A1 |
| 20200387797 | Ryan et al. | Dec 2020 | A1 |
| 20210028973 | Côté et al. | Jan 2021 | A1 |
| 20210257060 | Curtis et al. | Aug 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 1097829 | Jan 1995 | CN |
| 101482944 | Jul 2009 | CN |
| 103984938 | Aug 2014 | CN |
| 105144112 | Dec 2015 | CN |
| 105426411 | Mar 2016 | CN |
| 109359763 | Feb 2019 | CN |
| 2984568 | Feb 2016 | EP |
| 2006-129446 | May 2006 | JP |
| 2011071624 | Jun 2011 | WO |
| 2013016584 | Jan 2013 | WO |
| 2014169056 | Oct 2014 | WO |
| Entry |
|---|
| Esling et al., “Time-Series Data Mining,” in 45.1 ACM Computing Surveys 1-34 (2012). (Year: 2012). |
| Wang, “Online Monitoring and Prediction of Complex Time Series Events from Nonstationary Time Series Data,” doctoral dissertation, Rutgers U. (2012). (Year: 2012). |
| Chu, “A Piecewise Linear Approach to Modeling and Forecasting Demand for Macau Tourism,” in 32 Tourism Mgmt. 1414-20 (2011). (Year: 2011). |
| “OpenStack Compute Administration Manual”, Mar. 1, 2011 (Mar. 1, 2011), XP055207492, Retrieved from the Internet: URL:http://web.archive.org/web/20110708071910/http://docs.openstack.org/bexar/openstack-compute/admin/os-compute-admin-book-bexar.pdf [retrieved on Aug. 12, 2015]. |
| “OpenStack Object Storage Administrator Manual”, Jun. 2, 2011 (Jun. 2, 2011), XP055207490, Retrieved from the Internet: URL:http://web.archive.org/web/20110727190919/http://docs.openstack.org/cactus/openstack-object-storage/admin/os-objectstorage-adminguide-cactus.pdf [retrieved on Aug. 12, 2015]. |
| “Time Series Pattern Search: A tool to extract events from time series data”, available online at <https://www.ceadar.ie/pages/time-series-pattern-search/>, retrieved on Apr. 24, 2020, 4 pages. |
| Ahmed, Reservoir-based network traffic stream summarization for anomaly detection, Article in Pattern Analysis and Applications, Oct. 2017. |
| Alberto Zuin: “OpenNebula Setting up High Availability in OpenNebula with LVM”, May 2, 2011 (May 2, 2011), XP055207701, Retrieved from the Internet: URL:http://opennebula.org/setting-up-highavailability-in-opennebula-with-lvm/ [retrieved on Aug. 13, 2015] p. 1. |
| Andrew Beekhof: “Clusters from Scratch—Apache, DRBD and GFS2 Creating Active/Passive and Active/Active Clusters on Fedora 12”, Mar. 11, 2010 (Mar. 11, 2010), XP055207651, Retrieved from the Internet: URL:http://clusterlabs.org/doc/en-US/Pacemaker/1.0/pdf/Clusters from Scratch/Pacemaker-1.0-Clusters from Scratch-en-US.pdi [retrieved on Aug. 13, 2015]. |
| Anonymous: “High Availability for the Ubuntu Enterprise Cloud (UEC)—Cloud Controller (CLC)”, Feb. 19, 2011 (Feb. 19, 2011), XP055207708, Retrieved from the Internet: URL:http://blog.csdn.net/superxgl/article/details/6194473 [retrieved on Aug. 13, 2015] p. 1. |
| Charapko, Gorilla—Facebook's Cache for Time Series Data, http://charap.co/gorilla-facebooks-cache-for-monitoring-data/, Jan. 11, 2017. |
| Chris Bunch et al: “AppScale: Open-Source Platform-As-A-Service”, Jan. 1, 2011 (Jan. 1, 2011), XP055207440, Retrieved from the Internet: URL:http://128.111.41.26/research/tech reports/reports/2011-01 .pdf [retrieved on Aug. 12, 2015] pp. 2-6. |
| Davies, David L., et al., “A Cluster Separation measure”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-1, No. 2, Apr. 1979, pp. 224-227. |
| Dunning et al., Computing Extremely Accurate Quantiles Using t-Digests. |
| Faraz Rasheed, “A Framework for Periodic Outlier Pattern Detection in Time-Series Sequences,” May 2014, IEEE. |
| Greenwald et al. “Space-efficient online computation of quantile summaries.” ACM Proceedings of the 2001 SIGMOD international conference on Management of data pp. 58-66. |
| Greunen, “Forecasting Methods for Cloud Hosted Resources, a comparison,” 2015, 11th International Conference on Network and Service Management (CNSM), pp. 29-35 (Year: 2015). |
| Gueyoung Jung et al: “Performance and availability aware regeneration for cloud based multitier applications”, Dependable Systems and Networks (DSN), 2010 IEEE/IFIP International Conference on, IEEE, Piscataway, NJ, USA, Jun. 28, 2010 (Jun. 28, 2010), pp. 497-506. |
| Gunter et al., Log Summarization and Anomaly Detection for Troubleshooting Distributed Systems, Conference: 8th IEEE/ACM International Conference on Grid Computing (GRID 2007), Sep. 19-21, 2007, Austin, Texas, USA, Proceedings. |
| Hao et al., Visual Analytics of Anomaly Detection in Large Data Streams, Proc. SPIE 7243, Visualization and Data Analysis 2009, 10 pages. |
| Haugen et al., “Extracting Common Time Trends from Concurrent Time Series: Maximum Autocorrelation Factors with Applications”, Stanford University, Oct. 20, 2015, pp. 1-38. |
| Herbst, “Self-adaptive workload classification and forecasting for proactive resource provisioning”, 2014, ICPE'13, pp. 187-198 (Year: 2014). |
| Jain and Chlamtac, P-Square Algorithm for Dynamic Calculation of Quantiles and Histograms Without Storing Observations, ACM, Oct. 1985 (10 pages). |
| Jarvis, R. A., et al., “Clustering Using a Similarity Measure Based on Shared Neighbors”, IEEE Transactions on Computers, vol. C-22, No. 11, Nov. 1973, pp. 1025-1034. |
| Li et al., “Forecasting Web Page Views: Methods and Observations,” in 9 J. Machine Learning Res. 2217-50 (2008). (Year: 2008). |
| Lin, Xuemin, et al. “Continuously maintaining quantile summaries of the most recent n elements over a data stream,” IEEE, 2004. |
| Niino, Junichi, “Open Source Cloud Infrastructure ‘OpenStack’, its History and Scheme”, Available online at <http://www.publickey1.jp/blog/11/openstack_1.html>, Jun. 13, 2011, 18 pages (9 pages of English Translation and 9 pages of Original Document). |
| NPL: Web document dated Feb. 3, 2011, Title: OpenStack Compute, Admin Manual. |
| Nurmi D et al: “The Eucalyptus Open-Source Cloud-Computing System”, Cluster Computing and the Grid, 2009. CCGRID '09. 9th IEEE/ACM International Symposium on, IEEE, Piscataway, NJ, USA, May 18, 2009 (May 18, 2009), pp. 124-131. |
| Qiu, Hai, et al. “Anomaly detection using data clustering and neural networks.” Neural Networks, 2008. IJCNN 2008.(IEEE World Congress on Computational Intelligence). IEEE International Joint Conference on. IEEE, 2008. |
| Slipetskyy, Rostyslav, “Security Issues in OpenStack”, Maste s Thesis, Technical University of Denmark, Jun. 2011, p. 7 (entire document especially abstract). |
| Somlo, Gabriel, et al., “Incremental Clustering for Profile Maintenance in Information Gathering Web Agents”, Agents '01, Montreal, Quebec, Canada, May 28-Jun. 1, 2001, pp. 262-269. |
| Suntinger, “Trend-based similarity search in time-series data,” 2010, Second International Conference on Advances in Databases, Knowledge, and Data Applications, IEEE, pp. 97-106 (Year: 2010). |
| Szmit et al., “Usage of Modified Holt-Winters Method in the Anomaly Detection of Network Traffic: Case Studies”, Journal of Computer Networks and Communications, vol. 2012, Article ID 192913, Mar. 29, 2012, pp. 1-5. |
| Taylor et al., “Forecasting Intraday Time Series With Multiple Seasonal Cycles Using Parsimonious Seasonal Exponential Smoothing”, OMEGA, vol. 40, No. 6, Dec. 2012, pp. 748-757. |
| Voras et al., “Criteria for evaluation of Open Source Cloud Computing Solutions”, Information Technology Interfaces (ITI), Proceedings of the ITI 2011 33rd International Conference on Information Technology Interfaces, US, IEEE, Jun. 27-30, 2011, pp. 137-142. |
| Voras I et al: “Evaluating open-source cloud computing solutions”, MIPRO, 2011 Proceedings of the 34th International Convention, IEEE, May 23, 2011 (May 23, 2011), pp. 209-214. |
| Wilks, Samuel S. “Determination of sample sizes for setting tolerance limits,” The Annals of Mathematical Statistics 12.1 (1941): 91-96. |
| Willy Tarreau: “HAProxy Architecture Guide”, May 25, 2008 (May 25, 2008), XP055207566, Retrieved from the Internet: URL:http://www.haproxy.org/download/1.2/doc/architecture.txt [retrieved on Aug. 13, 2015]. |
| Yin, “System resource utilization analysis and prediction for cloud based applications under bursty workloads,” 2014, Information Sciences, vol. 279, pp. 338-357 (Year: 2014). |
| Yokoyama, Tetsuya, “Windows Server 2008, Certification Test, Part 5: Reliability”, Apr. 15, 2008, pp. 124-125 along with 4 pages of English Translation. |
| Guajardo, et al., “A model updating strategy for predicting time series with seasonal patterns” Applied Soft Computing, vol. 10 No. 1, 2010, pp. 276-283. |
| Laptev, “Generic and scalable framework for automated time-series anomaly detection,” 2015, in Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1939-1947 (Year 2015). |
| Lehmann, “Variability of Atmospheric Circulation Patterns Associated With Large Volume Changes of the Baltic Sea”, Adv. Sci. Res., 12, 219-225, 2015, www.adv-sci-res.net/12/219/2015/DOI: 10.5.194/ASR-12-219-2015 (Year: 2015). |
| Toreti, “A Novel Approach for the Detection of Inhomogeneities Affecting Climate Time Series”, Journal of Applied Meteorology and Climatology, vol. 51, Feb. 2012, DOI: 10.1175/JAMC-D-10-05033.1 (Year: 2012). |
| Zhai, “Spatial Variation and Trends in PDSI and SPI Indices and Their Relation to Streamflow in 10 Large Regions of China”, 2010, Journal of Climate, vol. 23, 2010 American Meteorological Society, DOI: 10.1175/2009JCLI2968.1 (Year: 2010). |
| Carvalho et al., “Long-term SLOs for reclaimed cloud computing resources,” 2014, in Proceedings of the ACM Symposium on Cloud Computing (SOCC '14). Association for Computing Machinery, New York, NY, USA, 1-13 (Year: 2014). |
| Ghosh, “Biting off safely more than you can chew: Predictive analytics for resource over-commit in iaas cloud,” 2012, in 2012 IEEE Fifth International Conference on Cloud Computing, pp. 25-32 (Year: 2012). |
| Kuhn, Markus, A summary of the international standard date and time notation (https://www.cl.cam.ac.uk/˜mgk25/iso-time.html), date retrieved from web.archive.org Oct. 13, 2018, 5 pages (Year: 2018). |
| Morgan, Derek P., Chapter 4: ISO 8601 Dates, Times, Datetimes, Durations, and Functions, The Essential Guide to SAS Dates and Times, Second Edition, Dec. 2014, 40 pages (Year: 2014). |
| Number | Date | Country | |
|---|---|---|---|
| 20210183120 A1 | Jun 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62301590 | Feb 2016 | US | |
| 62301585 | Feb 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15266979 | Sep 2016 | US |
| Child | 17186411 | US |
