Patent Application
20190349273
Cloud computing relates to the sharing of computing resources that are generally accessed via the Internet. In particular, a cloud computing infrastructure allows users, such as individuals and/or enterprises, to access a shared pool of computing resources, such as servers, storage devices, networks, applications, and/or other computing based services. By doing so, users are able to access computing resources on demand that are located at remote locations, which resources may be used to perform a variety computing functions (e.g., storing and/or processing large quantities of computing data). For enterprise and other organization users, cloud computing provides flexibility in accessing cloud computing resources without accruing large up-front costs, such as purchasing expensive network equipment or investing large amounts of time in establishing a private network infrastructure. Instead, by utilizing cloud computing resources, users are able redirect their resources to focus on their enterprise's core functions.
In modern communication networks, examples of cloud computing services a user may utilize include so-called infrastructure as a service (IaaS), software as a service (SaaS), and platform as a service (PaaS) technologies. IaaS is a model in which providers abstract away the complexity of hardware infrastructure and provide rapid, simplified provisioning of virtual servers and storage, giving enterprises access to computing capacity on demand. In such an approach, however, a user may be left to install and maintain platform components and applications. SaaS is a delivery model that provides software as a service rather than an end product. Instead of utilizing a local network or individual software installations, software is typically licensed on a subscription basis, hosted on a remote machine, and accessed by client customers as needed. For example, users are generally able to access a variety of enterprise and/or information technology (IT)-related software via a web browser. PaaS acts an extension of SaaS that goes beyond providing software services by offering customizability and expandability features to meet a user's needs. For example, PaaS can provide a cloud-based developmental platform for users to develop, modify, and/or customize applications and/or automating enterprise operations without maintaining network infrastructure and/or allocating computing resources normally associated with these functions.
In some cases, tracking one or more metrics associated with the cloud computing services may help to achieve more efficient allocation of cloud computing resources and/or a more streamlined cloud computing service.
A summary of certain embodiments disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be set forth below.
Information Technology (IT) networks may include a number of computing devices, server systems, databases, and the like that generate, collect, and store information. As increasing amounts of data representing vast resources become available, it becomes increasingly difficult to analyze the data, interact with the data, and/or provide reports for the data. The current embodiments enable customized widgets to be generated for such data, enabling a visualization of certain indicators for the data for rapid and/or real-time monitoring of the data.
For example, in an embodiment, a system configured to create analytics widgets in a guided widget creation workflow is associated with a computational instance of a remote platform that remotely manages a managed network. The system includes a database containing analytics data associated with the managed network, the analytics data defining indicators or metrics. The remote platform is configured to present a portion of the computational instance on a graphical user interface (GUI) via a display connected to a computing device having access to the computational instance. Incident data is collected over first and second periods of time. Incident data from the first period of time is used to train a plurality of predictive models, which may be variable in length. The predictive models are then used to predict incident data over the second period of time. The predictions from the plurality of models are then compared to the collected incident data over the second period of time. The predictive model that best predicted the collected data over the second period of time is selected and used to predict incident data over a third period of time. In some embodiments, the incident data may include a number of open incidents, a number of new incidents, a number of closed incidents, a total number of incidents, or some other value. In some embodiments, the predicted incident data may be constrained within an allowable range of values that is configurable by the user. The predicted incident data may also include a range of values for incident data within a set confidence interval. The collected incident data over the first and/or second periods of time and the predicted incident data over the third period of time may then be incorporated into a graphical display of a graphical user interface (e.g., a widget of a dashboard).
Various refinements of the features noted above may exist in relation to various aspects of the present disclosure. Further features may also be incorporated in these various aspects as well. These refinements and additional features may exist individually or in any combination. For instance, various features discussed below in relation to one or more of the illustrated embodiments may be incorporated into any of the above-described aspects of the present disclosure alone or in any combination. The brief summary presented above is intended only to familiarize the reader with certain aspects and contexts of embodiments of the present disclosure without limitation to the claimed subject matter.
Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:
One or more specific embodiments will be described below. In an effort to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and enterprise-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.
As used herein, the term “computing system” refers to a single electronic computing device that includes, but is not limited to a single computer, virtual machine, virtual container, host, server, laptop, and/or mobile device, or to a plurality of electronic computing devices working together to perform the function described as being performed on or by the computing system. As used herein, the term “medium” refers to one or more non-transitory, computer-readable physical media that together store the contents described as being stored thereon. Embodiments may include non-volatile secondary storage, read-only memory (ROM), and/or random-access memory (RAM). As used herein, the term “application” refers to one or more computing modules, programs, processes, workloads, threads and/or a set of computing instructions executed by a computing system. Example embodiments of an application include software modules, software objects, software instances and/or other types of executable code. As used herein, the terms alerts, incidents (INTs), changes (CHGs), and problems (PRBs) are used in accordance with the generally accepted use of the terminology for CMDBs. Moreover, the term “issues” with respect to a CI of a CMDB collectively refers to alerts, INTs, CHGs, and PRBs associated with the CI.
The disclosed subject matter includes techniques for predicting future values based on patterns observed in collected historical data. Specifically, the instant embodiments are directed to forecasting incident data in computer networks. However, it should be used that similar techniques may be used for a wide range of data collected from a wide range of data sources. In the present embodiments, incident data is collected over first and second periods of time. Incident data from the first period of time is used to train a plurality of predictive models. The predictive models are then used to predict incident data over the second period of time. The predictions from the plurality of models are then compared to the collected incident data over the second period of time. The predictive model that best predicted the collected data over the second period of time is selected and used to predict incident data over a third period of time. In some embodiments, the incident data may include a number of open incidents, a number of new incidents, a number of closed incidents, a total number of incidents, or some other value. In some embodiments, the predicted incident data may be constrained within an allowable range of values that is configurable by the user. The predicted incident data may also include a range of values for incident data within a set confidence interval. The collected incident data over the first and/or second periods of time and the predicted incident data over the third period of time may then be incorporated into a graphical display of a graphical user interface (e.g., a widget of a dashboard).
With the preceding in mind, the following figures relate to various types of generalized system architectures or configurations that may be employed to provide services to an organization in a multi-instance framework on which the present approaches may be employed. Correspondingly, these system and platform examples may also relate to systems and platforms on which the techniques discussed herein may be implemented or otherwise utilized. Turning now to
For the illustrated embodiment,
In
To utilize computing resources within the platform 16, network operators may choose to configure the data centers 18 using a variety of computing infrastructures. In one embodiment, one or more of the data centers 18 are configured using a multi-instance cloud architecture to provide every customer its own unique customer instance or instances. For example, a multi-instance cloud architecture could provide each customer instance with its own dedicated application server and dedicated database server. In other examples, the multi-instance cloud architecture could deploy a single physical or virtual server 26 and/or other combinations of physical and/or virtual servers 26, such as one or more dedicated web servers, one or more dedicated application servers, and one or more database servers, for each customer instance. In a multi-instance cloud architecture, multiple customer instances could be installed on one or more respective hardware servers, where each customer instance is allocated certain portions of the physical server resources, such as computing memory, storage, and processing power. By doing so, each customer instance has its own unique software stack that provides the benefit of data isolation, relatively less downtime for customers to access the platform 16, and customer-driven upgrade schedules. An example of implementing a customer instance within a multi-instance cloud architecture will be discussed in more detail below with reference to
In the depicted example, to facilitate availability of the client instance 102, the virtual servers 26A, 26B, 26C, 26D and virtual database servers 104A, 104B are allocated to two different data centers 18A, 18B, where one of the data centers 18 acts as a backup data center 18. In reference to
As shown in
Although
As may be appreciated, the respective architectures and frameworks discussed with respect to
With this in mind, and by way of background, it may be appreciated that the present approach may be implemented using one or more processor-based systems such as shown in
With this in mind, an example computer system may include some or all of the computer components depicted in
The one or more processors 202 may include one or more microprocessors capable of performing instructions stored in the memory 206. Additionally or alternatively, the one or more processors 202 may include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and/or other devices designed to perform some or all of the functions discussed herein without calling instructions from the memory 206.
With respect to other components, the one or more busses 204 includes suitable electrical channels to provide data and/or power between the various components of the computing system 200. The memory 206 may include any tangible, non-transitory, and computer-readable storage media. Although shown as a single block in
The discussion now turns to a mechanism for displaying system data, enabling interactivity with the system data, and reporting on the system data.
In some embodiments, it may be desirable to enable customized positioning and/or sizing of widgets 304. Accordingly, the dashboard 302 may be used to provide such features. As used herein, the term “dashboard” refers to a graphical-user-interface (GUI) screen where data-driven widgets 304 may be placed on the screen without being constrained to pre-defined containers 306 and/or static placement and/or size. In other words, for the dashboard 302, the widgets 304 may be dynamically moved to any location on the dashboard 302 without being constrained to pre-defined locations, as indicated by arrows 308. Further, the size of the widgets 304 may be dynamically altered in the dashboard 302, as indicated by sizing indicators 310 and arrows 312.
As there may be more flexibility in configuring a dashboard 302 over a homepage 300, it may be desirable in certain situations to convert a homepage 300 to a dashboard 302. Indeed, it may be burdensome to generate dashboards 302 from scratch after time and effort may have already been afforded to creating a homepage 300. Accordingly, in some embodiments, a conversion process 314 may be implemented to convert a homepage 300 to a dashboard 302.
The conversion process 314 may identify the widgets 304 found on the homepage 300 (block 316). For example, a computer-readable representation of the homepage 300 (e.g., a homepage object) may be traversed to identify each of the widgets 304 on the homepage 300.
Further, the conversion process 314 may identify the containers 306 and their associated sizes and placements for the identified widgets 304 found on the homepage 300 (block 318). For example, the computer-readable representation of the homepage 300 (e.g., a homepage object) may be traversed to identify each of containers 306 containing the widgets 304 on the homepage 300. Position and/or size attributes of the containers 306 may be identified by accessing object attributes of the computer-readable representation of the homepage 300.
Once the widgets 304 and the containers 306 and their attributes are identified. A corresponding dashboard 302 may be generated (block 320). For example, computer instructions may generate a computer-readable representation of the homepage 300, inserting the widgets 304 at the position and/or size identified by the container 306 attributes. Once the dashboard 302 is generated, it may be accessed and the size and position of the widgets 304 may be modified dynamically.
The widgets 304 may be independent data-driven software that perform particular tasks. For example, the widgets 304 may provide visualizations generated based upon datasets of the system, such as those present within database. In accordance with certain aspects of the present disclosure, the widgets 304 are generated according to a guided workflow presented as a part of a graphical user interface (GUI) 400, an example of which is illustrated in
However, these techniques may be used to predict future values for a wide range of statistics or metrics. For example, in the area of application portfolio management, these techniques may be used to predict average cost per user by week, month, quarter, or year. In the area of configuration management databases (CMDBs), these techniques may be used to predict the sum cost of configuration items (CIs), number of open changes to CIs, number of monitored CIs, etc. In the area of customer service case management these techniques may be used to predict number of cases closed per agent per month, summed re-assignment count of open cases, number of open cases not updated in the last 5 days, etc. In the area of discovery these techniques may be used to predict sum duration of jobs executed in a day. In the area of financial management these techniques may be used to predict total expenses, top spenders, annual planned budget, operational expenditure (OPEX), capital expenditure (CAPEX), etc. In the area of human resources these techniques may be used to predict average time taken for onboarding activities, new hire satisfaction survey results, summed duration of onboarding action items, etc. In the area of knowledge management these techniques may be used to predict number of published articles flagged, average article rating, summed length or published articles, etc. In the area of project portfolio management, these techniques may be used to predict the summed overdue age of project tasks, the project task backlog growth, etc. It should be understood, however, that the preceding enumerated examples
As shown in
In a configuration section 512 of the GUI 400, the attribute input fields 514 now include different chart type buttons 516. The GUI 400 displays the different chart types using the data set in the table 502 within the data visualization section 500 in response to a particular chart type button 516 being selected. By way of non-limiting example, the GUI 400 in
The target line 700 is a line of a target number of incidents as a function of date. The target band 700 may be user-defined, or may be user-defined as a function of the number of open incidents at a given date. The trend line 702 demonstrates the manner in which the data (e.g., incidents) trends over the range of dates (e.g., over time) and may be smoothed relative to the open incidents line 602. The prediction band 708 and the confidence band 710 are visualizations in regression analysis of the incident data. The comment box 712 indicates that a comment has been left for that day or data point. When the comment box 712 is selected, the GUI 400 may respond by displaying the comment (e.g., via a pop-up window). The maximum line 714 and the minimum line 716 indicate the maximum and minimum values, respectively, of the plotted incident data.
The forecast line 704 depicts how the incidents are forecast to increase, decrease, or remain constant at future dates. The forecast line 704 may be generated using one or more of a plurality of forecasting models based on past incident data over a period of time. The forecast range 706 includes a range of values on either side of the forecast line 704 in which the forecast values are predicted to fall within a set confidence level.
The predicted values from the various prediction models over the second period of time 806 are compared to the actual incident data over the second period of time (indicated by line 802) to determine which of the prediction models had predicted values that were closest to the actual incident data over the second period of time 806. The comparison may include determining the difference between the predicted values and the actual values for each model for each day and adding up the differences or averaging the differences to determine which prediction model best predicted the actual values. In other embodiments, the closest prediction model may be determined for each day and then the predicted model that was closest for the greatest number of days within the second time period 806 may be selected. Other techniques for selecting the closest prediction model may include, for example, root mean squared error (RMSE), Akaike information criterion (AIC), Bayes factor, Bayesian information criterion (BIC), cross validation, deviance information criterion (DIC), efficient determination criterion (EDC), focused information criterion (FIC), Hannan-Quinn information criterion, stepwise regression, Watanable-Akaike information criterion (WAIC), or some other technique for evaluating model fit.
Once one of the prediction models is selected, the prediction model is used to predict incident data for a third period of time 808 based on the incident data for the first period of time 804, the second period of time 806, both, or some other training data set. In some embodiments, an allowable range of values may be set. If the selected prediction model predicts a value below the minimum of the allowable range, the prediction value may be updated to the minimum value of the allowable range or flagged as outside of the allowable range. For example, a user may want to set a lower limit of the allowable range of incident values at zero as there cannot be a negative number of incidents. Similarly, if the selected prediction model predicts a value above the maximum of the allowable range, the prediction value may be updated to the maximum value of the allowable range or flagged as outside of the allowable range. The allowable range may be set by a user, an administrator, an algorithm, or in some other way.
In some embodiments, the selected prediction model may also be used to generate a range of possible values within a confidence level. The model predictions for the third period of time 808 are then graphed as the forecast line. As with the allowable range, the confidence level may be set by a user, an administrator, an algorithm, or in some other way.
As previously discussed, the plurality of prediction models may include, among other prediction models, a random forest prediction model, a drift prediction model, a naive seasonal prediction model, a naive seasonal drift prediction model, a linear prediction model, a seasonal trend loess prediction model, etc. The random forest prediction model is made up of a number of decision trees. Each decision tree takes a random subset of the training data and generates a unique prediction model based on the subset of the training data. The various unique prediction model associated with each decision tree are then combined (e.g., averaged) to form a larger prediction model (i.e., the random forest prediction model). In some embodiments, the most recent period/season may be temporarily removed from the training data set. A lag matrix may then be calculated and random forest model generated for each of a set of maximum lags (e.g., 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 20, 28, 30, 31, etc.). The maximum lag of the random forest prediction model with the lowest RMSE is then selected. A new random forest model may then be generated by using all of the data and the selected maximum lag value. In the instant embodiment, random forest models are generated using the statistical machine intelligence and learning engine (SMILE) library.
The drift prediction model uses linear regression to determine the slope and intercept of a line that passes through the first and last points of the data and then extends the line to the number of observations to be forecast. The naive seasonal drift prediction model is like the drift prediction model, but projects a seasonal component onto the drift prediction model's trend lines. The seasonal component is calculated by deducting the draft's trend line from the previous period/season of the data.
Prediction model analysis and selection may occur in a continuous, on-going fashion, or at discreet intervals (e.g., daily, weekly, bi-weekly, monthly, bi-monthly, quarterly, annually, etc.). The lengths of the various periods of time may also be adjusted by a user. For example, if a new email platform was implemented 30 days ago, it may be best to setup the system to only use data (e.g., the first period of time) after the new platform was implemented to train the prediction model. However, in other cases, it may be helpful to train the prediction model using as large a training data set as possible. In such cases, the first period of time (i.e., the time covered by the training data set) may be set to extend for a few months or possibly longer. Along these lines, in some cases it may be beneficial to extend the second period of time (i.e., the time period during which predictions from various prediction models are compared to actual data) to ensure that the most accurate prediction model is chosen. In some embodiments, prediction model analysis and selection may be triggered upon request by a user, or when the currently selected model misses a prediction or a series of predictions by some threshold amount.
In some embodiments, the user may override the automatic predictive model selection and force the system to use a predictive model selected by the user. In other embodiments, the user may provide a preference for one model over the others or weight one or more predictive models higher than others, such that the system may be configured to select a higher weighted predictive model unless one of the other lower weighted models performs significantly better than the higher weighted predictive model.
Some embodiments may implement a forecast data caching scheme. For example, forecasted values may be cached in a database table as a comma separated value (CSV) field. If a forecast cache entry is available for a time series, identified by a universally unique identifier (UUID), the forecasted values are available immediately. Otherwise, the forecasted values are calculated according to the settings (e.g., using the automatically selected prediction model). If any of the forecast configurations for a time series have changed, or a new data point has been recorded, or any of the existing data points are edited, then the respective forecast cache entries are invalidated. Changes in the data points may be detected, for example, by calculating a hash of data point values.
The graph window includes the open incidents line 602, in this case number of new incidents on a given day, up to a point in time 918, wherein the point in time is the time of the most recent measurement. The forecast line 704 starts at the point in time 918 and goes forward, plotting forecasted values for the number of new incidents on days following the point in time 918. As previously discussed, the user may utilize the forecast lower and upper limit drop-down menus 912, 914 to set lower and/or upper limits to forecast values. If the selected predictive model predicts a value below the lower limit or above the upper limit, the predicted value may just be replaced with the lower or upper limit. In the graph window 906 of
At block 1006, a plurality of predictive models is trained with machine learning techniques using the incident data from the first time period. As previously described, the predictive models may include, for example, a random forest prediction model, a drift prediction model, a naive seasonal prediction model, a naive seasonal drift prediction model, a linear prediction model, a seasonal trend loess prediction model, and/or one or more other predictive models. At block 1008, each of the plurality of trained predictive models are used to predict incident data over the second time period. At block 1010, the predictive model predictions for each predictive model are compared to the collected incident data over the second time period. At block 1012, the predictive model with predicted values closest to the collected incident values over the second time period is selected.
At block 1014, the selected predictive model is used to predict incident data over a third time period. The prediction may be based on the incident data from the first time period and/or the second time period, or some other time period. At block 1016, a graphical display is generated that includes the predicted incident data over the third period of time as a forecast. As shown in
The disclosed subject matter includes techniques for forecasting incident data in computer networks. Specifically, incident data is collected over first and second periods of time. Incident data from the first period of time is used to train a plurality of predictive models. The predictive models are then used to predict incident data over the second period of time. The predictions from the plurality of models are then compared to the collected incident data over the second period of time. The predictive model that best predicted the collected data over the second period of time is selected and used to predict incident data over a third period of time. In some embodiments, the incident data may include a number of open incidents, a number of new incidents, a number of closed incidents, a total number of incidents, or some other value. In some embodiments, the predicted incident data may be constrained within an allowable range of values that is configurable by the user. The predicted incident data may also include a range of values for incident data within a set confidence interval. The collected incident data over the first and/or second periods of time and the predicted incident data over the third period of time may then be incorporated into a graphical display of a graphical user interface (e.g., a widget of a dashboard).
The specific embodiments described above have been shown by way of example, and it should be understood that these embodiments may be susceptible to various modifications and alternative forms. It should be further understood that the claims are not intended to be limited to the particular forms disclosed, but rather to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure.
The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature that demonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical. Further, if any claims appended to the end of this specification contain one or more elements designated as “means for [perform]ing [a function] . . . ” or “step for [perform]ing [a function] . . . ”, it is intended that such elements are to be interpreted under 35 U.S.C. 112(f). However, for any claims containing elements designated in any other manner, it is intended that such elements are not to be interpreted under 35 U.S.C. 112(f).
