A major shift in the software industry exists where the majority of new software is built for and deployed on public cloud providers—providers that provide on-demand availability of computer system resources, such as for example data storage and computing power, over a network such as the Internet and without direct active management by a user. Cloud providers typically charge their customers for the services used on a monthly basis, and provide a breakdown of the costs generated by such services which are specific to the particular cloud provider.
Most software companies have customers that require company software, which in turn uses company computing resources. It is difficult to determine what the requirements for customer success translate to in terms of computing resources provided by cloud providers. What is needed is an improved method for analyzing the costs associated with customer requirements.
The present technology, roughly described, automatically correlates infrastructure usage and cost data to key performance indicators for a software system. The present system continuously monitors the software system to detect operational events, key performance indicators, service degradation, and other events of a software system. The events may include the start and stop times of resource usage, memory usage, processor cycles used, and other data that may be expressed as time series data. The key performance indicator data can include the time of a transaction, login process, or other transaction, a number of deployments, number of active users, pipeline minutes for deployments, and other metrics, for example metrics associated with operations initiated directly or indirectly by a user or customer request or service usage.
The present system then correlates network infrastructure resource usage with key performance indicator data. The correlation can be performed using one of several techniques, including but not limited to a statistical model that utilizes statistical correlation and co-variance analysis. A correlation score is determined for a particular set of KPI data and the network infrastructure data. If the correlation score satisfies a threshold, the correlation is determined to exist. Based on detected correlations, the present system can automatically predict future infrastructure requirements based on forecast KPI data, as well as the corresponding costs of the predicted infrastructure requirements. The data, correlations, and predictions can be reported to a user, for example through a dashboard with graphical, metric, and other elements.
In some instances, a method for automatically allocating resource usage to key performance indicators receives, by an application server, event data from one or more delegates installed in one or more computing resources. The event data includes business service specific resource events. The application server can receive performance data associated with an operation initiated by a user. The application server can then correlate an infrastructure usage based on the event data with the performance data over a period of time. The infrastructure usage correlated to the performance data can be reported to a remote device associated with a customer.
In some instances, a non-transitory computer readable storage medium has embodied thereon a program. The program is executable by a processor to perform a method for automatically allocating resource usage to key performance indicators. The method includes receiving, by an application server, event data from one or more delegates installed in one or more computing resources. The event data includes business service specific resource events. The application server can receive performance data associated with an operation initiated by a user. The application server can then correlate an infrastructure usage based on the event data with the performance data over a period of time. The infrastructure usage correlated to the performance data can be reported to a remote device associated with a customer.
A system for automatically allocating resource usage to key performance indicators includes a server. The server includes a memory, a processor, and one or more modules, where the one or more modules are stored in memory and executable by the processor. The modules are executable to receive event data from one or more delegates installed in one or more computing resources, the event data including business service specific resource events, receive performance data associated with an operation initiated by a user, correlate an infrastructure usage based on the event data with the performance data over a period of time, and report the infrastructure usage correlated to the performance data to a remote device associated with a customer.
The present system automatically correlates infrastructure usage and cost data to key performance indicators for a software system. The present system continuously monitors the software system to detect operational events, key performance indicators, service degradation, and other events of a software system. The events may include the start and stop times of resource usage, memory usage, processor cycles used, and other data that may be expressed as time series data. The key performance indicator data can include the time of a transaction, login process, or other transaction, a number of deployments, number of active users, pipeline minutes for deployments, and other metrics, for example metrics associated with operations initiated directly or indirectly by a user or customer request or service usage.
The present system then correlates network infrastructure resource usage with key performance indicator data. The correlation can be performed using one of several techniques, including but not limited to a statistical model that utilizes statistical correlation and co-variance analysis. A correlation score is determined for a particular set of KPI data and the network infrastructure data. If the correlation score satisfies a threshold, the correlation is determined to exist. Based on detected correlations, the present system can automatically predict future infrastructure requirements based on forecast KPI data, as well as the corresponding costs of the predicted infrastructure requirements. The data, correlations, and predictions can be reported to a user, for example through a dashboard with graphical, metric, and other elements.
Network 150 may include one or more private networks, public networks, intranets, the Internet, an intranet, wide-area networks, local area networks, cellular networks, radio-frequency networks, Wi-Fi networks, any other network which may be used to transmit data, and any combination of these networks.
Continuous monitoring system 160 may continuously detect service performance, events, trends, and so forth, in the performance or behavior of one or more applications within environment 110 in real-time, for example before or after a software update is delivered to an application. To detect service regression, monitoring system 160 may monitor the applications either directly through delegates installed on the applications themselves, such as delegates 122, 132, and 142, or by access to real-time streaming monitoring data (including metrics or other data) provided by application program monitoring system 140.
A delegate may include an agent or other code that is installed to an application or system (e.g., host) and can communicate with remote systems and applications such as continuous monitoring system 160. Each delegate may receive instructions and tasks from monitoring system 160, retrieve information and transmit the information periodically or based on other events to monitoring system 160, may install new code or update code on an application or system, and perform other tasks and operations. In some instances, delegates may be installed on an application program monitoring system, such as a monitoring system provided by AppDynamics, Inc., of San Francisco Calif., to retrieve and transmit a stream of application performance metrics to delivery monitoring system 160. In some instances, delegates may be provided on one or more servers of an environment 120, such as servers hosting application 130 and application 136, to monitor applications and servers that include new code and those that did not host any new code (e.g., control servers).
Continuous monitoring system 160 may provide continuous monitoring and continuous and customized correlation for a system. The continuous monitoring system can determine if there are any immediate or near-term issues, such as performance regression, and may provide reports and alerts based on the determination(s). The continuous monitoring system 160 may include a manager that manages tasks associated with the monitoring, utilization modules, clustering modules, a data store and other functionality. More details for a continuous monitoring system are discussed with respect to
As monitoring system 160 provides continuous delivery and monitoring of new code, it may provide updates, alerts, notifications, and other information through one or more user interfaces to a user 175 via client device 170. The continuous monitoring system 160 can receive application data from delegates 122, 132, and 142 within cloud service providers to obtain event data, as well as communicate with cloud computing service provider applications (e.g., application program interfaces, or APIs) to obtain pricing data. The event data and pricing data can be used with business service data to correlate infrastructure usage and KPI data. Continuous monitoring system is discussed in more detail with respect to
Client device 170 may be implemented as any computer that can receive and provide reports, alerts, and other data, such as for example through a user interface or dashboard, via a network browser, mobile application, or other application on a mobile device, smart phone, tablet, or any other computing machine. Reporting the status and results of continuous delivery service monitoring and automatically correlating network infrastructure resource usage with KPI data is discussed in more detail herein.
Batch processor 230 may receive event data and other data related to executing environments from event server 210. Batch processor may correlate and aggregate data and provide the data to environment data store 240 and time series database 250. Batch processor 250 is described in more detail with respect to
The cloud computing service providers each have their own pricing information and schedules. The pricing information relates to costs for products and services provided by each particular cloud computing service. Pricing server 220 may retrieve costs for the products provided by each cloud computing service and provide the data to batch processor 230.
Environment data store may include a list of environments associated with business services implemented by a client. The environments may include development operations, engineering, finance, and other environments. The environments and identification information for each environment is provided to batch processor 230, which can correlate the environments to a subset of the events received from event server 210. Batch processor 230 generates timeseries data for each combination of environments and infrastructure usage and provides timeseries data to timeseries data base 250.
API server 250 receives client business environments data from environment datastore 240 and the correlated data from timeseries database 250, and provides interface data to client device 170. The data may be provided to client device 170 as a content page, dashboard, or in some other format for display or other communication to a client.
Batch processor 230 includes event processing module 310, infrastructure cost module 320, key performance correlation module 330, and optimization module 340. Event processing module 310 receives event data from event server 210 and generates events from the event data. The events can be used to determine usage of infrastructure resources within particular environments. For example, the events may include a start time and stop time of usage for a virtual machine for a particular business service.
In some instances, different events may be detected for different cloud service providers, such as Kubernetes, Amazon Web Service, Microsoft Azure, Google Cloud Platform, and other services. For example, delegates may detect events such as the creation of a new deployment, a deployment autoscaling, pod container image update, service deployment, toggle of a feature flag, and other events. In some instances, for an Amazon Web Service, one or more delegates can detect events including but not limited to CloudWatch events (e.g., augmented AI events, auto-scaling events, batch events, and so forth), EC2 auto scaling events, EBS events, Config Events, EC2 state change events, and other events.
Infrastructure cost module 320 may receive pricing data from pricing server 220. The pricing data may be used to determine the cost of using a particular infrastructure resource for a cloud service provider. For example, pricing data can include an hourly cost of a virtual machine for a particular cloud service provider. In some instances, pricing data is retrieved for each service provider for which services or resources are used.
Key performance correlation module 330 may determine correlations between key performance indicators and resource event usage. For example, key performance correlation module may receive a list or other identification of key performance indicators, and receive data regarding those key performance indicators, for example through one or more delegates or other tracking or detection code. The key performance indicator data may be received periodically, such as for example every second, five seconds, 10 seconds, 30 seconds, one minute, or some other time period, or can be received based on the occurrence of an event, such as a detection of a new deployment. The key performance indicator data is received as time series data.
Key performance correlation module 330 can use algorithms and and/or other tools to correlate the KPI time series data with time series infrastructure resource data. The correlation can be performed in a variety of ways, including for example a statistical model analysis. A score may be generated by the KPI module 330 to determine the strength of the correlation. If the correlation score satisfies a threshold, the particular KPI and structure data can be considered closely correlated.
Optimization module 340 may process snapshots and trends of correlated infrastructure resources and KPI data to identify how to optimize use of a particular environment at the current time or in the future. For example, if a KPI forecast indicates a KPI will increase, the increase in the correlated infrastructure resources can be predicted. The actual available infrastructure resources can be compared to the predicted infrastructure resources required, and an alert can be generated if the actual infrastructure resources will not satisfy the predicted infrastructure resources.
Data is collected by the delegates installed in the customer environments at step 420. The delegates may be installed in a plurality of environments, for example within one or more cloud computing services. More detail for step 420 is discussed with respect to the method of
Collected event data is transmitted by delegates to an event server at step 430. The collected data is then received by the event server at step 440. Infrastructure cost data is received from one or more cloud providers by pricing server 220 at step 450. Infrastructure cost data may include pricing information for services subscribed to or purchased by the customer from a particular cloud provider.
Key performance indicator data is collected over a period of time at step 460. The KPI data may include one or more metrics, events, or other measurable elements that a company uses to determine how well a company is doing or progress towards an intended result. A KPI can create an analytical basis for decision-making and help focus attention on what matters to a particular company. In some instances, a KPI depends on how a company tracks use of its system by the customer. Examples of KPIs for a software as a service (SAS) company may include number of deployments, number of active users, pipeline minutes for deployments, time duration of a login, and other metrics. In some instances, other examples of a KPI may include a number of customers, how and when to use the software, deployments or verifications, and the number of infrastructures set up in an environment. In some instances, the present system may provide a number of KPIs by default. In some instances, one or more KPIs may be provided by a customer to the present system, and can be measured and tracked on an ongoing basis by the present system.
For each KPI, data may be collected over a period of time. The data may be collected periodically, such as every five seconds, 15 seconds, 30 seconds, minutes, two minutes, 10 minutes, hour, or some other period of time. In some instances, data may be collected in response to an event, such as the start time and end time of a login transaction. The collected data may be stored as a timeseries data.
KPI data is associated with infrastructure resource data at step 470. To associate the KPI data to infrastructure resource data, a correlation analysis may be performed between infrastructure resource usage and a KPI data over a period of time. Correlation may involve a statistical model or some other correlation process. Associating KPI data with infrastructure resource data is discussed in more detail with respect to the method of
Resource usage is optimized based on forecasted KPI data at step 480. In some instances, businesses may be aware of predicted customer traction or utilization of their services. This prediction may be in the form of a forecast of key performance indicators. Infrastructure resource usage may be determined and optimized based on the forecast KPI data. Optimization may include determining costs for the infrastructure usage associated with the forecast KPI data. Additionally, resource usage can be optimized based on predicted KPI trends. More details for optimizing resource usage based on forecast KPI data is discussed with respect to the method of
Correlation data may be reported at step 490. The correlation data may indicate the current KPI data and corresponding infrastructure resource data correlated to the KPI data. In some instances, were KPI data is received as incoming timeseries data or the data stream, the KPI data may vary as additional data comes in. In this instance, the corresponding infrastructure usage may also vary according to the varying KPI data that it is correlated with. The reporting of correlation data may be done through a dashboard, user interface, alerts, electronic messages, when some other format.
A determination is made as to whether the correlation for the selected KPI satisfies a threshold at step 620. In some instances, the correlation score may be compared to a threshold score to determine if a correlation is strong enough to consider reliable. For example, if the correlation score is higher than a threshold of 0.5, 0.6, or 0.7, the correlation may be determined to satisfy the threshold. The threshold may be set to any value based on system designer or administrator preference. If the correlation does not satisfy the threshold at step 620, the KPI is added to a list of uncorrelated KPIs at step 640. In some instances, the KPI may be analyzed with subsequent infrastructure usage data to determine if it correlates with later data. If the correlation does satisfy the threshold at step 620, the correlation and trend data between the KPI and infrastructure usage is stored at step 630. The stored data may later be used to report to a user and optimize subsequent infrastructure usage for forecasted KPI data.
Infrastructure usage may be extrapolated from the forecasted KPI data at step 740. When infrastructure usage is correlated to forecasted KPI data, the infrastructure usage corresponding to the forecast KPI data can be extrapolated for the correlation.
Costs for the extrapolated infrastructure usage may be determined at step 750. The costs may include financial costs, processing costs, memory costs, or some other associated costs of infrastructure. A determination is made as to whether an increase in extrapolated resource usage is greater than a threshold at step 760. The threshold may be set by an administrator, be tied to actual resources, or set based on other criteria. If the increase in the extrapolated resource usage is greater than a threshold, such as for example greater than a rate of increase, greater than the current resources available, or some other threshold of resource usage, an alert is triggered at step 770. The alert may indicate to an administrator that additional resources need to be acquired or activated in order to meet the extrapolated resource usage, or that the expected rate of increase or additional infrastructure may require additional components. The method then ends at step 780. If the extrapolated resource usage is not greater than the threshold, the method ends at step 780.
The components shown in
Mass storage device 830, which may be implemented with a magnetic disk drive, an optical disk drive, a flash drive, or other device, is a non-volatile storage device for storing data and instructions for use by processor unit 810. Mass storage device 830 can store the system software for implementing embodiments of the present invention for purposes of loading that software into main memory 820.
Portable storage device 840 operates in conjunction with a portable non-volatile storage medium, such as a floppy disk, compact disk or Digital video disc, USB drive, memory card or stick, or other portable or removable memory, to input and output data and code to and from the computer system 800 of
Input devices 860 provide a portion of a user interface. Input devices 860 may include an alpha-numeric keypad, such as a keyboard, for inputting alpha-numeric and other information, a pointing device such as a mouse, a trackball, stylus, cursor direction keys, microphone, touch-screen, accelerometer, and other input devices. Additionally, the system 800 as shown in
Display system 870 may include a liquid crystal display (LCD) or other suitable display device. Display system 870 receives textual and graphical information and processes the information for output to the display device. Display system 870 may also receive input as a touch-screen.
Peripherals 880 may include any type of computer support device to add additional functionality to the computer system. For example, peripheral device(s) 880 may include a modem or a router, printer, and other device.
The system of 800 may also include, in some implementations, antennas, radio transmitters and radio receivers 890. The antennas and radios may be implemented in devices such as smart phones, tablets, and other devices that may communicate wirelessly. The one or more antennas may operate at one or more radio frequencies suitable to send and receive data over cellular networks, Wi-Fi networks, commercial device networks such as a Bluetooth device, and other radio frequency networks. The devices may include one or more radio transmitters and receivers for processing signals sent and received using the antennas.
The components contained in the computer system 800 of
The foregoing detailed description of the technology herein has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments were chosen to best explain the principles of the technology and its practical application to thereby enable others skilled in the art to best utilize the technology in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the technology be defined by the claims appended hereto.
The present application is a continuation-in-part of U.S. patent application Ser. No. 16/831,811, filed on Mar. 27, 2020, titled “AUTOMATICALLY ALLOCATING NETWORK INFRASTRUCTURE RESOURCE COSTS WITH BUSINESS SERVICES,” the disclosure of which is incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
Parent | 16831811 | Mar 2020 | US |
Child | 16862507 | US |