Patent Application
20220239730
Techniques have been developed to detect anomalies in monolithic applications as a way to alert the user that malware might be present. Such techniques are not readily transportable to distributed applications, such as those deployed onto a Kubernetes® platform, which run as ephemeral containers within pods on different nodes of the Kubernetes platform. Detection of anomalies in distributed applications is quite different from detecting anomalies in monolithic applications for several reasons.
The distributed application is composed of several parts, each designed to execute a particular workload and thus behave differently from other parts. There is a need to understand the normal behavior of each part in order to detect whether a certain behavior is anomalous or not. By contrast, the monolithic application is a single unit of software and anomaly is detected when the software as a whole deviates from its expected behavior.
In distributed applications deployed onto a Kubernetes platform, data is distributed between pods and there is data traffic in and out of each pod. By contrast, data is maintained in a single place in a monolithic application and the only data traffic is data traffic in and out of the monolithic application. As a result, anomalous behavior in distributed applications may go undetected using anomalous detection techniques developed for monolithic applications which evaluate only the data traffic in and out of the applications for anomalous behavior.
Further, in distributed applications deployed onto a Kubernetes platform, pods in some sets, e.g., replica sets, are expected to behave similarly on average. In such situations, one of the pods in a set could behave anomalously or all of the pods in the set could behave anomalously. Anomalous detection techniques developed for monolithic applications may be unable to detect anomalous behavior when one of the pods in the set behave differently from other pods in the set.
One or more embodiments provide techniques for detecting anomalies in a distributed application. In particular, techniques described herein are able to detect anomalous behavior in a distributed application deployed onto a Kubernetes platform. The anomalous behavior may be detected in a set of pods executing one of the workloads of the distributed application or it may be detected in one of the pods in the set, based on a comparison with their respective past normal behavior. In addition, the anomaly may be detected in a pod based on a comparison with the behavior of other pods in the set.
Client cluster 100 is a Kubernetes cluster and to simplify the description, only pods 151 executing activity monitor 150, pods 161-163 executing Workload A, and pods 171-172 executing Workload B, all running in the Kubernetes cluster are shown. As is known in the art, pods run on nodes of the Kubernetes cluster, and one or more containers run inside pods. In addition, for illustrative purposes, Workload A and Workload B are depicted as workloads of the distributed application. In general, a distributed application may have any number of workloads.
In some embodiments, pods 161-163 form a replica set for executing Workload A and pods 171-172 form a replica set for executing Workload B. Pods of the same replica set are expected to behave similarly on average and are typically executed in different nodes. The idea behind creating a replica set with pods distributed across multiple nodes is to split the load of the of the workload computation among the multiple nodes. Therefore, in the embodiments where pods 161-163 form a replica set, each of pods 161-163 runs on different nodes of the Kubernetes cluster. Similarly, in the embodiments where pods 171-172 form a replica set, each of pods 171-172 runs on different nodes of the Kubernetes cluster.
One or more pods 151 of activity monitor 150 monitor data traffic between the pods, egress data traffic of the pods, and ingress data traffic of the pods, and collect activity data from the monitored data traffic, e.g., from packet headers of data packets that are transmitted or received by the pods. The collected activity data includes: port information (e.g., port number), protocol information (e.g., TCP), sender information (e.g., workload name for internal Kubernetes traffic or IP address for external non-Kubernetes traffic), and receiver information (e.g., workload name for internal, Kubernetes traffic or IP address for external non-Kubernetes traffic), of the data traffic into and out of the pods. Another one of pods 151 of activity monitor 150 then transmits the collected activity data to anomaly detection system 200, along with metadata that identifies the pod and the workload associated with the data traffic and whether the data traffic is egress data traffic, ingress data traffic, or inter-workload data traffic.
Anomaly detection system 200 includes databases and processes that are executed in one or more computing devices to perform anomaly detection according to embodiments. The databases include an activity data database 210, a model database 230, and an alerts database 250. The processes include an anomaly model creator 220, an anomaly tester 240, and an alerts service 260.
Anomaly detection system 200 may be provisioned in a public or private cloud computing environment, and even in the same data center in which client cluster 100 is provisioned. As such, the scope of embodiments is not limited to a particular computing environment of anomaly detection system 200. In addition, anomaly detection system 200 may be operated by a third party and connected to a plurality of client clusters of different organizations to whom the third party is providing anomaly detection services.
Activity data database 210 receives activity data from activity monitor 150 over a network 180 and stores the activity data in a structured manner for use by anomaly model creator 220. Anomaly model creator 220 examines the activity data stored by activity data database 210, and compiles the statistics in Table 1 from the activity data for each pod and each workload.
During a time period designated by the administrator of client cluster 100 or during any time period where client cluster 100 is expected to exhibit normal behavior, the compiled statistics are stored in a structured manner by model database 230 as a reference model. The administrator of client cluster 100 has the option of instructing anomaly detection system 200 to delete a reference model, and designate another time period for collecting the activity data for creating the reference model. The instruction to delete the reference model may be given, for example, when the administrator of client cluster 100 detects malicious or unusual activity in client cluster 100 (e.g., by applying techniques known in the art) during the time period in which activity data used in creating the reference model is being collected. After the reference model has been created and maintained (not deleted), anomaly detection system 200 performs anomaly detection for client cluster 100 by performing the following steps.
First, activity data database 210 stores the activity data received from activity monitor 150 over network 180 in a structured manner. Second, anomaly model creator 220 examines the activity data stored by activity data database 210, and compiles the statistics in Table 1 from the activity data for each pod and each workload. Third, anomaly tester 240 compares the statistics compiled for each workload and each pod against the reference model and determines whether there are deviations from the normal behavior represented by the reference model in any of the pods or any of workloads that need to be flagged as anomalies. Alerts database 250 stores the flagged anomalies in a structured manner, and alerts service 260 issues alerts indicating the anomalies to client cluster 100 according to preferences set by the administrator of client system 200.
In the embodiments, anomaly tester 240 flags each of the behaviors listed in Table 2 of a pod or a workload, as an anomaly:
The threshold percentage for flagging a behavior listed in Table 2 as an anomaly is configurable, e.g., according to a preference of the administrator of client computer 100 or by anomaly detection system 200. In addition, different threshold percentages may be set for different behaviors listed in Table 2.
Over time, the configuration of the distributed application may be modified so that a different number of pods execute the workloads of the distributed application. The configuration of the distributed application that is modified from that of
In some embodiments, in addition to or alternative to comparing the statistics compiled for each pod against the reference model, anomaly tester 240 compares the statistics compiled for a pod based on activity data collected during a current anomaly detection period against the statistics compiled for other pods in the same replica set as the pod based on activity data collected during the current anomaly detection period. Any behavior listed in Table 3 that is observed in the pod but not in the other pods is flagged as an anomaly. Alerts database 250 stores the flagged anomalies in a structured manner, and alerts service 260 issues alerts indicating the anomalies to client cluster 100 according to preferences set by the administrator of client system 200. Because the comparisons in these embodiments are made against statistics that are compiled based on activity data collected during the same anomaly detection period, they may be used as the basis for detecting anomalies even after modifications are made to the configuration of the distributed application.
The threshold percentage for flagging a behavior listed in Table 3 as an anomaly is configurable, e.g., according to a preference of the administrator of client computer 100 or by anomaly detection system 200. In addition, different threshold percentages may be set for different behaviors listed in Table 3.
If the received activity data is egress data (step 316, Yes), activity data database 210 at step 318 records in association with each of the pod ID and the workload ID, whether or not the data is transmitted to a public network (e.g., to public IP addresses) or to a private network (e.g., to private IP addresses). Then, activity data database 210 at step 319 records in association with each of the pod ID and the workload ID, the port, protocol, and destination information that are in the collected activity data.
If the received activity data is ingress data (step 316, No; step 320, Yes), activity data database 210 at step 322 records in association with each of the pod ID and the workload ID, whether or not the data is received from a public network (e.g., from public IP addresses) or from a private network (e.g., from private IP addresses).
If the received activity data is neither egress data nor ingress data (step 316, No; step 320; No), activity data 210 determines that the activity data is associated with inter-workload traffic and step 324 is executed for the inter-workload data. Step 324 is also executed for egress data (after step 319) and for ingress data (after step 322). The message rates and error rates that are computed and recorded by activity data database 210 include all of the message and error rates listed in Table 3. The process ends after step 324.
The method of
If the reference model is not valid for comparison because the distributed application has gone through a change in its configuration since the last update of the reference model (step 418, Yes), anomaly tester 240 skips step 420 and executes step 422. At step 422, anomaly tester 240 determines if an anomaly is detected by the comparison made at step 416 or step 420. If so (step 422, Yes), anomaly tester 240 at step 424 notifies alerts database 250 of the detected anomalies, in response to which alerts database 250 stores the detected anomalies in a structured manner, and alerts service 260 issues alerts indicating the anomalies to client cluster 100 according to preferences set by the administrator of client system 200. The process terminates after step 424 or if no anomaly is detected by the comparison made at step 416 or step 420 (step 422, No).
If the reference model is not valid for comparison because the configuration of the distributed application has been modified since the last update of the reference model, anomaly model creator 220 updates the reference model based on activity data generated after the configuration of the distributed application has been modified. After the update, anomaly tester 240 performs both the comparison of step 416 and the comparison of step 420.
The embodiments described herein may employ various computer-implemented operations involving data stored in computer systems. For example, these operations may require physical manipulation of physical quantities. Usually, though not necessarily, these quantities may take the form of electrical or magnetic signals, where the quantities or representations of the quantities can be stored, transferred, combined, compared, or otherwise manipulated. Such manipulations are often referred to in terms such as producing, identifying, determining, or comparing. Any operations described herein that form part of one or more embodiments may be useful machine operations.
One or more embodiments of the invention also relate to a device or an apparatus for performing these operations. The apparatus may be specially constructed for required purposes, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. Various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
The embodiments described herein may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc.
One or more embodiments of the present invention may be implemented as one or more computer programs or as one or more computer program modules embodied in computer readable media. The term computer readable medium refers to any data storage device that can store data which can thereafter be input to a computer system. Computer readable media may be based on any existing or subsequently developed technology that embodies computer programs in a manner that enables a computer to read the programs. Examples of computer readable media are hard drives, NAS systems, read-only memory (ROM), RAM, compact disks (CDs), digital versatile disks (DVDs), magnetic tapes, and other optical and non-optical data storage devices. A computer readable medium can also be distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
Although one or more embodiments of the present invention have been described in some detail for clarity of understanding, certain changes may be made within the scope of the claims. Accordingly, the described embodiments are to be considered as illustrative and not restrictive, and the scope of the claims is not to be limited to details given herein but may be modified within the scope and equivalents of the claims. In the claims, elements and/or steps do not imply any particular order of operation unless explicitly stated in the claims.
Virtualization systems in accordance with the various embodiments may be implemented as hosted embodiments, non-hosted embodiments, or as embodiments that blur distinctions between the two. Furthermore, various virtualization operations may be wholly or partially implemented in hardware. For example, a hardware implementation may employ a look-up table for modification of storage access requests to secure non-disk data.
Many variations, additions, and improvements are possible, regardless of the degree of virtualization. The virtualization software can therefore include components of a host, console, or guest OS that perform virtualization functions.
Plural instances may be provided for components, operations, or structures described herein as a single instance. Boundaries between components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention. In general, structures and functionalities presented as separate components in exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionalities presented as a single component may be implemented as separate components. These and other variations, additions, and improvements may fall within the scope of the appended claims.
This application is a divisional of U.S. patent application Ser. No. 17/033,520, filed Sep. 25, 2020, which application is incorporated by reference herein in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17033520 | Sep 2020 | US |
| Child | 17724270 | US |
