Distributed analytics engines are network visualization tools for displaying security and policy data for workloads and the connections between them in network datacenters. Such engines create large scale graphs in a user interface (UI) for visualizing the network and security posture of private and/or public datacenter networks (e.g., logical and/or physical networks). A network admin or security admin can leverage these engines to gain powerful insight into the workloads of a logical and/or physical network operating in one or more datacenters, allowing them to better protect the logical network. However, when a particular network instantiates many workloads (e.g., utilizing tens of thousands of VMs, bare-metal servers, and other types of compute resources), visualizing even a subset of these vertices (workloads that communicate with other workloads) and their flows can be quite confusing to the admins. The previous art does not provide a good way of visualizing the graphs of the UI, in a way that is scalable to tens of thousands of workloads, without significant or even overwhelming manual work for the admins.
In the existing art, admins can define their own groups of workloads and apply filters in order to enable more effective visualization. However, if the admin wants to view interactions between many individual computes, or if the admin has not defined a robust set of groups, then visualization at scale becomes an incomprehensible jumble of unrelated workloads and their connections rather than a useful visual reference. Therefore, there is a need in the art for a machine learning technique to automatically define groups of workloads into related clusters.
Some embodiments provide a mechanism to automatically group workloads of a logical and/or physical network into clusters of related workloads. The method of some embodiments displays consolidated workload data for a logical and/or physical network. The method, for each of multiple workloads: (1) receives a set of identifiers characterizing the workload; and (2) converts the set of identifiers to a vector representation of the workload. The method then identifies clusters of workloads based on the vector representations of the workloads. The method then displays the workloads grouped in the identified clusters and displays data flows between the clusters of workloads. Converting the set of identifiers to a vector representation of the workload may include applying a similarity metric to the set of identifiers.
The identifiers characterizing the workload may include a compute name of the workload and/or a set of identifying metadata (e.g., tags, labels, comments, annotations and/or other user provided descriptive values, etc.) of the workload. The similarity metrics used in some embodiments may include Jaro similarity metrics and/or Jaccard similarity metrics. Displaying data flows between the clusters of workloads includes, displaying data flows between a first workflow in a first cluster and a second workflow in a second cluster in some embodiments.
The method of some embodiments, identifies clusters of workloads, based on the vector representations of the workloads, by creating a matrix of the vector representations of the workloads in the workloads. Identifying clusters of workloads based on the vector representations of the workloads may also include reducing a dimensionality of the vectors in the matrix using principal component analysis (PCA). Identifying clusters of workloads based on the vector representations of the workloads may also include applying a clustering algorithm to the matrix. The clustering algorithm, in some embodiments, is a hierarchical density based spatial clustering of applications with noise (HDBSCAN) algorithm.
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, the Detailed Description, the Drawings, and the Claims is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, the Detailed Description, and the Drawings.
The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Some embodiments provide a mechanism to automatically group workloads of a logical and/or physical network into clusters of related workloads. The method of some embodiments displays consolidated workload data for a logical and/or physical network. The method, for each of multiple workloads: (1) receives a set of identifiers characterizing the workload; and (2) converts the set of identifiers to a vector representation of the workload. The method then identifies clusters of workloads based on the vector representations of the workloads. The method then displays the workloads grouped in the identified clusters and displays data flows between the clusters of workloads. Converting the set of identifiers to a vector representation of the workload may include applying a similarity metric to the set of identifiers.
The identifiers characterizing the workload may include a compute name of the workload and/or a set of identifying metadata of the workload. The similarity metrics used in some embodiments may include Jaro similarity metrics and/or Jaccard similarity metrics. Displaying data flows between the clusters of workloads includes displaying data flows between a first workflow in a first cluster and a second workflow in a second cluster in some embodiments.
The method of some embodiments identifies clusters of workloads, based on the vector representations of the workloads, by creating a matrix of the vector representations of the workloads in the workloads. Identifying clusters of workloads based on the vector representations of the workloads may also include reducing a dimensionality of the vectors in the matrix using principal component analysis (PCA). Identifying clusters of workloads based on the vector representations of the workloads may also include applying a clustering algorithm to the matrix. The clustering algorithm, in some embodiments, is a hierarchical density based spatial clustering of applications with noise (HDBSCAN) algorithm.
The connections between the workloads within clusters and between clusters are displayed according to the line patterns shown in the legend 140. Legend 140 includes patterns for unprotected connections, such as the connections between workloads within clusters 110 and 120, allowed connections, such as the connections between workloads of cluster 110 and cluster 130, and blocked connections, such as the connection between workload Win Baremetal-1 and Catalog-4. In the embodiment of
In some embodiments, the Security display interface 100 provides options to modify characteristics of workloads, resources available to workloads, number of simultaneous instances of particular types of workload, etc. Such options may be provided in some embodiments by a menu (e.g., a pop-up menu) activated by operation of a user interface device in relation to the workloads (e.g., a mouse button click/double-click, a touch screen selection, etc.). However, in other embodiments, other controls for such modifications are provided. Similarly, the Security display interface 100 of some embodiments includes options to modify the characteristics of connections between workloads. As with the workload modification options, these may be provided by a menu accessed with a selection by a user interface device or by some other controls. For example, in some embodiments, the Security display interface 100 of some embodiments includes an option to change the status of a displayed set of connections from blocked to unblocked or vice versa, with the interface creating or modifying underlying rules (e.g., firewall rules) to implement the change in status. Similarly, in some embodiments the number of connections between clusters can be limited to a maximum or minimum number of separate connections, limited to a maximum or minimum bandwidth, allocated reserved bandwidth, have additional or reduced security features applied, etc.
The set of workloads of a network are generally not static, but may have new workloads added or old workloads deleted or modified (e.g., a workload may be renamed or the workload identifying metadata may change). Therefore, the data collector 205 receives new workload data as the workload data changes. The data collector 205 sends the workload data to a manager 210.
In the illustrated embodiment, the manager 210 acts as a central control and middle box for data being passed to various modules 215-225 that store and analyze the workload data and also to the interface module 230 for the workload display. However, one of ordinary skill in the art will understand that in some embodiments, modules 215-225 pass data directly between them. The manager 210 sends the workload data (workload IDs, names, identifying metadata, etc.) to the analysis database 215. Additionally, in some embodiments, the analysis database 215 stores derived data generated by other modules (e.g., the vectors, generated from vector generator 220, which are analyzed to generate the clusters, and the clusters themselves produced by the cluster analyzer 225).
The manager 210 sends the workload data to a vector generator module 220, which converts the workload data into vectors and calculates a similarity score for each vector (See, e.g.,
The analysis database 215 thus has the data necessary to generate a display. An administrator activates the interface module 230 (e.g., through a GUI that can access the workload clustering application) to call up the display. The interface module 230 retrieves the workload and cluster data from the manager 210 (or in some embodiments, directly from the analysis database 215). The interface module sends the workload and cluster data to a current view module 235, which provides the current view to a display generator 240. The administrator is then able to adjust the view by interacting with a GUI that activates the interface module 230. For example, as shown in
The process 300 of
In the present practice of operating networks, the administrators who set up the workloads of the networks typically provide meaningful names for the workloads. For example, workloads handling the database of a catalog of goods and/or services might all be named “Catalog-DB-n”, where n is a number or other designation identifying a specific catalog database handling workload. Giving the workloads obviously meaningful names simplifies processes of troubleshooting and analysis of the network, which is why it is a common practice. However, even in cases where the names of workloads are not obviously meaningful, network administrators usually apply some non-arbitrary naming convention to the workload names (e.g., all catalog databases workload names might start with the same arbitrary combination of symbols that are distinct from other types of workload).
For networks where the workload names are not arbitrary, some embodiments of the present invention, group workloads into cluster by workload names. In some embodiment, a user is presented with an option of whether to group workloads into clusters based on workload names, in other embodiments, the processes of the invention automatically determine whether workload names are non-arbitrary.
m: number of matching characters between n0 and n1
|n0| and |n1|: length of compute names
t: half the number of transpositions
example: n0=hello, n1=yellow
However, other embodiments calculate name similarity values using other formulas.
The process 400 then generates (at 415) a similarity score for each workload based on the workload names. In some embodiments, generating the similarity score for a particular workload may include storing the calculated similarity scores for each name similarity calculation between the particular workload and each of the other workloads (e.g., as a vector associated with the particular workload). In other embodiments, generating the similarity score for each workload further includes some further mathematical, data aggregation, or data organizing of the name similarity metric data produced in operation 410. After generating the similarity scores for each workload, the process 400 then ends.
In some network settings, instead of or in addition to implementing meaningful/non-arbitrary names for workloads, the administrators supply identifying metadata (e.g., tags, labels, comments, annotations and/or other user provided descriptive values etc.) for each of multiple workloads. For example, a set of identifying metadata could include one item of identifying metadata that identifies a workload as applying to a DB, another item of identifying metadata that identifies it as applying to the products database, another item of identifying metadata identifying what application the workload is implementing, another item of identifying metadata that identifies what type of connections the workload needs, etc.
If the process 500 determines (at 510) the identifying metadata should be parsed and cleaned, then the process 500 parses and cleans (at 515) the identifying metadata to remove extraneous metadata items and/or extraneous parts of identifying metadata items. For example, to parse the identifying metadata items, the process 500 may identify distinct words appearing in an identifying metadata item, such as by identifying the words “database,” “merchandise,” and “food” in an identifying metadata item that includes a single string “database_merchandise_food”. Such parsing, in some embodiments, results in grouping into a cluster of workloads where not all workloads have the same exact identifying metadata item.
Operation 515 also cleans the identifying metadata of extraneous items (items that are less helpful or unhelpful in identifying useful clusters). Some embodiments have a minimum cluster size for the number of workloads in each cluster. The minimum cluster size may be set by a user or derived automatically by the automated clustering method. In such embodiments, identifying metadata items appearing in fewer workloads than the minimum cluster size may be cleaned from the identifying metadata of the workloads. For example, if the minimum cluster size is five workloads, and only two workloads are associated with a particular identifying metadata item, then that identifying metadata item might be cleaned from the metadata items to be used for determining how to cluster the workloads.
Even when a particular identifying metadata item is common, it may be removed in the cleaning operation as not being relevant enough to identifying related workload groups. For example, some workloads may be associated with an identifying metadata item that identifies the date or time that the workload was implemented. Several otherwise unrelated workloads might happen to have the same date or time. In such a case, an identifying metadata item specifying a particular date may be (1) more common than the number of workloads in the minimum cluster size, but (2) not helpful in separating the workloads into relevant groups. Therefore, in some embodiments, identifying metadata items specifying dates or times or similarly low or no relevance data are removed. Finally, an identifying metadata item may include both relevant and less relevant information (e.g., “database_merchandise_food_06_06_2021” that includes both a description of the workload and a date). The parsing and cleansing of the identifying metadata item, in some embodiments, may remove the less relevant information (here, the date) and leave the relevant information “database_merchandise_food” to be evaluated in later operations of process 500.
One of ordinary skill in the art will understand that in some embodiments, these identifying metadata items are not removed from association with the workloads themselves, only from the set of received identifiers used to identify clusters of related workloads. After parsing and cleaning the identifying metadata, or if the process 500 determined (at 510) that identifying metadata did not need to be parsed and cleaned, the process 500 performs (at 520) an identifying metadata similarity metric on the identifying metadata of the workloads. In some embodiment, the name similarity metric is a Jaccard similarity metric. In some embodiments, the Jaccard string similarity metric and/or some other similarity metric is applied to every set of workload identifying metadata in order to compare the similarity of each set of workload identifying metadata to each other set of workload identifying metadata.
Some embodiments use the following formula (e.g., comparing each workload name to the names of every other workload) to compute an identifying metadata similarity value.
T0: set of workload0's identifying metadata items
T1: set of workload1's identifying metadata items
example:
workload0 identifying metadata items {database, merchandise, shoes}
workload1 identifying metadata items {database, merchandise, food}
The process 500 then generates (at 525) a similarity score for each workload based on the workload identifying metadata. In some embodiments, generating the similarity score for a particular workload may include storing the calculated similarity scores for each identifying metadata similarity calculation between the particular workload and each of the other workloads (e.g., as a vector associated with the particular workload). In other embodiments, generating the similarity score for each workload further includes some further mathematical, data aggregation, or data organizing of the identifying metadata similarity metric data performed in operation 520. After generating the similarity scores for each workload, the process 500 then ends.
Although process 400 of
The process 600 performs (at 615) a name similarity metric on the names of the workloads (e.g., generating a similarity value between the name of each workload and the names of every other workload). In some embodiments, this is performed as described with respect to operation 410 of
The process 600, of
sn=workload name similarity (See, e.g., eq. 1)
st=workload identifying metadata similarity (See, e.g., eq. 3)
In some embodiments, generating the similarity score for a particular workload may include storing the calculated similarity scores for each combined similarity calculation between the particular workload and each of the other workloads (e.g., as a vector associated with the particular workload). In other embodiments, generating the similarity score for each workload further includes some further mathematical, data aggregation, or data organizing of the combined similarity metric data produced in operation 625. After generating the similarity scores for each workload, the process 600 then ends.
Once the vector creation has been performed and the similarity scores have been generated, some embodiments then identify clusters of the workloads (See, e.g., operation 315 of
The datacenter 800 also includes one or more servers 814 that implement network managers and/or controller to manage and control the service engines 830, VMs 805, and SFEs 812. The hosts and servers communicate with each other through the network. In some embodiments, one or more of the VMs 805 or servers 814 execute the above-described application clustering and visualization processes of some embodiments of the invention. These processes perform the automated cluster analysis described above and generate the user interface through which administrators can visualize the clusters workloads (e.g., as shown in
In some embodiments, a workload may be any of: an application running on a VM 805, an application running directly on a host 802, a container or Pod executing on a VM 805 or on a host 802, or some other element in the network of the datacenter 800. In some embodiments, the workloads connect to logical networks, physical networks, or some workloads on physical networks and some on logical networks. The automatic clustering processes of some embodiments shows workloads of a physical network, logical network, or both. The workloads shown by the automatic clustering application of some embodiments may be implemented by machines at a single physical location or multiple physical locations.
Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as computer-readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer-readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.
In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here, is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.
The bus 905 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system 900. For instance, the bus 905 communicatively connects the processing unit(s) 910 with the read-only memory 930, the system memory 925, and the permanent storage device 935.
From these various memory units, the processing unit(s) 910 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only-memory (ROM) 930 stores static data and instructions that are needed by the processing unit(s) 910 and other modules of the computer system. The permanent storage device 935, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 900 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 935.
Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device 935. Like the permanent storage device 935, the system memory 925 is a read-and-write memory device. However, unlike storage device 935, the system memory 925 is a volatile read-and-write memory, such as random access memory. The system memory 925 stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 925, the permanent storage device 935, and/or the read-only memory 930. From these various memory units, the processing unit(s) 910 retrieve instructions to execute and data to process, in order to execute the processes of some embodiments.
The bus 905 also connects to the input and output devices 940 and 945. The input devices 940 enable the user to communicate information and select commands to the computer system 900. The input devices 940 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 945 display images generated by the computer system 900. The output devices 945 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD), that display images generated by the computer system. Some embodiments include devices such as touchscreens that function as both input and output devices 940 and 945.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage, and memory, that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms “display” or “displaying” mean displaying on an electronic device. As used in this specification, the terms “computer-readable medium,” “computer-readable media,” and “machine-readable medium” are entirely restricted to tangible, physical objects, that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral or transitory signals.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For instance, several of the above-described embodiments deploy gateways in public cloud datacenters. However, in other embodiments, the gateways are deployed in a third-party's private cloud datacenters (e.g., datacenters that the third-party uses to deploy cloud gateways for different entities in order to deploy virtual networks for these entities). Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.
Number | Date | Country | |
---|---|---|---|
63209402 | Jun 2021 | US |