Patent Grant
11196628
The use of containers has changed the way applications are packaged and deployed, with monolithic applications being replaced by microservice-based applications. Here, the application is broken down into multiple, loosely coupled services running in containers, with each service implementing a specific, well-defined part of the application. However, the use of containers also introduces new challenges, in that the fleet of containers need to be managed and all these services and containers need to communicate with each other.
Management of the containers is addressed by container orchestration systems, such as Docker Swarm, Apache Mesos, or Kubernetes, the latter of which has become a de-facto choice for container orchestration. Kubernetes clusters can be run in an on-premises datacenter or in any public cloud (e.g., as a managed service or by bringing-up your own cluster on compute instances). These Kubernetes clusters may include both Linux and Windows hosts, and thus to enable networking between the entities of a cluster it is important to have a single network plugin that can be easily run on any platform and any cloud. It is also highly desirable to be able to configure global security policies across these clusters and provide centralized visibility (e.g., with simple debugging and troubleshooting features).
Some embodiments provide debugging and troubleshooting techniques for a container network interface plugin (e.g., for a Kubernetes cluster). Specifically, some embodiments map networking and/or network policy data to cluster abstractions in order to present this networking and/or network policy data to users (e.g., network administrators, application developers, etc.). This networking and/or network policy data may include flow table entries, ongoing network connections, flow tracing information, etc.
Some embodiments provide this mapping data in the context of a container network interface plugin. In some embodiments, a container cluster (e.g., a Kubernetes cluster) includes numerous nodes (e.g., virtual machines (VMs), physical host servers), each of which executes one or more pods (e.g., lightweight VMs on which parts of an application execute). The container network interface (CNI) plugin of some embodiments includes a set of modules that execute on each node to provide networking and security functionality for the pods. Specifically, some embodiments operate a flow-based managed forwarding element (e.g., Open vSwitch) on each node, to which each of the pods connect. In addition, a CNI agent executes on each node (i) to configure the forwarding element and (ii) to handle troubleshooting requests. In addition, a CNI controller separate from the nodes provides configuration data (e.g., forwarding information, network policy to be enforced) to the agents, which use this configuration data to configure the forwarding elements on their respective nodes. Distributed services (e.g., for aggregating troubleshooting information from multiple nodes) may also execute within the cluster.
In this context, the agent of some embodiments on a particular container host (e.g., a Kubernetes node) is capable of identifying flow entries used by the managed forwarding element of the particular container host related to a particular container cluster concept (e.g., a Kubernetes abstraction, such as a pod or network policy) and mapping elements of these flow entries to different Kubernetes concepts (including those in the request). Flow entries in flow-based forwarding elements include, in some embodiments, (i) a set of match conditions against which data messages are compared and (ii) a set of actions for the forwarding element to perform on data messages that match the match conditions.
When an agent receives a request for flow entries that relate to a particular Kubernetes concept (e.g., to a specific network policy), the agent identifies flow entries realized by the forwarding element executing on its container host that match the request. For example, for specific network policies or network policy rules, flow entries include a specific identifier in one of the match or action fields (e.g., a conjunction identifier, for conjunctive flow entries). Specific pods can be identified by network addresses (or data link addresses) used in flow entries (e.g., as match conditions). For each identified flow entry that matches the request, the agent generates mapping data that maps elements of the flow entry to specific Kubernetes concepts (e.g., pods, network policies, rules, etc.). For instance, matches over table identifiers, network addresses, and other conditions may be indicative of specific network policies and/or network policy rules, pods, nodes, etc. Raw flow entry data may be difficult for a network administrator or application developer to understand, so the generated mapping data is provided along with each flow entry for presentation to the requesting user. In different embodiments, this data is provided to the controller or directly to a user interface (e.g., a command line interface) from which the request was received.
In some embodiments, either the agent or another module operating on each of the container hosts along with the agent (e.g., a connection exporter module), also monitors ongoing connections being processed by the forwarding element. This module retrieves data about these ongoing connections, maps the data to Kubernetes concepts implemented in the cluster, and exports the ongoing connection information along with the mapped Kubernetes concepts. The connection data typically includes at least source and destination network addresses and transport layer ports as well as transport layer protocol, and may also include information about the status of the ongoing connection (e.g., the number of packets sent in either direction, the amount of data sent, any related connections, etc.). In some embodiments, the module maps the source and destination addresses to specific Kubernetes concepts (e.g., pods executing on that container host or other container hosts, distributed services, etc.). In addition, in some embodiments, the module identifies network policies that are relevant to each ongoing connection and exports this information along with the other connection and mapping data.
In some embodiments, the forwarding element uses a separate connection tracker module to store state regarding ongoing connections (because the flow entries used by the forwarding element are otherwise stateless). This connection tracker module stores the connection information (e.g., source and destination network addresses, source and destination transport layer ports, protocol, amounts of packets and data transferred, connection state). In some embodiments, the connection tracker stores this information for both incoming and outgoing directions of a connection. For flow entries (e.g., network policy flow entries) that require connection state information, the forwarding element retrieves this connection state information from the connection tracker module.
The connection exporter module on a host container, in some embodiments, retrieves the ongoing connections from the connection tracker module (e.g., at regular intervals) and maps this data to the Kubernetes concepts as described above. In some embodiments, the connection exporter module exports the connection data to a connection aggregator executing in the Kubernetes cluster (e.g., as a distributed service). This connection data may be exported as, e.g., IPFIX data, with the mapping information stored as metadata (e.g., as IPFIX information elements). In addition, in some embodiments, the connection exporter module also maintains metrics tied to the Kubernetes concepts, such as the number of connections per pod, per pod namespace, or per node, the number of connections to which each different network policy is applied, or the number of connections that meet a specific TCP connection state (e.g., only having the TCP_SYN flag).
The connection aggregator receives connection data (with mapping data for the Kubernetes concepts) from the flow exporter modules on multiple nodes of the cluster. In some embodiments, the connection aggregator also adds additional mapping data for the Kubernetes cluster concepts (e.g., information that is not efficient for the connection exporter module to add). This connection aggregator may also compare and combine data for the same ongoing connections from source and destination nodes. In addition, either the connection aggregator or a separate network visualizer analyzes the aggregated connection data and uses this data to generate a cluster visualization. This cluster visualization may show pods, nodes, services, etc. of the cluster, as well as the connections between these entities.
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description and the Drawings is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description and the Drawing, but rather are to be defined by the appended claims, because the claimed subject matters can be embodied in other specific forms without departing from the spirit of the subject matters.
The novel features of the invention are set forth in the appended claims. However, for purpose of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Some embodiments provide debugging and troubleshooting techniques for a container network interface plugin (e.g., for a Kubernetes cluster). Specifically, some embodiments map networking and/or network policy data to cluster abstractions in order to present this networking and/or network policy data to users (e.g., network administrators, application developers, etc.). This networking and/or network policy data may include flow table entries, ongoing network connections, flow tracing information, etc.
Some embodiments provide this mapping data in the context of a container network interface plugin for a container cluster (e.g., a Kubernetes cluster).
In some embodiments, the nodes 105 are virtual machines or physical host servers that host pods 135, as well as various entities that enable the pods 135 to run on the node 105. As shown, these various entities include a kubelet 140, a CNI plugin 145, a kube-proxy 150, iptables 155, a daemonset 160 which includes a CNI agent 165 and one or more Open vSwitch (OVS) daemons 170, and an OVS bridge 175. The pods, in some embodiments, are lightweight virtual machines (VMs) or other data compute nodes (DCNs) that encapsulate one or more containers. Pods may wrap a single container or a number of related containers (e.g., containers for the same application) that share resources. In some embodiments, each pod 135 includes storage resources for the containers as well as a network address (e.g., an IP address) at which the pod can be reached.
The kubelet 140 is a standard Kubernetes agent that runs on each node in a cluster to manage containers running in the pods 135. Similarly, the kube-proxy 150 is a network proxy that runs on each node in a Kubernetes cluster to maintain network rules and manage the iptables 155 or another datapath implementation. In some embodiments, the iptables 155 are configured by the kube-proxy 150, but not actually used for networking as the OVS daemons 170 and OVS bridge 175 are instead used for handling networking between pods 135.
The kubelet 140 on each node 105 executes the CNI plugin 145 on that node, in some embodiments. In some embodiments, the CNI plugin 145 is a gRPC client that issues remote procedure calls to the CNI agent 165 on the node for each container network interface command, which returns the result of those commands (or an error) to CNI plugin 145.
The daemonset 160 is a type of pod that is instantiated on all of the nodes 105 of the cluster, in this case to execute the CNI agent 165 and the OVS daemons 170. The CNI agent 165 on a particular node is responsible for managing the OVS instantiation on the node (i.e., the OVS daemons 170 and the OVS bridge 175), the operation of which is described below. In some embodiments, the CNI agent 165 executes in a separate container of the daemonset 160 than the OVS daemons 170.
Whenever a new pod 135 is to be created on a node 105, the CNI agent 165 receives instructions from the CNI plugin 145 about the creation of the new pod. The CNI agent 165 creates the pod's network interface (e.g., a veth interface or other virtual ethernet device/adapter), allocates a network address (e.g., IP address) for the interface, and connects the interface to the OVS bridge 175 (e.g., via a corresponding network interface on the OVS bridge 175). The CNI agent 165 also generates and installs the necessary networking flow entries for this network address in one of the OVS daemons 170 (i.e., in ovs-vswitchd, the daemon responsible for processing data messages for which flow entries are not cached in the OVS bridge 175). Flow entries in flow-based forwarding elements such as OVS include, in some embodiments, (i) a set of match conditions against which data packets are compared and (ii) a set of actions for the forwarding element to perform on data packets that match the match conditions.
In some embodiments, the agent 165 also receives updates from the Kubernetes API server 120 regarding the creation of new nodes 105 (either by monitoring for updates at the Kubernetes API server 120, or by the server 120 pushing such updates to all of the agents 165), so that it can create a new tunnel to each remote node in the cluster in OVS (e.g., by creating new flow entries for tunneling to the remote node). In addition, the agent 165 receives updates from the CNI controller 115 when network policies are created, removed, or modified (again, either by monitoring for updates at the CNI controller 115, or by the controller 115 pushing such updates to all of the agents 165 that require the updates). The agent 165 generates and installs new flow entries in OVS to implement these network policies for the local pods 135. Furthermore, as described in greater detail below, the CNI agent 165 handles certain types of troubleshooting and/or monitoring requests (e.g., flow entry realization requests, flow tracing requests, etc.).
OVS, as mentioned, includes the OVS bridge 175 as well as OVS daemons 170. These daemons 170, in some embodiments, include ovsdb-server (for handling configuration of the OVS bridge via database tables) and ovs-vswitchd. The latter daemon, as mentioned, processes data messages for which the OVS bridge 175 does not have flow entries. The OVS bridge 175 on a particular node 105 includes a classifier that processes data messages (i) between two pods 135 on its node 105, (ii) between a pod 135 on its node 105 and another pod or service on a different node in the cluster, and (iii) between a pod 135 on its node 105 and an external address. The OVS bridge 175 stores cached flow entries for faster processing of these data messages without involving the OVS daemon ovs-vswitchd when possible.
However, when the OVS bridge 175 receives a data message for which it does not have a cached flow entry (e.g., the first data message of a connection), the bridge 175 passes the data message to the ovs-vswitchd daemon 170. This user space classifier daemon processes the data message through a set of staged flow tables (or subtables of a single flow table) that store flow entries, typically requiring multiple lookups over the course of several stages. The daemon uses these results to generate a cache flow entry for similar data messages (e.g., for the same connection and/or for similar connections) and provides this cache flow entry to the OVS bridge 175 so that the bridge can more efficiently process subsequent data messages for the same or similar connections.
The CNI agent 165 creates the OVS bridge 175 on its node, as well as the virtual ethernet device/adapter (e.g., veth pairs) for each pod 135. In addition, the CNI agent 165 creates an internal port on the OVS bridge 175 as the gateway for the node's pod subnet, as well as a tunnel port for creating overlay tunnels to other nodes (e.g., using VXLAN, Geneve, STT, etc.). In some embodiments, each node 105 in the cluster 100 is assigned a subnet for its pods, and all of the pods 135 are assigned a network address (i.e., by the CNI agent 165) in the subnet of their node 105 (referred to herein as a “pod subnet” or “node pod subnet”). The agent 165 also assigns the gateway port a network address in the node's pod subnet (e.g., the first IP address in a classless inter-domain routing (CIDR) block allocated for the node).
When one pod 135 sends a data message to a pod or service on another node, the OVS bridge 175 on the source node applies any network policies, performs forwarding (e.g., based on the destination network address of the data message), and encapsulates the data message to be sent to the appropriate node. The agent 165 is responsible for generating and installing flow entries in the OVS daemon 170 (ovs-vswitchd) that map destination network addresses (i.e., pod subnets of nodes) to the appropriate encapsulation tunnel addresses (which typically correspond to network addresses of node network interfaces (which are different from and outside of the pod subnets allocated for the nodes).
Outside of the nodes 105, the cluster 100 also includes a CNI controller 115, which in some embodiments is implemented as a Kubernetes deployment 110. The CNI controller 115 receives configuration changes (e.g., to network policy, pod, or namespace resources) from the Kubernetes API 120 (e.g., by monitoring the Kubernetes API 120). The controller 115 computes policy rules that can be translated to OVS flow entries and provides these policy rules to the agents 165 on the nodes 105. In some embodiments, the controller 115 identifies which nodes 105 host pods 135 to which each rule applies, and only disseminate the rules to the agents 165 on the appropriate nodes 105.
The controller 115 also exposes a REST API to enable the CNI command line interface (CLI) 130 to access the controller 115. The CLI 130, in some embodiments, queries information from (i) the controller 115 and (ii) the agents 165. A user (e.g., an administrator, application developer, etc.) can use CLI queries to retrieve basic runtime information from the controller 115 and/or agents 165. In addition, as described further below, the users may also request information about the flow entries realized on the nodes, request data about ongoing connections within the cluster, and/or initiate flow trace operations through the CLI 130.
Lastly, the Octant UI 125 is an existing user interface tool for developers to use to gain insight into how applications run on a Kubernetes cluster. The CNI plugin 180 for the Octant UI 125 enables a user to view the health and basic runtime information for the controller 115 and the agents 165 in the Octant UI 125. In some embodiments, the plugin 180 accesses the Kubernetes API 120 to retrieve this information via custom resource definitions (CRDs) in the API 120 that are created by the controller 115 and each agent 165 to populate their health and runtime information.
The CNI aspects of the cluster (i.e., the controller 115, Octant CNI plugin 180, CLI 130, plugins 145, agents 165, and OVS modules 170 and 175) can be deployed in some embodiments via a single YAML manifest in some embodiments. Just as Kubernetes supports multiple platforms (e.g., Linux and Windows) for the nodes, the CNI plugin of some embodiments can also be adapted to support multiple platforms. Specifically, for Windows nodes, some embodiments directly connect containers to the physical network through an external Hyper-V switch. In this configuration, OVS operates as an extension of the Hyper-V switch that is bound to a host networking system (HNS) network. This allows the OVS bridge to process data messages sent to and from the containers on the pods. To maintain the host connectivity, the original network configuration on the physical NIC is moved to the OVS bridge in some such embodiments. In addition, in some such embodiments, both the agent 165 and the OVS daemons 170 are run as processes rather than as containers.
When using the CNI of some embodiments in the cloud, this CNI may operate in tandem with or as an alternative to a default CNI provided by the cloud provider. For instance, the CNI of some embodiments may operate in a policy-only mode, in which case the default cloud provider CNI is responsible for performing IP address management (IPAM) operations and allocating secondary IP addresses from virtual private cloud (VPC) subnets. The cloud provider CNI is already aware of the underlying VPC network and configuration. In this mode, the CNI of some embodiments does not perform any encapsulation, as all inter-node traffic is served by the VPC network. Instead, the CNI of this invention enforces Kubernetes network policies on the pods in the cluster. In addition, in some embodiments, the CNI implements its own policy specification using CRDs, thereby providing additional security features in the cloud.
In such a managed cloud, the pod network is known to the physical underlay network, and therefore (as mentioned) there is no need to perform encapsulation. In this mode, an additional bridge (referred to as a transport bridge, rather than the integration bridge shown in
Some embodiments allow the CNI configuration to be modified in such a way that multiple CNIs can be invoked, in a specified order, to perform various tasks involved in the life cycle of a network interface created for a container. In the cloud case here, the cloud provider CNI is invoked first to allocate an IP and MAC address from the VPC subnet. The cloud provider CNI then passes that information in the network configuration to the CNI of this invention. This CNI is then invoked with this network configuration, to create the required interface(s) for the container and program the necessary flows in OVS.
It should be understood that the cluster shown in
In these contexts, the CNI agent of some embodiments on a particular node is capable of identifying flow entries used by the forwarding element of the particular node (i.e., the OVS instance on the node) related to a particular container cluster concept (e.g., a Kubernetes abstraction, such as a pod or network policy) requested by a user. The CNI agent maps elements of these flow entries to different Kubernetes concepts, including those in the request, and returns them to the requestor. In general, OVS provides useful tools that allow a user to view bridge configurations as well as raw flow entries. However, a typical Kubernetes application developer is not familiar with OVS flow entries (or the tools for viewing OVS data), which can make it more difficult for them to understand and troubleshoot the network. However, if the agent can correlate the Kubernetes abstractions (e.g., pods, services, nodes, network policies) to the internal state of the CNI (e.g., via the flow entries realized in the OVS instances), this allows such a user to identify whether the Kubernetes abstractions are properly realized.
As shown, the process 200 begins by receiving (at 205) a request for information about flow entries associated with a particular Kubernetes concept in a cluster. The request may relate to a particular network policy (i.e., a declared Kubernetes network policy), or a specific entity in the cluster (e.g., a particular pod, node, or service). In addition, some embodiments allow more complex requests, such as a request for all flow entries relating to any network policy that are applied to a specific pod. In different embodiments, this request may be received at the CNI agent directly from the command line interface tool associated with the CNI (or a different interface with which a developer or administrator interacts) or via the centralized CNI controller (e.g., based on a request to the controller from the CLI tool).
Returning to the
Next, the process 200 selects (at 215) one of the identified flow entries that is responsive to the request. The process 200 maps (at 220) relevant match fields of the selected flow entry and/or actions specified by the selected flow entry to Kubernetes concepts in the cluster. As with the Kubernetes concepts specified in the request, the elements of the flow entry may be mapped to pods, network policies and/or network policy rules, etc. For instance, matches over table identifiers might be indicative of a specific network policy, matches over specific addresses might be indicative of specific pods, conjunction identifiers specified in actions might be indicative of a specific policy rule, etc.
The process 200 then determines (at 225) whether additional identified flow entries remain. If more flow entries remain, the process 200 returns to 215 to select another flow entry identified as responsive to the request for processing. It should be understood that the process 200 is a conceptual process, and the agent may use other methods to map all of the flow entries to Kubernetes cluster concepts. For instance, the agent of some embodiments processes some or all of the flow entries in parallel.
Once the mapping data is generated for all of the relevant flow entries, the process provides (at 230) the raw flow entry data along with the mapping data to the requestor. The process 200 then ends. As shown in
As mentioned,
The CLI tool of some embodiments enables retrieving other information about the network policies in addition to the flow entries. For a given network policy, the CLI tool can retrieve from the controller the rules for implementing that policy, the groups of addresses to which the policy relates, the pods to which the policy is applied, and the nodes on which the policy is applied. Similarly, for a specific pod, in some embodiments the CLI tool can retrieve a list of all of the network policies applied to the pod.
With information indicating on which nodes the policy is specified, the user can then use the CLI tool to request the flows for realizing the policy from one or more of those nodes, as in
In this example, the first flow entry 705 reads cookie=0x8501000094e9d817, table=90, priority=200, ip, nw_src=172.100.0.3 actions=conjunction (1,1/3). This raw flow entry data may be difficult for a network administrator or application developer to understand, but the report provides the generated mapping data along with the flow entry 705 (and the other flow entries) for presentation to the requestor. This mapping data indicates the table (or sub-table) of OVS to which the flow entry belongs (ingressRuleTable (NetworkPolicy ingress rules), to which table=90 maps). In addition, the mapping data specifies that this is a network policy conjunction flow that matches network policy rule conditions, and specifies the particular network policy (web-app/web-app-policy) and policy rule (rule 1). All of this information enables the user to properly contextualize the flow entry. Lastly, the mapping data indicates that the match condition ip,nw_src=172.100.0.3 means that the flow entry matches over the source IP address 172.100.0.3, which corresponds to a specific web client. Finally, the mapping data provided in the report indicates that the flow entry 705 is realized on the node.
The mapping data for flow entries 710 and 715 is similar to that for flow entry 705. The data for the second flow entry 710 indicates that the match condition ip,reg1=0x4 means that the flow entry matches on packets specifically sent to the pod web-server-sp7z2, which operates on the node k8s2. The data for the third flow entry 715 indicates that the match condition tcp, tp_dst=80 means that the flow entry matches on packets with a TCP port number of 80 (for http traffic).
Finally, the mapping data for the fourth flow entry 720 indicates that the flow entry belongs to the same table as the other three. This flow entry is different than the previous flow entries in that its match condition is a conjunction identifier conj_id=1, which is met only when the match conditions of entries for all of the conjunction dimensions have been met (i.e., the first three flow entries 705-715 have been matched). In this case, the network policy passes the data traffic because the relevant network policy rule is an allow rule. The mapping information indicates this information, as well as noting the specific network policy and rule. Furthermore, the mapping information indicates the number of packets and total bytes that have been allowed as per this flow entry. In some embodiments, the CNI agent stores data mapping these conjunction identifiers to network policy rules, so as to be able to identify the particular flow entries that implement a given network policy or policy rule.
In general, OVS supports using multiple protocols to export connection information, such as IPFIX, NetFlow, and sFlow. The CNI of some embodiments enables this export for traffic between the pods (which is governed by the CNI rules). The connection data can be exported to any number of different collectors, analyzers, and visualization consoles that support these protocols. However, standard connection exporting functionality has some limitations, in that (i) no context is added to the exported connection data to correlate flows with applications or Kubernetes abstractions, (ii) policy enforcement information is not included in the data, and (iii) the standard export mechanisms are based on packet sampling, which could generate too much flow traffic to the collector while nevertheless missing some connections.
Instead, the CNI of some embodiments provides an efficient way to export ongoing connections correlated to Kubernetes concepts and associated with network policy information, so that the consumers (e.g., a policy analytics engine, visualization solution, or direct user observation) can more easily identify the patterns of the connections within the cluster as well as the network policies and specific network policy rules that impact the different connections. Thus, some embodiments collect connection information from the data plane, append Kubernetes context to the connection information, and export the connection data (with the appended context) using, e.g., IPFIX. The context added to a connection may include source pod, source node, destination pod, destination node, destination service (if the connection is between a pod and a service in the cluster), and ingress and/or egress network policy and policy rules.
To accomplish this, in some embodiments, either the agent or another module operating on each of the container hosts along with the agent (e.g., a connection exporter module) monitors ongoing connections being processed by the forwarding element. This module retrieves data about these ongoing connections, maps the data to Kubernetes concepts implemented in the cluster, and exports the ongoing connection information along with the mapped Kubernetes concepts.
To enable connection monitoring within the cluster, in addition to these various components each node 805 executes a connection tracker 880, and within the daemonset 860 a flow exporter module 885 also operates. Furthermore, the cluster 800 includes a set of one or more flow aggregation, monitoring, and visualization servers 830.
The connection tracker 880 is a module used by the OVS bridge 875 (and, in some cases, the OVS daemons 870 as well) to store state regarding ongoing connections. The flow entries used by OVS are otherwise stateless, but in some embodiments can specify actions that request data from the connection tracker 880. The connection tracker 880 receives data about connections from the OVS bridge 875 when they are initiated, and stores this connection information (e.g., source and destination network addresses, source and destination transport layer ports, protocol, amounts of packets, and data transferred, connection state) for ongoing connections. In some embodiments, the connection tracker 880 stores this information for both incoming and outgoing directions of a connection, as well as related connections. This allows, for example, network policies to specify that a particular pod (or group of pods) cannot initiate connections with external addresses, but can only respond to such connections (or vice versa). Further information about the operation of the connection tracker 880 of some embodiments can be found in U.S. Patent Publication 2019/0149516, which is incorporated herein by reference.
The flow exporter 885 may be part of the CNI agent 865, or a separate module (e.g., operating in a separate container in the daemonset 860) in different embodiments. The flow exporter 885 retrieves the ongoing connections from the connection tracker module (e.g., at regular intervals) and maps this data to the Kubernetes concepts. In some embodiments, the flow exporter 885 polls the connection tracker 880 at 5 or 10 second intervals. The mapping operation is similar to that described above for flow entries in some embodiments and uses the cluster data stored by the CNI agent 865. The retrieved connection data includes, in some embodiments, source and destination network addresses and transport layer ports. In addition, the connection data also includes information about the status of the ongoing connection (e.g., the number of packets sent in either direction, the amount of data sent, any related connections, etc.).
The flow exporter 885 uses information from the CNI agent 865 to map connection data to the available Kubernetes cluster data, in some embodiments (as further described below, some Kubernetes cluster data may not be available to the CNI agent on a node, and is added at a centralized aggregator). As described above, the CNI agent 865 stores data that maps, e.g., IP addresses to specific pods in the cluster. This data can also include mapping of pods to specific nodes (so that the flow exporter 885 can export the connection information along with source and destination nodes in addition to source and destination pods).
Furthermore, the flow exporter also identifies network policies that are applied to each connection in some embodiments. As described above, the network policies are implemented using conjunctive match flow entries in some embodiments, and therefore policy rules can be identified by the conjunction identifiers. In some embodiments, the OVS bridge 875 is configured to provide to the connection tracker 880 the conjunction identifiers for flows that are applied to each connection. In other embodiments, other types of unique identifiers (e.g., UUIDs) are embedded in the OVS flow entries that implement network policy rules, and these unique identifiers are provided to the connection tracker 880.
The flow exporter 885 exports this connection information with mapping data to the flow aggregation, monitoring, and visualization components 830. In some embodiments, the connection information is exported in IPFIX format, with the Kubernetes concepts included as metadata (e.g., as IPFIX Information Elements). In some embodiments, rather than exporting the same connection numerous times, the flow exporter 885 tracks the connection lifecycle based on the regular updates from the connection tracker, and only sends updates to the components 830 when a flow is created or deleted (in order to reduce traffic).
The flow aggregation, monitoring, and visualization components 830 may execute within the Kubernetes cluster 800 (e.g., as a distributed service) or outside of the cluster, and may operate as a single component or as multiple separate components. For instance, some embodiments include an aggregator within the cluster that aggregates connection data (e.g., IPFIX data) from multiple nodes, then forwards this aggregated data to one or more collectors and/or visualizers outside of the cluster that are accessed by users (e.g., application developers and/or administrators).
In addition, some embodiments add Kubernetes mapping data if that data is not available at the CNI agent 865. In some cases, it may not be efficient for every agent in a cluster to store information about every Kubernetes concept in the cluster. For example, in some embodiments, the CNI agent 865 only has the pod names for each IP address, but does not store information mapping every pod in the cluster to the node on which that pod executes. Similarly, the CNI agent 865 might only store information for its local pods and not for all of the remote pods, and might not store information about distributed services in the cluster. In some such embodiments, the aggregator 830 adds its own Kubernetes concepts mapping data (e.g., source and destination node names, distributed services data) in addition to the mapping data received from the flow exporters 885.
The additional monitoring and/or visualization components 830 may be part of a larger network management and control system, in some embodiments. For instance, in some embodiments, a flow aggregation component 830 in the cluster 800 provides the aggregated data to a network management and control system visualization (e.g., NSX Intelligence). Such a management and control system can push global policies (e.g., via the CLI tool 825 or another adaptor executing in the cluster) as well as monitor network and policy realization status. These global policies are enforced by the CNI either by conversion to Kubernetes network policies or as separate flow entries, in different embodiments. This allows the network management and control system to manage multiple federated Kubernetes clusters in some embodiments, each running the same CNI plugins. The management and control system collects cluster health and statistics, as well as log bundles of the CNI components from any connected cluster in some embodiments.
The network analysis and visualization tools of such a management and control system can also be extended to consume the connection data exported by the flow exporter 885 and flow aggregation component in the cluster. Such a tool can visualize all of the connections and applied network policies within the cluster. In addition, advanced connection monitoring, policy analytics, and recommendation features of such a tool may be enabled for a Kubernetes cluster.
The GUI 900 also shows ongoing (or recent) connections between the pods, as determined by the flow exporter and aggregator features within the cluster. Some embodiments, as is the case here, visualize different connections differently (e.g., different types of lines, different colors, etc.), and use arrows to show the initiation direction of the connections. In some embodiments, the GUI 900 may also be customizable to show the nodes on which each pod executes (or group the pods by node), pod IP addresses, or other information about the pods. The GUI 900 may also include options to filter out certain pods (i.e., to only show certain groups, to remove services, etc.) or connections, in some embodiments, as well as to display information about the policies in place.
As shown, the process 1000 begins by identifying (at 1005) the expiration of a timer for polling a connection tracker module. In some embodiments, the connection exporter module uses a timer in order to ensure that the connection tracker is polled at regular intervals (e.g., every 5 seconds, every 10 seconds, every minute, etc.). It should also be understood that some embodiments may retrieve connection data form the connection tracker for other reasons (e.g., an external request for the data), either as an alternative to or in addition to regular polling.
In response to identifying the expiration of the timer (or based on another event), the process 1000 retrieves (at 1010) ongoing connection data from the connection tracker module on its node. This connection data typically includes at least source and destination network addresses and transport layer ports and may also include information about the status of the ongoing connection (e.g., the number of packets sent in either direction, the amount of data sent, any related connections, etc.).
As mentioned,
Returning to
The process 1000 then selects (at 1020) one of the identified new connections, and maps (at 1025) the source and destination addresses of the selected connection to Kubernetes concepts in the cluster (as available to the flow exporter). In some embodiments, the flow exporter maps the source and destination addresses to pods executing on its node or on other nodes, and possibly also to the nodes on which the identified pods execute.
The process 1000 also identifies (at 1030) network policies that are applied to the selected connection on the node. As described above, the connection tracker stores identifiers for the network policy rules applied to packets belonging to the connection in some embodiments. These may be conjunction identifiers (for conjunctive match flow entries), UUIDs associated with the rule, etc. The CNI agent on the node stores the mappings of these identifiers to the network policy rules, and so can identify the policy rules (and thus the network policies) applied to packets of each connection. As noted above, in some embodiments, the flow exporter only maps a portion of the Kubernetes cluster information to the connections (i.e., the portion for which the CNI agent on the node includes the mapping data). In such embodiments, the aggregator to which the connection and mapping data is reported also adds additional Kubernetes cluster mapping data to the connection information.
In some embodiments, the flow exporter queries the CNI agent for the cluster mapping data. For instance, the flow exporter of some embodiments sends the source and destination IP addresses of each new connection to the CNI agent, which returns the names of the pods associated with those addresses (and, if storing the information, the names of the nodes on which those pods execute). For network policies, the flow exporter sends the identifier(s) associated with the connection, and the CNI agent returns the policy rules (and their associated network policies) that correspond to those identifiers.
Returning to
Once the mapping data has been generated for any new connections, the process 1000 identifies (at 1040) any expired (i.e., torn down) connections. The connection exporter may identify these expired connections based on the connection state provided by the connection tracker (e.g., in
Finally, the process reports (at 1045) the new and expired connection data along with the Kubernetes cluster mapping data. As described above, in some embodiments the connection exporter exports this data to an aggregation service operating in the same cluster (e.g., as a service). This aggregation service compares and combines data for the same ongoing connections from source and destination nodes and may add additional Kubernetes information to the data (e.g., node information).
The connection aggregator 1145, as described above, may also (i) supplement the mapping data received from the flow exporters 1125 and 1130 and (ii) provide the connection and mapping data to additional monitoring and/or visualization components in the cluster or outside of the cluster (e.g., components of a network management and control system). These components may visualize all of the connections and applied network policies within the cluster. In addition, advanced connection monitoring, policy analytics, and recommendation features of such a tool may be enabled for a Kubernetes cluster.
In some embodiments, the agents on the nodes in a cluster also enable flow tracing and relate flow entries matched in a flow tracing operation to Kubernetes concepts. Flow tracing operations allow one or more forwarding elements in the cluster to simulate the processing of a data message with pre-specified characteristics in order for an administrator or application developer to determine which flow entries act upon the data message (either using a simulation mechanism provided by the forwarding element or by injecting a flow tracing data message having the specified characteristics). These flow tracing operations and mapping of matched flow entries to Kubernetes concepts are described in more detail in concurrently-filed U.S. patent application Ser. No. 17/006,845, filed Aug. 30, 2020, which is incorporated herein by reference.
The bus 1405 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 1400. For instance, the bus 1405 communicatively connects the processing unit(s) 1410 with the read-only memory 1430, the system memory 1425, and the permanent storage device 1435.
From these various memory units, the processing unit(s) 1410 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments.
The read-only-memory (ROM) 1430 stores static data and instructions that are needed by the processing unit(s) 1410 and other modules of the electronic system. The permanent storage device 1435, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 1400 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 1435.
Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device. Like the permanent storage device 1435, the system memory 1425 is a read-and-write memory device. However, unlike storage device 1435, the system memory is a volatile read-and-write memory, such a random-access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 1425, the permanent storage device 1435, and/or the read-only memory 1430. From these various memory units, the processing unit(s) 1410 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.
The bus 1405 also connects to the input and output devices 1440 and 1445. The input devices enable the user to communicate information and select commands to the electronic system. The input devices 1440 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 1445 display images generated by the electronic system. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer readable medium,” “computer readable media,” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.
This specification refers throughout to computational and network environments that include virtual machines (VMs). However, virtual machines are merely one example of data compute nodes (DCNs) or data compute end nodes, also referred to as addressable nodes. DCNs may include non-virtualized physical hosts, virtual machines, containers that run on top of a host operating system without the need for a hypervisor or separate operating system, and hypervisor kernel network interface modules.
VMs, in some embodiments, operate with their own guest operating systems on a host using resources of the host virtualized by virtualization software (e.g., a hypervisor, virtual machine monitor, etc.). The tenant (i.e., the owner of the VM) can choose which applications to operate on top of the guest operating system. Some containers, on the other hand, are constructs that run on top of a host operating system without the need for a hypervisor or separate guest operating system. In some embodiments, the host operating system uses name spaces to isolate the containers from each other and therefore provides operating-system level segregation of the different groups of applications that operate within different containers. This segregation is akin to the VM segregation that is offered in hypervisor-virtualized environments that virtualize system hardware, and thus can be viewed as a form of virtualization that isolates different groups of applications that operate in different containers. Such containers are more lightweight than VMs.
Hypervisor kernel network interface modules, in some embodiments, is a non-VM DCN that includes a network stack with a hypervisor kernel network interface and receive/transmit threads. One example of a hypervisor kernel network interface module is the vmknic module that is part of the ESXi™ hypervisor of VMware, Inc.
It should be understood that while the specification refers to VMs, the examples given could be any type of DCNs, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, the example networks could include combinations of different types of DCNs in some embodiments.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. In addition, a number of the figures (including
| Number | Date | Country | Kind |
|---|---|---|---|
| PCT/CN2020/105523 | Jul 2020 | WO | international |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5224100 | Lee et al. | Jun 1993 | A |
| 5245609 | Ofek et al. | Sep 1993 | A |
| 5265092 | Soloway et al. | Nov 1993 | A |
| 5504921 | Dev et al. | Apr 1996 | A |
| 5550816 | Hardwick et al. | Aug 1996 | A |
| 5729685 | Chatwani et al. | Mar 1998 | A |
| 5751967 | Raab et al. | May 1998 | A |
| 5781534 | Perlman et al. | Jul 1998 | A |
| 6104699 | Holender et al. | Aug 2000 | A |
| 6104700 | Haddock et al. | Aug 2000 | A |
| 6141738 | Munter et al. | Oct 2000 | A |
| 6219699 | McCloghrie et al. | Apr 2001 | B1 |
| 6430160 | Smith et al. | Aug 2002 | B1 |
| 6456624 | Eccles et al. | Sep 2002 | B1 |
| 6512745 | Abe et al. | Jan 2003 | B1 |
| 6539432 | Taguchi et al. | Mar 2003 | B1 |
| 6658002 | Ross et al. | Dec 2003 | B1 |
| 6680934 | Cain | Jan 2004 | B1 |
| 6721334 | Ketcham | Apr 2004 | B1 |
| 6785843 | McRae et al. | Aug 2004 | B1 |
| 6941487 | Balakrishnan et al. | Sep 2005 | B1 |
| 6963585 | Pennec et al. | Nov 2005 | B1 |
| 6999454 | Crump | Feb 2006 | B1 |
| 7013342 | Riddle | Mar 2006 | B2 |
| 7062559 | Yoshimura et al. | Jun 2006 | B2 |
| 7079544 | Wakayama et al. | Jul 2006 | B2 |
| 7180856 | Breslau et al. | Feb 2007 | B1 |
| 7197572 | Matters et al. | Mar 2007 | B2 |
| 7200144 | Terrell et al. | Apr 2007 | B2 |
| 7209439 | Rawlins et al. | Apr 2007 | B2 |
| 7243143 | Bullard | Jul 2007 | B1 |
| 7283473 | Arndt et al. | Oct 2007 | B2 |
| 7315985 | Gauvin et al. | Jan 2008 | B1 |
| 7342916 | Das et al. | Mar 2008 | B2 |
| 7391771 | Orava et al. | Jun 2008 | B2 |
| 7450598 | Chen et al. | Nov 2008 | B2 |
| 7463579 | Lapuh et al. | Dec 2008 | B2 |
| 7478173 | Delco | Jan 2009 | B1 |
| 7483370 | Dayal et al. | Jan 2009 | B1 |
| 7555002 | Arndt et al. | Jun 2009 | B2 |
| 7577131 | Joseph et al. | Aug 2009 | B2 |
| 7590133 | Hatae et al. | Sep 2009 | B2 |
| 7602723 | Mandato et al. | Oct 2009 | B2 |
| 7606260 | Oguchi et al. | Oct 2009 | B2 |
| 7627692 | Pessi | Dec 2009 | B2 |
| 7633955 | Saraiya et al. | Dec 2009 | B1 |
| 7639625 | Kaminsky et al. | Dec 2009 | B2 |
| 7643488 | Khanna et al. | Jan 2010 | B2 |
| 7649851 | Takashige et al. | Jan 2010 | B2 |
| 7706266 | Plamondon | Apr 2010 | B2 |
| 7710874 | Balakrishnan et al. | May 2010 | B2 |
| 7729245 | Breslau et al. | Jun 2010 | B1 |
| 7760735 | Chen et al. | Jul 2010 | B1 |
| 7764599 | Doi et al. | Jul 2010 | B2 |
| 7792987 | Vohra et al. | Sep 2010 | B1 |
| 7802000 | Huang et al. | Sep 2010 | B1 |
| 7808919 | Nadeau et al. | Oct 2010 | B2 |
| 7808929 | Wong et al. | Oct 2010 | B2 |
| 7818452 | Matthews et al. | Oct 2010 | B2 |
| 7826482 | Minei et al. | Nov 2010 | B1 |
| 7839847 | Nadeau et al. | Nov 2010 | B2 |
| 7885276 | Lin | Feb 2011 | B1 |
| 7936770 | Frattura et al. | May 2011 | B1 |
| 7937438 | Miller et al. | May 2011 | B1 |
| 7937492 | Kompella et al. | May 2011 | B1 |
| 7948986 | Ghosh et al. | May 2011 | B1 |
| 7953865 | Miller et al. | May 2011 | B1 |
| 7991859 | Miller et al. | Aug 2011 | B1 |
| 7995483 | Bayar et al. | Aug 2011 | B1 |
| 8024478 | Patel | Sep 2011 | B2 |
| 8027354 | Portolani et al. | Sep 2011 | B1 |
| 8031606 | Memon et al. | Oct 2011 | B2 |
| 8031633 | Bueno et al. | Oct 2011 | B2 |
| 8046456 | Miller et al. | Oct 2011 | B1 |
| 8054832 | Shukla et al. | Nov 2011 | B1 |
| 8055789 | Richardson et al. | Nov 2011 | B2 |
| 8060875 | Lambeth | Nov 2011 | B1 |
| 8131852 | Miller et al. | Mar 2012 | B1 |
| 8149737 | Metke et al. | Apr 2012 | B2 |
| 8155028 | Abu-Hamdeh et al. | Apr 2012 | B2 |
| 8161270 | Parker et al. | Apr 2012 | B1 |
| 8166201 | Richardson et al. | Apr 2012 | B2 |
| 8199750 | Schultz et al. | Jun 2012 | B1 |
| 8223668 | Allan et al. | Jul 2012 | B2 |
| 8224931 | Brandwine et al. | Jul 2012 | B1 |
| 8224971 | Miller et al. | Jul 2012 | B1 |
| 8254273 | Kaminsky et al. | Aug 2012 | B2 |
| 8265062 | Tang et al. | Sep 2012 | B2 |
| 8265075 | Pandey | Sep 2012 | B2 |
| 8281067 | Stolowitz | Oct 2012 | B2 |
| 8290137 | Yurchenko et al. | Oct 2012 | B2 |
| 8306043 | Breslau et al. | Nov 2012 | B2 |
| 8312129 | Miller et al. | Nov 2012 | B1 |
| 8339959 | Moisand et al. | Dec 2012 | B1 |
| 8339994 | Gnanasekaran et al. | Dec 2012 | B2 |
| 8345558 | Nicholson et al. | Jan 2013 | B2 |
| 8351418 | Zhao et al. | Jan 2013 | B2 |
| 8359576 | Prasad et al. | Jan 2013 | B2 |
| 8456984 | Ranganathan et al. | Jun 2013 | B2 |
| 8504718 | Wang et al. | Aug 2013 | B2 |
| 8565108 | Marshall et al. | Oct 2013 | B1 |
| 8571031 | Davies et al. | Oct 2013 | B2 |
| 8611351 | Gooch et al. | Dec 2013 | B2 |
| 8612627 | Brandwine | Dec 2013 | B1 |
| 8625594 | Safrai et al. | Jan 2014 | B2 |
| 8625603 | Ramakrishnan et al. | Jan 2014 | B1 |
| 8625616 | Vobbilisetty et al. | Jan 2014 | B2 |
| 8644188 | Brandwine et al. | Feb 2014 | B1 |
| 8645952 | Biswas et al. | Feb 2014 | B2 |
| 8750288 | Nakil et al. | Jun 2014 | B2 |
| 8762501 | Kempf et al. | Jun 2014 | B2 |
| 8806005 | Miri et al. | Aug 2014 | B2 |
| 8837300 | Nedeltchev et al. | Sep 2014 | B2 |
| 8838743 | Lewites et al. | Sep 2014 | B2 |
| 8929221 | Breslau et al. | Jan 2015 | B2 |
| 9059926 | Akhter et al. | Jun 2015 | B2 |
| 9197529 | Ganichev et al. | Nov 2015 | B2 |
| 9226220 | Banks et al. | Dec 2015 | B2 |
| 9258195 | Pendleton et al. | Feb 2016 | B1 |
| 9280448 | Farrell et al. | Mar 2016 | B2 |
| 9282019 | Ganichev et al. | Mar 2016 | B2 |
| 9344349 | Ganichev et al. | May 2016 | B2 |
| 9407580 | Ganichev et al. | Aug 2016 | B2 |
| 9602334 | Benny | Mar 2017 | B2 |
| 9860151 | Ganichev et al. | Jan 2018 | B2 |
| 9898317 | Nakil et al. | Feb 2018 | B2 |
| 10044581 | Russell | Aug 2018 | B1 |
| 10181993 | Ganichev et al. | Jan 2019 | B2 |
| 10200306 | Nhu et al. | Feb 2019 | B2 |
| 10469342 | Lenglet et al. | Nov 2019 | B2 |
| 10608887 | Jain et al. | Mar 2020 | B2 |
| 10778557 | Ganichev et al. | Sep 2020 | B2 |
| 10805239 | Nhu et al. | Oct 2020 | B2 |
| 20010020266 | Kojima et al. | Sep 2001 | A1 |
| 20010043614 | Viswanadham et al. | Nov 2001 | A1 |
| 20020093952 | Gonda | Jul 2002 | A1 |
| 20020194369 | Rawlins et al. | Dec 2002 | A1 |
| 20030041170 | Suzuki | Feb 2003 | A1 |
| 20030058850 | Rangarajan et al. | Mar 2003 | A1 |
| 20040073659 | Rajsic et al. | Apr 2004 | A1 |
| 20040098505 | Clemmensen | May 2004 | A1 |
| 20040186914 | Shimada | Sep 2004 | A1 |
| 20040267866 | Carollo et al. | Dec 2004 | A1 |
| 20040267897 | Hill et al. | Dec 2004 | A1 |
| 20050018669 | Arndt et al. | Jan 2005 | A1 |
| 20050027881 | Figueira et al. | Feb 2005 | A1 |
| 20050053079 | Havala | Mar 2005 | A1 |
| 20050083953 | May | Apr 2005 | A1 |
| 20050111445 | Wybenga et al. | May 2005 | A1 |
| 20050120160 | Plouffe et al. | Jun 2005 | A1 |
| 20050132044 | Guingo et al. | Jun 2005 | A1 |
| 20050182853 | Lewites et al. | Aug 2005 | A1 |
| 20050220096 | Friskney et al. | Oct 2005 | A1 |
| 20050232230 | Nagami et al. | Oct 2005 | A1 |
| 20060002370 | Rabie et al. | Jan 2006 | A1 |
| 20060026225 | Canali et al. | Feb 2006 | A1 |
| 20060028999 | Iakobashvili et al. | Feb 2006 | A1 |
| 20060029056 | Perera et al. | Feb 2006 | A1 |
| 20060037075 | Frattura et al. | Feb 2006 | A1 |
| 20060174087 | Hashimoto et al. | Aug 2006 | A1 |
| 20060187908 | Shimozono et al. | Aug 2006 | A1 |
| 20060193266 | Siddha et al. | Aug 2006 | A1 |
| 20060206655 | Chappell et al. | Sep 2006 | A1 |
| 20060218447 | Garcia et al. | Sep 2006 | A1 |
| 20060221961 | Basso et al. | Oct 2006 | A1 |
| 20060282895 | Rentzis et al. | Dec 2006 | A1 |
| 20060291388 | Amdahl et al. | Dec 2006 | A1 |
| 20070050763 | Kagan et al. | Mar 2007 | A1 |
| 20070055789 | Claise et al. | Mar 2007 | A1 |
| 20070064673 | Bhandaru et al. | Mar 2007 | A1 |
| 20070097982 | Wen et al. | May 2007 | A1 |
| 20070156919 | Potti et al. | Jul 2007 | A1 |
| 20070260721 | Bose et al. | Nov 2007 | A1 |
| 20070286185 | Eriksson et al. | Dec 2007 | A1 |
| 20070297428 | Bose et al. | Dec 2007 | A1 |
| 20080002579 | Lindholm et al. | Jan 2008 | A1 |
| 20080002683 | Droux et al. | Jan 2008 | A1 |
| 20080049614 | Briscoe et al. | Feb 2008 | A1 |
| 20080049621 | McGuire et al. | Feb 2008 | A1 |
| 20080049786 | Ram et al. | Feb 2008 | A1 |
| 20080059556 | Greenspan et al. | Mar 2008 | A1 |
| 20080071900 | Hecker et al. | Mar 2008 | A1 |
| 20080086726 | Griffith et al. | Apr 2008 | A1 |
| 20080112551 | Forbes et al. | May 2008 | A1 |
| 20080159301 | Heer | Jul 2008 | A1 |
| 20080240095 | Basturk | Oct 2008 | A1 |
| 20090010254 | Shimada | Jan 2009 | A1 |
| 20090100298 | Lange et al. | Apr 2009 | A1 |
| 20090109973 | Ilnicki | Apr 2009 | A1 |
| 20090123903 | Weitenberner | May 2009 | A1 |
| 20090150527 | Tripathi et al. | Jun 2009 | A1 |
| 20090245138 | Sapsford et al. | Oct 2009 | A1 |
| 20090249213 | Murase et al. | Oct 2009 | A1 |
| 20090292858 | Lambeth et al. | Nov 2009 | A1 |
| 20090327903 | Smith et al. | Dec 2009 | A1 |
| 20100128623 | Dunn et al. | May 2010 | A1 |
| 20100131636 | Suri et al. | May 2010 | A1 |
| 20100188976 | Rahman et al. | Jul 2010 | A1 |
| 20100214949 | Smith et al. | Aug 2010 | A1 |
| 20100232435 | Jabr et al. | Sep 2010 | A1 |
| 20100254385 | Sharma et al. | Oct 2010 | A1 |
| 20100275199 | Smith et al. | Oct 2010 | A1 |
| 20100306408 | Greenberg et al. | Dec 2010 | A1 |
| 20110022695 | Dalal et al. | Jan 2011 | A1 |
| 20110075664 | Lambeth et al. | Mar 2011 | A1 |
| 20110085557 | Gnanasekaran et al. | Apr 2011 | A1 |
| 20110085559 | Chung et al. | Apr 2011 | A1 |
| 20110085563 | Kotha et al. | Apr 2011 | A1 |
| 20110128959 | Bando et al. | Jun 2011 | A1 |
| 20110137602 | Desineni et al. | Jun 2011 | A1 |
| 20110194567 | Shen | Aug 2011 | A1 |
| 20110202920 | Takase | Aug 2011 | A1 |
| 20110261825 | Ichino | Oct 2011 | A1 |
| 20110299413 | Chatwani et al. | Dec 2011 | A1 |
| 20110299534 | Koganti et al. | Dec 2011 | A1 |
| 20110299537 | Saraiya et al. | Dec 2011 | A1 |
| 20110305167 | Koide | Dec 2011 | A1 |
| 20110317559 | Kern et al. | Dec 2011 | A1 |
| 20110317696 | Aldrin et al. | Dec 2011 | A1 |
| 20120079478 | Galles et al. | Mar 2012 | A1 |
| 20120159454 | Barham et al. | Jun 2012 | A1 |
| 20120182992 | Cowart et al. | Jul 2012 | A1 |
| 20120275331 | Benkö et al. | Nov 2012 | A1 |
| 20120287791 | Xi et al. | Nov 2012 | A1 |
| 20130024579 | Zhang et al. | Jan 2013 | A1 |
| 20130031233 | Feng et al. | Jan 2013 | A1 |
| 20130041934 | Annamalaisami et al. | Feb 2013 | A1 |
| 20130044636 | Koponen et al. | Feb 2013 | A1 |
| 20130054761 | Kempf et al. | Feb 2013 | A1 |
| 20130058346 | Sridharan et al. | Mar 2013 | A1 |
| 20130067067 | Miri et al. | Mar 2013 | A1 |
| 20130125120 | Zhang et al. | May 2013 | A1 |
| 20130163427 | Beliveau et al. | Jun 2013 | A1 |
| 20130163475 | Beliveau et al. | Jun 2013 | A1 |
| 20130332602 | Nakil et al. | Dec 2013 | A1 |
| 20130332983 | Koorevaar et al. | Dec 2013 | A1 |
| 20130339544 | Mithyantha | Dec 2013 | A1 |
| 20140019639 | Ueno | Jan 2014 | A1 |
| 20140029451 | Nguyen | Jan 2014 | A1 |
| 20140115578 | Cooper et al. | Apr 2014 | A1 |
| 20140119203 | Sundaram et al. | May 2014 | A1 |
| 20140126418 | Brendel et al. | May 2014 | A1 |
| 20140157405 | Joll et al. | Jun 2014 | A1 |
| 20140177633 | Manula et al. | Jun 2014 | A1 |
| 20140195666 | Dumitriu et al. | Jul 2014 | A1 |
| 20140207926 | Benny | Jul 2014 | A1 |
| 20140219086 | Cantu′ et al. | Aug 2014 | A1 |
| 20140281030 | Cui et al. | Sep 2014 | A1 |
| 20140282823 | Rash et al. | Sep 2014 | A1 |
| 20140304393 | Annamalaisami et al. | Oct 2014 | A1 |
| 20150016286 | Ganichev et al. | Jan 2015 | A1 |
| 20150016287 | Ganichev et al. | Jan 2015 | A1 |
| 20150016298 | Ganichev et al. | Jan 2015 | A1 |
| 20150016469 | Ganichev et al. | Jan 2015 | A1 |
| 20150043378 | Bardgett et al. | Feb 2015 | A1 |
| 20150180755 | Zhang et al. | Jun 2015 | A1 |
| 20150263899 | Tubaltsev et al. | Sep 2015 | A1 |
| 20150281036 | Sun et al. | Oct 2015 | A1 |
| 20160105333 | Lenglet et al. | Apr 2016 | A1 |
| 20160119204 | Murasato et al. | Apr 2016 | A1 |
| 20160149791 | Ganichev et al. | May 2016 | A1 |
| 20160226741 | Ganichev et al. | Aug 2016 | A1 |
| 20170317954 | Masurekar et al. | Nov 2017 | A1 |
| 20170358111 | Madsen | Dec 2017 | A1 |
| 20180062939 | Kulkarni et al. | Mar 2018 | A1 |
| 20180102959 | Ganichev et al. | Apr 2018 | A1 |
| 20180123903 | Holla et al. | May 2018 | A1 |
| 20180262447 | Nhu et al. | Sep 2018 | A1 |
| 20180262594 | Nhu et al. | Sep 2018 | A1 |
| 20180373961 | Wang et al. | Dec 2018 | A1 |
| 20190109769 | Jain et al. | Apr 2019 | A1 |
| 20190140931 | Ganichev et al. | May 2019 | A1 |
| 20200067799 | Lenglet et al. | Feb 2020 | A1 |
| 20200106744 | Miriyala | Apr 2020 | A1 |
| 20210029059 | Nhu et al. | Jan 2021 | A1 |
| 20210051109 | Chitalia et al. | Feb 2021 | A1 |
| 20210218630 | Lu et al. | Jul 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 1154601 | Nov 2001 | EP |
| 2002141905 | May 2002 | JP |
| 2003069609 | Mar 2003 | JP |
| 2003124976 | Apr 2003 | JP |
| 2003318949 | Nov 2003 | JP |
| 9506989 | Mar 1995 | WO |
| 2012126488 | Sep 2012 | WO |
| 2013184846 | Dec 2013 | WO |
| 2015005968 | Jan 2015 | WO |
| Entry |
|---|
| Non-Published commonly Owned U.S. Appl. No. 17/006,845, filed Aug. 30, 2020, 70 pages, VMware, Inc. |
| Non-Published commonly Owned Related U.S. Appl. No. 17/006,846 with similar specification, filed Aug. 30, 2020, 47 pages, VMware, Inc. |
| Non-Published commonly Owned U.S. Appl. No. 17/068,655, filed Oct. 12, 2020, 63 pages, Nicira, Inc. |
| Phaal, Peter, et al., “sFlow Version 5,” Jul. 2004, 46 pages, available at http://www.sflow.org/sflow_version_5.txt. |
| Phan, Doantam, et al., “Visual Analysis of Network Flow Data with Timelines and Event Plots,” VizSEC 2007, Month Unknown 2007, 16 pages. |
