Patent Application
20240031264
Benefit is claimed under 35 U.S.C. 119(a)-(d) to Foreign Application Serial No. 202241042205 filed in India entitled “MONITORING PERFORMANCE OF APPLICATIONS WITH RESPECT TO SOFTWARE DEFINED WIDE AREA NETWORK EDGE DEVICES”, on Jul. 22, 2022, by VMware, Inc., which is herein incorporated in its entirety by reference for all purposes.
The present disclosure relates generally to software defined wide area networks (SD-WANs). More specifically, the present disclosure relates to systems and methods for monitoring performance of applications with respect to SD-WAN edge devices.
Wide area networks (WANs) are commonly employed to provide end user access to telecommunications network hardware over large geographic distances. WANs often suffer from certain drawbacks, however, such as difficulties in integrating multiple service providers to cover these large geographic distances. Software defined WANs (SD-WANs) have found more recent acceptance as one approach to addressing these drawbacks and improving network performance. More specifically, SD-WANs virtualize applications and other services, decoupling networking hardware from management and operation of such applications/services. In this manner, SD-WAN technology provides improved branch agility, scalability, and more secure and reliable access to applications hosted in datacenters or the cloud. In an effort to increase the acceptance of SD-WANs, ongoing efforts exist to improve their performance, as well as the performance of their applications.
In some embodiments of this disclosure, systems and methods are described for more effectively monitoring the performance of application programs with respect to their associated SD-WAN edge devices. Variables are formulated for measuring the performance of individual applications accessed from an individual edge device. This allows for measurement of performance on both a per-application and a per-edge device basis. In this manner, SD-WAN connectivity issues may be measured with increased granularity, allowing for more effective determination of root causes of performance issues and more effective troubleshooting.
A computer implemented method for monitoring application performance with respect to edge devices is described, and includes: receiving data flow records for a plurality of SD-WAN edge devices each permitting access to a plurality of application programs, the data flow records comprising data flow information for each application program accessed from each SD-WAN edge device; for each application program accessed from each SD-WAN edge device, extracting the respective data flow information so as to form a data flow variable describing data flow of a particular application program accessed from a particular SD-WAN edge device; determining, from one or more of the data flow variables, whether anomalous behavior of a particular application program accessed from a particular SD-WAN edge device has occurred; and transmitting an alert to a user if the anomalous behavior has occurred.
A non-transitory computer-readable storage medium is described. The computer-readable storage medium includes instructions configured to be executed by one or more processors of a computing device and to cause the computing device to carry out steps that include: receiving data flow records for a plurality of SD-WAN edge devices each permitting access to a plurality of application programs, the data flow records comprising data flow information for each application program accessed from each SD-WAN edge device; for each application program accessed from each SD-WAN edge device, extracting the respective data flow information so as to form a data flow variable describing data flow of a particular application program accessed from a particular SD-WAN edge device; determining, from one or more of the data flow variables, whether anomalous behavior of a particular application program accessed from a particular SD-WAN edge device has occurred; and transmitting an alert to a user if the anomalous behavior has occurred.
Other aspects and advantages of embodiments of the disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings which illustrate, by way of example, the principles of the described embodiments.
The disclosure will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements.
Certain details are set forth below to provide a sufficient understanding of various embodiments of the disclosure. However, it will be clear to one skilled in the art that embodiments of the disclosure may be practiced without one or more of these particular details, or with other details. Moreover, the particular embodiments of the present disclosure described herein are provided by way of example and should not be used to limit the scope of the disclosure to these particular embodiments. In other instances, hardware components, network architectures, and/or software operations have not been shown in detail in order to avoid unnecessarily obscuring the disclosure.
In some embodiments of this disclosure, systems and methods are described for more effectively measuring the performance of SD-WAN applications and their associated edge devices. Variables are formulated for measuring the performance of individual applications accessed from an individual edge device. In some embodiments of the disclosure, flow records such as Internet Protocol Flow Information Export (IPFIX) records are collected. Records specific to a particular application and a particular edge device from which the application is accessed may be extracted and assembled into a variable which illustrates the performance of that application from that edge device over time.
In this manner, variables may be assembled which allow for measurement of performance on both a per-application and a per-edge device basis. Accordingly, SD-WAN connectivity issues may be measured with increased granularity, allowing for more effective determination of root causes of performance issues and more effective troubleshooting.
Each of hosts 102, 112, 122 and 132 are capable of running virtualization software 108, 118, 128 and 138, respectively. The virtualization software can run within a virtual machine (VM) and includes management tools for starting, stopping and managing various virtual machines running on the host. For example, host 102 can be configured to stop or suspend operations of virtual machines 104 or 106 utilizing virtualization software 108. Virtualization software 108, commonly referred to as a hypervisor, can also be configured to start new virtual machines or change the amount of processing or memory resources from host hardware 110 that are assigned to one or more VMs running on host 102. Host hardware 110 includes one or more processors, memory, storage resources, I/O ports and the like that are configured to support operation of VMs running on host 102. In some embodiments, a greater amount of processing, memory or storage resources of host hardware 110 is allocated to operation of VM 104 than to VM 106. This may be desirable when, e.g., VM 104 is running a larger number of services or running on a more resource intensive operating system than VM 106. Clients 150 and 160 are positioned outside server cluster 100 and can request access to services running on server cluster 100 via network 170. Responding to the request for access and interacting with clients 150 and 160 can involve interaction with a single service or in other cases may involve multiple smaller services cooperatively interacting to provide information requested by clients 150 and/or 160.
Hosts 102, 112, 122 and 132, which make up server cluster 100, can also include or have access to a storage area network (SAN) that can be shared by multiple hosts. The SAN is configured to provide storage resources as known in the art. In some embodiments, the SAN can be used to store log data or flow records generated during operation of server cluster 100. While description is made herein with respect to the operation of individual ones of hosts 110-140, it will be appreciated that other hosts 110-140 provide analogous functionality, respectively. Log data and flow records may also be stored in one or more host 102, 112, 122 and 132 memories.
In some embodiments, the illustrated SD-WAN network, which may also be referred to as a virtual network, can connect multiple branch sites to each other and to resources of a centralized datacenter. In this example, the SD-WAN 200 is created for connecting the branch sites 230-236 to each other and to resources 240, 242 of the datacenter 205, via the sets of hubs 212-216 (of which there may be any number, and which may also be referred to herein as forwarding hub nodes) of the hub cluster 210. The SD-WAN 200 is established by a controller cluster (not shown), the sets of hubs 212-216, and four exemplary edge nodes 220-226, one in each of the branch sites 230-236.
The edge nodes 220-226 in some embodiments are edge devices such as edge machines (e.g., VMs 104, 106, etc., containers, programs executing on computers, etc.) and/or standalone appliances that operate at multi-computer locations of the particular entity (e.g., at an office or datacenter of the entity) to connect the computers at their respective locations to the hubs 212-216 and other edge nodes (if so configured). Thus, for example, each branch site 230-236 may have a server cluster 100 that contains or generates its respective edge node 220-226 as, e.g., one or more VMs 104, 106, etc. In some embodiments, the edge nodes 220-226 are clusters of edge nodes at each of the branch sites 230-236. In other embodiments, the edge nodes 220-226 are deployed to each of the branch sites 230-236 as high-availability pairs such that one edge node in the pair is the active edge node and the other edge node in the pair is the standby edge node that can take over as the active edge node in case of failover. Also, in this example, the sets of hubs 212-216 are deployed as machines (e.g., VMs 204 and 206 or containers) in the same public datacenter 205. In other embodiments, the hubs may be deployed in different public datacenters.
In some embodiments of the disclosure, the hubs 212-216 are multi-tenant forwarding elements that can be used to establish secure connection links (e.g., tunnels) with edge nodes 220-226 at SD-WAN network 200 multi-computer sites, such as branch sites (branch offices), datacenters (e.g., third party datacenters), or the like. For example, the sets of hubs 212-216 in the cluster 210 provide access from each of the branch sites 230-236 to each of the other branch sites 230-236, as well as to other remote datacenters (not shown), via the connection links 250, which terminate at the cluster 210 as shown. These multi-computer sites are often at different physical locations (e.g., different buildings, different cities, different states, etc.), according to some embodiments. In some embodiments, the forwarding hub nodes 212-216 can be deployed as physical nodes or virtual nodes. Additionally, the forwarding hub nodes 212-216 can be deployed on a datacenter premises in some embodiments, while in other embodiments, the forwarding hub nodes 212-216 can be deployed on a cloud (e.g., as a set of virtual edges configured as a cluster). Thus, in some embodiments, datacenter 205 may house a server cluster 100 that contains or generates hub cluster 210, and its respective hubs 212-216, e.g., as one or more VMs 104, 106, etc.
In the example of
The edge nodes 220-226 are forwarding elements that exchange packets with one or more hubs 212-216 and/or other edge nodes 220-226 through one or more secure connection links 250, according to some embodiments. In this example, all secure connection links 250 of the edge nodes 220-226 are with the sets of hubs 212-216.
The sets of hubs 212-216 in some embodiments provide the branch sites 230-236 with access to compute, storage, and service resources of the datacenter 205 such as the resources 240-242, as well as those of other datacenters. Examples of such resources 240-242 include compute machines (e.g., virtual machines and/or containers providing server operations), storage machines (e.g., database servers), and middlebox service operations (e.g., firewall services, load balancing services, encryption services, etc.). In some embodiments, the connections 250 between the branch sites 230-236 and the datacenter hubs 212-216 are secure encrypted connections that encrypt packets exchanged between the edge nodes 220-226 of the branch sites 230-236 and the hubs 212-216. Examples of secure encrypted connections used in some embodiments include VPN (virtual private network) connections, or secure IPsec (Internet Protocol security) connections. It is also noted that in some embodiments, connections 250 may be any pathways suitable for data exchange between any network elements, secured or otherwise.
In some embodiments, multiple secure connection links (e.g., multiple secure tunnels) can be established between an edge node 220-226 and a hub 212-216. When multiple such links are defined between an edge node 220-226 and a hub 212-216, each secure connection link, in some embodiments, is associated with a different physical network link between the edge node 220-226 and an external network. For instance, to access external networks in some embodiments, an edge node 220-226 has one or more commercial broadband Internet links (e.g., a cable mode and a fiber optic link) to access the Internet, a wireless cellular link (e.g., a 5G LTE network), etc.
In some embodiments, each secure connection link between a hub 212-216 and an edge node 220-226 is formed as a VPN tunnel between the respective hub 212-216 and edge node 220-226. As described above, the set of hubs 212-216 also connects the edge nodes 220-226 to other remote datacenters. In some embodiments, these connections are through secure VPN tunnels. The collection of the edge nodes 220-226, hubs 212-216, and secure connections between the edge nodes 220-226, hubs 212-216, and other remote datacenters forms the SD-WAN 200, through which users may access desired resources such as applications or services.
In some embodiments of the disclosure, edge nodes 220-226 forward packets to hubs 212-216 based on hub-selection rules that each identify a set of one or more hubs (e.g., the sets of hubs 212-216) of the datacenter 205 for receiving one or more flows from the branch sites 230-236. In some embodiments, the edge nodes 220-226 use flow attributes of received packets to identify hub-selection rules. The edge nodes 220-226 identify hub-selection rules for received packets by matching flow attributes of the received packets with the match criteria of the hub-selection, which associate the match criteria with one or more identifiers of one or more forwarding hub nodes of the datacenter 205, according to some embodiments. In this manner, hubs 212-216 determine the appropriate destinations of received packets and forward them to the specified destination, e.g., resources 240-242, a remote datacenter, or any other specified destination.
Collector 310 may contain a configuration collector module 355 and a metrics collector module 360, which may be implemented as programs, or dedicated hardware modules. In operation, configuration collector module 355 may read packets transmitted/received by an edge device 220-226, and compile packet flow information, or flow records, read from the packets. As one example, collector module 355 may read and compile IPFIX data from IP traffic of an edge device 220-226. Such data may be any information contained in any packet of IP traffic, including but not limited to packet source addresses, destination addresses, application identifiers, source/destination bytes, and the like. Metrics collector 360 may also analyze IP traffic to determine desired information or metrics therefrom which may be useful in characterizing the data flow of applications and/or edge devices, including for example a number of lost packets, round trip time (RTT), flow direction, and any other IP traffic statistics or information that may be desired.
Platform 320 may receive flow records from collector module 355 and metrics from metrics collector 360. From this information, platform 320 may compile any desired variables or quantities characterizing the performance of individual applications accessed via its edge device 220-226. In exemplary operation, configuration processor and storage module 370 may process the flow records from collector module 355 to extract that flow data which is specific to one particular application and the particular edge device 220-226 it is accessed through. This extracted flow data may then be transmitted to analytics engine 375. In addition, metrics processor and storage module 365 may process metrics compiled by metrics collector 360 to add to the corresponding application- and edge-specific flow data determined by configuration processor 370. In one embodiment of the disclosure, configuration processor 370 may analyze destination data and/or application IDs (extracted from IP traffic by configuration collector 355, as above) to identify those flow records for a specific application and a specific edge device 220-226 through which the application is accessed. These specific flow records may then be copied, and analytics engine 375 may append the corresponding metrics from metrics processor 365 thereto. As above, appended metrics may include such quantities as RTT and packet loss for packets traveling to/from that specific application through that specific edge device. In this manner, analytics engine 375 may generate a variable containing a continuous record of flow data specific to particular applications and the specific edge devices through which they are accessed.
Analytics engine 375 then performs any desired analysis of these variables, i.e., application- and edge device-specific flow records and associated metrics, to determine whether the performance of a particular application or edge device has been compromised or is problematic in some manner. In some embodiments, analytics engine 375 may generate and transmit alerts 330 to users or any other desired entity when the variables indicate compromised performance of an application or edge device. Analytics engine 375 may also generate graphs 340 graphically illustrating variables or components thereof over time, as well as lists 350 of entities that may be impacted by compromised performance of specific applications or edge devices. In this manner, flow monitoring system 300 may identify specific applications and edge devices of an SD-WAN environment whose performance is compromised in some manner, and alert users or other desired entities accordingly.
Platform 320 may then extract data flow information for that particular edge device 220-226 and particular application programs accessed from that edge device 220-226 (Step 410). In some embodiments, platform 320 may extract, from the stream of IP traffic packet data generated by collector 310, that packet data which is specific to a particular application accessed from that edge device 220-226. This extracted flow data may then be deemed a data flow variable characterizing data flow of one particular application as accessed from one particular edge device (Step 420). Metrics from metrics processor 365 may be appended to these variables as desired, to improve the characterization of data flow for applications and their associated edge devices 220-226.
The process of
Analytics engine 375 may then analyze the variables to determine whether any show anomalous behavior of an application and/or edge device (Step 450). Variables may be analyzed in any manner, using any criteria, to determine whether they have detected anomalous behavior of any application or edge device. As one example, variables may be deemed to indicate anomalous behavior, and trigger an alert, when one or more of their values deviates from a predetermined baseline or historic average value by some predetermined amount that is deemed significant. For instance, a median absolute deviation technique may be employed to capture data flow variable values over any one or more desired periods of time, and determine the median and median absolute deviation thereof. Data flow variable values that exceed the median by greater than the median absolute deviation, or a predetermined multiple of the median absolute deviation, may be deemed to represent anomalous behavior and trigger transmission of an alert 330 (Step 460) or the like. As another example, variables may be deemed to indicate anomalous behavior, and trigger an alert, when one or more of their values exhibits a sudden change greater than some predetermined amount, or exceeds/falls below a predetermined threshold. Embodiments of the disclosure contemplate any method or approach for determining anomalous behavior from values of determined data flow variables.
If no anomalous behavior is detected, or after an alert is transmitted, the process returns to Step 400 to repeat Steps 400-450, so as to update data flow variables and continue monitoring of applications and edge devices. In some embodiments, the process of
In some SD-WAN applications, it is often desirable to quickly identify problematic application programs or edge devices, in order to restore them to proper operation and avoid compromising user experiences. One of ordinary skill in the art will observe that the above described data flow variables may be used to rapidly identify application programs or edge devices that are exhibiting anomalous behavior.
Analytics engine 375 then analyzes each data flow variable to detect anomalous behavior of each application or edge device. Analytics engine 375 may analyze these data flow variables in any manner. In some embodiments, analytics engine 375 may analyze the data flow variables for the same application across multiple edge devices (Step 510). That is, as one application may be accessed by different users through different edge devices, the analytics engine 375 may analyze the data flow variables corresponding to each different edge device's traffic for the same application. Accordingly, if anomalous behavior occurs for the same application on each different edge device, i.e., if the data flow variables for the same application display anomalous behavior across each edge device, analytics engine 375 may transmit an alert indicating that the performance of that application has been compromised (Step 520). In some embodiments, analytics engine 375 may also, or alternatively, compile and transmit a list 350 of entities impacted by the detected anomalous behavior. Subsequent remedial actions may then focus on that particular application, rather than first investigating particular edge devices or the like.
If no particular application displays anomalous behavior across multiple edge devices, analytics engine 375 may analyze the data flow variables corresponding to each different application's traffic on the same edge device (Step 530). If anomalous behavior occurs across applications on the same edge device, i.e., if the data flow variables for the same edge device display anomalous behavior across the traffic handled by that edge device, analytics engine 375 may transmit an alert indicating that the performance of that edge device has been compromised (Step 540). In some embodiments, analytics engine 375 may also, or alternatively, compile and transmit a list 350 of entities impacted by the detected anomalous behavior. Subsequent remedial actions may then focus on that particular edge device, rather than first investigating particular applications or the like.
As above, variables such as Edge1-App1 and Edge2-App2 may be used to quickly identify anomalous or problematic behavior of an application and/or edge device. In some embodiments, anomalous or problematic behavior may be determined from quantities such as packet drop and RTT. In some embodiments, anomalous behavior may be determined according to deviations in historical median values of these quantities. For instance, various sensitivity levels may be established, so that the system of embodiments of the disclosure may have greater or lesser sensitivity to changes in variable values before anomalous behavior is found and, e.g., an alert is issued. Sensitivity levels may be implemented in any manner. As one example, packet drop sensitivity may be based on percentage deviation in value from a historical median, while RTT sensitivity may be based on a number of standard deviations from a historical mean RTT value.
In one embodiment of the disclosure, packet drop sensitivity levels may be implemented as a high sensitivity having >1% increase in packet drop from a historic average or median value, with a medium sensitivity having >3% increase in packet drop and a low sensitivity requiring >5% increase in packet drop. Thus, for example, when systems of the disclosure are set to low sensitivity, a 5% increase in packet drop values of a variable such as Edge1-App1 would be required to trigger an alert, while a setting of high sensitivity would require only a 1% increase in packet drop to trigger an alert. It is noted that any number and type of sensitivity levels may be employed, each using any numerical value as its threshold percentage.
Similarly, RTT sensitivity levels may be implemented as a high sensitivity requiring >2 standard deviation increase in RTT from a historic mean or other average value, with a medium sensitivity requiring >2.5 standard deviation increase in RTT and a low sensitivity requiring >3 standard deviation increase to trigger an alert. It is noted that any number and type of sensitivity levels may be employed, each using any numerical value as its threshold number of standard deviations.
The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the described embodiments. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the described embodiments. Thus, the foregoing descriptions of specific embodiments are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. It will be apparent to one of ordinary skill in the art that many modifications and variations are possible in view of the above teachings.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202241042205 | Jul 2022 | IN | national |
