Patent Grant
11489720
Today, scaling and serving high influxes of traffic and requests is necessary in a rapidly growing world of Internet network infrastructure. Traffic patterns can vary depending on various factors such as application, time of day, region, etc., which has led to a transition to virtualization from traditional hardware appliances in order to cater to the varying traffic patterns. As public datacenters offered by multiple cloud service providers (CSPs) become more popular and widespread, virtual network functions (VNFs), and/or other types of tenant deployable elements, that were previously deployed on private datacenters are now being migrated to the CSPs, which offer various resource element types (e.g., resource elements that offer different compute, network, and storage options).
However, the performance metrics published by these CSPs are often simplistic and fall short of providing necessary information that is crucial to deployment and elasticity of the VNFs. As a result, several challenges arise including determining the appropriate resource element type to meet the performance needs of various VNFs, dimensioning the deployment (e.g., determining the number of instances of the resource element type needed and determining an availability set for fault tolerance), determining whether the published SLAs (service-level agreements) are adhered to, determining the scale-in/-out triggers for different resource element types, etc.
Some embodiments of the invention provide a method for evaluating multiple candidate resource elements that are candidates for deploying a set of one or more tenant deployable elements in a public cloud. For each particular tenant deployable element, the method deploys in the public cloud at least one instance of each of a set of one or more candidate resource elements and at least one agent to execute on the deployed resource element instance. The method communicates with each deployed agent to collect metrics for quantifying performance of the agent's respective resource element instance. The method then aggregates the collected metrics in order to generate a report that quantifies performance of each candidate resource element in the set of candidate resource elements for deploying the particular tenant deployable element in the public cloud.
In some embodiments, the generated reports are used for each particular tenant deployable element to select a candidate resource element to use to deploy the particular tenant deployable element in the public cloud. Also, in some embodiments, first and second types of candidate resource elements are candidates for one particular tenant deployable element, and by quantifying the performance of the first and second candidate resource elements, the report specifies either the first or second candidate resource element as a better resource element for deploying the particular tenant deployable element than the other candidate resource element. In addition to selecting which candidate resource element to deploy, some embodiments also use the generated report to determine a number of instances of the candidate resource element to deploy for the particular tenant deployable element in the public cloud. In some embodiments, to deploy the candidate resource element instance(s), a resource element instance is selected from a pool of pre-allocated resource elements in the public cloud, while in other embodiments, one or more new instances of the resource element are spun up for deployment.
The candidate resource elements, in some embodiments, also include different sub-types of candidate resource elements. In some embodiments, these different sub-types perform a same set of operations for the tenant deployable resource, but consume different amounts of resources on host computers, such as processor resources, memory resources, storage resources, and ingress/egress bandwidth. For example, in some embodiments, the tenant deployable element is a workload or service machine for execution on a host computer, and the different sub-types of candidate resource elements perform a set of operations of the workload or service machine, but consume different amounts of memory. The selected candidate resource element, in some embodiments, is selected based on whether these amounts meet a guaranteed SLA, or whether the number of instances of the selected candidate resource elements it takes to meet the SLA based on these amounts is fewer than the number of instances of other candidate resource elements it takes to meet the SLA. Alternatively, or conjunctively, different resource elements of the same resource element type, in some embodiments, perform different sets of operations.
The collected metrics, in some embodiments, include metrics such as throughput (e.g., in bits per second, in bytes per second, etc.), packets per second, connections per second, requests per second, transactions per second, transmission controller protocol (TCP) SYN arrival rate, number of open TCP connections, number of established TCP connections, and number of secure socket layer (SSL) transactions. In some embodiments, the metrics are collected based on a set of variables (e.g., variables specified in a request) such as cloud service provider (CSP) (e.g., Amazon AWS, Microsoft Azure, etc.), region, availability zone, resource element type, time of day, payload size, payload type, and encryption and authentication types. For example, the metrics in some embodiments may be collected for a particular resource element type in a public cloud provided by a particular CSP in a particular region during a particular time of day (e.g., during peak business hours for the particular region).
In some embodiments, the resource element types include compute resource elements (e.g., virtual machines (VMs), containers, middlebox service, nodes, and pods), networking resource elements (e.g., switches, routers, firewalls, load balancers, and network address translators (NATs)), and storage resource elements (e.g., databases, datastores, etc.). Examples of tenant deployable elements, in some embodiments, include load balancers, firewalls, intrusion detection systems, deep packet inspectors (DPIs), and network address translators (NATs).
In some embodiments, a controller or controller cluster directs each deployed agent to perform a set of performance-related tests on the agent's respective resource element instance to collect metrics associated with the agent's respective resource element instance. The controller cluster, in some embodiments, also configures each deployed agent to provide the collected metrics to the controller cluster, which aggregates the collected metrics to generate the report. In some embodiments, the controller cluster configures the agents to provide the collected metrics to the controller cluster by recording the metrics in a database accessible to the controller cluster so that the controller cluster can retrieve the metrics from the database for aggregation. In some such embodiments, the controller cluster stores the generated report in the database, and retrieves the generated report (and other reports) from the database in order to respond to requests for metrics, and requests to identify and deploy additional resource element instances in the public cloud and in other public clouds, according to some embodiments.
Also, in some embodiments, the controller cluster monitors the deployed resource elements and modifies these deployed resource elements based on evaluations of both real-time (i.e., current) and historical metrics. In some embodiments, the controller cluster modifies the deployed resource elements by scaling-up or scaling-down the number of instances of the deployed resource element. For example, the controller cluster scales-up or scales-down the number of instances periodically, in some embodiments, to ensure a guaranteed SLA is met during normal hours and during peak hours (i.e., by scaling-up the number of instances during peak hours, and scaling back down the number of instances during normal hours).
The controller cluster, in some embodiments, operates in the same public cloud as the agents, while in other embodiments, the controller cluster operates in another cloud (public or private). When the controller cluster operates in another cloud, in some embodiments, at least one agent is deployed in the other cloud and communicates with each other agent deployed in the public cloud to perform at least one performance-related test for which both agents (i.e., the agent in the public cloud and the agent in the other cloud) collect metric data.
In some embodiments, the deployed agents and the controller cluster implement a framework for evaluating a set of one or more public clouds and one or more resource elements in the set of public clouds as candidates for deploying tenant deployable elements. The requests, in some embodiments, are received from users through a user interface provided by the controller cluster. Alternatively, or conjunctively, the requests in some embodiments are received from network elements through a representational state transfer (REST) endpoint provided by the controller cluster.
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, the Detailed Description, the Drawings, and the Claims is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, the Detailed Description, and the Drawings.
The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Some embodiments of the invention provide a method for evaluating multiple candidate resource elements that are candidates for deploying a set of one or more tenant deployable elements in a public cloud. For each particular tenant deployable element, the method deploys in the public cloud at least one instance of each of a set of one or more candidate resource elements and at least one agent to execute on the deployed resource element instance. The method communicates with each deployed agent to collect metrics for quantifying performance of the agent's respective resource element instance. The method then aggregates the collected metrics in order to generate a report that quantifies performance of each candidate resource element in the set of candidate resource elements for deploying the particular tenant deployable element in the public cloud.
As illustrated, the virtual network 100 includes a controller 110 (or controller cluster) and a client resource 120 within the framework 105, and a virtual machine (VM) 125 within the public datacenter 140. The client resource 120 can be a client-controlled VM operating in the framework 105. While the controller and the client resource 120 are visually represented together within the framework 105, the controller and client resource in some embodiments are located at different sites. For example, the controller 110 in some embodiments may be located at a first private datacenter, while the client resource 120 is located at a second private datacenter.
The virtual network 100 in some embodiments is established for a particular entity. An example of an entity for which such a virtual network can be established includes a business entity (e.g., a corporation), a non-profit entity (e.g., a hospital, a research organization, etc.), and an education entity (e.g., a university, a college, etc.), or any other type of entity. Examples of public cloud providers include Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, etc., while examples of entities include a company (e.g., corporation, partnership, etc.), an organization (e.g., a school, a non-profit, a government entity, etc.), etc. In some embodiments, the virtual network 100 is a Software-Defined Wide Area Network (SDWAN) that span multiple different public cloud datacenters in different geographic locations.
The client resource 120 and the VM 125 in some embodiments can be resource elements of any resource element type and include various combinations of CPU (central processing unit), memory, storage, and networking capacity. While the client resource elements 120 and the VM 125 are illustrated and described herein as instances of VMs, in other embodiments, these resource can be containers, pods, compute nodes, and other types of VMs (e.g., service VMs). As shown, the client resource 120 include a data gathering (“DG”) agent 130 and the VM 125 includes a DG agent 135 (a DG agents is also referred to herein as a “agent”).
Additionally, the controller 110 includes an orchestration component 115. In some embodiments, the client resource 120, the VM 125, and the agents 130 and 135 are deployed by the orchestration component 115 of the controller 110 for the purpose of performing performance-related tests and collecting performance metrics (e.g., key performance indicators (KPIs)) during those tests. Also, in some embodiments, the orchestration component may deploy additional resource elements of a same resource element type, or different resource element type(s), in the public cloud datacenter 140, as well as in other public cloud datacenters (not shown), as will be further described below.
In some embodiments, the agents 130 and 135 perform individual tests at their respective sites, and perform tests between the sites along the connection links 150. Different performance-related tests can be used to measure different metrics, in some embodiments. Examples of different metrics that can be measured using the performance-related tests include throughput (e.g., in bits per second, bytes per second, etc.), packets per second, connections per second, requests per second, transactions per second, TCP SYN arrival rate, number of open TCP connections, number of established TCP connections, and secure sockets layer (SSL) transactions. In some embodiments, performance metrics other than those indicated herein may also be collected. Also, in some embodiments, different metric types can be collected for different types of resource elements. For instance, the metrics collected for a load balancer may be different by one or more metric types than the metrics collected for a DPI.
As the agents 130 and 135 perform the tests and collect metrics, they send the collected metrics to the controller 110 for aggregation and analysis, in some embodiments. In the network 100, the agents 130 and 135 are illustrated with links 155 leading back to the controller 110 along which the metrics are sent. While illustrated as individual connection links, the links 150 and 155 are sets of multiple connection links, with paths across these multiple connection links, in some embodiments.
In some embodiments, rather than sending the metrics directly to the controller, the agents push the collected metrics to a time-series database where the metrics are recorded and accessed by the controller for aggregation and publication.
As illustrated, the collected metrics include time of day, resource element type, region/zone, payload type, payload size, and encryption/authentication modes. In some embodiments, the collected metrics can include additional or fewer metrics than those shown, as well as different metrics than those shown. As the metrics are gathered in the public cloud datacenter 205, they are pushed to the time-series database 220 along the path 240, and recorded in the database.
Once the collected metrics have been recorded in the time-series database 220, the controller 230 can access the collected metrics to aggregate them, and record the aggregated metrics in the database. In some embodiments, the REST endpoint 234 of the controller 230 provides a front end for publishing information, and serves published REST APIs. Additionally, the UI 232 provides a way for users to query information and receive query results, as well as to subscribe and receive standard and/or custom alerts, according to some embodiments. In some embodiments, the information from the database is used for capacity planning, dimensioning, and defining scale-in/scale-out, especially during peak hours in order to efficiently manage both the load and resource elements.
In some embodiments, the queries can be directed toward specific metrics (e.g., time of day, resource element type, region/zone, payload type, payload size, and encryption/authentication modes). For example, a query might seek to determine the packets per second from a first resource element type belonging to a first CSP in a first region to a second resource element type of a second CSP in a second region during a specified time period (e.g., 8:00 AM to 11:00 AM). Additional query examples can include a query to determine the average connections per second for a particular resource element type during a specific month of the year, and a query to determine variance in throughput on a specific day of the week for a resource element instance that claims a particular speed.
The process 300 starts (at 310) by deploying at least one agent in each of multiple public cloud datacenters (PCDs). The controller in some embodiments deploys the agents in each PCD to execute on resource elements in each PCD. In some embodiments, the controller executes in a particular cloud datacenter, and deploys at least one agent to execute within that same particular cloud datacenter. The controller, agents, and resource elements on which the agents are deployed make up a data gathering and measurement framework.
The process communicates (at 320) with each deployed agent in each PCD to collect metrics for quantifying performance of each PCD for deploying a set of one or more resource elements. For example, the controller 110 in some embodiments communicates with the deployed agents in each PCD in order to direct the deployed agents to perform one or more performance-related tests and to collect metrics associated with the performance-related tests. In some embodiments, the controller also directs the at least one agent deployed within the same particular cloud datacenter as the controller to communicate with each other agent deployed in each other PCD to perform one or more performance-related tests to quantify performance of each PCD.
The process receives (at 330) collected metrics from the agents in each of the multiple PCDs. For example, in addition to performing performance-related tests and collecting metrics to quantify the performance of the PCDs and/or resource elements in the PCDs, each agent in some embodiments is configured to provide the collected metrics to the controller. As described above with reference to the traffic stream 200, the agents in some embodiments provide the collected metrics to the controller by recording the metrics in a time-series database for retrieval by the controller.
The process then aggregates (at 340) the collected metrics received from the deployed agents. The collected metrics, in some embodiments, are associated with the PCDs as well as resource elements deployed in the PCDs. For example, in some embodiments, the agents are deployed on different resource elements in the different PCDs, and collect metrics to quantify the performance of the different resource elements in the different PCDs, in addition to collecting metrics to quantify the performance of the different PCDs. In some embodiments, each deployed agent communicates with at least one other agent within the agent's respective PCD, and at least one other agent external to the agent's respective PCD, in order to collect metrics both inside and outside of the agent's respective PCD. The controller in some embodiments aggregates the collected metrics based on PCD association and/or resource element type association.
The process uses (at 350) the aggregated metrics in order to generate reports for analyzing in order to quantify the performance of each PCD. In some embodiments, the controller 230 stores the generated reports in the time-series database 220. The controller retrieves the generated reports from the time-series database in some embodiments for use in responding to queries for metrics associated with PCDs, resource elements, and/or a combination of PCDs and resource elements. The queries, in some embodiments, are received from users through the UI 232, or from network elements (e.g., other tenant deployable elements) through the REST endpoint 234.
The process then uses (at 360) the generated reports to deploy resource elements to the PCDs. In some embodiments, the process uses the generated reports to deploy resource elements to the PCDs according to requests to identify and deploy resource elements. Like the queries for metrics, the requests to identify and deploy resource elements to the PCDs can be received by the controller from users through a UI or from tenant deployable elements through a REST endpoint. Following 360, the process returns to 310 to continue deploying agents in different PCDs to continue collecting metrics.
When the controller determines at 420 that the information being queried is available, the process transitions to 430 to retrieve the queried information. The process then proceeds to 470. Otherwise, when the controller determines that the queried information is not available, the process transitions to 440 to direct agents to run tests to collect real-time metrics (i.e., current metrics) needed to measure and provide the queried information.
Next, the controller receives, at 450, the collected metrics. For example, the controller in some embodiments can retrieve the metrics from the database after the agents have pushed said metrics to the database. The controller then aggregates, at 460, the collected metrics with a set of historical metrics (e.g., also retrieved from the database) to measure and generate the requested information. For instance, the controller may aggregate the collected metrics with historical metrics associated with the same or similar resource element types.
After generating the requested information, the controller responds to the query at 470 with the requested information. When the source of the query is a tenant deployable element (e.g., a VNF or cloud-native network function), for example, the controller can respond via the REST endpoint. Alternatively, when the source of the query is a user, the controller can respond via the UI, according to some embodiments. The process 400 then ends.
The figure illustrates three different performance-related tests being performed by the framework 505. In a first test, the client resource element 520 has several connections 550 to the VM 522, and the framework determines the number of connections that the VM can handle per second. In performing this test, the client resource element 520 continues to send connection requests to the VM 522, in some embodiments, until the VM becomes overloaded. In some embodiments, this test is performed multiple times according to multiple different sets of parameters, and, as a result, can be used to calculate, e.g., the average number of connections per second a particular VM can handle (e.g., a threshold number of connections per second). As will be discussed further below, different types of resource elements can include different sub-types of the resource elements which consume different amounts of resources (e.g., host computer resources), in some embodiments. In some such embodiments, the different sub-types may be associated with different metrics.
In a second test between the client resource element 520 and the VM 524, multiple packets 560 are sent along the connection link 565. The framework in turn determines the number of packets per second that the link 565 or the VM 524 can handle. The client resource element 520 can continue to send multiple packets to the VM 524 until the VM becomes overloaded (e.g., when packets begin to drop). Like the first test, the framework can perform this second test according to different sets of parameters (e.g., for different resource element types, different regions, different time periods, etc.).
In a third test between the client resource element 520 and the VM 526, the client resource element is illustrated as sending a SYN message 570 to the VM 526 along the connection link 575. Timestamps T1 and T2 are shown on either end of the connection link 575 to represent the sent and received times of the SYN message, and are used to determine the SYN arrival rate.
As the agents 540-546 collect the metrics from these tests, the agents push the collected metrics to the controller (i.e., to the database) for aggregation. In some embodiments, each of the tests illustrated is performed for each of the VMs. Also, in some embodiments, the tests can be performed between the different VMs of the various CSPs to measure performance between CSPs.
In some embodiments, the controller 110 manages resource elements deployed in public cloud datacenters based on real-time and historical performance metrics associated with the resource elements. In some embodiments, the controller monitors a particular resource element deployed in a particular public cloud datacenter (PCD). The controller identifies a set of performance metric values that correspond to a specified subset of performance metric types that are associated with the particular resource element and the particular PCD (e.g., CPU usage by a VM running in the PCD). The controller evaluates the identified set of performance metric values based on a set of guaranteed performance metric values, and modifies the particular resource element based on the evaluation (e.g., by deploying additional resource element instances of the particular resource element.)
The current CPU usage in some embodiments is the current CPU usage by the resource element as reported in a cloud environment. In some embodiments, the detected application state changes are a result of CPU usage by the resource element exceeding a threshold. To make this determination, some embodiments compare current (i.e., real-time) CPU usage of the resource element with historical or baseline CPU usage for the resource element to identify anomalies/discrepancies in the current CPU usage.
When the process determines at 620 that the CPU usage of the resource element does not exceed the threshold, the process transitions to 650 to determine whether one or more characteristic metrics associated with the resource element exceed a threshold. Otherwise, when the process determines at 620 that the CPU usage of the resource element does exceed the threshold, the process transitions to 630 to scale-out the number of instances of the resource element deployed in the cloud environment (i.e., to help distribute the load). In some embodiments, the process scales-out the number of instances by spinning up additional instances of the resource element to deploy. Alternatively, or conjunctively, some embodiments select additional resource element instances from a pre-allocated pool of resource element instances. The process then transitions to 640 to determine whether the application state changes are persisting.
When the process determines at 640 that the application state changes are no longer persisting (i.e., scaling out the number of instances of the resource element has resolved the issue), the process ends. Otherwise, when the process determines at 640 that the application state changes are still persisting, the process transitions to 650 to determine whether one or more characteristic metrics of the resource element (e.g., time of day, resource element type, region/zone, payload type, payload size, and encryption/authentication modes) exceed a threshold. In some embodiments, the detected state change is due to exceeding a threshold associated with one or more key performance metrics specific to the traffic pattern being served by a particular instance of a resource element. For example, the controller in some embodiments can determine that a guaranteed SLA is not being met by a particular resource element type, and in turn, provide additional instances of that type of resource element in order to meet the guaranteed SLA.
When the process determines at 650 that no characteristic metrics exceed the threshold, the process transitions to 680 to adjust the resource element instance's current placement. Otherwise, when the process determines at 650 that one or more characteristic metrics have exceeded the threshold, the process transitions to 660 to scale-out the number of resource element instances. The process then determines, at 670, whether the application state changes are still persisting (i.e., despite the additional resource element instances). When the process determines at 670 that the state changes are no longer persisting, the process ends.
Alternatively, when the process determines at 670 that the state changes are persisting, the process transitions to 680 to adjust the current placement of the resource element instance(s). Some embodiments, for example, change a resource element instance's association from one host to another host (e.g., to mitigate connection issues experienced by the former host). Alternatively, or conjunctively, some embodiments adjust the placement of the resource element instance from one public cloud datacenter to another public cloud datacenter. As another alternative, some embodiments upgrade the resource element instance to a larger resource element instance on the same public cloud datacenter. After the resource element instance's current placement is adjusted at 680, the process ends.
In addition to responding to queries for different metrics and reports, the data gathering framework of some embodiments also receives and responds to queries directed to identifying resource element types for implementing tenant deployable elements and identifying public cloud datacenters in which instances of the identified resource element types should be deployed. For example, a query in some embodiments can include a request to identify a resource element type from a set of resource element types for deployment in one of two or more public cloud datacenters of two or more different CSPs. In some embodiments, the request specifies a set of criteria for identifying the resource element type and selecting the public cloud datacenter (e.g., the resource element type must be able to handle N number of connection requests per second).
The process 700 starts when the controller receives a request to deploy a resource element. The process selects (at 710) a particular resource element of a particular resource element type to deploy. In some embodiments, the process identifies the particular resource element of the particular resource element type to deploy by identifying a resource element type for implementing a particular tenant deployable element. Such a tenant deployable element in some embodiments may be a load balancer, a firewall, an intrusion detection system (IDS), a deep packet inspector (DPI), and network address translator (NAT).
The process identifies (at 720) a subset of metric types based on the particular resource element type to use to assess a set of public clouds for deploying the first resource element. In some embodiments, the subset of metric types is specified in the request to deploy the particular resource element, while in other embodiments, the process identifies from available or possible metric types a subset of metric types that are relevant to the particular resource element type as the subset of metric types.
The process retrieves (at 730) a particular set of metric values collected for the identified subset of metric types. In some embodiments, the metric values are retrieved by having one or more agents (e.g., the agents 540-546) perform the process 300 to collect the metrics or metric values associated with the particular resource element type. Alternatively, some embodiments retrieve the metric values from a database. The metric values collected by the agents, in some embodiments, include throughput (e.g., in bits per second, in bytes per second, etc.), packets per second, connections per second, requests per second, transactions per second, transmission control protocol (TCP) SYN arrival rate, number of open TCP connections, and number of established TCP connections.
The process uses the retrieved metric values to assess (at 740) the set of public clouds as candidate public clouds for deploying the selected resource element. In some embodiments, each candidate public cloud is assessed based on its own set of metric values for the identified subset of metric types for the particular resource element type (i.e., metric values collected for both the particular resource element type and the candidate public cloud). For example, in the virtual network 500 described above, the metrics collected by the agents 542-546 can include metrics associated with each VM 522-526 and their respective public clouds 532-536, in some embodiments.
Based on the assessment, the process selects (at 750) a particular public cloud from the set of public clouds for deploying the selected resource element. In some embodiments, the candidate public cloud having the best set of metric values for the identified subset of metric types for the selected resource element type compared to other candidate public clouds is selected. Alternatively, or conjunctively, the controller cluster in some embodiments provides the metrics to a user (e.g., network administrator) through a UI in the form of a report, and receives a selection through the UI from the user. In some embodiments, the selection includes an identifier for the selected public cloud.
The process deploys (at 760) the selected resource element of the particular resource element type to the selected particular public cloud. In some embodiments, the deployed particular resource element is a resource element instance selected from a pool of pre-allocated resource element instances of the particular resource element type in the selected public cloud. Alternatively, or conjunctively, some embodiments spin up new instances of the resource element for deployment.
The process then determines (at 770) whether there are any additional resource elements to evaluate for deployment. When the process determines that there are additional resource elements to evaluate, the process transitions to 780 to select another resource element. In some embodiments, the additional resource element selected for evaluation is a second resource element of a second resource element type. After selecting the second resource element, the process returns to 720 to identify a subset of metric types based on the second resource element. The subset of metric types identified for the second resource element may differ from the subset of metric types identified for the other resource element of the other type by at least one metric type, according to some embodiments. Additionally, in some embodiments, the second resource element performs different functions than the other resource element, while in other embodiments, the resource elements perform the same functions. In some embodiments, a second particular public cloud that is provided by a different CSP than the particular public cloud selected for the other resource element is then selected from the set of public clouds for deploying the second resource element of the second resource element type.
Returning to process 700, when the process instead determines (at 770) that there are no additional resource elements to evaluate, the process 700 ends. The data gathering and measurement framework described herein has many use cases, several of which are described above. To elaborate further on these novel use cases, and provide other novel use cases, additional novel processes for using the data gathering and measurement framework to intelligently deploy resources, and scale these resources, in a public cloud will be described below.
The resource elements, in some embodiments, are a second set of resource elements that is identical to a first set of resource elements that already exist in the public cloud. In some embodiments, the controller cluster deploys the second set of resource elements to collect metrics and use the metrics to test the environment (i.e., public cloud environment) and modify the first set of resource elements accordingly. For example, in some embodiments, the first and second sets of resource elements are first and second sets of machines that are similarly configured (i.e., the second set of machines are configured like the first set of machines) deployed on the same or similar host computers in the public cloud. In other embodiments, the resource elements are existing resource elements that are actively serving a particular tenant.
The process communicates (at 820) with the deployed agents to generate performance metrics regarding the set of resource elements. For example, in some embodiments, the controller cluster directs the agents to perform a set of performance-related tests in order to generate the performance metrics. In some embodiments, the controller cluster instructs the agents to perform specific tests to generate specific types of metrics (e.g., based on the type of resource elements in the set), while in other embodiments, the controller cluster instructs the agents to perform a set of default performance tests intended to capture a wide variety of metrics.
As described above, the agents perform the performance-related tests, in some embodiments, by communicating with other agents in other cloud datacenters. In some embodiments, the agents communicate with each other by sending data messages and collecting operational metrics related to the sent, and/or received, data messages. When the resource elements are the second set of resource elements corresponding to the existing first set of resource elements, in some embodiments, the data messages used in the performance tests are data messages similar to those sent and/or received by the existing first set of resource elements. In some embodiments, the data messages are sent to, and received from, other elements both inside of, and external to, the public cloud in which the resource elements are deployed.
The process then analyzes (at 830) the generated performance metrics. Each deployed resource element, in some embodiments, is associated with a guaranteed SLA, and the controller cluster, or a set of designated servers, analyzes the generated performance metrics by comparing guaranteed performance metric values specified by the SLA for the set of resource elements with the generated performance metrics to determine whether the guaranteed performance metric values are being met by the particular resource element. Alternatively, or conjunctively, the controller cluster in some embodiments analyzes the generated performance metrics by comparing them with historical performance metrics retrieved from a database (e.g., database 232) and associated with the set of resource elements and/or associated with other resource elements of the same type to identify fluctuations or changes in performance.
Based on the analysis, the process determines (at 840) whether any modifications to the deployment of the set of resource elements are needed. In some embodiments, for example, the controller cluster may determine that the performance of the set of resource elements has degraded, improved, or remained consistent when compared to historical performance metrics from the database. Similarly, the controller cluster in some embodiments may determine that the performance of the set of resource elements meets, does not meet, or exceeds a guaranteed SLA.
When the process determines (at 840) that no modifications to the deployment of the set of resource elements are needed (i.e., the analysis did not indicate performance issues), the process ends. Otherwise, when the process determines that modifications to the set of resource elements are needed, the process transitions to 850 to modify the deployment of the set of resources based on the analysis. As described above for the process 700, the set of resource elements can be modified by scaling out the number of instances of the resource elements in the set, in some embodiments, and/or by adjusting the placement of the particular resource element (e.g., by placing the particular resource element on another host). In some embodiments, the process 800 modifies the particular resource element by removing the particular resource element and replacing it with a different resource element. The different resource element, in some embodiments, is of a different resource element type, or a different resource element sub-type. Following 850, the process 800 ends.
The process then selects (at 920) a tenant deployable element from the set, and identifies (at 930) a set of one or more candidate resource elements for deploying the selected tenant deployable element in the public cloud. The candidate resource elements, in some embodiments, include different types of resource elements that are candidates for deploying the selected tenant deployable element. Examples of candidate resource elements of some embodiments include compute resource elements (e.g., virtual machines (VMs), containers, middlebox service, nodes, and pods), networking resource elements (e.g., switches, routers, firewalls, load balancers, and network address translators (NATs)), and storage resource elements (e.g., databases, datastores, etc.).
In some embodiments, the different types of candidate resource elements also include different sub-types of candidate resource elements. For example, the set of candidate resource elements for the selected tenant deployable element can include first and second candidate resource elements that are of the same type, and that perform the same set of operations of the selected tenant deployable element, but are considered different sub-types due to differences in the amounts of resources they consume (i.e., resources of the host computers on which they are deployed in the public cloud). These host-computer-resources in some embodiments include compute resources, memory resources, and storage resources.
In the public cloud, the process deploys (at 940) at least one instance of each of identified candidate resource element in the set and at least one agent to execute on the deployed resource element instance. The deployed agents, in some embodiments, are configured to run performance-related tests on their respective candidate resource elements in order to generate and collect performance-related metrics. In some embodiments, at least one agent is deployed in another cloud (e.g., a private cloud datacenter of the tenant) to allow for cross-cloud performance tests, such as testing the connections per second of a particular candidate resource element. The at least one agent in the other cloud is deployed in the same cloud as the controller cluster, in some embodiments.
The process communicates (at 950) with each deployed agent to collect metrics for quantifying performance of the agent's respective resource element instance. In some embodiments, communicating with the deployed agents includes configuring the agents to perform the tests mentioned above, and to provide metrics collected in associated with these test to the controller cluster. The agents of some embodiments are configured to provide the collected metrics to the controller cluster by recording the metrics to a database accessible to the controller cluster (e.g., as described above for
The process aggregates (at 960) the collected metrics in order to generate a report that quantifies performance of the agent's respective resource element instance. As described above, the collected metrics, in some embodiments, include metrics such as throughput (e.g., in bits per second, in bytes per second, etc.), packets per second, connections per second, requests per second, transactions per second, transmission controller protocol (TCP) SYN arrival rate, number of open TCP connections, number of established TCP connections, and number of secure socket layer (SSL) transactions. In some embodiments, the controller cluster stores the generated report in a database for later use.
Based on the generated report, the process selects (at 970) a candidate resource element from the set for deploying the selected tenant deployable element. In some embodiments, the controller cluster selects the candidate resource element based on criteria specified in the request to deploy the set of tenant deployable elements, or based on which candidate resource element is the best fit for meeting a guaranteed SLA. The selection, in some embodiments, also includes determining a number of instances of the candidate resource element to deploy for the selected tenant deployable element. Alternatively, or conjunctively, the controller cluster in some embodiments provides the generated report to a user (e.g., to network administrator through the UI) to allow the user to select which candidate resource element to deploy. In some such embodiments, the controller cluster may provide recommendations in the report as to which candidate resource element should be selected.
The process determines (at 980) whether there are any additional tenant deployable elements in the set to select. When the process determines that there are additional tenant deployable elements to select (i.e., for evaluating candidate resource elements for deploying the tenant deployable elements), the process returns to 920 to select a tenant deployable element from the set. Otherwise, when the process determines (at 980) that there are no additional tenant deployable resources in the set to select the process ends.
In some embodiments, rather than, or in addition to evaluating multiple candidate resource elements that are candidates for deploying multiple tenant deployable elements, the controller cluster performs the process 900 to evaluate multiple candidate resource elements for deploying a single tenant deployable element in a single public cloud.
The process identifies (at 1020) multiple candidate resource elements for implementing the particular tenant deployable element in each of the first and second public cloud datacenters. Multiple candidate resource elements exist in each of the first and second public cloud datacenters for the particular tenant deployable element, according to some embodiments, while in other embodiments, only one candidate resource element exists in either one, or both, of the datacenters. In some embodiments, the particular tenant deployable element is a VNF and all of the candidate resource elements are VMs. Alternatively, in some embodiments, the particular tenant deployable element is a cloud-native network function and the candidate resource elements are containers.
For each candidate resource element in the first public cloud datacenter, the process identifies (at 1030) a first set of performance metrics associated with the candidate resource element. For each candidate resource element in the second public cloud datacenter, the process identifies (at 1040) a second set of performance metrics associated with the candidate resource element. The performance metrics associated with the candidate resource elements, in some embodiments, are retrieved by the controller cluster from a database (e.g., the database 232).
In some embodiments, a particular candidate resource element that exists in both the first and second public cloud datacenters may be referred to differently within each public cloud datacenter. In some such embodiments, the controller cluster may include a mapping between the different names of the particular candidate resource element in order to ensure the correct metrics are retrieved. Also, in some embodiments, such as when no performance metrics associated with one or more of the candidate resources are stored in the database, or when the stored metrics do not include current metrics, the controller cluster performs the process 300 to collect performance metrics for each candidate resource element for which no performance metrics are stored.
The process evaluates (at 1050) the first and second sets of metrics to select a candidate resource element to implement the particular tenant deployable element in either the first or second public cloud datacenter. In some embodiments, the controller makes this selection based on which candidate resource element/public cloud datacenter combination has the best overall metrics, while in other embodiments, the controller makes this selection based on which candidate resource element/public cloud datacenter combination has the best metrics compared to a set of desired metrics or other criteria provided with the request to implement the particular tenant deployable resource. The specified criteria, in some embodiments, can include performance criteria (e.g., a specified threshold value or range for a particular performance metric), non-performance criteria (e.g., CSP identifier, region identifier, availability zone identifier, resource element type, time of day, payload size, payload type, and encryption and authentication types), or a combination of both performance and non-performance criteria.
The process then uses (at 1060) the selected resource element to implement the particular tenant deployable element in either the first or second public cloud datacenter. The process then ends. In some embodiments, rather than making the selection itself as part of an automated process, the controller cluster generates a report identifying the performance metrics associated with the candidate resource elements and provides the report to a user (e.g., to a network administrator through a UI) to enable the user to manually make a selection. The controller cluster in some embodiments may provide recommendations in the report as to which candidate resource element should be selected. In some such embodiments, the controller cluster receives an identifier of the user-selected resource element through the UI.
In the second stage 1102, each of the agents 1140-1146 are shown providing metrics (i.e., performance metrics collected by the agents during the test in stage 1101) to the controller 1110. While not shown, the agents in some embodiments provide the metrics to the controller by recording the metrics in a database accessible to the controller, as also described in some of the embodiments above. Also, while illustrated as being co-located in the same private cloud 1115, the controller 1110 and the client VM 1120 in other embodiments execute in different locations (e.g., different clouds, different datacenters, etc.). In still other embodiments, the controller 1110 executes in one of the cloud datacenters 1132-1136. Additionally, while this example illustrates VM instances being deployed, other embodiments can include other types of resource elements, such as containers and pods.
Next, the controller aggregates the received metrics in stage 1103 in order to select one of the cloud datacenters provided by one of the CSPs for deploying the VM (i.e., resource element). Finally, in stage 1104, the orchestration component 1112 of the controller 1110 deploys the VM instance 1124 in the selected cloud datacenter 1134, while the remaining cloud datacenters are illustrated with dotted lines to indicate they were not selected for deploying the VM instance.
Similar to
The resource element types in some embodiments include a variety of resource element types, while in other embodiments, the resource element types are resource element sub-types defined by an amount of resources consumed by the resource element (i.e., resources of the host computer on which the resource element executes). Examples of consumable resources include processing resources, storage resources, and memory resources, according to some embodiments. Accordingly, while the resource element instance types described in this example are illustrated and described as sub-types of VMs (i.e., VMs that consume different amounts of host-computer-resources), other embodiments include sub-types of other resource element types (e.g., sub-types of containers), while still other embodiments include a variety of different resource element types and resource element sub-types (e.g., a combination of VM instance sub-types and container instance sub-types).
In some embodiments, the resource element types depend on the type of tenant deployable element that the resource elements are implementing, and/or the types of operations performed by the tenant deployable element. For example, the tenant deployable element can be a workload or service machine, a forwarding element that has to be deployed on a machine executing on a host computer or deployed as a forwarding appliance, or a middlebox service element.
In the second stage 1202, each of the agents 1240-1248 are shown providing metrics (i.e., metrics collected during the tests in stage 1201) to the controller 1210 in the private cloud 1215. As described above for
In the next stage 1203, the controller aggregates the metrics received from the agents in order to select a VM type to deploy in the cloud datacenter 1230. In some embodiments, the controller aggregates the metrics and generates a report identifying the selected VM type and stores the report in the database for later use (e.g., to respond to queries for metrics). Also, in some embodiments, the controller provides the aggregated metrics to users through a UI (e.g., the UI 232 of the controller 200 described above) in response to users subscribing to receive metrics and reports.
In the final stage 1204, the orchestration component 1212 of the controller 1210 deploys a VM instance 1228 of the selected VM type 4 in the cloud datacenter 1230, while the other VM types are illustrated with dashed outlines to indicate these types were not selected for deployment. While only one instance of the VM 1228 is shown, multiple instances of the selected VM type are deployed in some embodiments.
As mentioned above for
The process identifies (at 1320) first and second sets of performance metric values for the first and second resource elements to evaluate. The first and second sets of performance metric values, in some embodiments, are metric values of the same metric types and are retrieved from a database by the controller cluster. In some embodiments, the controller cluster performs the process 900 in order to collect metric values for the candidate resource elements when there are no metric values associated with the candidate resource elements in the database.
The process evaluates (at 1330) the first and second sets of performance metric values. In some embodiments, the controller evaluates the first and second sets of performance metric values by comparing them to each other. Also, in some embodiments, the controller cluster compares the sets of performance metric values with a guaranteed SLA, or other criteria (e.g., other criteria specified in a request to deploy the tenant deployable element).
Based on this evaluation, the process selects (at 1340) the first or second candidate resource element to implement the tenant deployable element in the public cloud. The selected candidate resource element, in some embodiments, is the candidate resource element having the closest performance metrics to those specified in the guaranteed SLA, while in other embodiments, the selected candidate resource element is the candidate resource element that best matches the criteria specified in the request. In still other embodiments, the selected candidate resource element is the candidate resource element with the overall best performance metric values. Also, in some embodiments, the controller cluster provides the evaluated performance metrics to a user in the form of a report through a UI to enable the user to make the selection. The report, in some embodiments, includes a suggestion for which candidate resource element should be selected. In some embodiments, the controller cluster receives an identifier of the user-selected candidate resource element through the UI. The process then deploys (at 1350) the selected candidate resource element to implement the tenant deployable element in the public cloud. Following 1350, the process ends.
Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.
In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.
The bus 1405 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system 1400. For instance, the bus 1405 communicatively connects the processing unit(s) 1410 with the read-only memory 1430, the system memory 1425, and the permanent storage device 1435.
From these various memory units, the processing unit(s) 1410 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only-memory (ROM) 1430 stores static data and instructions that are needed by the processing unit(s) 1410 and other modules of the computer system. The permanent storage device 1435, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 1400 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 1435.
Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device. Like the permanent storage device 1435, the system memory 1425 is a read-and-write memory device. However, unlike storage device 1435, the system memory is a volatile read-and-write memory, such as random access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 1425, the permanent storage device 1435, and/or the read-only memory 1430. From these various memory units, the processing unit(s) 1410 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.
The bus 1405 also connects to the input and output devices 1440 and 1445. The input devices enable the user to communicate information and select commands to the computer system. The input devices 1440 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 1445 display images generated by the computer system. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as touchscreens that function as both input and output devices.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms “display” or “displaying” mean displaying on an electronic device. As used in this specification, the terms “computer readable medium,” “computer readable media,” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral or transitory signals.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202141027333 | Jun 2021 | IN | national |
| 202141027388 | Jun 2021 | IN | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5652751 | Sharony | Jul 1997 | A |
| 5909553 | Campbell et al. | Jun 1999 | A |
| 6154465 | Pickett | Nov 2000 | A |
| 6157648 | Voit et al. | Dec 2000 | A |
| 6201810 | Masuda et al. | Mar 2001 | B1 |
| 6363378 | Conklin et al. | Mar 2002 | B1 |
| 6445682 | Weitz | Sep 2002 | B1 |
| 6744775 | Beshai et al. | Jun 2004 | B1 |
| 6976087 | Westfall et al. | Dec 2005 | B1 |
| 7003481 | Banka et al. | Feb 2006 | B2 |
| 7280476 | Anderson | Oct 2007 | B2 |
| 7313629 | Nucci et al. | Dec 2007 | B1 |
| 7320017 | Kurapati et al. | Jan 2008 | B1 |
| 7373660 | Guichard et al. | May 2008 | B1 |
| 7581022 | Griffin et al. | Aug 2009 | B1 |
| 7680925 | Sathyanarayana et al. | Mar 2010 | B2 |
| 7681236 | Tamura et al. | Mar 2010 | B2 |
| 7962458 | Holenstein et al. | Jun 2011 | B2 |
| 8094575 | Vadlakonda et al. | Jan 2012 | B1 |
| 8094659 | Arad | Jan 2012 | B1 |
| 8111692 | Ray | Feb 2012 | B2 |
| 8141156 | Mao et al. | Mar 2012 | B1 |
| 8224971 | Miller et al. | Jul 2012 | B1 |
| 8228928 | Parandekar et al. | Jul 2012 | B2 |
| 8243589 | Trost et al. | Aug 2012 | B1 |
| 8259566 | Chen et al. | Sep 2012 | B2 |
| 8274891 | Averi et al. | Sep 2012 | B2 |
| 8301749 | Finklestein et al. | Oct 2012 | B1 |
| 8385227 | Downey | Feb 2013 | B1 |
| 8566452 | Goodwin et al. | Oct 2013 | B1 |
| 8630291 | Shaffer et al. | Jan 2014 | B2 |
| 8661295 | Khanna et al. | Feb 2014 | B1 |
| 8724456 | Hong et al. | May 2014 | B1 |
| 8724503 | Johnsson et al. | May 2014 | B2 |
| 8745177 | Kazerani et al. | Jun 2014 | B1 |
| 8799504 | Capone et al. | Aug 2014 | B2 |
| 8804745 | Sinn | Aug 2014 | B1 |
| 8806482 | Nagargadde et al. | Aug 2014 | B1 |
| 8855071 | Sankaran et al. | Oct 2014 | B1 |
| 8856339 | Mestery et al. | Oct 2014 | B2 |
| 8964548 | Keralapura et al. | Feb 2015 | B1 |
| 8989199 | Sella et al. | Mar 2015 | B1 |
| 9009217 | Nagargadde et al. | Apr 2015 | B1 |
| 9055000 | Ghosh et al. | Jun 2015 | B1 |
| 9060025 | Xu | Jun 2015 | B2 |
| 9071607 | Twitchell, Jr. | Jun 2015 | B2 |
| 9075771 | Gawali et al. | Jul 2015 | B1 |
| 9135037 | Petrescu-Prahova et al. | Sep 2015 | B1 |
| 9137334 | Zhou | Sep 2015 | B2 |
| 9154327 | Marino et al. | Oct 2015 | B1 |
| 9203764 | Shirazipour et al. | Dec 2015 | B2 |
| 9306949 | Richard et al. | Apr 2016 | B1 |
| 9323561 | Ayala et al. | Apr 2016 | B2 |
| 9336040 | Dong et al. | May 2016 | B2 |
| 9354983 | Yenamandra et al. | May 2016 | B1 |
| 9356943 | Lopilato et al. | May 2016 | B1 |
| 9379981 | Zhou et al. | Jun 2016 | B1 |
| 9413724 | Xu | Aug 2016 | B2 |
| 9419878 | Hsiao et al. | Aug 2016 | B2 |
| 9432245 | Sorenson et al. | Aug 2016 | B1 |
| 9438566 | Zhang et al. | Sep 2016 | B2 |
| 9450817 | Bahadur et al. | Sep 2016 | B1 |
| 9450852 | Chen et al. | Sep 2016 | B1 |
| 9462010 | Stevenson | Oct 2016 | B1 |
| 9467478 | Khan et al. | Oct 2016 | B1 |
| 9485163 | Fries et al. | Nov 2016 | B1 |
| 9521067 | Michael et al. | Dec 2016 | B2 |
| 9525564 | Lee | Dec 2016 | B2 |
| 9559951 | Sajassi et al. | Jan 2017 | B1 |
| 9563423 | Pittman | Feb 2017 | B1 |
| 9602389 | Maveli et al. | Mar 2017 | B1 |
| 9608917 | Anderson et al. | Mar 2017 | B1 |
| 9608962 | Chang | Mar 2017 | B1 |
| 9621460 | Mehta et al. | Apr 2017 | B2 |
| 9641551 | Kariyanahalli | May 2017 | B1 |
| 9648547 | Hart et al. | May 2017 | B1 |
| 9665432 | Kruse et al. | May 2017 | B2 |
| 9686127 | Ramachandran et al. | Jun 2017 | B2 |
| 9715401 | Devine et al. | Jul 2017 | B2 |
| 9717021 | Hughes et al. | Jul 2017 | B2 |
| 9722815 | Mukundan et al. | Aug 2017 | B2 |
| 9747249 | Cherian et al. | Aug 2017 | B2 |
| 9755965 | Yadav et al. | Sep 2017 | B1 |
| 9787559 | Schroeder | Oct 2017 | B1 |
| 9807004 | Koley et al. | Oct 2017 | B2 |
| 9819540 | Bahadur et al. | Nov 2017 | B1 |
| 9819565 | Djukic et al. | Nov 2017 | B2 |
| 9825822 | Holland | Nov 2017 | B1 |
| 9825911 | Brandwine | Nov 2017 | B1 |
| 9825992 | Xu | Nov 2017 | B2 |
| 9832128 | Ashner et al. | Nov 2017 | B1 |
| 9832205 | Santhi et al. | Nov 2017 | B2 |
| 9875355 | Williams | Jan 2018 | B1 |
| 9906401 | Rao | Feb 2018 | B1 |
| 9930011 | Clemons, Jr. et al. | Mar 2018 | B1 |
| 9935829 | Miller et al. | Apr 2018 | B1 |
| 9942787 | Tillotson | Apr 2018 | B1 |
| 10038601 | Becker et al. | Jul 2018 | B1 |
| 10057183 | Salle et al. | Aug 2018 | B2 |
| 10057294 | Xu | Aug 2018 | B2 |
| 10135789 | Mayya et al. | Nov 2018 | B2 |
| 10142226 | Wu et al. | Nov 2018 | B1 |
| 10178032 | Freitas | Jan 2019 | B1 |
| 10187289 | Chen et al. | Jan 2019 | B1 |
| 10200264 | Menon et al. | Feb 2019 | B2 |
| 10229017 | Zou et al. | Mar 2019 | B1 |
| 10237123 | Dubey et al. | Mar 2019 | B2 |
| 10250498 | Bales et al. | Apr 2019 | B1 |
| 10263832 | Ghosh | Apr 2019 | B1 |
| 10320664 | Nainar et al. | Jun 2019 | B2 |
| 10320691 | Matthews et al. | Jun 2019 | B1 |
| 10326830 | Singh | Jun 2019 | B1 |
| 10348767 | Lee et al. | Jul 2019 | B1 |
| 10355989 | Panchal et al. | Jul 2019 | B1 |
| 10425382 | Mayya et al. | Sep 2019 | B2 |
| 10454708 | Mibu | Oct 2019 | B2 |
| 10454714 | Mayya et al. | Oct 2019 | B2 |
| 10461993 | Turabi et al. | Oct 2019 | B2 |
| 10498652 | Mayya et al. | Dec 2019 | B2 |
| 10511546 | Singarayan et al. | Dec 2019 | B2 |
| 10523539 | Mayya et al. | Dec 2019 | B2 |
| 10550093 | Ojima et al. | Feb 2020 | B2 |
| 10554538 | Spohn et al. | Feb 2020 | B2 |
| 10560431 | Chen et al. | Feb 2020 | B1 |
| 10565464 | Han et al. | Feb 2020 | B2 |
| 10567519 | Mukhopadhyaya et al. | Feb 2020 | B1 |
| 10574528 | Mayya et al. | Feb 2020 | B2 |
| 10594516 | Cidon | Mar 2020 | B2 |
| 10594659 | El-Moussa et al. | Mar 2020 | B2 |
| 10608844 | Cidon et al. | Mar 2020 | B2 |
| 10637889 | Ermagan et al. | Apr 2020 | B2 |
| 10666460 | Cidon et al. | May 2020 | B2 |
| 10666497 | Tahhan | May 2020 | B2 |
| 10686625 | Cidon et al. | Jun 2020 | B2 |
| 10693739 | Naseri et al. | Jun 2020 | B1 |
| 10749711 | Mukundan et al. | Aug 2020 | B2 |
| 10778466 | Cidon et al. | Sep 2020 | B2 |
| 10778528 | Mayya et al. | Sep 2020 | B2 |
| 10805114 | Cidon et al. | Oct 2020 | B2 |
| 10805272 | Mayya et al. | Oct 2020 | B2 |
| 10819564 | Turabi et al. | Oct 2020 | B2 |
| 10826775 | Moreno et al. | Nov 2020 | B1 |
| 10841131 | Cidon et al. | Nov 2020 | B2 |
| 10911374 | Kumar et al. | Feb 2021 | B1 |
| 10938693 | Mayya et al. | Mar 2021 | B2 |
| 10951529 | Duan et al. | Mar 2021 | B2 |
| 10958479 | Cidon et al. | Mar 2021 | B2 |
| 10959098 | Cidon et al. | Mar 2021 | B2 |
| 10992558 | Silva et al. | Apr 2021 | B1 |
| 10992568 | Michael et al. | Apr 2021 | B2 |
| 10999100 | Cidon et al. | May 2021 | B2 |
| 10999137 | Cidon et al. | May 2021 | B2 |
| 10999165 | Cidon et al. | May 2021 | B2 |
| 11005684 | Cidon | May 2021 | B2 |
| 11018995 | Cidon et al. | May 2021 | B2 |
| 11044190 | Ramaswamy et al. | Jun 2021 | B2 |
| 11050588 | Mayya et al. | Jun 2021 | B2 |
| 11050644 | Hegde et al. | Jun 2021 | B2 |
| 11071005 | Shen et al. | Jul 2021 | B2 |
| 11089111 | Markuze et al. | Aug 2021 | B2 |
| 11095612 | Oswal et al. | Aug 2021 | B1 |
| 11102032 | Cidon et al. | Aug 2021 | B2 |
| 11108851 | Kurmala et al. | Aug 2021 | B1 |
| 11115347 | Gupta et al. | Sep 2021 | B2 |
| 11115426 | Pazhyannur et al. | Sep 2021 | B1 |
| 11115480 | Markuze et al. | Sep 2021 | B2 |
| 11121962 | Michael et al. | Sep 2021 | B2 |
| 11121985 | Cidon et al. | Sep 2021 | B2 |
| 11128492 | Sethi et al. | Sep 2021 | B2 |
| 11153230 | Cidon et al. | Oct 2021 | B2 |
| 11171885 | Cidon et al. | Nov 2021 | B2 |
| 11212140 | Mukundan et al. | Dec 2021 | B2 |
| 11212238 | Cidon et al. | Dec 2021 | B2 |
| 11223514 | Mayya et al. | Jan 2022 | B2 |
| 11245641 | Ramaswamy et al. | Feb 2022 | B2 |
| 11252079 | Michael et al. | Feb 2022 | B2 |
| 11252105 | Cidon et al. | Feb 2022 | B2 |
| 11252106 | Cidon et al. | Feb 2022 | B2 |
| 11258728 | Cidon et al. | Feb 2022 | B2 |
| 11381474 | Kumar et al. | Jul 2022 | B1 |
| 20020085488 | Kobayashi | Jul 2002 | A1 |
| 20020087716 | Mustafa | Jul 2002 | A1 |
| 20020198840 | Banka et al. | Dec 2002 | A1 |
| 20030061269 | Hathaway et al. | Mar 2003 | A1 |
| 20030088697 | Matsuhira | May 2003 | A1 |
| 20030112766 | Riedel et al. | Jun 2003 | A1 |
| 20030112808 | Solomon | Jun 2003 | A1 |
| 20030126468 | Markham | Jul 2003 | A1 |
| 20030161313 | Jinmei et al. | Aug 2003 | A1 |
| 20030189919 | Gupta et al. | Oct 2003 | A1 |
| 20030202506 | Perkins et al. | Oct 2003 | A1 |
| 20030219030 | Gubbi | Nov 2003 | A1 |
| 20040059831 | Chu et al. | Mar 2004 | A1 |
| 20040068668 | Lor et al. | Apr 2004 | A1 |
| 20040165601 | Liu et al. | Aug 2004 | A1 |
| 20040224771 | Chen et al. | Nov 2004 | A1 |
| 20050078690 | DeLangis | Apr 2005 | A1 |
| 20050154790 | Nagata et al. | Jul 2005 | A1 |
| 20050172161 | Cruz et al. | Aug 2005 | A1 |
| 20050195754 | Nosella | Sep 2005 | A1 |
| 20050265255 | Kodialam et al. | Dec 2005 | A1 |
| 20060002291 | Alicherry et al. | Jan 2006 | A1 |
| 20060114838 | Mandavilli et al. | Jun 2006 | A1 |
| 20060171365 | Borella | Aug 2006 | A1 |
| 20060182034 | Klinker et al. | Aug 2006 | A1 |
| 20060182035 | Vasseur | Aug 2006 | A1 |
| 20060193247 | Naseh et al. | Aug 2006 | A1 |
| 20060193252 | Naseh et al. | Aug 2006 | A1 |
| 20070064604 | Chen et al. | Mar 2007 | A1 |
| 20070064702 | Bates et al. | Mar 2007 | A1 |
| 20070083727 | Johnston et al. | Apr 2007 | A1 |
| 20070091794 | Filsfils et al. | Apr 2007 | A1 |
| 20070103548 | Carter | May 2007 | A1 |
| 20070115812 | Hughes | May 2007 | A1 |
| 20070121486 | Guichard et al. | May 2007 | A1 |
| 20070130325 | Lesser | Jun 2007 | A1 |
| 20070162639 | Chu et al. | Jul 2007 | A1 |
| 20070177511 | Das et al. | Aug 2007 | A1 |
| 20070237081 | Kodialam et al. | Oct 2007 | A1 |
| 20070260746 | Mirtorabi et al. | Nov 2007 | A1 |
| 20070268882 | Breslau et al. | Nov 2007 | A1 |
| 20080002670 | Bugenhagen et al. | Jan 2008 | A1 |
| 20080049621 | McGuire et al. | Feb 2008 | A1 |
| 20080055241 | Goldenberg et al. | Mar 2008 | A1 |
| 20080080509 | Khanna et al. | Apr 2008 | A1 |
| 20080095187 | Jung et al. | Apr 2008 | A1 |
| 20080117930 | Chakareski et al. | May 2008 | A1 |
| 20080144532 | Chamarajanagar et al. | Jun 2008 | A1 |
| 20080181116 | Kavanaugh et al. | Jul 2008 | A1 |
| 20080219276 | Shah | Sep 2008 | A1 |
| 20080240121 | Xiong et al. | Oct 2008 | A1 |
| 20090013210 | McIntosh et al. | Jan 2009 | A1 |
| 20090125617 | Kiessig et al. | May 2009 | A1 |
| 20090141642 | Sun | Jun 2009 | A1 |
| 20090154463 | Hines et al. | Jun 2009 | A1 |
| 20090247204 | Sennett et al. | Oct 2009 | A1 |
| 20090274045 | Meier et al. | Nov 2009 | A1 |
| 20090276657 | Wetmore et al. | Nov 2009 | A1 |
| 20090303880 | Maltz et al. | Dec 2009 | A1 |
| 20100008361 | Guichard et al. | Jan 2010 | A1 |
| 20100017802 | Lojewski | Jan 2010 | A1 |
| 20100046532 | Okita | Feb 2010 | A1 |
| 20100061379 | Parandekar et al. | Mar 2010 | A1 |
| 20100080129 | Strahan et al. | Apr 2010 | A1 |
| 20100088440 | Banks et al. | Apr 2010 | A1 |
| 20100091823 | Retana et al. | Apr 2010 | A1 |
| 20100107162 | Edwards et al. | Apr 2010 | A1 |
| 20100118727 | Draves et al. | May 2010 | A1 |
| 20100118886 | Saavedra | May 2010 | A1 |
| 20100165985 | Sharma et al. | Jul 2010 | A1 |
| 20100191884 | Holenstein et al. | Jul 2010 | A1 |
| 20100223621 | Joshi et al. | Sep 2010 | A1 |
| 20100226246 | Proulx | Sep 2010 | A1 |
| 20100290422 | Haigh et al. | Nov 2010 | A1 |
| 20100309841 | Conte | Dec 2010 | A1 |
| 20100309912 | Mehta et al. | Dec 2010 | A1 |
| 20100322255 | Hao et al. | Dec 2010 | A1 |
| 20100332657 | Elyashev et al. | Dec 2010 | A1 |
| 20110007752 | Silva et al. | Jan 2011 | A1 |
| 20110032939 | Nozaki et al. | Feb 2011 | A1 |
| 20110040814 | Higgins | Feb 2011 | A1 |
| 20110075674 | Li et al. | Mar 2011 | A1 |
| 20110107139 | Middlecamp et al. | May 2011 | A1 |
| 20110110370 | Moreno et al. | May 2011 | A1 |
| 20110141877 | Xu et al. | Jun 2011 | A1 |
| 20110142041 | Imai | Jun 2011 | A1 |
| 20110153909 | Dong | Jun 2011 | A1 |
| 20110235509 | Szymanski | Sep 2011 | A1 |
| 20110255397 | Kadakia et al. | Oct 2011 | A1 |
| 20120008630 | Ould-Brahim | Jan 2012 | A1 |
| 20120027013 | Napierala | Feb 2012 | A1 |
| 20120136697 | Peles et al. | May 2012 | A1 |
| 20120157068 | Eichen et al. | Jun 2012 | A1 |
| 20120173694 | Yan et al. | Jul 2012 | A1 |
| 20120173919 | Patel et al. | Jul 2012 | A1 |
| 20120182940 | Taleb et al. | Jul 2012 | A1 |
| 20120221955 | Raleigh et al. | Aug 2012 | A1 |
| 20120227093 | Shalzkamer et al. | Sep 2012 | A1 |
| 20120250682 | Vincent et al. | Oct 2012 | A1 |
| 20120250686 | Vincent et al. | Oct 2012 | A1 |
| 20120281706 | Agarwal et al. | Nov 2012 | A1 |
| 20120287818 | Corti et al. | Nov 2012 | A1 |
| 20120300615 | Kempf et al. | Nov 2012 | A1 |
| 20120307659 | Yamada | Dec 2012 | A1 |
| 20120317270 | Vrbaski et al. | Dec 2012 | A1 |
| 20120317291 | Wolfe | Dec 2012 | A1 |
| 20130019005 | Hui et al. | Jan 2013 | A1 |
| 20130021968 | Reznik et al. | Jan 2013 | A1 |
| 20130044764 | Casado et al. | Feb 2013 | A1 |
| 20130051237 | Ong | Feb 2013 | A1 |
| 20130051399 | Zhang et al. | Feb 2013 | A1 |
| 20130054763 | Merwe et al. | Feb 2013 | A1 |
| 20130086267 | Gelenbe et al. | Apr 2013 | A1 |
| 20130097304 | Asthana | Apr 2013 | A1 |
| 20130103834 | Dzerve et al. | Apr 2013 | A1 |
| 20130124718 | Griffith et al. | May 2013 | A1 |
| 20130124911 | Griffith et al. | May 2013 | A1 |
| 20130124912 | Griffith et al. | May 2013 | A1 |
| 20130128889 | Mathur et al. | May 2013 | A1 |
| 20130142201 | Kim et al. | Jun 2013 | A1 |
| 20130170354 | Takashima et al. | Jul 2013 | A1 |
| 20130173788 | Song | Jul 2013 | A1 |
| 20130182712 | Aguayo et al. | Jul 2013 | A1 |
| 20130185729 | Vasic | Jul 2013 | A1 |
| 20130191688 | Agarwal et al. | Jul 2013 | A1 |
| 20130238782 | Zhao et al. | Sep 2013 | A1 |
| 20130242718 | Zhang | Sep 2013 | A1 |
| 20130254599 | Katkar et al. | Sep 2013 | A1 |
| 20130258839 | Wang et al. | Oct 2013 | A1 |
| 20130258847 | Zhang et al. | Oct 2013 | A1 |
| 20130266015 | Qu et al. | Oct 2013 | A1 |
| 20130266019 | Qu et al. | Oct 2013 | A1 |
| 20130283364 | Chang et al. | Oct 2013 | A1 |
| 20130286846 | Atlas et al. | Oct 2013 | A1 |
| 20130297611 | Moritz et al. | Nov 2013 | A1 |
| 20130297770 | Zhang | Nov 2013 | A1 |
| 20130301469 | Suga | Nov 2013 | A1 |
| 20130301642 | Radhakrishnan et al. | Nov 2013 | A1 |
| 20130308444 | Sem-Jacobsen et al. | Nov 2013 | A1 |
| 20130315242 | Wang et al. | Nov 2013 | A1 |
| 20130315243 | Huang et al. | Nov 2013 | A1 |
| 20130329548 | Nakil et al. | Dec 2013 | A1 |
| 20130329601 | Yin et al. | Dec 2013 | A1 |
| 20130329734 | Chesla et al. | Dec 2013 | A1 |
| 20130346470 | Obstfeld et al. | Dec 2013 | A1 |
| 20140019604 | Twitchell, Jr. | Jan 2014 | A1 |
| 20140019750 | Dodgson et al. | Jan 2014 | A1 |
| 20140040975 | Raleigh et al. | Feb 2014 | A1 |
| 20140064283 | Balus et al. | Mar 2014 | A1 |
| 20140071832 | Johnsson et al. | Mar 2014 | A1 |
| 20140092907 | Sridhar et al. | Apr 2014 | A1 |
| 20140108665 | Arora et al. | Apr 2014 | A1 |
| 20140112171 | Pasdar | Apr 2014 | A1 |
| 20140115584 | Mudigonda et al. | Apr 2014 | A1 |
| 20140123135 | Huang et al. | May 2014 | A1 |
| 20140126418 | Brendel et al. | May 2014 | A1 |
| 20140156818 | Hunt | Jun 2014 | A1 |
| 20140156823 | Liu et al. | Jun 2014 | A1 |
| 20140164560 | Ko et al. | Jun 2014 | A1 |
| 20140164617 | Jalan et al. | Jun 2014 | A1 |
| 20140173113 | Vemuri et al. | Jun 2014 | A1 |
| 20140173331 | Martin et al. | Jun 2014 | A1 |
| 20140181824 | Saund et al. | Jun 2014 | A1 |
| 20140208317 | Nakagawa | Jul 2014 | A1 |
| 20140219135 | Li et al. | Aug 2014 | A1 |
| 20140223507 | Xu | Aug 2014 | A1 |
| 20140229210 | Sharifian et al. | Aug 2014 | A1 |
| 20140244851 | Lee | Aug 2014 | A1 |
| 20140258535 | Zhang | Sep 2014 | A1 |
| 20140269690 | Tu | Sep 2014 | A1 |
| 20140279862 | Dietz et al. | Sep 2014 | A1 |
| 20140280499 | Basavaiah et al. | Sep 2014 | A1 |
| 20140317440 | Biermayr et al. | Oct 2014 | A1 |
| 20140321277 | Lynn, Jr. et al. | Oct 2014 | A1 |
| 20140337500 | Lee | Nov 2014 | A1 |
| 20140341109 | Cartmell et al. | Nov 2014 | A1 |
| 20140372582 | Ghanwani et al. | Dec 2014 | A1 |
| 20150003240 | Drwiega et al. | Jan 2015 | A1 |
| 20150016249 | Mukundan et al. | Jan 2015 | A1 |
| 20150029864 | Raileanu et al. | Jan 2015 | A1 |
| 20150039744 | Niazi et al. | Feb 2015 | A1 |
| 20150046572 | Cheng et al. | Feb 2015 | A1 |
| 20150052247 | Threefoot et al. | Feb 2015 | A1 |
| 20150052517 | Raghu et al. | Feb 2015 | A1 |
| 20150056960 | Egner et al. | Feb 2015 | A1 |
| 20150058917 | Xu | Feb 2015 | A1 |
| 20150088942 | Shah | Mar 2015 | A1 |
| 20150089628 | Lang | Mar 2015 | A1 |
| 20150092603 | Aguayo et al. | Apr 2015 | A1 |
| 20150096011 | Watt | Apr 2015 | A1 |
| 20150124603 | Ketheesan et al. | May 2015 | A1 |
| 20150134777 | Onoue | May 2015 | A1 |
| 20150139238 | Pourzandi et al. | May 2015 | A1 |
| 20150146539 | Mehta et al. | May 2015 | A1 |
| 20150163152 | Li | Jun 2015 | A1 |
| 20150169340 | Haddad et al. | Jun 2015 | A1 |
| 20150172121 | Farkas et al. | Jun 2015 | A1 |
| 20150172169 | DeCusatis et al. | Jun 2015 | A1 |
| 20150188823 | Williams et al. | Jul 2015 | A1 |
| 20150189009 | Bemmel | Jul 2015 | A1 |
| 20150195178 | Bhattacharya et al. | Jul 2015 | A1 |
| 20150201036 | Nishiki et al. | Jul 2015 | A1 |
| 20150222543 | Song | Aug 2015 | A1 |
| 20150222638 | Morley | Aug 2015 | A1 |
| 20150236945 | Michael et al. | Aug 2015 | A1 |
| 20150236962 | Veres et al. | Aug 2015 | A1 |
| 20150244617 | Nakil et al. | Aug 2015 | A1 |
| 20150249644 | Xu | Sep 2015 | A1 |
| 20150257081 | Ramanujan et al. | Sep 2015 | A1 |
| 20150271056 | Chunduri et al. | Sep 2015 | A1 |
| 20150271104 | Chikkamath et al. | Sep 2015 | A1 |
| 20150271303 | Neginhal et al. | Sep 2015 | A1 |
| 20150281004 | Kakadia et al. | Oct 2015 | A1 |
| 20150312142 | Barabash et al. | Oct 2015 | A1 |
| 20150312760 | O'Toole | Oct 2015 | A1 |
| 20150317169 | Sinha et al. | Nov 2015 | A1 |
| 20150334025 | Rader | Nov 2015 | A1 |
| 20150334696 | Gu et al. | Nov 2015 | A1 |
| 20150341271 | Gomez | Nov 2015 | A1 |
| 20150349978 | Wu et al. | Dec 2015 | A1 |
| 20150350907 | Timariu et al. | Dec 2015 | A1 |
| 20150358236 | Roach et al. | Dec 2015 | A1 |
| 20150363221 | Terayama et al. | Dec 2015 | A1 |
| 20150363733 | Brown | Dec 2015 | A1 |
| 20150365323 | Duminuco et al. | Dec 2015 | A1 |
| 20150372943 | Hasan et al. | Dec 2015 | A1 |
| 20150372982 | Herle et al. | Dec 2015 | A1 |
| 20150381407 | Wang et al. | Dec 2015 | A1 |
| 20150381493 | Bansal et al. | Dec 2015 | A1 |
| 20160020844 | Hart et al. | Jan 2016 | A1 |
| 20160021597 | Hart et al. | Jan 2016 | A1 |
| 20160035183 | Buchholz et al. | Feb 2016 | A1 |
| 20160036924 | Koppolu et al. | Feb 2016 | A1 |
| 20160036938 | Aviles et al. | Feb 2016 | A1 |
| 20160037434 | Gopal et al. | Feb 2016 | A1 |
| 20160072669 | Saavedra | Mar 2016 | A1 |
| 20160072684 | Manuguri et al. | Mar 2016 | A1 |
| 20160080502 | Yadav et al. | Mar 2016 | A1 |
| 20160105353 | Cociglio | Apr 2016 | A1 |
| 20160105392 | Thakkar et al. | Apr 2016 | A1 |
| 20160105471 | Nunes et al. | Apr 2016 | A1 |
| 20160105488 | Thakkar et al. | Apr 2016 | A1 |
| 20160117185 | Fang et al. | Apr 2016 | A1 |
| 20160134461 | Sampath et al. | May 2016 | A1 |
| 20160134528 | Lin et al. | May 2016 | A1 |
| 20160134591 | Liao et al. | May 2016 | A1 |
| 20160142373 | Ossipov | May 2016 | A1 |
| 20160150055 | Choi | May 2016 | A1 |
| 20160164832 | Bellagamba et al. | Jun 2016 | A1 |
| 20160164914 | Madhav et al. | Jun 2016 | A1 |
| 20160173338 | Wolting | Jun 2016 | A1 |
| 20160191363 | Haraszti et al. | Jun 2016 | A1 |
| 20160191374 | Singh et al. | Jun 2016 | A1 |
| 20160192403 | Gupta et al. | Jun 2016 | A1 |
| 20160197834 | Luft | Jul 2016 | A1 |
| 20160197835 | Luft | Jul 2016 | A1 |
| 20160198003 | Luft | Jul 2016 | A1 |
| 20160210209 | Verkaik et al. | Jul 2016 | A1 |
| 20160212773 | Kanderholm et al. | Jul 2016 | A1 |
| 20160218947 | Hughes et al. | Jul 2016 | A1 |
| 20160218951 | Masseur et al. | Jul 2016 | A1 |
| 20160255169 | Kovvuri et al. | Sep 2016 | A1 |
| 20160261493 | Li | Sep 2016 | A1 |
| 20160261495 | Xia et al. | Sep 2016 | A1 |
| 20160261506 | Hegde et al. | Sep 2016 | A1 |
| 20160261639 | Xu | Sep 2016 | A1 |
| 20160269298 | Li et al. | Sep 2016 | A1 |
| 20160269926 | Sundaram | Sep 2016 | A1 |
| 20160285736 | Gu | Sep 2016 | A1 |
| 20160308762 | Teng et al. | Oct 2016 | A1 |
| 20160315912 | Mayya et al. | Oct 2016 | A1 |
| 20160323377 | Einkauf et al. | Nov 2016 | A1 |
| 20160328159 | Coddington et al. | Nov 2016 | A1 |
| 20160330111 | Manghirmalani et al. | Nov 2016 | A1 |
| 20160352588 | Subbarayan et al. | Dec 2016 | A1 |
| 20160353268 | Senarath et al. | Dec 2016 | A1 |
| 20160359738 | Sullenberger et al. | Dec 2016 | A1 |
| 20160366187 | Kamble | Dec 2016 | A1 |
| 20160371153 | Dornemann | Dec 2016 | A1 |
| 20160380886 | Blair et al. | Dec 2016 | A1 |
| 20160380906 | Hodique et al. | Dec 2016 | A1 |
| 20170005986 | Bansal et al. | Jan 2017 | A1 |
| 20170006499 | Hampel et al. | Jan 2017 | A1 |
| 20170012870 | Blair et al. | Jan 2017 | A1 |
| 20170019428 | Cohn | Jan 2017 | A1 |
| 20170026283 | Williams et al. | Jan 2017 | A1 |
| 20170026355 | Mathaiyan et al. | Jan 2017 | A1 |
| 20170034046 | Cai et al. | Feb 2017 | A1 |
| 20170034052 | Chanda et al. | Feb 2017 | A1 |
| 20170034129 | Sawant et al. | Feb 2017 | A1 |
| 20170048296 | Ramalho et al. | Feb 2017 | A1 |
| 20170053258 | Carney et al. | Feb 2017 | A1 |
| 20170055131 | Kong et al. | Feb 2017 | A1 |
| 20170063674 | Maskalik et al. | Mar 2017 | A1 |
| 20170063782 | Jain et al. | Mar 2017 | A1 |
| 20170063794 | Jain et al. | Mar 2017 | A1 |
| 20170064005 | Lee | Mar 2017 | A1 |
| 20170093625 | Pera et al. | Mar 2017 | A1 |
| 20170097841 | Chang et al. | Apr 2017 | A1 |
| 20170104653 | Badea et al. | Apr 2017 | A1 |
| 20170104755 | Arregoces et al. | Apr 2017 | A1 |
| 20170109212 | Gaurav et al. | Apr 2017 | A1 |
| 20170118173 | Arramreddy et al. | Apr 2017 | A1 |
| 20170123939 | Maheshwari et al. | May 2017 | A1 |
| 20170126516 | Tiagi et al. | May 2017 | A1 |
| 20170126564 | Mayya et al. | May 2017 | A1 |
| 20170134186 | Mukundan et al. | May 2017 | A1 |
| 20170134520 | Abbasi et al. | May 2017 | A1 |
| 20170139789 | Fries et al. | May 2017 | A1 |
| 20170142000 | Cai et al. | May 2017 | A1 |
| 20170149637 | Banikazemi et al. | May 2017 | A1 |
| 20170155557 | Desai et al. | Jun 2017 | A1 |
| 20170163473 | Sadana et al. | Jun 2017 | A1 |
| 20170171310 | Gardner | Jun 2017 | A1 |
| 20170180220 | Leckey | Jun 2017 | A1 |
| 20170181210 | Nadella et al. | Jun 2017 | A1 |
| 20170195161 | Ruel et al. | Jul 2017 | A1 |
| 20170195169 | Mills et al. | Jul 2017 | A1 |
| 20170201585 | Doraiswamy et al. | Jul 2017 | A1 |
| 20170207976 | Rovner et al. | Jul 2017 | A1 |
| 20170214545 | Cheng et al. | Jul 2017 | A1 |
| 20170214701 | Hasan | Jul 2017 | A1 |
| 20170223117 | Messerli et al. | Aug 2017 | A1 |
| 20170237710 | Mayya et al. | Aug 2017 | A1 |
| 20170257260 | Govindan et al. | Sep 2017 | A1 |
| 20170257309 | Appanna | Sep 2017 | A1 |
| 20170264496 | Ao et al. | Sep 2017 | A1 |
| 20170279717 | Bethers et al. | Sep 2017 | A1 |
| 20170279803 | Desai et al. | Sep 2017 | A1 |
| 20170280474 | Vesterinen et al. | Sep 2017 | A1 |
| 20170288987 | Pasupathy et al. | Oct 2017 | A1 |
| 20170289002 | Ganguli et al. | Oct 2017 | A1 |
| 20170295264 | Touitou et al. | Oct 2017 | A1 |
| 20170302565 | Ghobadi et al. | Oct 2017 | A1 |
| 20170310641 | Jiang et al. | Oct 2017 | A1 |
| 20170310691 | Vasseur et al. | Oct 2017 | A1 |
| 20170317954 | Masurekar et al. | Nov 2017 | A1 |
| 20170317969 | Masurekar et al. | Nov 2017 | A1 |
| 20170317974 | Masurekar et al. | Nov 2017 | A1 |
| 20170337086 | Zhu et al. | Nov 2017 | A1 |
| 20170339054 | Yadav et al. | Nov 2017 | A1 |
| 20170339070 | Chang et al. | Nov 2017 | A1 |
| 20170364419 | Lo | Dec 2017 | A1 |
| 20170366445 | Nemirovsky et al. | Dec 2017 | A1 |
| 20170366467 | Martin et al. | Dec 2017 | A1 |
| 20170373950 | Szilagyi | Dec 2017 | A1 |
| 20170374174 | Evens et al. | Dec 2017 | A1 |
| 20180006995 | Bickhart et al. | Jan 2018 | A1 |
| 20180007005 | Chanda et al. | Jan 2018 | A1 |
| 20180007123 | Cheng et al. | Jan 2018 | A1 |
| 20180013636 | Seetharamaiah et al. | Jan 2018 | A1 |
| 20180014051 | Phillips et al. | Jan 2018 | A1 |
| 20180020035 | Boggia et al. | Jan 2018 | A1 |
| 20180034668 | Mayya et al. | Feb 2018 | A1 |
| 20180041425 | Zhang | Feb 2018 | A1 |
| 20180062875 | Tumuluru | Mar 2018 | A1 |
| 20180062914 | Boutros et al. | Mar 2018 | A1 |
| 20180062917 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180063036 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180063193 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180063233 | Park | Mar 2018 | A1 |
| 20180063743 | Tumuluru et al. | Mar 2018 | A1 |
| 20180069924 | Tumuluru et al. | Mar 2018 | A1 |
| 20180074909 | Bishop et al. | Mar 2018 | A1 |
| 20180077081 | Lauer et al. | Mar 2018 | A1 |
| 20180077202 | Xu | Mar 2018 | A1 |
| 20180084081 | Kuchibhotla et al. | Mar 2018 | A1 |
| 20180097725 | Wood et al. | Apr 2018 | A1 |
| 20180114569 | Strachan et al. | Apr 2018 | A1 |
| 20180123910 | Fitzgibbon | May 2018 | A1 |
| 20180131608 | Jiang et al. | May 2018 | A1 |
| 20180131615 | Zhang | May 2018 | A1 |
| 20180131720 | Hobson et al. | May 2018 | A1 |
| 20180145899 | Rao | May 2018 | A1 |
| 20180159796 | Wang et al. | Jun 2018 | A1 |
| 20180159856 | Gujarathi | Jun 2018 | A1 |
| 20180167378 | Kostyukov et al. | Jun 2018 | A1 |
| 20180176073 | Dubey et al. | Jun 2018 | A1 |
| 20180176082 | Katz et al. | Jun 2018 | A1 |
| 20180176130 | Banerjee et al. | Jun 2018 | A1 |
| 20180213472 | Ishii et al. | Jul 2018 | A1 |
| 20180219765 | Michael et al. | Aug 2018 | A1 |
| 20180219766 | Michael et al. | Aug 2018 | A1 |
| 20180234300 | Mayya et al. | Aug 2018 | A1 |
| 20180260125 | Botes et al. | Sep 2018 | A1 |
| 20180262468 | Kumar et al. | Sep 2018 | A1 |
| 20180270104 | Zheng et al. | Sep 2018 | A1 |
| 20180278541 | Wu et al. | Sep 2018 | A1 |
| 20180287907 | Kulshreshtha et al. | Oct 2018 | A1 |
| 20180295101 | Gehrmann | Oct 2018 | A1 |
| 20180295529 | Jen et al. | Oct 2018 | A1 |
| 20180302286 | Mayya et al. | Oct 2018 | A1 |
| 20180302321 | Manthiramoorthy et al. | Oct 2018 | A1 |
| 20180307851 | Lewis | Oct 2018 | A1 |
| 20180316606 | Sung et al. | Nov 2018 | A1 |
| 20180351855 | Sood et al. | Dec 2018 | A1 |
| 20180351862 | Jeganathan et al. | Dec 2018 | A1 |
| 20180351863 | Vairavakkalai et al. | Dec 2018 | A1 |
| 20180351882 | Jeganathan et al. | Dec 2018 | A1 |
| 20180367445 | Bajaj | Dec 2018 | A1 |
| 20180373558 | Chang et al. | Dec 2018 | A1 |
| 20180375744 | Mayya et al. | Dec 2018 | A1 |
| 20180375824 | Mayya et al. | Dec 2018 | A1 |
| 20180375967 | Pithawala et al. | Dec 2018 | A1 |
| 20190013883 | Vargas et al. | Jan 2019 | A1 |
| 20190014038 | Ritchie | Jan 2019 | A1 |
| 20190020588 | Twitchell, Jr. | Jan 2019 | A1 |
| 20190020627 | Yuan | Jan 2019 | A1 |
| 20190028378 | Houjyo et al. | Jan 2019 | A1 |
| 20190028552 | Johnson et al. | Jan 2019 | A1 |
| 20190036808 | Shenoy et al. | Jan 2019 | A1 |
| 20190036810 | Michael et al. | Jan 2019 | A1 |
| 20190036813 | Shenoy et al. | Jan 2019 | A1 |
| 20190046056 | Khachaturian et al. | Feb 2019 | A1 |
| 20190058657 | Chunduri et al. | Feb 2019 | A1 |
| 20190058709 | Kempf et al. | Feb 2019 | A1 |
| 20190068470 | Mirsky | Feb 2019 | A1 |
| 20190068493 | Ram et al. | Feb 2019 | A1 |
| 20190068500 | Hira | Feb 2019 | A1 |
| 20190075083 | Mayya et al. | Mar 2019 | A1 |
| 20190103990 | Cidon et al. | Apr 2019 | A1 |
| 20190103991 | Cidon et al. | Apr 2019 | A1 |
| 20190103992 | Cidon et al. | Apr 2019 | A1 |
| 20190103993 | Cidon et al. | Apr 2019 | A1 |
| 20190104035 | Cidon et al. | Apr 2019 | A1 |
| 20190104049 | Cidon et al. | Apr 2019 | A1 |
| 20190104050 | Cidon et al. | Apr 2019 | A1 |
| 20190104051 | Cidon et al. | Apr 2019 | A1 |
| 20190104052 | Cidon et al. | Apr 2019 | A1 |
| 20190104053 | Cidon et al. | Apr 2019 | A1 |
| 20190104063 | Cidon et al. | Apr 2019 | A1 |
| 20190104064 | Cidon et al. | Apr 2019 | A1 |
| 20190104109 | Cidon et al. | Apr 2019 | A1 |
| 20190104111 | Cidon et al. | Apr 2019 | A1 |
| 20190104413 | Cidon et al. | Apr 2019 | A1 |
| 20190109769 | Jain et al. | Apr 2019 | A1 |
| 20190132221 | Boutros et al. | May 2019 | A1 |
| 20190140889 | Mayya et al. | May 2019 | A1 |
| 20190140890 | Mayya et al. | May 2019 | A1 |
| 20190158371 | Dillon et al. | May 2019 | A1 |
| 20190158605 | Markuze et al. | May 2019 | A1 |
| 20190199539 | Deng et al. | Jun 2019 | A1 |
| 20190220703 | Prakash et al. | Jul 2019 | A1 |
| 20190238364 | Boutros et al. | Aug 2019 | A1 |
| 20190238446 | Barzik et al. | Aug 2019 | A1 |
| 20190238449 | Michael et al. | Aug 2019 | A1 |
| 20190238450 | Michael et al. | Aug 2019 | A1 |
| 20190238483 | Marichetty et al. | Aug 2019 | A1 |
| 20190268421 | Markuze et al. | Aug 2019 | A1 |
| 20190268973 | Bull et al. | Aug 2019 | A1 |
| 20190280962 | Michael et al. | Sep 2019 | A1 |
| 20190280963 | Michael et al. | Sep 2019 | A1 |
| 20190280964 | Michael et al. | Sep 2019 | A1 |
| 20190306197 | Degioanni | Oct 2019 | A1 |
| 20190313907 | Khachaturian et al. | Oct 2019 | A1 |
| 20190319847 | Nahar et al. | Oct 2019 | A1 |
| 20190334813 | Raj et al. | Oct 2019 | A1 |
| 20190334820 | Zhao | Oct 2019 | A1 |
| 20190342219 | Liu et al. | Nov 2019 | A1 |
| 20190356736 | Narayanaswamy et al. | Nov 2019 | A1 |
| 20190364099 | Thakkar et al. | Nov 2019 | A1 |
| 20190364456 | Yu | Nov 2019 | A1 |
| 20190372888 | Michael et al. | Dec 2019 | A1 |
| 20190372889 | Michael et al. | Dec 2019 | A1 |
| 20190372890 | Michael et al. | Dec 2019 | A1 |
| 20190394081 | Tahhan | Dec 2019 | A1 |
| 20200014609 | Hockett et al. | Jan 2020 | A1 |
| 20200014615 | Michael et al. | Jan 2020 | A1 |
| 20200014616 | Michael et al. | Jan 2020 | A1 |
| 20200014661 | Mayya et al. | Jan 2020 | A1 |
| 20200014663 | Chen et al. | Jan 2020 | A1 |
| 20200021514 | Michael et al. | Jan 2020 | A1 |
| 20200021515 | Michael et al. | Jan 2020 | A1 |
| 20200036624 | Michael et al. | Jan 2020 | A1 |
| 20200044943 | Bor-Yaliniz et al. | Feb 2020 | A1 |
| 20200059420 | Abraham | Feb 2020 | A1 |
| 20200059459 | Abraham et al. | Feb 2020 | A1 |
| 20200092207 | Sipra et al. | Mar 2020 | A1 |
| 20200097327 | Beyer et al. | Mar 2020 | A1 |
| 20200099659 | Cometto et al. | Mar 2020 | A1 |
| 20200106696 | Michael et al. | Apr 2020 | A1 |
| 20200106706 | Mayya et al. | Apr 2020 | A1 |
| 20200119952 | Mayya et al. | Apr 2020 | A1 |
| 20200127905 | Mayya et al. | Apr 2020 | A1 |
| 20200127911 | Gilson et al. | Apr 2020 | A1 |
| 20200153701 | Mohan et al. | May 2020 | A1 |
| 20200153736 | Liebherr et al. | May 2020 | A1 |
| 20200162407 | Tillotson | May 2020 | A1 |
| 20200169473 | Rimar et al. | May 2020 | A1 |
| 20200177503 | Hooda et al. | Jun 2020 | A1 |
| 20200177550 | Valluri et al. | Jun 2020 | A1 |
| 20200177629 | Hooda et al. | Jun 2020 | A1 |
| 20200186471 | Shen et al. | Jun 2020 | A1 |
| 20200195557 | Duan et al. | Jun 2020 | A1 |
| 20200204460 | Schneider et al. | Jun 2020 | A1 |
| 20200213212 | Dillon et al. | Jul 2020 | A1 |
| 20200213224 | Cheng et al. | Jul 2020 | A1 |
| 20200218558 | Sreenath et al. | Jul 2020 | A1 |
| 20200235990 | Janakiraman et al. | Jul 2020 | A1 |
| 20200235999 | Mayya et al. | Jul 2020 | A1 |
| 20200236046 | Jain et al. | Jul 2020 | A1 |
| 20200244721 | S et al. | Jul 2020 | A1 |
| 20200252234 | Ramamoorthi et al. | Aug 2020 | A1 |
| 20200259700 | Bhalla et al. | Aug 2020 | A1 |
| 20200267184 | Vera-Schockner | Aug 2020 | A1 |
| 20200280587 | Janakiraman et al. | Sep 2020 | A1 |
| 20200287819 | Theogaraj et al. | Sep 2020 | A1 |
| 20200287976 | Theogaraj et al. | Sep 2020 | A1 |
| 20200296011 | Jain et al. | Sep 2020 | A1 |
| 20200296026 | Michael et al. | Sep 2020 | A1 |
| 20200314006 | Mackie et al. | Oct 2020 | A1 |
| 20200314614 | Moustafa et al. | Oct 2020 | A1 |
| 20200322230 | Natal et al. | Oct 2020 | A1 |
| 20200336336 | Sethi et al. | Oct 2020 | A1 |
| 20200344143 | Faseela et al. | Oct 2020 | A1 |
| 20200344163 | Gupta et al. | Oct 2020 | A1 |
| 20200351188 | Arora et al. | Nov 2020 | A1 |
| 20200358878 | Bansal et al. | Nov 2020 | A1 |
| 20200366530 | Mukundan et al. | Nov 2020 | A1 |
| 20200366562 | Mayya et al. | Nov 2020 | A1 |
| 20200382345 | Zhao et al. | Dec 2020 | A1 |
| 20200382387 | Pasupathy et al. | Dec 2020 | A1 |
| 20200412576 | Kondapavuluru et al. | Dec 2020 | A1 |
| 20200413283 | Shen et al. | Dec 2020 | A1 |
| 20210006482 | Hwang et al. | Jan 2021 | A1 |
| 20210006490 | Michael et al. | Jan 2021 | A1 |
| 20210029019 | Kottapalli | Jan 2021 | A1 |
| 20210029088 | Mayya et al. | Jan 2021 | A1 |
| 20210036888 | Makkalla et al. | Feb 2021 | A1 |
| 20210036987 | Mishra et al. | Feb 2021 | A1 |
| 20210067372 | Cidon et al. | Mar 2021 | A1 |
| 20210067373 | Cidon et al. | Mar 2021 | A1 |
| 20210067374 | Cidon et al. | Mar 2021 | A1 |
| 20210067375 | Cidon et al. | Mar 2021 | A1 |
| 20210067407 | Cidon et al. | Mar 2021 | A1 |
| 20210067427 | Cidon et al. | Mar 2021 | A1 |
| 20210067442 | Sundararajan et al. | Mar 2021 | A1 |
| 20210067461 | Cidon et al. | Mar 2021 | A1 |
| 20210067464 | Cidon et al. | Mar 2021 | A1 |
| 20210067467 | Cidon et al. | Mar 2021 | A1 |
| 20210067468 | Cidon et al. | Mar 2021 | A1 |
| 20210105199 | C H et al. | Apr 2021 | A1 |
| 20210112034 | Sundararajan et al. | Apr 2021 | A1 |
| 20210126830 | R. et al. | Apr 2021 | A1 |
| 20210126853 | Ramaswamy et al. | Apr 2021 | A1 |
| 20210126854 | Guo et al. | Apr 2021 | A1 |
| 20210126860 | Ramaswamy et al. | Apr 2021 | A1 |
| 20210144091 | C H et al. | May 2021 | A1 |
| 20210160169 | Shen et al. | May 2021 | A1 |
| 20210160813 | Gupta et al. | May 2021 | A1 |
| 20210184952 | Mayya et al. | Jun 2021 | A1 |
| 20210184966 | Ramaswamy et al. | Jun 2021 | A1 |
| 20210184983 | Ramaswamy et al. | Jun 2021 | A1 |
| 20210194814 | Roux et al. | Jun 2021 | A1 |
| 20210226880 | Ramamoorthy et al. | Jul 2021 | A1 |
| 20210234728 | Cidon et al. | Jul 2021 | A1 |
| 20210234775 | Devadoss et al. | Jul 2021 | A1 |
| 20210234786 | Devadoss et al. | Jul 2021 | A1 |
| 20210234804 | Devadoss et al. | Jul 2021 | A1 |
| 20210234805 | Devadoss et al. | Jul 2021 | A1 |
| 20210235312 | Devadoss et al. | Jul 2021 | A1 |
| 20210235313 | Devadoss et al. | Jul 2021 | A1 |
| 20210266262 | Subramanian et al. | Aug 2021 | A1 |
| 20210279069 | Salgaonkar et al. | Sep 2021 | A1 |
| 20210314289 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210328835 | Mayya et al. | Oct 2021 | A1 |
| 20210336880 | Gupta et al. | Oct 2021 | A1 |
| 20210377109 | Shrivastava et al. | Dec 2021 | A1 |
| 20210377156 | Michael et al. | Dec 2021 | A1 |
| 20210392060 | Silva et al. | Dec 2021 | A1 |
| 20210392070 | Tootaghaj et al. | Dec 2021 | A1 |
| 20210399920 | Sundararajan et al. | Dec 2021 | A1 |
| 20210399978 | Michael et al. | Dec 2021 | A9 |
| 20210400113 | Markuze et al. | Dec 2021 | A1 |
| 20210409277 | Jeuk | Dec 2021 | A1 |
| 20220006726 | Michael et al. | Jan 2022 | A1 |
| 20220006751 | Ramaswamy et al. | Jan 2022 | A1 |
| 20220006756 | Ramaswamy et al. | Jan 2022 | A1 |
| 20220035673 | Markuze et al. | Feb 2022 | A1 |
| 20220038370 | Vasseur et al. | Feb 2022 | A1 |
| 20220038557 | Markuze et al. | Feb 2022 | A1 |
| 20220094644 | Cidon et al. | Mar 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 1926809 | Mar 2007 | CN |
| 102577270 | Jul 2012 | CN |
| 102811165 | Dec 2012 | CN |
| 104956329 | Sep 2015 | CN |
| 106656847 | May 2017 | CN |
| 110447209 | Nov 2019 | CN |
| 111198764 | May 2020 | CN |
| 1912381 | Apr 2008 | EP |
| 3041178 | Jul 2016 | EP |
| 3509256 | Jul 2019 | EP |
| 2010233126 | Oct 2010 | JP |
| 2017059991 | Mar 2017 | JP |
| 2574350 | Feb 2016 | RU |
| 03073701 | Sep 2003 | WO |
| 2007016834 | Feb 2007 | WO |
| 2012167184 | Dec 2012 | WO |
| 2016061546 | Apr 2016 | WO |
| 2017083975 | May 2017 | WO |
| 2019070611 | Apr 2019 | WO |
| 2019094522 | May 2019 | WO |
| 2020012491 | Jan 2020 | WO |
| 2020018704 | Jan 2020 | WO |
| 2020091777 | May 2020 | WO |
| 2020101922 | May 2020 | WO |
| 2020112345 | Jun 2020 | WO |
| 2021040934 | Mar 2021 | WO |
| 2021118717 | Jun 2021 | WO |
| 2021150465 | Jul 2021 | WO |
| 2021211906 | Oct 2021 | WO |
| 2022005607 | Jan 2022 | WO |
| Entry |
|---|
| Alsaeedi, Mohammed, et al., “Toward Adaptive and Scalable OpenFlow-SDN Flow Control: A Survey,” IEEE Access, Aug. 1, 2019, 34 pages, vol. 7, IEEE, retrieved from https://ieeexplore.ieee.org/document/8784036. |
| Alvizu, Rodolfo, et al., “SDN-Based Network Orchestration for New Dynamic Enterprise Networking Services,” 2017 19th International Conference on Transparent Optical Networks, Jul. 2-6, 2017, 4 pages, IEEE, Girona, Spain. |
| Barozet, Jean-Marc, “Cisco SDWAN,” Deep Dive, Dec. 2017, 185 pages, Cisco, Retreived from https://www.coursehero.com/file/71671376/Cisco-SDWAN-Deep-Divepdf/. |
| Bertaux, Lionel, et al., “Software Defined Networking and Virtualization for Broadband Satellite Networks,” IEEE Communications Magazine, Mar. 18, 2015, 7 pages, vol. 53, IEEE, retrieved from https://ieeexplore.ieee.org/document/7060482. |
| Cox, Jacob H., et al., “Advancing Software-Defined Networks: A Survey,” IEEE Access, Oct. 12, 2017, 40 pages, vol. 5, IEEE, retrieved from https://ieeexplore.ieee.org/document/8066287. |
| Del Piccolo, Valentin, et al., “A Survey of Network Isolation Solutions for Multi-Tenant Data Centers,” IEEE Communications Society, Apr. 20, 2016, vol. 18, No. 4, 37 pages, IEEE. |
| Duan, Zhenhai, et al., “Service Overlay Networks: SLAs, QoS, and Bandwidth Provisioning,” IEEE/ACM Transactions on Networking, Dec. 2003, 14 pages, vol. 11, IEEE, New York, NY, USA. |
| Fortz, Bernard, et al., “Internet Traffic Engineering by Optimizing OSPF Weights,” Proceedings IEEE INFOCOM 2000, Conference on Computer Communications, Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies, Mar. 26-30, 2000, 11 pages, IEEE, Tel Aviv, Israel, Israel. |
| Francois, Frederic, et al., “Optimizing Secure SDN-enabled Inter-Data Centre Overlay Networks through Cognitive Routing,” 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), Sep. 19-21, 2016, 10 pages, IEEE, London, UK. |
| Guo, Xiangyi, et al., (U.S. Appl. No. 62/925,193), filed Oct. 23, 2019, 26 pages. |
| Huang, Cancan, et al., “Modification of Q.SD-WAN,” Rapporteur Group Meeting—Doc, Study Period 2017-2020, Q4/11-DOC1 (190410), Study Group 11, Apr. 10, 2019, 19 pages, International Telecommunication Union, Geneva, Switzerland. |
| Lasserre, Marc, et al., “Framework for Data Center (DC) Network Virtualization,” RFC 7365, Oct. 2014, 26 pages, IETF. |
| Li, Shengru, et al., “Source Routing with Protocol-oblivious Forwarding (POF) to Enable Efficient e-Health Data Transfers,” 2016 IEEE International Conference on Communications (ICC), May 22-27, 2016, 6 pages, IEEE, Kuala Lumpur, Malaysia. |
| Lin, Weidong, et al., “Using Path Label Routing in Wide Area Software-Defined Networks with Open Flow,” 2016 International Conference on Networking and Network Applications, Jul. 2016, 6 pages, IEEE. |
| Long, Feng, “Research and Application of Cloud Storage Technology in University Information Service,” Chinese Excellent Masters' Theses Full-text Database, Mar. 2013, 72 pages, China Academic Journals Electronic Publishing House, China. |
| Michael, Nithin, et al., “HALO: Hop-by-Hop Adaptive Link-State Optimal Routing,” IEEE/ACM Transactions on Networking, Dec. 2015, 14 pages, vol. 23, No. 6, IEEE. |
| Ming, Gao, et al., “A Design of SD-WAN-Oriented Wide Area Network Access,” 2020 International Conference on Computer Communication and Network Security (CCNS), Aug. 21-23, 2020, 4 pages, IEEE, Xi'an, China. |
| Mishra, Mayank, et al., “Managing Network Reservation for Tenants in Oversubscribed Clouds,” 2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems, Aug. 14-16, 2013, 10 pages, IEEE, San Francisco, CA, USA. |
| Mudigonda, Jayaram, et al., “NetLord: A Scalable Multi-Tenant Network Architecture for Virtualized Datacenters,” Proceedings of the ACM SIGCOMM 2011 Conference, Aug. 15-19, 2011, 12 pages, ACM, Toronto, Canada. |
| Non-Published Commonly Owned Related International Patent Application PCT/US2022/011729 with similar specification, filed Jan. 7, 2022, 49 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/103,614, filed Nov. 24, 2020, 38 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/143,092, filed Jan. 6, 2021, 42 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/143,094, filed Jan. 6, 2021, 42 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/194,038, filed Mar. 5, 2021, 35 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/227,016, filed Apr. 9, 2021, 37 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/227,044, filed Apr. 9, 2021, 37 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,327, filed Jun. 18, 2021, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,333, filed Jun. 18, 2021, 47 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,340, filed Jun. 18, 2021, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,342, filed Jun. 18, 2021, 47 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,345, filed Jun. 18, 2021, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/384,735, filed Jul. 24, 2021, 62 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/384,736, filed Jul. 24, 2021, 63 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/384,737, filed Jul. 24, 2021, 63 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/384,738, filed Jul. 24, 2021, 62 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/510,862, filed Oct. 26, 2021, 46 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/517,639, filed Nov. 2, 2021, 46 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/517,641, filed Nov. 2, 2021, 46 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/562,890, filed Dec. 27, 2021, 36 pages, Nicira, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/568,795, filed Jan. 5, 2022, 34 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/569,517 with similar specification, filed Jan. 6, 2022, 49 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/569,519 with similar specification, filed Jan. 6, 2022, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/569,522 with similar specification, filed Jan. 6, 2022, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/569,523 with similar specification, filed Jan. 6, 2022, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/569,524 with similar specification, filed Jan. 6, 2022, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/569,526, filed Jan. 6, 2022, 27 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/572,583, filed Jan. 10, 2022, 33 pages, Nicira, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 15/803,964, filed Nov. 6, 2017, 15 pages, The Mode Group. |
| Noormohammadpour, Mohammad, et al., “DCRoute: Speeding up Inter-Datacenter Traffic Allocation while Guaranteeing Deadlines,” 2016 IEEE 23rd International Conference on High Performance Computing (HiPC), Dec. 19-22, 2016, 9 pages, IEEE, Hyderabad, India. |
| PCT International Search Report and Written Opinion of Commonly Owned International Patent Application PCT/US2022/011729, dated Apr. 4, 2022, 12 pages, International Searching Authority (EPO). |
| Ray, Saikat, et al., “Always Acyclic Distributed Path Computation,” University of Pennsylvania Department of Electrical and Systems Engineering Technical Report, May 2008, 16 pages, University of Pennsylvania ScholarlyCommons. |
| Sarhan, Soliman Abd Elmonsef, et al., “Data Inspection in SDN Network,” 2018 13th International Conference on Computer Engineering and Systems (ICCES), Dec. 18-19, 2018, 6 pages, IEEE, Cairo, Egypt. |
| Tootaghaj, Diman Zad, et al., “Homa: An Efficient Topology and Route Management Approach in SD-WAN Overlays,” IEEE INFOCOM 2020—IEEE Conference on Computer Communications, Jul. 6-9, 2020, 10 pages, IEEE, Toronto, ON, Canada. |
| Webb, Kevin C., et al., “Blender: Upgrading Tenant-Based Data Center Networking,” 2014 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS), Oct. 20-21, 2014, 11 pages, IEEE, Marina del Rey, CA, USA. |
| Xie, Junfeng, et al., A Survey of Machine Learning Techniques Applied to Software Defined Networking (SDN): Research Issues and Challenges, IEEE Communications Surveys & Tutorials, Aug. 23, 2018, 38 pages, vol. 21, Issue 1, IEEE. |
| Yap, Kok-Kiong, et al., “Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering,” SIGCOMM '17: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, Aug. 21-25, 2017, 14 pages, Los Angeles, CA. |
| Barozet, Jean-Marc, “Cisco SD-WAN as a Managed Service,” BRKRST-2558, Jan. 27-31, 2020, 98 pages, Cisco, Barcelona, Spain, retrieved from https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKRST-2558.pdf. |
