Methods for application defined virtual network service among multiple transport in SD-WAN

Information

  • Patent Grant
  • 11729065
  • Patent Number
    11,729,065
  • Date Filed
    Thursday, May 5, 2022
    2 years ago
  • Date Issued
    Tuesday, August 15, 2023
    10 months ago
Abstract
Some embodiments provide a method of selecting data links for an application in a network. The method receives, from a machine implementing the application, a set of identifiers of required link characteristics. Based on at least one of the identifiers, the method selects a transport group that includes a set of optional links matching the identifiers. From the selected transport group, the method selects a link matching the set of identifiers.
Description

In recent years, datacenters have come to use a large variety of types of data links to connect to external networks. These links may be wired or wireless. They may use specific types of wired connection such as Ethernet, DSL, etc. These links may have other distinguishing characteristics such as available bandwidth, expected error rates, etc. However, in the prior art, datacenters have not been able to provide applications running on machines of the data center (e.g., running on machines implemented by hosts of the datacenter) with an option to identify particular link characteristics that the application requires for most efficient functioning. Accordingly, there is a need in the art for a method that allows applications to define their link requirements and to identify a link for the application to use from groups of links selected for some defining characteristic(s).


BRIEF SUMMARY

Some embodiments provide a method of selecting data links for an application in a network. The method receives, from a machine implementing the application, a set of identifiers of required link characteristics. In some embodiments, the applications hosted on the machines leverage the Generic Networking Virtualization Encapsulation (Geneve) protocol to specify custom messages in the type-length-value (TLV) format, encapsulating each of the application packets, which can carry a specification of link characteristics towards edge forwarding nodes. Based on at least one of the identifiers, the method selects a transport group that includes a set of optional links matching the identifiers. From the selected transport group, the method selects a link matching the set of identifiers.


The method of various embodiments selects the transport group based on different characteristics. For example, the method may select the transport group based on one or more identifiers of: (1) a security and/or encryption characteristic of links in the transport group, (2) a filtering capability of links in the transport group, (3) an identifier of a presence of proxies on the links in the transport group, (4) an identifier of a lowest maximum transmission unit (MTU) size of links of the transport group, (5) a minimum throughput of the links of the transport group, (6) a maximum rate of packet drops of links of the transport group, and/or (7) a maximum rate of interface errors of links of the transport group.


The method of some embodiments also sends an identifier of the application and the selected link for the application to an edge node of the logical network. The edge node may then identify packets of the application and route the packets of the application to the selected link.


The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, the Detailed Description, the Drawings, and the Claims is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, the Detailed Description, and the Drawings.





BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.



FIG. 1 illustrates an example of a virtual network that is created for a particular entity using a hub that is deployed in a public cloud datacenter of a public cloud provider.



FIG. 2 illustrates a datacenter with multiple links to one or more external networks.



FIG. 3 conceptually illustrates a process of some embodiments for providing transport group identifiers to link policy analyzers of edge nodes.



FIG. 4 conceptually illustrates a process of some embodiments for selecting a link for an application.



FIG. 5 illustrates the multiple applications using links selected by a link policy analyzer.



FIG. 6 illustrates a link requirement set of some embodiments.



FIG. 7 illustrates a set of transport group identifiers of some embodiments.



FIG. 8 conceptually illustrates a computer system with which some embodiments of the invention are implemented.





DETAILED DESCRIPTION

In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.


Some embodiments provide a method of selecting data links for an application in a network. The method receives, from a machine implementing the application, a set of identifiers of required link characteristics. In some embodiments, the applications hosted on the machines leverage the Geneve protocol to specify custom messages in the TLV format, encapsulating each of the application packets, which can carry a specification of link characteristics towards edge forwarding nodes. Based on at least one of the identifiers, the method selects a transport group that includes a set of optional links matching the identifiers. From the selected transport group, the method selects a link matching the set of identifiers.


The method of various embodiments selects the transport group based on different characteristics. For example, the method may select the transport group based on one or more identifiers of: (1) a security and/or encryption characteristic of links in the transport group, (2) a filtering capability of links in the transport group, (3) an identifier of a presence of proxies on the links in the transport group, (4) an identifier of a lowest maximum transmission unit (MTU) size of links of the transport group, (5) a minimum throughput of the links of the transport group, (6) a maximum rate of packet drops of links of the transport group, and/or (7) a maximum rate of interface errors of links of the transport group.


The method of some embodiments also sends an identifier of the application and the selected link for the application to an edge node of the logical network. The edge node may then identify packets of the application and route the packets of the application to the selected link.


As used in this document, data messages refer to a collection of bits in a particular format sent across a network. One of ordinary skill in the art will recognize that the term data message may be used herein to refer to various formatted collections of bits that may be sent across a network, such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Also, as used in this document, references to L2, L3, L4, and L7 layers (or layer 2, layer 3, layer 4, layer 7) are references, respectively, to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the OSI (Open System Interconnection) layer model.



FIG. 1 illustrates an example of a virtual network 100 that is created for a particular entity using SD-WAN forwarding elements deployed at branch sites (sometimes called “branches” or “sites”), datacenters, and public clouds. Examples of entities include a company (e.g., corporation, partnership, etc.), an organization (e.g., a school, a non-profit, a government entity, etc.), etc. In some embodiments, an SD-WAN is the application of software based network technologies that virtualize WAN connections. An SD-WAN decouples network software services from underlying hardware to create a virtualized network overlay. An SD-WAN in some embodiments connects different sites (e.g., different buildings, or locations in different neighborhood, cities, states, countries, continents, etc.) through software tools that deploy forwarding elements in the cloud (e.g., one or more public clouds, such as public clouds provided by Amazon Web Services (AWS), Microsoft, Google, and/or one or more private clouds) and connect these forwarding elements through route records.


The SD-WANs of some embodiments employ a hub and spoke architecture, in which the hubs serve as focal/intermediary points for connecting edge forwarding elements (in some embodiments, the edge forwarding elements could be routers, gateways, or other routing devices) at branch sites that serve as the spokes of the SD-WAN architecture. The branches themselves may be implemented as sites to support manufacturing, Points of Sale (POS), medical facilities such as hospitals and clinics, or other scenarios. In some embodiments, hubs act as a central point of management for some or all connected branch sites. Hubs of some embodiments are set up by a centralized management plane orchestrator. The orchestrator notifies all the edge forwarding elements on the branches about the hubs, and the edge forwarding elements build secure overlay (in some embodiments, multi-path) tunnels to the hubs. The hubs themselves include edge forwarding elements, typically deployed in datacenters to allow branches to access the datacenters' resources and to route traffic within and outside the SD-WAN.


The edge forwarding elements connect to each other either directly or through a hub (meaning traffic from one branch site would go through that site's edge forwarding element to a hub forwarding element at a datacenter, and this hub forwarding element would then relay the traffic to another branch site through that site's edge forwarding element). Similarly, in some embodiments, traffic from branches passes through a hub, then out of the SD-WAN, over an external network to an external (outside the SD-WAN) machine.


In FIG. 1, the SD-WAN forwarding elements include cloud gateway 105 and SD-WAN forwarding elements 130, 132, 134, and 136. The cloud gateway (CGW) in some embodiments is a forwarding element that is in a private or public datacenter 110. The CGW 105 in some embodiments has secure connection links (e.g., tunnels) with edge forwarding elements (e.g., SD-WAN edge forwarding elements (FEs) 130, 132, 134, and 136) at the particular entity's multi-machine sites (e.g., SD-WAN edge sites 120, 122, and 124), such as branch offices, datacenters, etc. These multi-machine sites are often at different physical locations (e.g., different buildings, different cities, different states, etc.) and are referred to below as multi-machine sites or nodes.


Four multi-machine sites 120-126 are illustrated in FIG. 1, with three of them being branch sites 120-124, and one being a datacenter 126. Each branch site is shown to include an edge forwarding node 130-134, while the datacenter site 126 is shown to include a hub forwarding node 136. Each branch site 120-124 and the datacenter 126 includes resources 150-156 respectively. These resources 150-156 may include servers, hosts, routers, switches, and/or other physical or logical elements (e.g., virtual machines, containers, etc.). The resources 150-156 may communicate with resources of other branches and/or other resources outside of their own site through the forwarding elements 130-136, respectively. The datacenter SD-WAN forwarding node 136 is also referred to as a hub node 136 because in some embodiments this forwarding node can be used to connect (e.g., through a VPN tunnel) to other edge forwarding nodes of the branch sites 120-124. The hub node 136 in some embodiments provides services (e.g., middlebox services) for packets that it forwards from one branch site to another branch site. The hub node 136 also provides access to the datacenter resources 156, as further described below.


Each edge forwarding element (e.g., SD-WAN edge FEs 130-134) exchanges data messages with one or more cloud gateways 105 through one or more connection links 115 (e.g., multiple connection links available at the edge forwarding element). In some embodiments, these connection links include secure and unsecure connection links, while in other embodiments they only include secure connection links. As shown by edge node 134 and gateway 105, multiple secure connection links (e.g., multiple secure tunnels that are established over multiple physical links) can be established between one edge node and a gateway.


When multiple such links are defined between an edge node and a gateway, each secure connection link in some embodiments is associated with a different physical network link between the edge node and an external network. For instance, to access external networks, an edge node in some embodiments has one or more commercial broadband Internet links (e.g., a cable modem, a fiber optic link) to access the Internet, an MPLS (multiprotocol label switching) link to access external networks through an MPLS provider's network, and/or a wireless cellular link (e.g., a 5G LTE network). In some embodiments, the different physical links between the edge node 134 and the cloud gateway 105 are the same type of links (e.g., are different MPLS links).


In some embodiments, one edge forwarding node 130-134 can also have multiple direct links 115 (e.g., secure connection links established through multiple physical links) to another edge forwarding node 130-134, and/or to a datacenter hub node 136. Again, the different links in some embodiments can use different types of physical links or the same type of physical links. Also, in some embodiments, a first edge forwarding node of a first branch site can connect to a second edge forwarding node of a second branch site (1) directly through one or more links 115, or (2) through a cloud gateway or datacenter hub to which the first edge forwarding node connects through two or more links 115. Hence, in some embodiments, a first edge forwarding node (e.g., 134) of a first branch site (e.g., 124) can use multiple SD-WAN links 115 to reach a second edge forwarding node (e.g., 130) of a second branch site (e.g., 120), or a hub forwarding node 136 of a datacenter site 126.


The cloud gateway 105 in some embodiments is used to connect two SD-WAN forwarding nodes 130-136 through at least two secure connection links 115 between the gateway 105 and the two forwarding elements at the two SD-WAN sites (e.g., branch sites 120-124 or datacenter site 126). In some embodiments, the cloud gateway 105 also provides network data from one multi-machine site to another multi-machine site (e.g., provides the accessible subnets of one site to another site). Like the cloud gateway 105, the hub forwarding element 136 of the datacenter 126 in some embodiments can be used to connect two SD-WAN forwarding nodes 130-134 of two branch sites through at least two secure connection links 115 between the hub 136 and the two forwarding elements at the two branch sites 120-124.


In some embodiments, each secure connection link between two SD-WAN forwarding nodes (i.e., CGW 105 and edge forwarding nodes 130-136) is formed as a VPN (virtual private network) tunnel between the two forwarding nodes. In this example, the collection of the SD-WAN forwarding nodes (e.g., forwarding elements 130-136 and cloud gateway 105) and the secure connections between the forwarding nodes forms the virtual network 100 for the particular entity that spans at least public or private cloud datacenter 110 to connect the branch and datacenter sites 120-126.


In some embodiments, secure connection links are defined between gateways in different public cloud datacenters to allow paths through the virtual network to traverse from one public cloud datacenter to another, while no such links are defined in other embodiments. Also, in some embodiments, the cloud gateway 105 is a multi-tenant gateway that is used to define other virtual networks for other entities (e.g., other companies, organizations, etc.). Some such embodiments use tenant identifiers to create tunnels between a gateway and edge forwarding element of a particular entity, and then use tunnel identifiers of the created tunnels to allow the gateway to differentiate data message flows that it receives from edge forwarding elements of one entity from data message flows that it receives along other tunnels of other entities. In other embodiments, gateways are single-tenant and are specifically deployed to be used by just one entity.



FIG. 1 illustrates a cluster of controllers 140 that serves as a central point for managing (e.g., defining and modifying) configuration data that is provided to the edge nodes 130-0136 and/or gateways to configure some or all of the operations. In some embodiments, this controller cluster 140 is in one or more public cloud datacenters, while in other embodiments it is in one or more private datacenters. In some embodiments, the controller cluster 140 has a set of manager servers that define and modify the configuration data, and a set of controller servers that distribute the configuration data to the edge forwarding elements (FEs), hubs and/or gateways. In some embodiments, the controller cluster 140 directs edge forwarding elements and hubs to use certain gateways (i.e., assigns a gateway to the edge forwarding elements and hubs). The controller cluster 140 also provides next hop forwarding rules in some embodiments.



FIG. 2 illustrates a datacenter 200 with multiple links 230 to one or more external networks 235. The datacenter 200 includes host computers 202A and 202B and an edge node 215. The edge node 215 connects the datacenter 200 to external networks 235 through multiple links 230. Each of the links 230 may have some or all of their characteristics (e.g., MTU bandwidth, error rate, presence of proxies on the link, etc.) different from the characteristics of the other links 230. The host computers 202A and 202B implement machines (e.g., VMs) 205A-205C. The machines 205A-205C in turn implement applications 210A-210C, respectively. The applications 210A-210C communicate with external networks 235 (e.g., to machines on the external networks) by accessing one or more of the links 230 through the SD-WAN edge FE 220 of the edge node 215. The applications 210A-210C may further communicatively connect to applications or other software or hardware of machines of another datacenter 250 (or other network connected computer or set of computers) through the external networks 235.


As further described with respect to FIG. 3, the characteristics of available links may change over time. To respond to these changes, in some embodiments, the SD-WAN edge FE 220, of FIG. 2, updates the transport groups dynamically. In some such embodiments, the configuration of the transport groups is facilitated by the network controller 240. Further, in some embodiments, reports detailing the results of the link monitoring and/or transport group monitoring are provided to an end user or administrator so that the end user or administrator can react to changes in the link characteristics or transport groups if necessary.


The host computers 202A and 202B send link requirements 212A-212C for each app 210A-210C to the SD-WAN edge FE 220, which in turn sends the link requirements to a link policy analyzer 225. The link policy analyzer 225 also receives transport group identifiers 245 from a network controller 240. The process of some embodiments for generating the transport group identifiers 245 is described with respect to FIG. 3. The process of selecting a link for a particular application, performed by the link policy analyzer 225 is described with respect to FIG. 4. Characteristics of the link requirements for applications of some embodiments are described with respect to FIG. 6. Characteristics of the transport group identifiers of some embodiments are described with respect to FIG. 7.



FIG. 3 conceptually illustrates a process 300 of some embodiments for providing transport group identifiers to link policy analyzers of edge nodes. In some embodiments, the process 300 is performed by a network controller such as network controller 240 of FIG. 2. The process 300, of FIG. 3, identifies (at 305) characteristics of available links. For example, the process 300 may identify characteristics such as throughput, error rate, security features, etc. Characteristics of links are further described with respect to FIGS. 6 and 7, below. The process 300 then groups (at 310) the available links (e.g., links 230 of FIG. 2 that SD-WAN FE 220 uses to communicate with external networks) into multiple transport groups.


In the process 300 of FIG. 3, links are assigned to a particular transport group based on the links having the characteristics that define that transport group. In some embodiments, the defining characteristics of each transport group include at least one characteristic that is not based on the physical nature of the links (e.g., not based on whether the link is wireless, wired, or using a particular type of physical interface). The transport groups of some embodiments are not exclusive, so any particular link may be assigned to multiple transport groups so long as the link has the defining characteristic of each transport group to which it is assigned. A defining characteristic of a transport group could be a threshold value (i.e., a minimum or maximum value of some characteristic of the links in the group). A defining characteristic could also be the presence or absence of some feature, such as the presence of a particular security protocol that is applied to some links or the absence of proxies on some links.


Once the links are assigned to the transport groups, the process 300 provides (at 315) the transport group identifiers to link policy analyzers of edge nodes. In some embodiments, the transport group identifiers specify the defining characteristic(s) of each transport group, as well as identifiers of the links in each transport group. The identifier of each link of a transport group, in some embodiments, is accompanied by identifiers of additional characteristics of the link. In other embodiments, the link is identified as being a link of that transport group, but no identifiers of additional characteristics of the link are sent with the transport group identifier. The process 300 then ends.


Although for the sake of description, the process 300 is shown as a linear flow chart with a start and an end, in some embodiments, the process 300 is repeated as long as applications need links. That is, the state of the available links is dynamic as various link characteristics may change over time (e.g., throughput may improve or degrade, features may be added or removed in response to network conditions, etc.). These changes may qualify or disqualify the links for inclusion in different transport groups over time. Therefore the links in some embodiments are repeatedly or continuously monitored and the transport group members are adjusted in accord with changed characteristics of the monitored links. Thus, such embodiments produce more resilient matches between the required link characteristics for the applications and their assigned links at any given point of time.


In some embodiments, the active link monitoring continuously keeps track of various link characteristics and metrics. In other embodiments, this mechanism is extended to actively probe and monitor link state/attributes related to various links and transport groups.


Once the process 300 is complete, a link policy analyzer (e.g., link policy analyzer 225 of FIG. 2) uses the transport group identifiers (e.g., transport group identifiers 245 of FIG. 2) and other data to select links for each application using the SD-WAN edge FE served by the link policy analyzer. FIG. 4 conceptually illustrates a process 400 of some embodiments for selecting a link for an application. The process 400 in some embodiments is provided by a link policy analyzer that acts as a separate module from the SD-WAN edge FE. However, one of ordinary skill in the art will understand that in other embodiments, the link policy analyzer may be a subsystem of an SD-WAN edge FE, or may be a subsystem of some other component present in a datacenter.


The process 400 receives (at 405) the transport group identifiers (e.g., from a network controller or other component implementing process 300 of FIG. 3). The process 400, of FIG. 4, also receives (at 410) a set of identifiers of required link characteristics for applications operating on VMs of host computers. Identifiers of required link characteristics of some embodiments are further described with respect to FIG. 6. In some embodiments, the applications themselves supply the required link characteristics (e.g., pre-programed required link characteristics or required link characteristics derived from an analysis of network conditions and/or other data by the applications). In other embodiments, the VM or the host computer determines the required link characteristics from a database of required link characteristics for specific applications or by analyzing the application and/or traffic generated by the application and/or in response to the application.


The process 400, of FIG. 4, selects (at 415), based on at least one required link characteristic, a transport group. For example, the defining characteristic of a transport group may be a specified minimum MTU of the links. An MTU is the size of the largest packet that can be sent on a network path without fragmenting the packet. The required link characteristic for an application could be a minimum MTU. In that example, a transport group with a minimum MTU that was at or greater than the minimum MTU of the required link characteristic would be selected by the process 400.


Once a particular transport group was selected, the process 400 would select (at 420), from the particular transport group, a link matching the rest of the required link characteristics for that application (if any). In some embodiments, when more than one link in a transport group satisfies all required link characteristics of an application, some default characteristic of the links is used to determine which of multiple satisfactory links is used. That is, in some embodiments, the network controller implements an attribute/characteristic hierarchy for a set of links, based on the set of required or preferred link characteristics of the incoming data and/or the application. The link analyzer of such embodiments ranks the importance of at least a subset of those characteristics when determining which transport group/link to assign the data from a particular application to. In some embodiments, the ranking may be based on the specific requirements for a particular application and in order of decreasing importance of the characteristics to that application. For example, an application that requires high throughput but is tolerant of a high error rate would use a link ranking list that placed the throughput ranking of the links in a transport group above the error rate ranking. The node then sends the data to the most appropriate transport group link based on the progressive ranking of each characteristic. The link analyzers of some embodiments track changes to link characteristics (e.g., determining when throughput of a link drops, when error rates increase, etc.).


In some embodiment, an application may be assigned to a different link if the characteristics of the link it had been using and/or the characteristics of the new link have changed so as to make the new link a better match for the characteristics required/preferred for the app. Additionally, in some embodiments, the rules for determining the hierarchy of links for an application could be adjusted dynamically (e.g., by the network controller or the link analyzer) throughout a period of time depending on particular circumstances or external pressures going on. In still other embodiments, one of multiple satisfactory links may be chosen at random.


In some embodiments, the required link characteristics may include preferences as well as absolute requirements. For example, a set of required link characteristics may include a requirement that a link include a particular security protocol, and a preference for a link with an MTU of at least 1300. In that example, the link policy analyzer would provide a link with that security protocol and minimum MTU if such a link were available, but would still provide a link with that security protocol and a lower MTU if a link with both the required security protocol and preferred minimum MTU were not available.


In some embodiments, when no link with all required link characteristics is available, the process 400 will provide an error message informing a user or network administrator that no satisfactory links are available for a particular application. In other embodiments when no link with all required link characteristics is available, the process 400 will select (at 420) a link that meets as many of the required link characteristics as possible. The process then ends.


The following are examples of transport groups and applications that may select particular transport groups. In the example case, there are 6 links/interfaces (Link 1, Link 2, Link 3, Link 4, Link 5, and Link 6). Links 1-3 provide larger MTUs (1450 bytes or above) while links 4-6 provide smaller MTUs (300-400 bytes). Real-time transport protocol (RTP) is a kind of data traffic which typically has small to medium sized packets (e.g., at or under 300 bytes). An application which primarily sends such RTP data traffic could use any of the links in a transport group called TG1 (which includes links with a minimum MTU of 300 and thus includes Link 1, Link 2, Link 3, Link 4, Link 5, and Link 6). In contrast, a bulk transaction type TCP application typically sends large sized packets (e.g., 1450 bytes). An application that sends such TCP traffic could require links with large MTUs. Such an application could use a transport group called TG2 (which includes links with a minimum MTU of 1450 and thus includes only Link 1, Link 2, and Link 3, which provide larger MTUs) so that the application can access maximum throughput. Within the transport group, some embodiments make use of an Adaptive Path MTU which would be the minimum native MTU within the transport group.


These link characteristic-based groups define a set of links that are acceptable for use with an application which has requirements matching the transport group's defining characteristics. This alerts the link policy analyzer that no links outside the transport group should be used for a particular application. However, within a particular transport group, the link policy analyzer of some embodiments may quickly switch the selected link for an application based on changing network conditions. For example, if the defining characteristic of a transport group is minimum path MTU for all links within a particular group, for an application that requires at least that minimum value, any link in the group is appropriate to switch the application to in the event that the originally assigned link becomes inoperative or otherwise undesirable. Similarly, in some embodiments, any TCP based applications are subject to maximum segment size (MSS) adjustments based on an adaptive path MTU within the transport group, which effectively improves and maximizes throughput utilization for the application traffic.



FIG. 5 illustrates the multiple applications using links selected by a link policy analyzer. Application 210A has been assigned to link 510. Applications 210B and 210C have been assigned to link 520. Each of the applications 210A-210C connects to the external network through SD-WAN edge FE 220. SD-WAN edge FE 220 identifies packets sent from each of the application 210A-210C and sends the packets to one or more external networks 235 through the link assigned to that application. In this embodiment, more than one application can be assigned to the same link, such as applications 210B and 210C being assigned to link 520. Such an assignment could be because both application 210B and application 210C have required link characteristics that resulted in both applications being assigned to the same transport group or because link 520 is included in multiple transport groups, with application 210B assigned to one transport group and application 210C assigned to another transport group.



FIG. 6 illustrates a link requirement set 600 of some embodiments. The link requirements set 600 is one example of a structure of a link requirements data set. The link requirements set 600 includes an application identifier 601, minimum bandwidth 602, minimum MTU 604, maximum allowed number of proxies 606, security protocols 608 and 610, and other features 612 and 614. Any or all of these values may be required for a link for the application associated with link requirements set 600.


Although the link requirement set 600 is shown as an ordered data structure with the specific type of requirements identified by their location in the data structure, one of ordinary skill in the art will understand that other orders are possible within the scope of the invention as well as alternate data structures such as using a code to identify a type of data requirement and a number to represent a particular magnitude of that requirement (e.g., an identifier such as “02” to identify a requirement as a MTU requirement followed by the number 1450 to indicate the minimum required value for the MTU).



FIG. 7 illustrates a set of transport group identifiers 700 of some embodiments. The transport group identifiers set 700 is one example of a structure of a transport group identifiers data set. Transport group identifiers set 700 includes a group identifier 701, minimum bandwidth 702, minimum MTU 704, maximum allowed number of proxies 706, security protocols 708 and 710, and other features 712 and 714. Any or all of these values may be used as the defining characteristics of the transport group defined by the transport group identifiers set 700. In addition to the specified defining characteristics of the transport group, the transport group identifiers set 700, of some embodiments, includes, for the links, link identifiers 721A-721B, link minimum bandwidth 722A-722B, a link minimum MTU 724A-724B, a link maximum allowed number of proxies 726A-726B, link security protocols 728A-728B and 730A-730B, and other link features 732A-732B and 734A-734B.


In FIG. 7, the transport group identifiers set 700 includes identifiers of multiple characteristics of the available links, such as characteristics 722A-734A of link1, in some embodiments, the transport group identifiers set 700 only contains characteristics of the group (e.g., characteristics 701-714) and identifiers of the individual links (e.g., 721A) rather than the characteristics of each link. In some such embodiments, the link characteristics are still provided by the network controller, but in separate data sets from the transport group identifiers. Such a separation of the link characteristics from the transport group identifiers would be more efficient in cases where multiple transport groups included the same links. Rather than sending all link characteristics in each transport group that includes the link, the network controller would send a set of link characteristics once, and the link policy analyzer would use the link identifiers (e.g., 721A-721B) to retrieve the link characteristics for the identified link from the set of link characteristics.


Although the transport group identifiers set 700 is shown as an ordered data structure with the specific type of transport group characteristics identified by their location in the data structure, one of ordinary skill in the art will understand that other orders are possible within the scope of the invention as well as alternate data structures such as using a code to identify a type of data characteristic and a number to represent a particular magnitude of that characteristic (e.g., an identifier such as “02” to identify a requirement as a MTU requirement followed by the number 1450 to indicate the minimum required value for the MTU).


This specification refers throughout to computational and network environments that include virtual machines (VMs). However, virtual machines are merely one example of data compute nodes (DCNs) or data compute end nodes, also referred to as addressable nodes. DCNs may include non-virtualized physical hosts, virtual machines, containers that run on top of a host operating system without the need for a hypervisor or separate operating system, and hypervisor kernel network interface modules. Therefore, it should be understood that where the specification refers to VMs, the examples given could be any type of DCNs, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, the example networks could include combinations of different types of DCNs in some embodiments.


Although the above figures show the transport group identifiers being generated by a network controller and sent to a link policy analyzer, in other embodiments, the link policy analyzer itself or the SD-WAN edge FE generates the transport croup identifiers based on the links connected to the SD-WAN edge FE. In other embodiments, the network controller identifies the link characteristics to the link policy analyzer and then the link policy analyzer generates the transport group identifiers. In still other embodiments, the network controller identifies the link characteristics to the SD-WAN edge FE, which then generates the transport group identifiers or forwards the link characteristics to the link policy analyzer (which then generates the transport group identifiers).


VMs, in some embodiments, operate with their own guest operating systems on a host using resources of the host virtualized by virtualization software (e.g., a hypervisor, virtual machine monitor, etc.). The tenant (i.e., the owner of the VM) can choose which applications to operate on top of the guest operating system. Some containers, on the other hand, are constructs that run on top of a host operating system without the need for a hypervisor or separate guest operating system. In some embodiments, the host operating system uses name spaces to isolate the containers from each other and therefore provides operating-system level segregation of the different groups of applications that operate within different containers. This segregation is akin to the VM segregation that is offered in hypervisor-virtualized environments that virtualize system hardware, and thus can be viewed as a form of virtualization that isolates different groups of applications that operate in different containers. Such containers are more lightweight than VMs.


Hypervisor kernel network interface modules, in some embodiments, are non-VM DCNs that include a network stack with a hypervisor kernel network interface and receive/transmit threads. One example of a hypervisor kernel network interface module is the vmknic module that is part of the ESXi™ hypervisor of VMware, Inc.


Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as computer-readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer-readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.


In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.



FIG. 8 conceptually illustrates a computer system 800 with which some embodiments of the invention are implemented. The computer system 800 can be used to implement any of the above-described hosts, controllers, gateway and edge forwarding elements. As such, it can be used to execute any of the above-described processes. This computer system 800 includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. Computer system 800 includes a bus 805, processing unit(s) 810, a system memory 825, a read-only memory 830, a permanent storage device 835, input devices 840, and output devices 845.


The bus 805 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system 800. For instance, the bus 805 communicatively connects the processing unit(s) 810 with the read-only memory 830, the system memory 825, and the permanent storage device 835.


From these various memory units, the processing unit(s) 810 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only-memory (ROM) 830 stores static data and instructions that are needed by the processing unit(s) 810 and other modules of the computer system. The permanent storage device 835, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 800 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 835.


Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device 835. Like the permanent storage device 835, the system memory 825 is a read-and-write memory device. However, unlike storage device 835, the system memory 825 is a volatile read-and-write memory, such as random access memory. The system memory 825 stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 825, the permanent storage device 835, and/or the read-only memory 830. From these various memory units, the processing unit(s) 810 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.


The bus 805 also connects to the input and output devices 840 and 845. The input devices 840 enable the user to communicate information and select commands to the computer system 800. The input devices 840 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 845 display images generated by the computer system 800. The output devices 845 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as touchscreens that function as both input and output devices 840 and 845.


Finally, as shown in FIG. 8, bus 805 also couples computer system 800 to a network 865 through a network adapter (not shown). In this manner, the computer 800 can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet), or a network of networks (such as the Internet). Any or all components of computer system 800 may be used in conjunction with the invention.


Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.


While the above discussion primarily refers to microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.


As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms “display” or “displaying” mean displaying on an electronic device. As used in this specification, the terms “computer-readable medium,” “computer-readable media,” and “machine-readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral or transitory signals.


While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For instance, several of the above-described embodiments deploy gateways in public cloud datacenters. However, in other embodiments, the gateways are deployed in a third-party's private cloud datacenters (e.g., datacenters that the third-party uses to deploy cloud gateways for different entities in order to deploy virtual networks for these entities). Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.

Claims
  • 1. A method of selecting, on a per-application basis, one of a plurality of data links used by an edge forwarding node at a first site connected to a set of one or more other sites through a software-defined wide area network (SD-WAN), the method comprising: at the edge forwarding node: receiving, from a machine executing an application, a set of one or more required link characteristics;based on the received set of required link characteristics, selecting a transport group comprising a set of one or more optional links matching the required link characteristics;from the selected transport group, selecting one of the links in the transport group; andusing the selected link to forward data messages associated with the application.
  • 2. The method of claim 1, wherein the received set of required link characteristics is associated with an identifier of the application that is received at the edge forwarding node.
  • 3. The method of claim 2, wherein the edge forwarding node identifies packets from the application and routes the packets from the application to the selected link.
  • 4. The method of claim 1, wherein the edge forwarding node comprises a link analyzer and an edge forwarding element.
  • 5. The method of claim 4, wherein the edge forwarding element is a router.
  • 6. The method of claim 1, wherein the transport group is selected based on at least a required link characteristic comprising a lowest maximum transmission unit (MTU) size of links of the transport group.
  • 7. The method of claim 1, wherein the transport group is selected based on at least a required link characteristic comprising a minimum throughput of the links of the transport group.
  • 8. The method of claim 1, wherein the transport group is selected based on a required link characteristic comprising a maximum rate of packet drops of links of the transport group.
  • 9. The method of claim 1, wherein the transport group is selected based on a required link characteristic comprising maximum rate of interface errors of links of the transport group.
  • 10. The method of claim 1, wherein the transport group is selected based on at least a required link characteristic comprising a security and/or encryption characteristic of links in the transport group.
  • 11. The method of claim 1, wherein the transport group is selected based on at least a required link characteristic comprising a filtering capability of links in the transport group.
  • 12. The method of claim 1, wherein the transport group is selected based on at least a required link characteristic comprising a presence of proxies on the links in the transport group.
  • 13. The method of claim 1, wherein the link is selected based on a default ranking characteristic of the link.
  • 14. The method of claim 1, wherein the link is selected based on one or more ranking characteristics specific to the application, wherein the ranking characteristics are weighted by the importance of the characteristics to that application.
  • 15. The method of claim 1, wherein the link is selected at random from the links in the transport group.
  • 16. A non-transitory machine readable medium storing a program that when executed by at least one processing unit selects, on a per-application basis, one of a plurality of data links used by an edge forwarding node at a first site connected to a set of one or more other sites through a software-defined wide area network (SD-WAN), the program comprising sets of instructions for: at the edge forwarding node: receiving, from a machine executing an application, a set of one or more required link characteristics;based on the received set of required link characteristics, selecting a transport group comprising a set of one or more optional links matching the required link characteristics;from the selected transport group, selecting one of the links in the transport group; andusing the selected link to forward data messages associated with the application.
  • 17. The non-transitory machine readable medium of claim 16, wherein the received set of required link characteristics is associated with an identifier of the application that is received at the edge forwarding node.
  • 18. The non-transitory machine readable medium of claim 17, wherein the edge forwarding node identifies packets from the application and routes the packets from the application to the selected link.
  • 19. The non-transitory machine readable medium of claim 16, wherein the edge forwarding node comprises a link analyzer and an edge forwarding element.
  • 20. The non-transitory machine readable medium of claim 16, wherein the link is selected based on a default ranking characteristic of the link.
Priority Claims (1)
Number Date Country Kind
202141020681 May 2021 IN national
US Referenced Citations (890)
Number Name Date Kind
5652751 Sharony Jul 1997 A
5909553 Campbell et al. Jun 1999 A
6154465 Pickett Nov 2000 A
6157648 Voit et al. Dec 2000 A
6201810 Masuda et al. Mar 2001 B1
6363378 Conklin et al. Mar 2002 B1
6445682 Weitz Sep 2002 B1
6744775 Beshai et al. Jun 2004 B1
6976087 Westfall et al. Dec 2005 B1
7003481 Banka et al. Feb 2006 B2
7280476 Anderson Oct 2007 B2
7313629 Nucci et al. Dec 2007 B1
7320017 Kurapati et al. Jan 2008 B1
7373660 Guichard et al. May 2008 B1
7581022 Griffin et al. Aug 2009 B1
7680925 Sathyanarayana et al. Mar 2010 B2
7681236 Tamura et al. Mar 2010 B2
7962458 Holenstein et al. Jun 2011 B2
8094575 Vadlakonda et al. Jan 2012 B1
8094659 Arad Jan 2012 B1
8111692 Ray Feb 2012 B2
8141156 Mao et al. Mar 2012 B1
8224971 Miller et al. Jul 2012 B1
8228928 Parandekar et al. Jul 2012 B2
8243589 Trost et al. Aug 2012 B1
8259566 Chen et al. Sep 2012 B2
8274891 Averi et al. Sep 2012 B2
8301749 Finklestein et al. Oct 2012 B1
8385227 Downey Feb 2013 B1
8566452 Goodwin et al. Oct 2013 B1
8630291 Shaffer et al. Jan 2014 B2
8661295 Khanna et al. Feb 2014 B1
8724456 Hong et al. May 2014 B1
8724503 Johnsson et al. May 2014 B2
8745177 Kazerani et al. Jun 2014 B1
8797874 Yu et al. Aug 2014 B2
8799504 Capone et al. Aug 2014 B2
8804745 Sinn Aug 2014 B1
8806482 Nagargadde et al. Aug 2014 B1
8855071 Sankaran et al. Oct 2014 B1
8856339 Mestery et al. Oct 2014 B2
8964548 Keralapura et al. Feb 2015 B1
8989199 Sella et al. Mar 2015 B1
9009217 Nagargadde et al. Apr 2015 B1
9055000 Ghosh et al. Jun 2015 B1
9060025 Xu Jun 2015 B2
9071607 Twitchell, Jr. Jun 2015 B2
9075771 Gawali et al. Jul 2015 B1
9100329 Jiang et al. Aug 2015 B1
9135037 Petrescu-Prahova et al. Sep 2015 B1
9137334 Zhou Sep 2015 B2
9154327 Marino et al. Oct 2015 B1
9203764 Shirazipour et al. Dec 2015 B2
9225591 Beheshti-Zavareh et al. Dec 2015 B2
9306949 Richard et al. Apr 2016 B1
9323561 Ayala et al. Apr 2016 B2
9336040 Dong et al. May 2016 B2
9354983 Yenamandra et al. May 2016 B1
9356943 Lopilato et al. May 2016 B1
9379981 Zhou et al. Jun 2016 B1
9413724 Xu Aug 2016 B2
9419878 Hsiao et al. Aug 2016 B2
9432245 Sorenson et al. Aug 2016 B1
9438566 Zhang et al. Sep 2016 B2
9450817 Bahadur et al. Sep 2016 B1
9450852 Chen et al. Sep 2016 B1
9462010 Stevenson Oct 2016 B1
9467478 Khan et al. Oct 2016 B1
9485163 Fries et al. Nov 2016 B1
9521067 Michael et al. Dec 2016 B2
9525564 Lee Dec 2016 B2
9559951 Sajassi et al. Jan 2017 B1
9563423 Pittman Feb 2017 B1
9602389 Maveli et al. Mar 2017 B1
9608917 Anderson et al. Mar 2017 B1
9608962 Chang Mar 2017 B1
9614748 Battersby et al. Apr 2017 B1
9621460 Mehta et al. Apr 2017 B2
9641551 Kariyanahalli May 2017 B1
9648547 Hart et al. May 2017 B1
9665432 Kruse et al. May 2017 B2
9686127 Ramachandran et al. Jun 2017 B2
9715401 Devine et al. Jul 2017 B2
9717021 Hughes et al. Jul 2017 B2
9722815 Mukundan et al. Aug 2017 B2
9747249 Cherian et al. Aug 2017 B2
9755965 Yadav et al. Sep 2017 B1
9787559 Schroeder Oct 2017 B1
9807004 Koley et al. Oct 2017 B2
9819540 Bahadur et al. Nov 2017 B1
9819565 Djukic et al. Nov 2017 B2
9825822 Holland Nov 2017 B1
9825911 Brandwine Nov 2017 B1
9825992 Xu Nov 2017 B2
9832128 Ashner et al. Nov 2017 B1
9832205 Santhi et al. Nov 2017 B2
9875355 Williams Jan 2018 B1
9906401 Rao Feb 2018 B1
9930011 Clemons, Jr. et al. Mar 2018 B1
9935829 Miller et al. Apr 2018 B1
9942787 Tillotson Apr 2018 B1
9996370 Khafizov et al. Jun 2018 B1
10038601 Becker et al. Jul 2018 B1
10057183 Salle et al. Aug 2018 B2
10057294 Xu Aug 2018 B2
10116593 Sinn et al. Oct 2018 B1
10135789 Mayya et al. Nov 2018 B2
10142226 Wu et al. Nov 2018 B1
10178032 Freitas Jan 2019 B1
10178037 Appleby et al. Jan 2019 B2
10187289 Chen et al. Jan 2019 B1
10200264 Menon et al. Feb 2019 B2
10229017 Zou et al. Mar 2019 B1
10237123 Dubey et al. Mar 2019 B2
10250498 Bales et al. Apr 2019 B1
10263832 Ghosh Apr 2019 B1
10320664 Nainar et al. Jun 2019 B2
10320691 Matthews et al. Jun 2019 B1
10326830 Singh Jun 2019 B1
10348767 Lee et al. Jul 2019 B1
10355989 Panchal et al. Jul 2019 B1
10425382 Mayya et al. Sep 2019 B2
10454708 Mibu Oct 2019 B2
10454714 Mayya et al. Oct 2019 B2
10461993 Turabi et al. Oct 2019 B2
10498652 Mayya et al. Dec 2019 B2
10511546 Singarayan et al. Dec 2019 B2
10523539 Mayya et al. Dec 2019 B2
10550093 Ojima et al. Feb 2020 B2
10554538 Spohn et al. Feb 2020 B2
10560431 Chen et al. Feb 2020 B1
10565464 Han et al. Feb 2020 B2
10567519 Mukhopadhyaya et al. Feb 2020 B1
10574528 Mayya et al. Feb 2020 B2
10594516 Cidon et al. Mar 2020 B2
10594591 Houjyo et al. Mar 2020 B2
10594659 El-Moussa et al. Mar 2020 B2
10608844 Cidon et al. Mar 2020 B2
10637889 Ermagan et al. Apr 2020 B2
10666460 Cidon et al. May 2020 B2
10666497 Tahhan et al. May 2020 B2
10686625 Cidon et al. Jun 2020 B2
10693739 Naseri et al. Jun 2020 B1
10715427 Raj et al. Jul 2020 B2
10749711 Mukundan et al. Aug 2020 B2
10778466 Cidon et al. Sep 2020 B2
10778528 Mayya et al. Sep 2020 B2
10778557 Ganichev et al. Sep 2020 B2
10805114 Cidon et al. Oct 2020 B2
10805272 Mayya et al. Oct 2020 B2
10819564 Turabi et al. Oct 2020 B2
10826775 Moreno et al. Nov 2020 B1
10841131 Cidon et al. Nov 2020 B2
10911374 Kumar et al. Feb 2021 B1
10938693 Mayya et al. Mar 2021 B2
10951529 Duan et al. Mar 2021 B2
10958479 Cidon et al. Mar 2021 B2
10959098 Cidon et al. Mar 2021 B2
10992558 Silva et al. Apr 2021 B1
10992568 Michael et al. Apr 2021 B2
10999100 Cidon et al. May 2021 B2
10999137 Cidon et al. May 2021 B2
10999165 Cidon et al. May 2021 B2
10999197 Hooda et al. May 2021 B2
11005684 Cidon May 2021 B2
11018995 Cidon et al. May 2021 B2
11044190 Ramaswamy et al. Jun 2021 B2
11050588 Mayya et al. Jun 2021 B2
11050644 Hegde et al. Jun 2021 B2
11071005 Shen et al. Jul 2021 B2
11089111 Markuze et al. Aug 2021 B2
11095612 Oswal et al. Aug 2021 B1
11102032 Cidon et al. Aug 2021 B2
11108851 Kurmala et al. Aug 2021 B1
11115347 Gupta et al. Sep 2021 B2
11115426 Pazhyannur et al. Sep 2021 B1
11115480 Markuze et al. Sep 2021 B2
11121962 Michael et al. Sep 2021 B2
11121985 Cidon et al. Sep 2021 B2
11128492 Sethi et al. Sep 2021 B2
11153230 Cidon et al. Oct 2021 B2
11171885 Cidon et al. Nov 2021 B2
11212140 Mukundan et al. Dec 2021 B2
11212238 Cidon et al. Dec 2021 B2
11223514 Mayya et al. Jan 2022 B2
11245641 Ramaswamy et al. Feb 2022 B2
11252079 Michael et al. Feb 2022 B2
11252105 Cidon et al. Feb 2022 B2
11252106 Cidon et al. Feb 2022 B2
11258728 Cidon et al. Feb 2022 B2
11310170 Cidon et al. Apr 2022 B2
11323307 Mayya et al. May 2022 B2
11349722 Mayya et al. May 2022 B2
11363124 Markuze et al. Jun 2022 B2
11374904 Mayya et al. Jun 2022 B2
11375005 Rolando et al. Jun 2022 B1
11381474 Kumar et al. Jul 2022 B1
11381499 Ramaswamy et al. Jul 2022 B1
11388086 Ramaswamy et al. Jul 2022 B1
11394640 Ramaswamy et al. Jul 2022 B2
11418997 Devadoss et al. Aug 2022 B2
11438789 Devadoss et al. Sep 2022 B2
11444865 Ramaswamy et al. Sep 2022 B2
11444872 Mayya et al. Sep 2022 B2
11477127 Ramaswamy et al. Oct 2022 B2
11489720 Kempanna et al. Nov 2022 B1
11489783 Ramaswamy et al. Nov 2022 B2
11509571 Ramaswamy et al. Nov 2022 B1
11516049 Cidon et al. Nov 2022 B2
11533248 Mayya et al. Dec 2022 B2
11575591 Ramaswamy et al. Feb 2023 B2
11575600 Markuze et al. Feb 2023 B2
11582144 Ramaswamy et al. Feb 2023 B2
20020075542 Kumar et al. Jun 2002 A1
20020085488 Kobayashi Jul 2002 A1
20020087716 Mustafa Jul 2002 A1
20020152306 Tuck Oct 2002 A1
20020198840 Banka et al. Dec 2002 A1
20030050061 Wu et al. Mar 2003 A1
20030061269 Hathaway et al. Mar 2003 A1
20030088697 Matsuhira May 2003 A1
20030112766 Riedel et al. Jun 2003 A1
20030112808 Solomon Jun 2003 A1
20030126468 Markham Jul 2003 A1
20030161313 Jinmei et al. Aug 2003 A1
20030189919 Gupta et al. Oct 2003 A1
20030202506 Perkins et al. Oct 2003 A1
20030219030 Gubbi Nov 2003 A1
20040059831 Chu et al. Mar 2004 A1
20040068668 Lor et al. Apr 2004 A1
20040165601 Liu et al. Aug 2004 A1
20040224771 Chen et al. Nov 2004 A1
20050078690 DeLangis Apr 2005 A1
20050149604 Navada Jul 2005 A1
20050154790 Nagata et al. Jul 2005 A1
20050172161 Cruz et al. Aug 2005 A1
20050195754 Nosella Sep 2005 A1
20050210479 Andjelic Sep 2005 A1
20050265255 Kodialam et al. Dec 2005 A1
20060002291 Alicherry et al. Jan 2006 A1
20060114838 Mandavilli et al. Jun 2006 A1
20060171365 Borella Aug 2006 A1
20060182034 Klinker et al. Aug 2006 A1
20060182035 Vasseur Aug 2006 A1
20060193247 Naseh et al. Aug 2006 A1
20060193252 Naseh et al. Aug 2006 A1
20070050594 Augsburg et al. Mar 2007 A1
20070064604 Chen et al. Mar 2007 A1
20070064702 Bates et al. Mar 2007 A1
20070083727 Johnston et al. Apr 2007 A1
20070091794 Filsfils et al. Apr 2007 A1
20070103548 Carter May 2007 A1
20070115812 Hughes May 2007 A1
20070121486 Guichard et al. May 2007 A1
20070130325 Lesser Jun 2007 A1
20070162619 Aloni et al. Jul 2007 A1
20070162639 Chu et al. Jul 2007 A1
20070177511 Das et al. Aug 2007 A1
20070237081 Kodialam et al. Oct 2007 A1
20070260746 Mirtorabi et al. Nov 2007 A1
20070268882 Breslau et al. Nov 2007 A1
20080002670 Bugenhagen et al. Jan 2008 A1
20080049621 McGuire et al. Feb 2008 A1
20080055241 Goldenberg et al. Mar 2008 A1
20080080509 Khanna et al. Apr 2008 A1
20080095187 Jung et al. Apr 2008 A1
20080117930 Chakareski et al. May 2008 A1
20080144532 Chamarajanagar et al. Jun 2008 A1
20080181116 Kavanaugh et al. Jul 2008 A1
20080219276 Shah Sep 2008 A1
20080240121 Xiong et al. Oct 2008 A1
20080263218 Beerends et al. Oct 2008 A1
20090013210 McIntosh et al. Jan 2009 A1
20090028092 Rothschild Jan 2009 A1
20090125617 Kiessig et al. May 2009 A1
20090141642 Sun Jun 2009 A1
20090154463 Hines et al. Jun 2009 A1
20090182874 Morford et al. Jul 2009 A1
20090247204 Sennett et al. Oct 2009 A1
20090268605 Campbell et al. Oct 2009 A1
20090274045 Meier et al. Nov 2009 A1
20090276657 Wetmore et al. Nov 2009 A1
20090303880 Maltz et al. Dec 2009 A1
20100008361 Guichard et al. Jan 2010 A1
20100017802 Lojewski Jan 2010 A1
20100046532 Okita Feb 2010 A1
20100061379 Parandekar et al. Mar 2010 A1
20100080129 Strahan et al. Apr 2010 A1
20100088440 Banks et al. Apr 2010 A1
20100091782 Hiscock Apr 2010 A1
20100091823 Retana et al. Apr 2010 A1
20100107162 Edwards et al. Apr 2010 A1
20100118727 Draves et al. May 2010 A1
20100118886 Saavedra May 2010 A1
20100165985 Sharma et al. Jul 2010 A1
20100191884 Holenstein et al. Jul 2010 A1
20100223621 Joshi et al. Sep 2010 A1
20100226246 Proulx Sep 2010 A1
20100290422 Haigh et al. Nov 2010 A1
20100309841 Conte Dec 2010 A1
20100309912 Mehta et al. Dec 2010 A1
20100322255 Hao et al. Dec 2010 A1
20100332657 Elyashev et al. Dec 2010 A1
20110007752 Silva et al. Jan 2011 A1
20110032939 Nozaki et al. Feb 2011 A1
20110040814 Higgins Feb 2011 A1
20110075674 Li et al. Mar 2011 A1
20110107139 Middlecamp et al. May 2011 A1
20110110370 Moreno et al. May 2011 A1
20110141877 Xu et al. Jun 2011 A1
20110142041 Imai Jun 2011 A1
20110153909 Dong Jun 2011 A1
20110235509 Szymanski Sep 2011 A1
20110255397 Kadakia et al. Oct 2011 A1
20110302663 Prodan et al. Dec 2011 A1
20120008630 Ould-Brahim Jan 2012 A1
20120027013 Napierala Feb 2012 A1
20120039309 Evans et al. Feb 2012 A1
20120099601 Haddad et al. Apr 2012 A1
20120136697 Peles et al. May 2012 A1
20120140935 Kruglick Jun 2012 A1
20120157068 Eichen et al. Jun 2012 A1
20120173694 Yan et al. Jul 2012 A1
20120173919 Patel et al. Jul 2012 A1
20120182940 Taleb et al. Jul 2012 A1
20120221955 Raleigh et al. Aug 2012 A1
20120227093 Shalzkamer et al. Sep 2012 A1
20120240185 Kapoor et al. Sep 2012 A1
20120250682 Vincent et al. Oct 2012 A1
20120250686 Vincent et al. Oct 2012 A1
20120281706 Agarwal et al. Nov 2012 A1
20120287818 Corti et al. Nov 2012 A1
20120300615 Kempf et al. Nov 2012 A1
20120307659 Kamada Dec 2012 A1
20120317270 Vrbaski et al. Dec 2012 A1
20120317291 Wolfe Dec 2012 A1
20130019005 Hui et al. Jan 2013 A1
20130021968 Reznik et al. Jan 2013 A1
20130044764 Casado et al. Feb 2013 A1
20130051237 Ong Feb 2013 A1
20130051399 Zhang et al. Feb 2013 A1
20130054763 Merwe et al. Feb 2013 A1
20130086267 Gelenbe et al. Apr 2013 A1
20130097304 Asthana et al. Apr 2013 A1
20130103729 Cooney et al. Apr 2013 A1
20130103834 Dzerve et al. Apr 2013 A1
20130117530 Kim et al. May 2013 A1
20130124718 Griffith et al. May 2013 A1
20130124911 Griffith et al. May 2013 A1
20130124912 Griffith et al. May 2013 A1
20130128889 Mathur et al. May 2013 A1
20130142201 Kim et al. Jun 2013 A1
20130170354 Takashima et al. Jul 2013 A1
20130173788 Song Jul 2013 A1
20130182712 Aguayo et al. Jul 2013 A1
20130185729 Vasic et al. Jul 2013 A1
20130191688 Agarwal et al. Jul 2013 A1
20130223226 Narayanan et al. Aug 2013 A1
20130223454 Dunbar et al. Aug 2013 A1
20130238782 Zhao et al. Sep 2013 A1
20130242718 Zhang Sep 2013 A1
20130254599 Katkar et al. Sep 2013 A1
20130258839 Wang et al. Oct 2013 A1
20130258847 Zhang et al. Oct 2013 A1
20130266015 Qu et al. Oct 2013 A1
20130266019 Qu et al. Oct 2013 A1
20130283364 Chang et al. Oct 2013 A1
20130286846 Atlas et al. Oct 2013 A1
20130297611 Moritz et al. Nov 2013 A1
20130297770 Zhang Nov 2013 A1
20130301469 Suga Nov 2013 A1
20130301642 Radhakrishnan et al. Nov 2013 A1
20130308444 Sem-Jacobsen et al. Nov 2013 A1
20130315242 Wang et al. Nov 2013 A1
20130315243 Huang et al. Nov 2013 A1
20130329548 Nakil et al. Dec 2013 A1
20130329601 Yin et al. Dec 2013 A1
20130329734 Chesla et al. Dec 2013 A1
20130346470 Obstfeld et al. Dec 2013 A1
20140016464 Shirazipour et al. Jan 2014 A1
20140019604 Twitchell, Jr. Jan 2014 A1
20140019750 Dodgson et al. Jan 2014 A1
20140040975 Raleigh et al. Feb 2014 A1
20140064283 Balus et al. Mar 2014 A1
20140071832 Johnsson et al. Mar 2014 A1
20140092907 Sridhar et al. Apr 2014 A1
20140108665 Arora et al. Apr 2014 A1
20140112171 Pasdar Apr 2014 A1
20140115584 Mudigonda et al. Apr 2014 A1
20140122559 Branson et al. May 2014 A1
20140123135 Huang et al. May 2014 A1
20140126418 Brendel et al. May 2014 A1
20140156818 Hunt Jun 2014 A1
20140156823 Liu et al. Jun 2014 A1
20140160935 Zecharia et al. Jun 2014 A1
20140164560 Ko et al. Jun 2014 A1
20140164617 Jalan et al. Jun 2014 A1
20140164718 Schaik et al. Jun 2014 A1
20140173113 Vemuri et al. Jun 2014 A1
20140173331 Martin et al. Jun 2014 A1
20140181824 Saund et al. Jun 2014 A1
20140208317 Nakagawa Jul 2014 A1
20140219135 Li et al. Aug 2014 A1
20140223507 Xu Aug 2014 A1
20140229210 Sharifian et al. Aug 2014 A1
20140244851 Lee Aug 2014 A1
20140258535 Zhang Sep 2014 A1
20140269690 Tu Sep 2014 A1
20140279862 Dietz et al. Sep 2014 A1
20140280499 Basavaiah et al. Sep 2014 A1
20140317440 Biermayr et al. Oct 2014 A1
20140321277 Lynn, Jr. et al. Oct 2014 A1
20140337500 Lee Nov 2014 A1
20140337674 Ivancic et al. Nov 2014 A1
20140341109 Cartmell et al. Nov 2014 A1
20140365834 Stone et al. Dec 2014 A1
20140372582 Ghanwani et al. Dec 2014 A1
20150003240 Drwiega et al. Jan 2015 A1
20150016249 Mukundan et al. Jan 2015 A1
20150029864 Raileanu et al. Jan 2015 A1
20150039744 Niazi et al. Feb 2015 A1
20150046572 Cheng et al. Feb 2015 A1
20150052247 Threefoot et al. Feb 2015 A1
20150052517 Raghu et al. Feb 2015 A1
20150056960 Egner et al. Feb 2015 A1
20150058917 Xu Feb 2015 A1
20150088942 Shah Mar 2015 A1
20150089628 Lang Mar 2015 A1
20150092603 Aguayo et al. Apr 2015 A1
20150096011 Watt Apr 2015 A1
20150100958 Banavalikar et al. Apr 2015 A1
20150124603 Ketheesan et al. May 2015 A1
20150134777 Onoue May 2015 A1
20150139238 Pourzandi et al. May 2015 A1
20150146539 Mehta et al. May 2015 A1
20150163152 Li Jun 2015 A1
20150169340 Haddad et al. Jun 2015 A1
20150172121 Farkas et al. Jun 2015 A1
20150172169 DeCusatis et al. Jun 2015 A1
20150188823 Williams et al. Jul 2015 A1
20150189009 Bemmel Jul 2015 A1
20150195178 Bhattacharya et al. Jul 2015 A1
20150201036 Nishiki et al. Jul 2015 A1
20150222543 Song Aug 2015 A1
20150222638 Morley Aug 2015 A1
20150236945 Michael et al. Aug 2015 A1
20150236962 Veres et al. Aug 2015 A1
20150244617 Nakil et al. Aug 2015 A1
20150249644 Xu Sep 2015 A1
20150257081 Ramanujan et al. Sep 2015 A1
20150271056 Chunduri et al. Sep 2015 A1
20150271104 Chikkamath et al. Sep 2015 A1
20150271303 Neginhal et al. Sep 2015 A1
20150281004 Kakadia et al. Oct 2015 A1
20150312142 Barabash et al. Oct 2015 A1
20150312760 O'Toole Oct 2015 A1
20150317169 Sinha et al. Nov 2015 A1
20150326426 Luo et al. Nov 2015 A1
20150334025 Rader Nov 2015 A1
20150334696 Gu et al. Nov 2015 A1
20150341271 Gomez Nov 2015 A1
20150349978 Wu et al. Dec 2015 A1
20150350907 Timariu et al. Dec 2015 A1
20150358236 Roach et al. Dec 2015 A1
20150363221 Terayama et al. Dec 2015 A1
20150363733 Brown Dec 2015 A1
20150365323 Duminuco et al. Dec 2015 A1
20150372943 Hasan et al. Dec 2015 A1
20150372982 Herle et al. Dec 2015 A1
20150381407 Wang et al. Dec 2015 A1
20150381493 Bansal et al. Dec 2015 A1
20160020844 Hart et al. Jan 2016 A1
20160021597 Hart et al. Jan 2016 A1
20160035183 Buchholz et al. Feb 2016 A1
20160036924 Koppolu et al. Feb 2016 A1
20160036938 Aviles et al. Feb 2016 A1
20160037434 Gopal et al. Feb 2016 A1
20160072669 Saavedra Mar 2016 A1
20160072684 Manuguri et al. Mar 2016 A1
20160080502 Yadav et al. Mar 2016 A1
20160105353 Cociglio Apr 2016 A1
20160105392 Thakkar et al. Apr 2016 A1
20160105471 Nunes et al. Apr 2016 A1
20160105488 Thakkar et al. Apr 2016 A1
20160117185 Fang et al. Apr 2016 A1
20160134461 Sampath et al. May 2016 A1
20160134528 Lin et al. May 2016 A1
20160134591 Liao et al. May 2016 A1
20160142373 Ossipov May 2016 A1
20160150055 Choi May 2016 A1
20160164832 Bellagamba et al. Jun 2016 A1
20160164914 Madhav et al. Jun 2016 A1
20160173338 Wolting Jun 2016 A1
20160191363 Haraszti et al. Jun 2016 A1
20160191374 Singh et al. Jun 2016 A1
20160192403 Gupta et al. Jun 2016 A1
20160197834 Luft Jul 2016 A1
20160197835 Luft Jul 2016 A1
20160198003 Luft Jul 2016 A1
20160205071 Cooper et al. Jul 2016 A1
20160210209 Verkaik et al. Jul 2016 A1
20160212773 Kanderholm et al. Jul 2016 A1
20160218947 Hughes et al. Jul 2016 A1
20160218951 Vasseur et al. Jul 2016 A1
20160255169 Kovvuri et al. Sep 2016 A1
20160255542 Hughes et al. Sep 2016 A1
20160261493 Li Sep 2016 A1
20160261495 Xia et al. Sep 2016 A1
20160261506 Hegde et al. Sep 2016 A1
20160261639 Xu Sep 2016 A1
20160269298 Li et al. Sep 2016 A1
20160269926 Sundaram Sep 2016 A1
20160285736 Gu Sep 2016 A1
20160301471 Kunz et al. Oct 2016 A1
20160308762 Teng et al. Oct 2016 A1
20160315912 Mayya et al. Oct 2016 A1
20160323377 Einkauf et al. Nov 2016 A1
20160328159 Coddington et al. Nov 2016 A1
20160330111 Manghirmalani et al. Nov 2016 A1
20160337202 Ben-Itzhak et al. Nov 2016 A1
20160352588 Subbarayan et al. Dec 2016 A1
20160353268 Senarath et al. Dec 2016 A1
20160359738 Sullenberger et al. Dec 2016 A1
20160366187 Kamble Dec 2016 A1
20160371153 Dornemann Dec 2016 A1
20160378527 Zamir Dec 2016 A1
20160380886 Blair et al. Dec 2016 A1
20160380906 Hodique et al. Dec 2016 A1
20170005986 Bansal et al. Jan 2017 A1
20170006499 Hampel et al. Jan 2017 A1
20170012870 Blair et al. Jan 2017 A1
20170019428 Cohn Jan 2017 A1
20170026273 Yao et al. Jan 2017 A1
20170026283 Williams et al. Jan 2017 A1
20170026355 Mathaiyan et al. Jan 2017 A1
20170034046 Cai et al. Feb 2017 A1
20170034052 Chanda et al. Feb 2017 A1
20170034129 Sawant et al. Feb 2017 A1
20170048296 Ramalho et al. Feb 2017 A1
20170053258 Carney et al. Feb 2017 A1
20170055131 Kong et al. Feb 2017 A1
20170063674 Maskalik et al. Mar 2017 A1
20170063782 Jain et al. Mar 2017 A1
20170063794 Jain et al. Mar 2017 A1
20170064005 Lee Mar 2017 A1
20170075710 Prasad et al. Mar 2017 A1
20170093625 Pera et al. Mar 2017 A1
20170097841 Chang et al. Apr 2017 A1
20170104653 Badea et al. Apr 2017 A1
20170104755 Arregoces et al. Apr 2017 A1
20170109212 Gaurav et al. Apr 2017 A1
20170118067 Vedula Apr 2017 A1
20170118173 Arramreddy et al. Apr 2017 A1
20170123939 Maheshwari et al. May 2017 A1
20170126516 Tiagi et al. May 2017 A1
20170126564 Mayya et al. May 2017 A1
20170134186 Mukundan et al. May 2017 A1
20170134520 Abbasi et al. May 2017 A1
20170139789 Fries et al. May 2017 A1
20170142000 Cai et al. May 2017 A1
20170149637 Banikazemi et al. May 2017 A1
20170155557 Desai et al. Jun 2017 A1
20170155590 Dillon et al. Jun 2017 A1
20170163473 Sadana et al. Jun 2017 A1
20170171310 Gardner Jun 2017 A1
20170180220 Leckey et al. Jun 2017 A1
20170181210 Nadella et al. Jun 2017 A1
20170195161 Ruel et al. Jul 2017 A1
20170195169 Mills et al. Jul 2017 A1
20170201585 Doraiswamy et al. Jul 2017 A1
20170207976 Rovner et al. Jul 2017 A1
20170214545 Cheng et al. Jul 2017 A1
20170214701 Hasan Jul 2017 A1
20170223117 Messerli et al. Aug 2017 A1
20170236060 Ignatyev Aug 2017 A1
20170237710 Mayya et al. Aug 2017 A1
20170257260 Govindan et al. Sep 2017 A1
20170257309 Appanna Sep 2017 A1
20170264496 Ao et al. Sep 2017 A1
20170279717 Bethers et al. Sep 2017 A1
20170279741 Elias et al. Sep 2017 A1
20170279803 Desai et al. Sep 2017 A1
20170280474 Vesterinen et al. Sep 2017 A1
20170288987 Pasupathy et al. Oct 2017 A1
20170289002 Ganguli et al. Oct 2017 A1
20170289027 Ratnasingham Oct 2017 A1
20170295264 Touitou et al. Oct 2017 A1
20170302565 Ghobadi et al. Oct 2017 A1
20170310641 Jiang et al. Oct 2017 A1
20170310691 Vasseur et al. Oct 2017 A1
20170317954 Masurekar et al. Nov 2017 A1
20170317969 Masurekar et al. Nov 2017 A1
20170317974 Masurekar et al. Nov 2017 A1
20170324628 Dhanabalan Nov 2017 A1
20170337086 Zhu et al. Nov 2017 A1
20170339054 Yadav et al. Nov 2017 A1
20170339070 Chang et al. Nov 2017 A1
20170364419 Lo Dec 2017 A1
20170366445 Nemirovsky et al. Dec 2017 A1
20170366467 Martin et al. Dec 2017 A1
20170373950 Szilagyi et al. Dec 2017 A1
20170374174 Evens et al. Dec 2017 A1
20180006995 Bickhart et al. Jan 2018 A1
20180007005 Chanda et al. Jan 2018 A1
20180007123 Cheng et al. Jan 2018 A1
20180013636 Seetharamaiah et al. Jan 2018 A1
20180014051 Phillips et al. Jan 2018 A1
20180020035 Boggia et al. Jan 2018 A1
20180034668 Mayya et al. Feb 2018 A1
20180041425 Zhang Feb 2018 A1
20180062875 Tumuluru Mar 2018 A1
20180062914 Boutros et al. Mar 2018 A1
20180062917 Chandrashekhar et al. Mar 2018 A1
20180063036 Chandrashekhar et al. Mar 2018 A1
20180063193 Chandrashekhar et al. Mar 2018 A1
20180063233 Park Mar 2018 A1
20180063743 Tumuluru et al. Mar 2018 A1
20180069924 Tumuluru et al. Mar 2018 A1
20180074909 Bishop et al. Mar 2018 A1
20180077081 Lauer et al. Mar 2018 A1
20180077202 Xu Mar 2018 A1
20180084081 Kuchibhotla et al. Mar 2018 A1
20180097725 Wood et al. Apr 2018 A1
20180114569 Strachan et al. Apr 2018 A1
20180123910 Fitzgibbon May 2018 A1
20180123946 Ramachandran et al. May 2018 A1
20180131608 Jiang et al. May 2018 A1
20180131615 Zhang May 2018 A1
20180131720 Hobson et al. May 2018 A1
20180145899 Rao May 2018 A1
20180159796 Wang et al. Jun 2018 A1
20180159856 Gujarathi Jun 2018 A1
20180167378 Kostyukov et al. Jun 2018 A1
20180176073 Dubey et al. Jun 2018 A1
20180176082 Katz et al. Jun 2018 A1
20180176130 Banerjee et al. Jun 2018 A1
20180213472 Ishii et al. Jul 2018 A1
20180219765 Michael et al. Aug 2018 A1
20180219766 Michael et al. Aug 2018 A1
20180234300 Mayya et al. Aug 2018 A1
20180248790 Tan et al. Aug 2018 A1
20180260125 Botes et al. Sep 2018 A1
20180262468 Kumar et al. Sep 2018 A1
20180270104 Zheng et al. Sep 2018 A1
20180278541 Wu et al. Sep 2018 A1
20180287907 Kulshreshtha et al. Oct 2018 A1
20180295101 Gehrmann Oct 2018 A1
20180295529 Jen et al. Oct 2018 A1
20180302286 Mayya et al. Oct 2018 A1
20180302321 Manthiramoorthy et al. Oct 2018 A1
20180307851 Lewis Oct 2018 A1
20180316606 Sung et al. Nov 2018 A1
20180351855 Sood et al. Dec 2018 A1
20180351862 Jeganathan et al. Dec 2018 A1
20180351863 Vairavakkalai et al. Dec 2018 A1
20180351882 Jeganathan et al. Dec 2018 A1
20180367445 Bajaj Dec 2018 A1
20180373558 Chang et al. Dec 2018 A1
20180375744 Mayya et al. Dec 2018 A1
20180375824 Mayya et al. Dec 2018 A1
20180375967 Pithawala et al. Dec 2018 A1
20190013883 Vargas et al. Jan 2019 A1
20190014038 Ritchie Jan 2019 A1
20190020588 Twitchell, Jr. Jan 2019 A1
20190020627 Yuan Jan 2019 A1
20190028378 Houjyo et al. Jan 2019 A1
20190028552 Johnson et al. Jan 2019 A1
20190036808 Shenoy et al. Jan 2019 A1
20190036810 Michael et al. Jan 2019 A1
20190036813 Shenoy et al. Jan 2019 A1
20190046056 Khachaturian et al. Feb 2019 A1
20190058657 Chunduri et al. Feb 2019 A1
20190058709 Kempf et al. Feb 2019 A1
20190068470 Mirsky Feb 2019 A1
20190068493 Ram et al. Feb 2019 A1
20190068500 Hira Feb 2019 A1
20190075083 Mayya et al. Mar 2019 A1
20190103990 Cidon et al. Apr 2019 A1
20190103991 Cidon et al. Apr 2019 A1
20190103992 Cidon et al. Apr 2019 A1
20190103993 Cidon et al. Apr 2019 A1
20190104035 Cidon et al. Apr 2019 A1
20190104049 Cidon et al. Apr 2019 A1
20190104050 Cidon et al. Apr 2019 A1
20190104051 Cidon et al. Apr 2019 A1
20190104052 Cidon et al. Apr 2019 A1
20190104053 Cidon et al. Apr 2019 A1
20190104063 Cidon et al. Apr 2019 A1
20190104064 Cidon et al. Apr 2019 A1
20190104109 Cidon et al. Apr 2019 A1
20190104111 Cidon et al. Apr 2019 A1
20190104413 Cidon et al. Apr 2019 A1
20190109769 Jain et al. Apr 2019 A1
20190132221 Boutros et al. May 2019 A1
20190132234 Dong et al. May 2019 A1
20190140889 Mayya et al. May 2019 A1
20190140890 Mayya et al. May 2019 A1
20190158371 Dillon et al. May 2019 A1
20190158605 Markuze et al. May 2019 A1
20190199539 Deng et al. Jun 2019 A1
20190220703 Prakash et al. Jul 2019 A1
20190238364 Boutros et al. Aug 2019 A1
20190238446 Barzik et al. Aug 2019 A1
20190238449 Michael et al. Aug 2019 A1
20190238450 Michael et al. Aug 2019 A1
20190238483 Marichetty et al. Aug 2019 A1
20190268421 Markuze et al. Aug 2019 A1
20190268973 Bull et al. Aug 2019 A1
20190278631 Bernat et al. Sep 2019 A1
20190280962 Michael et al. Sep 2019 A1
20190280963 Michael et al. Sep 2019 A1
20190280964 Michael et al. Sep 2019 A1
20190306197 Degioanni Oct 2019 A1
20190306282 Masputra et al. Oct 2019 A1
20190313907 Khachaturian et al. Oct 2019 A1
20190319847 Nahar et al. Oct 2019 A1
20190327109 Guichard et al. Oct 2019 A1
20190334813 Raj et al. Oct 2019 A1
20190334820 Zhao Oct 2019 A1
20190342201 Singh Nov 2019 A1
20190342219 Liu et al. Nov 2019 A1
20190356736 Narayanaswamy et al. Nov 2019 A1
20190364099 Thakkar et al. Nov 2019 A1
20190364456 Yu Nov 2019 A1
20190372888 Michael et al. Dec 2019 A1
20190372889 Michael et al. Dec 2019 A1
20190372890 Michael et al. Dec 2019 A1
20190394081 Tahhan et al. Dec 2019 A1
20200014609 Hockett et al. Jan 2020 A1
20200014615 Michael et al. Jan 2020 A1
20200014616 Michael et al. Jan 2020 A1
20200014661 Mayya et al. Jan 2020 A1
20200014663 Chen et al. Jan 2020 A1
20200021514 Michael et al. Jan 2020 A1
20200021515 Michael et al. Jan 2020 A1
20200036624 Michael et al. Jan 2020 A1
20200044943 Bor-Yaliniz et al. Feb 2020 A1
20200044969 Hao et al. Feb 2020 A1
20200059420 Abraham Feb 2020 A1
20200059457 Raza et al. Feb 2020 A1
20200059459 Abraham et al. Feb 2020 A1
20200092207 Sipra et al. Mar 2020 A1
20200097327 Beyer et al. Mar 2020 A1
20200099625 Yigit et al. Mar 2020 A1
20200099659 Cometto et al. Mar 2020 A1
20200106696 Michael et al. Apr 2020 A1
20200106706 Mayya et al. Apr 2020 A1
20200119952 Mayya et al. Apr 2020 A1
20200127905 Mayya et al. Apr 2020 A1
20200127911 Gilson et al. Apr 2020 A1
20200153701 Mohan et al. May 2020 A1
20200153736 Liebherr et al. May 2020 A1
20200159661 Keymolen et al. May 2020 A1
20200162407 Tillotson May 2020 A1
20200169473 Rimar et al. May 2020 A1
20200177503 Hooda et al. Jun 2020 A1
20200177550 Vallur et al. Jun 2020 A1
20200177629 Hooda et al. Jun 2020 A1
20200186471 Shen et al. Jun 2020 A1
20200195557 Duan et al. Jun 2020 A1
20200204460 Schneider et al. Jun 2020 A1
20200213212 Dillon et al. Jul 2020 A1
20200213224 Cheng et al. Jul 2020 A1
20200218558 Sreenath et al. Jul 2020 A1
20200235990 Janakiraman et al. Jul 2020 A1
20200235999 Mayya et al. Jul 2020 A1
20200236046 Jain et al. Jul 2020 A1
20200241927 Yang et al. Jul 2020 A1
20200244721 S et al. Jul 2020 A1
20200252234 Ramamoorthi et al. Aug 2020 A1
20200259700 Bhalla et al. Aug 2020 A1
20200267184 Vera-Schockner Aug 2020 A1
20200280587 Janakiraman et al. Sep 2020 A1
20200287819 Theogaraj et al. Sep 2020 A1
20200287976 Theogaraj et al. Sep 2020 A1
20200296011 Jain et al. Sep 2020 A1
20200296026 Michael et al. Sep 2020 A1
20200301764 Thoresen et al. Sep 2020 A1
20200314006 Mackie et al. Oct 2020 A1
20200314614 Moustafa et al. Oct 2020 A1
20200322230 Natal et al. Oct 2020 A1
20200322287 Connor et al. Oct 2020 A1
20200336336 Sethi et al. Oct 2020 A1
20200344143 Faseela et al. Oct 2020 A1
20200344163 Gupta et al. Oct 2020 A1
20200351188 Arora et al. Nov 2020 A1
20200358878 Bansal et al. Nov 2020 A1
20200366530 Mukundan et al. Nov 2020 A1
20200366562 Mayya et al. Nov 2020 A1
20200382345 Zhao et al. Dec 2020 A1
20200382387 Pasupathy et al. Dec 2020 A1
20200412483 Tan et al. Dec 2020 A1
20200412576 Kondapavuluru et al. Dec 2020 A1
20200413283 Shen et al. Dec 2020 A1
20210006482 Hwang et al. Jan 2021 A1
20210006490 Michael et al. Jan 2021 A1
20210029019 Kottapalli Jan 2021 A1
20210029088 Mayya et al. Jan 2021 A1
20210036888 Makkalla et al. Feb 2021 A1
20210036987 Mishra et al. Feb 2021 A1
20210067372 Cidon et al. Mar 2021 A1
20210067373 Cidon et al. Mar 2021 A1
20210067374 Cidon et al. Mar 2021 A1
20210067375 Cidon et al. Mar 2021 A1
20210067407 Cidon et al. Mar 2021 A1
20210067427 Cidon et al. Mar 2021 A1
20210067442 Sundararajan et al. Mar 2021 A1
20210067461 Cidon et al. Mar 2021 A1
20210067464 Cidon et al. Mar 2021 A1
20210067467 Cidon et al. Mar 2021 A1
20210067468 Cidon et al. Mar 2021 A1
20210073001 Rogers et al. Mar 2021 A1
20210092062 Dhanabalan et al. Mar 2021 A1
20210105199 H et al. Apr 2021 A1
20210112034 Sundararajan et al. Apr 2021 A1
20210126830 R. et al. Apr 2021 A1
20210126853 Ramaswamy et al. Apr 2021 A1
20210126854 Guo et al. Apr 2021 A1
20210126860 Ramaswamy Apr 2021 A1
20210144091 H et al. May 2021 A1
20210160169 Shen et al. May 2021 A1
20210160813 Gupta et al. May 2021 A1
20210176255 Hill et al. Jun 2021 A1
20210184952 Mayya et al. Jun 2021 A1
20210184966 Ramaswamy et al. Jun 2021 A1
20210184983 Ramaswamy et al. Jun 2021 A1
20210194814 Roux et al. Jun 2021 A1
20210226880 Ramamoorthy et al. Jul 2021 A1
20210234728 Cidon et al. Jul 2021 A1
20210234775 Devadoss et al. Jul 2021 A1
20210234786 Devadoss et al. Jul 2021 A1
20210234804 Devadoss et al. Jul 2021 A1
20210234805 Devadoss et al. Jul 2021 A1
20210235312 Devadoss et al. Jul 2021 A1
20210235313 Devadoss Jul 2021 A1
20210266262 Subramanian et al. Aug 2021 A1
20210279069 Salgaonkar et al. Sep 2021 A1
20210314289 Chandrashekhar et al. Oct 2021 A1
20210328835 Mayya et al. Oct 2021 A1
20210336880 Gupta et al. Oct 2021 A1
20210377109 Shrivastava et al. Dec 2021 A1
20210377156 Michael et al. Dec 2021 A1
20210392060 Silva et al. Dec 2021 A1
20210392070 Tootaghaj et al. Dec 2021 A1
20210399920 Sundararajan et al. Dec 2021 A1
20210399978 Michael et al. Dec 2021 A9
20210400113 Markuze et al. Dec 2021 A1
20210409277 Jeuk et al. Dec 2021 A1
20220006726 Michael et al. Jan 2022 A1
20220006751 Ramaswamy et al. Jan 2022 A1
20220006756 Ramaswamy Jan 2022 A1
20220035673 Markuze et al. Feb 2022 A1
20220038370 Vasseur et al. Feb 2022 A1
20220038557 Markuze et al. Feb 2022 A1
20220045927 Liu et al. Feb 2022 A1
20220052928 Sundararajan et al. Feb 2022 A1
20220086035 Devaraj et al. Mar 2022 A1
20220094644 Cidon et al. Mar 2022 A1
20220123961 Mukundan et al. Apr 2022 A1
20220131740 Mayya et al. Apr 2022 A1
20220131807 Srinivas Apr 2022 A1
20220141184 Oswal et al. May 2022 A1
20220158923 Ramaswamy et al. May 2022 A1
20220158924 Ramaswamy et al. May 2022 A1
20220166713 Markuze et al. May 2022 A1
20220210041 Gandhi et al. Jun 2022 A1
20220210042 Gandhi Jun 2022 A1
20220231949 Ramaswamy et al. Jul 2022 A1
20220231950 Ramaswamy et al. Jul 2022 A1
20220232411 Vijayakumar et al. Jul 2022 A1
20220239596 Kumar et al. Jul 2022 A1
20220294701 Mayya et al. Sep 2022 A1
20220335027 Seshadri et al. Oct 2022 A1
20220337553 Mayya et al. Oct 2022 A1
20220353152 Ramaswamy Nov 2022 A1
20220353171 Ramaswamy et al. Nov 2022 A1
20220353175 Ramaswamy et al. Nov 2022 A1
20220353182 Ramaswamy et al. Nov 2022 A1
20220353190 Ramaswamy et al. Nov 2022 A1
20220407773 Kempanna et al. Dec 2022 A1
20220407774 Kempanna et al. Dec 2022 A1
20220407790 Kempanna et al. Dec 2022 A1
20220407820 Kempanna et al. Dec 2022 A1
20220407915 Kempanna et al. Dec 2022 A1
20230006929 Mayya et al. Jan 2023 A1
20230025586 Rolando et al. Jan 2023 A1
20230026330 Rolando et al. Jan 2023 A1
20230026865 Rolando et al. Jan 2023 A1
20230028872 Ramaswamy Jan 2023 A1
20230039869 Ramaswamy et al. Feb 2023 A1
Foreign Referenced Citations (45)
Number Date Country
1926809 Mar 2007 CN
102577270 Jul 2012 CN
102811165 Dec 2012 CN
104956329 Sep 2015 CN
106230650 Dec 2016 CN
106656847 May 2017 CN
110447209 Nov 2019 CN
111198764 May 2020 CN
1912381 Apr 2008 EP
2538637 Dec 2012 EP
2763362 Aug 2014 EP
3041178 Jul 2016 EP
3297211 Mar 2018 EP
3509256 Jul 2019 EP
3346650 Nov 2019 EP
2010233126 Oct 2010 JP
2014200010 Oct 2014 JP
2017059991 Mar 2017 JP
2017524290 Aug 2017 JP
2574350 Feb 2016 RU
03073701 Sep 2003 WO
2007016834 Feb 2007 WO
2012167184 Dec 2012 WO
2015092565 Jun 2015 WO
2016061546 Apr 2016 WO
2016123314 Aug 2016 WO
2017083975 May 2017 WO
2019070611 Apr 2019 WO
2019094522 May 2019 WO
2020012491 Jan 2020 WO
2020018704 Jan 2020 WO
2020091777 May 2020 WO
2020101922 May 2020 WO
2020112345 Jun 2020 WO
2021040934 Mar 2021 WO
2021118717 Jun 2021 WO
2021150465 Jul 2021 WO
2021211906 Oct 2021 WO
2022005607 Jan 2022 WO
2022154850 Jul 2022 WO
2022159156 Jul 2022 WO
2022231668 Nov 2022 WO
2022235303 Nov 2022 WO
2022265681 Dec 2022 WO
2023009159 Feb 2023 WO
Non-Patent Literature Citations (59)
Entry
Non-Published Commonly Owned U.S. Appl. No. 18/083,536, filed Dec. 18, 2022, 27 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 18/102,685, filed Jan. 28, 2023, 124 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 18/102,687, filed Jan. 28, 2023, 172 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 18/102,688, filed Jan. 28, 2023, 49 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 18/102,689, filed Jan. 28, 2023, 46 pages, VMware, Inc.
Taleb, Tarik, “D4.1 Mobile Network Cloud Component Design,” Mobile Cloud Networking, Nov. 8, 2013, 210 pages, MobileCloud Networking Consortium, retrieved from http://www.mobile-cloud-networking.eu/site/index.php?process=download&id=127&code=89d30565cd2ce087d3f8e95f9ad683066510a61f.
Valtulina, Luca, “Seamless Distributed Mobility Management (DMM) Solution in Cloud Based LTE Systems,” Master Thesis, Nov. 2013, 168 pages, University of Twente, retrieved from http://essay.utwente.nl/64411/1/Luca_Valtulina_MSc_Report_final.pdf.
Alsaeedi, Mohammed, et al., “Toward Adaptive and Scalable OpenFlow-SDN Flow Control: A Survey,” IEEE Access, Aug. 1, 2019, 34 pages, vol. 7, IEEE, retrieved from https://ieeexplore.ieee.org/document/8784036.
Alvizu, Rodolfo, et al., “SDN-Based Network Orchestration for New Dynamic Enterprise Networking Services,” 2017 19th International Conference on Transparent Optical Networks, Jul. 2-6, 2017, 4 pages, IEEE, Girona, Spain.
Barozet, Jean-Marc, “Cisco SD-WAN as a Managed Service,” BRKRST-2558, Jan. 27-31, 2020, 98 pages, Cisco, Barcelona, Spain, retrieved from https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKRST-2558.pdf.
Barozet, Jean-Marc, “Cisco SDWAN,” Deep Dive, Dec. 2017, 185 pages, Cisco, Retreived from https://www.coursehero.com/file/71671376/Cisco-SDWAN-Deep-Divepdf/.
Bertaux, Lionel, et al., “Software Defined Networking and Virtualization for Broadband Satellite Networks,” IEEE Communications Magazine, Mar. 18, 2015, 7 pages, vol. 53, IEEE, retrieved from https://ieeexplore.ieee.org/document/7060482.
Cox, Jacob H., et al., “Advancing Software-Defined Networks: A Survey,” IEEE Access, Oct. 12, 2017, 40 pages, vol. 5, IEEE, retrieved from https://ieeexplore.ieee.org/document/8066287.
Del Piccolo, Valentin, et al., “A Survey of Network Isolation Solutions for Multi-Tenant Data Centers,” IEEE Communications Society, Apr. 20, 2016, vol. 18, No. 4, 37 pages, IEEE.
Duan, Zhenhai, et al., “Service Overlay Networks: SLAs, QoS, and Bandwidth Provisioning,” IEEE/ACM Transactions on Networking, Dec. 2003, 14 pages, vol. 11, IEEE, New York, NY, USA.
Fortz, Bernard, et al., “Internet Traffic Engineering by Optimizing OSPF Weights,” Proceedings IEEE INFOCOM 2000, Conference on Computer Communications, Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies, Mar. 26-30, 2000, 11 pages, IEEE, Tel Aviv, Israel, Israel.
Francois, Frederic, et al., “Optimizing Secure SDN-enabled Inter-Data Centre Overlay Networks through Cognitive Routing,” 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), Sep. 19-21, 2016, 10 pages, IEEE, London, UK.
Guo, Xiangyi, et al., (U.S. Appl. No. 62/925,193), filed Oct. 23, 2019, 26 pages.
Huang, Cancan, et al., “Modification of Q.SD-WAN,” Rapporteur Group Meeting—Doc, Study Period 2017-2020, Q4/11-DOC1 (190410), Study Group 11, Apr. 10, 2019, 19 pages, International Telecommunication Union, Geneva, Switzerland.
Jivorasetkul, Supalerk, et al., “End-to-End Header Compression over Software-Defined Networks: a Low Latency Network Architecture,” 2012 Fourth International Conference on Intelligent Networking and Collaborative Systems, Sep. 19-21, 2012, 2 pages, IEEE, Bucharest, Romania.
Lasserre, Marc, et al., “Framework for Data Center (DC) Network Virtualization,” RFC 7365, Oct. 2014, 26 pages, IETF.
Li, Shengru, et al., “Source Routing with Protocol-oblivious Forwarding (POF) to Enable Efficient e-Health Data Transfers,” 2016 IEEE International Conference on Communications (ICC), May 22-27, 2016, 6 pages, IEEE, Kuala Lumpur, Malaysia.
Lin, Weidong, et al., “Using Path Label Routing in Wide Area Software-Defined Networks with Open Flow,” 2016 International Conference on Networking and Network Applications, Jul. 2016, 6 pages, IEEE.
Long, Feng, “Research and Application of Cloud Storage Technology in University Information Service,” Chinese Excellent Masters' Theses Full-text Database, Mar. 2013, 72 pages, China Academic Journals Electronic Publishing House, China.
Michael, Nithin, et al., “HALO: Hop-by-Hop Adaptive Link-State Optimal Routing,” IEEE/ACM Transactions on Networking, Dec. 2015, 14 pages, vol. 23, No. 6, IEEE.
Ming, Gao, et al., “A Design of SD-WAN-Oriented Wide Area Network Access,” 2020 International Conference on Computer Communication and Network Security (CCNS), Aug. 21-23, 2020, 4 pages, IEEE, Xi'an, China.
Mishra, Mayank, et al., “Managing Network Reservation for Tenants in Oversubscribed Clouds,” 2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems, Aug. 14-16, 2013, 10 pages, IEEE, San Francisco, CA, USA.
Mudigonda, Jayaram, et al., “NetLord: A Scalable Multi-Tenant Network Architecture for Virtualized Datacenters,” Proceedings of the ACM SIGCOMM 2011 Conference, Aug. 15-19, 2011, 12 pages, ACM, Toronto, Canada.
Non-Published Commonly Owned U.S. Appl. No. 17/351,327, filed Jun. 18, 2021, 48 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/351,342, filed Jun. 18, 2021, 47 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/351,345, filed Jun. 18, 2021, 48 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/384,735, filed Jul. 24, 2021, 62 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/384,736, filed Jul. 24, 2021, 63 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/384,738, filed Jul. 24, 2021, 62 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/569,517, filed Jan. 6, 2022, 49 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/569,519, filed Jan. 6, 2022, 48 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/569,520, filed Jan. 6, 2022, 50 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/569,522, filed Jan. 6, 2022, 48 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/569,523, filed Jan. 6, 2022, 48 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/569,524, filed Jan. 6, 2022, 48 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/569,526, filed Jan. 6, 2022, 27 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/574,225, filed Jan. 12, 2022, 56 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/574,236, filed Jan. 12, 2022, 54 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/695,264, filed Mar. 15, 2022, 28 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/735,029, filed May 2, 2022, 36 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/827,972, filed May 30, 2022, 30 pages, Nicira, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/850,112, filed Jun. 27, 2022, 41 pages, Nicira, Inc.
Non-Published Commonly Owned U.S. Appl. No. 15/803,964, filed Nov. 6, 2017, 15 pages, The Mode Group.
Noormohammadpour, Mohammad, et al., “DCRoute: Speeding up Inter-Datacenter Traffic Allocation while Guaranteeing Deadlines,” 2016 IEEE 23rd International Conference on High Performance Computing (HiPC), Dec. 19-22, 2016, 9 pages, IEEE, Hyderabad, India.
Ray, Saikat, et al., “Always Acyclic Distributed Path Computation,” University of Pennsylvania Department of Electrical and Systems Engineering Technical Report, May 2008, 16 pages, University of Pennsylvania ScholarlyCommons.
Sarhan, Soliman Abd Elmonsef, et al., “Data Inspection in SDN Network,” 2018 13th International Conference on Computer Engineering and Systems (ICCES), Dec. 18-19, 2018, 6 pages, IEEE, Cairo, Egypt.
Tootaghaj, Diman Zad, et al., “Homa: An Efficient Topology and Route Management Approach in SD-WAN Overlays,” IEEE INFOCOM 2020—IEEE Conference on Computer Communications, Jul. 6-9, 2020, 10 pages, IEEE, Toronto, ON, Canada.
Webb, Kevin C., et al., “Blender: Upgrading Tenant-Based Data Center Networking,” 2014 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS), Oct. 20-21, 2014, 11 pages, IEEE, Marina del Rey, CA, USA.
Xie, Junfeng, et al., A Survey of Machine Learning Techniques Applied to Software Defined Networking (SDN): Research Issues and Challenges, IEEE Communications Surveys & Tutorials, Aug. 23, 2018, 38 pages, vol. 21, Issue 1, IEEE.
Yap, Kok-Kiong, et al., “Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering,” SIGCOMM '17: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, Aug. 21-25, 2017, 14 pages, Los Angeles, CA.
Non-Published Commonly Owned U.S. Appl. No. 17/943,147, filed Sep. 12, 2022, 42 pages, Nicira, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/967,795, filed Oct. 17, 2022, 39 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 17/976,784, filed Oct. 29, 2022, 55 pages, VMware, Inc.
Zakurdaev, Gieorgi, et al., “Dynamic On-Demand Virtual Extensible LAN Tunnels via Software-Defined Wide Area Networks,” 2022 IEEE 12th Annual Computing and Communication Workshop and Conference, Jan. 26-29, 2022, 6 pages, IEEE, Las Vegas, NV, USA.
Related Publications (1)
Number Date Country
20220360500 A1 Nov 2022 US