Patent Grant
11375005
Software-Defined Wide Area Network (SD-WAN) is a technology that simplifies wide area networking through centralized control of the networking hardware or software that directs traffic across a wide area network (WAN). It also allows organizations to combine or replace private WAN connections with Internet, broadband, Long Term Evolution (LTE), and/or 5G connections. The central controller sets policies, prioritizes, optimizes, and routes WAN traffic, and selects the best link and path dynamically for optimum performance. SD-WAN vendors may offer security functions with their SD-WAN virtual or physical appliances, which are typically deployed at datacenters or branch offices.
Secure Access Service Edge (SASE) is a security framework that provides WAN security as a cloud service to the source of connection (e.g., user, device, branch office, IoT devices, edge computing locations) rather than an enterprise datacenter. Security is based on identity, real-time context, and enterprise security and compliance policies. An identity may be attached to anything from a person/user to a device, branch office, cloud service, application, IoT system, or an edge computing location. Typically, SASE incorporates SD-WAN as part of a cloud service that also delivers mobile access and a full security stack delivered from a local point of presence or PoP (e.g., routers, switches, servers, and other devices necessary for traffic to cross over networks.) SASE converges the connectivity and security stacks and moves them to the network edge. A security stack that once resided in appliances in the datacenter or in branch locations on the perimeter is installed in the cloud as a converged, integrated stack, which can also be referred to as a SASE stack.
Some embodiments provide a cloud native solution or software-defined wide area network (SD-WAN) environment that hides network virtualization management user interface components. Specifically, a SD-WAN orchestrator performs or drives network virtualization management operations such as provisioning tenants, configuring network services, and supporting operations in the SD-WAN environment. The network virtualization management deployment is partitioned among tenants using constructs such as tenant service routers (T1-SRs) and provider service routers (T0-SRs) so that all traffic can be policed appropriately.
In some embodiments, edge security services such as L4-7 firewalls, URL filtering, TLS proxy, IDS/IPS, etc., that are managed by the network virtualization management deployment are made available in the SD-WAN environment so security services can be applied to classify and police user traffic in the SD-WAN. In some embodiments, cloud gateways forward SD-WAN traffic to managed service nodes to apply security services. Network traffic is encapsulated with corresponding metadata to ensure that services can be performed according to the desired policy. Point-to-point tunnels are established between cloud gateways and the managed service nodes to transport the metadata to the managed service nodes using a particular overlay logical network. Virtual network identifiers (VNIs) in the transported metadata are used by the managed service nodes to identify tenants/policies. A managed service node receiving a packet uses the appropriate tenant-level service routers (or T1-SRs) based on the VNI to apply the prescribed services for the tenant, and the resulting traffic is returned to the cloud gateway that originated the traffic.
In some embodiments, the network virtualization management deployment provides stateful active-active (A/A) high availability services for SASE to protect against hardware failures in a PoP. Specifically, a pair of managed service nodes in a same PoP are configured to jointly provide stateful network security services in A/A configuration. When one managed service node in the pair fails, the other managed service node takes over by assuming the tunnel endpoint and the service states of the failed managed service node.
In some embodiments, the T1-SRs and T0-SRs have uplink and downlink connections with an external network. In some embodiments, a managed service node implementing a T0-SR and one or more T1-SRs performs two layers of address translation on packet traffic going to the external network. The two layers of address translation is for ensuring that the response traffic from the external network can successfully arrive back at the managed service node.
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description, the Drawings, and the Claims is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description, and the Drawings, but rather are to be defined by the appended claims, because the claimed subject matters can be embodied in other specific forms without departing from the spirit of the subject matters.
The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Network virtualization management (e.g., VMware NSX®) is normally deployed on premises, points-of-presence (PoPs), or in a virtual private cloud environment where the same administrative entity operates the deployment. However, in secure access service edge (SASE) use cases, a single network virtualization management deployment is expected to be shared by multiple customers/tenants and is expected to be cloud-based. Typically, users are not concerned with the location of the various network virtualization management components, which are consumable as a homogeneous entity regardless of their physical placement.
Some embodiments provide a cloud native solution or SD-WAN environment that hides network virtualization management user interface components (e.g., APIs). Specifically, a SD-WAN orchestrator (e.g., VeloCloud Orchestrator® or VCO) performs or drives network virtualization management operations such as provisioning tenants, configuring network services, and supporting operations in the SD-WAN environment. The network virtualization management deployment is partitioned among tenants using constructs such as tenant service routers (T1-SRs) and provider service routers (T0-SRs) so that all traffic can be policed appropriately.
In some embodiments, edge security services such as L4-7 firewalls, URL filtering, TLS proxy, IDS/IPS, etc., that are managed by the network virtualization management deployment are made available in the SD-WAN environment so security services can be applied to classify and police user traffic in the SD-WAN. In some embodiments, cloud gateways (also referred to as SD-WAN gateways, e.g., VeloCloud® Gateways, or VCGs) forward SD-WAN traffic to managed service nodes (that are managed by network virtualization management) to apply security services. Network traffic is encapsulated with corresponding metadata to ensure that services can be performed according to the desired policy. Point-to-point tunnels are established between cloud gateways and the managed service nodes to transport the metadata to the managed service nodes using a particular overlay logical network (e.g., VMware Geneve®). Virtual network identifiers (VNIs) in the transported metadata are used by the managed service nodes to identify tenants/policies. A managed service node receiving a packet (or other types of data messages) uses appropriate tenant-level service routers (or T1-SRs) based on the VNI to apply the prescribed services for the tenant, and the resulting traffic is returned to the cloud gateway that originated the traffic. This operation is also referred to as data plane stitching.
As illustrated in
The SD-WAN orchestrator 105 configures and/or provisions SD-WAN components such as cloud gateways 111-113 (also referred to as SD-WAN gateways, e.g., VeloCloud Gateways® or VCGs) and cloud edges 121-124 (also referred to as SD-WAN edges, e.g., VeloCloud Edges®, or VCEs.) The cloud gateways (VCGs) are hosted in PoPs in the cloud, and these PoPs may be physically located around the world. Different traffic streams in the SD-WAN are sent to the cloud gateways and they route the traffic to their destinations, such as cloud datacenters or corporate datacenters.
The cloud gateways perform optimizations between themselves and the cloud edges. The cloud edges (VCEs) can be configured to use cloud gateways which are physically nearby for better performance. Cloud edges are devices placed in the branch offices and datacenters. It can terminate multiple WAN connections and steer traffic over them for the best performance and reliability. A cloud edge device may provide support for various routing protocols such as Open Shortest Path First (OSPF) and Border Gateway Protocol (BGP), along with static routing, with an IP service-level agreement (SLA). It can also perform functionalities of legacy routers.
As illustrated, the cloud gateways 111-113 are physically located in different parts of the underlying physical network to route network traffic between various datacenters, branch offices, and service providers that participate in the SD-WAN 100. The cloud edge 121 is configured to extend the SD-WAN 100 into a branch office 131, the cloud edge 122 is configured to extend the SD-WAN 100 into a branch office 132, the cloud edge 123 is configured to extend the SD-WAN 100 into a datacenter 133, and the cloud edge 124 is configured to extend the SD-WAN 100 into a datacenter 134. Each of the cloud edges 121-124 use one or more physically proximate cloud gateways 111-113 to route traffic through the SD-WAN 100. In the example of
In some embodiments, the orchestrater 105 is part of a cloud-hosted centralized management system (e.g., VMware SD-WAN Orchestrator®, or VCO), which may be hosted by a management cluster that exists either within a same PoP or across multiple different PoPs. In some embodiments, the cloud edges 121-124 connect to the SD-WAN orchestrator 105 and download their configurations from it. The SD-WAN orchestrator 105 also provide visibility into the performance of the various SD-WAN components and aid in their troubleshooting. In some embodiments, the network virtualization management software exposes a set of APIs that can be used by the SD-WAN orchestrator 105 to drive the network virtualization management deployment to e.g., control the managed service node 141, define security policies, and drive configuration of security services in the managed service nodes.
In the SD-WAN environment 100, a managed service node 141 makes security services from security service provider 135 available to tenants or customers of the SD-WAN 100. A tenant may be a customer of the SD-WAN provider or a subdivision of a customer (e.g. a business unit, a site, etc.). More generally, boundaries of tenant segments are defined along different security postures. In some embodiments, managed service nodes are for providing security and gateway services that cannot be run on distributed routers. The managed service nodes may apply security services on E-W traffic from other network entities (e.g., cloud gateways) of the same PoP. These managed service nodes may also perform edge services such as N-S routing (traffic to and from external network), load balancing, DHCP, VPN, NAT, etc. In some embodiments, the managed service nodes are running as a virtual machine (VM) or data compute node (DCN) at a host machine running virtualization software or a hypervisor such as VMware ESX®. These managed service nodes are controlled by the network virtualization management deployment (e.g., VMware NSX-T® Edge). The orchestrator 105 communicates with network virtualization management deployment to configure the managed service node 141.
In some embodiments, the orchestration scheme of the SD-WAN 100 has multiple tiers. A first tier of the orchestration scheme handles user facing interactions (labeled as “orchestration user interface”). A second, intermediate tier (labeled as “orchestration intermediate layer”) handles the orchestrator's interactions with each PoP, including communicating with network virtualization management (e.g., NSX-T®), virtualization software (e.g., ESX®), and server management (e.g., vCenter®). The intermediate tier may also handle any rules translation between different domains, etc.
A SD-WAN may serve multiple different tenants. In some embodiments, the SD-WAN is partitioned into tenant segments (also referred to as velo segments or SD-WAN segments), each tenant segment is for conducting the traffic of a tenant. In the example, the SD-WAN 100 has three tenant segments A, B, and C. Each tenant segment may span across multiple datacenters and/or branch offices. In the example of
In some embodiments, the network virtualization management deployment provides network entities to apply the different sets of security policies to packet traffic of different tenant segments. In some embodiments, dedicated tenant-level (T1) entities are defined to apply security policies to individual tenant segments. In some embodiments, for each tenant segment, one or more dedicated tenant-level service routers (T1-SR) are used as processing pipelines to apply the policies to the packets of the tenant segment.
As mentioned, in the SD-WAN 100, traffic from datacenters and branch offices is sent to cloud edges and cloud gateways. In some embodiments, the cloud gateways send the traffic it receives from cloud edges to the policy-applying T1-SRs. In some embodiments, these T1-SRs are implemented or provisioned in service nodes (e.g., the managed service node 141) managed by the network virtualization management (deployed in PoPs). In some embodiments, the managed service node uses metadata embedded or encapsulated in the packet traffic to identify the tenant segment that the traffic belongs to, or the policies to be applied, and hence which T1-SR should be used to perform the security services. In some embodiments, the managed service node sends the T1-SR-processed packet traffic back to where it originated (e.g., the cloud gateway that sent the packet traffic to the managed service node). In some embodiments, the managed service node forwards the processed packet traffic directly to a destination without going back to the cloud gateway.
In some embodiments, each T1-SR serves a single VNI, which is mapped to a single tenant segment or a set of security policies. In some embodiments, multiple T1-SRs may serve traffic for a same VNI (or tenant segment). For example, the SD-WAN orchestrator 105 may provision a single T1-SR to serve traffic for a portion of a tenant segment. The SD-WAN orchestrator 105 may provision a T1-SR to handle traffic of a tenant segment from a single cloud edge. The SD-WAN orchestrator 105 may provision a first T1-SR to apply a first security policy and a second T1-SR to apply a second security policy for a particular tenant segment. In some embodiments, when the capacity for a single tenant segment or customer exceeds the throughput of an edge node or an edge node pair, the orchestrator 105 may provision additional managed service nodes or T1-SRs to serve traffic for the same VNI.
In some embodiments, the managed service node 141 provides custom stitching logic 230 between the cloud gateways and the T1-SRs, as well as uplink logic 240 between T1-SRs and the Internet. In some embodiments, the stitching logic 230 is for encapsulation and decapsulation of packets as well as demultiplexing traffic, and the uplink logic 240 is for applying routing and source network address translation (SNAT) on traffic going into the Internet. In some embodiments, the SD-WAN orchestrator 105 provisions a provider-level (T0) service router (T0-SR) shared by different tenants in the managed service node 141 to implement the function of the stitching logic 230 and the uplink logic 240. In some embodiments, each packet arriving at the managed service node 141 is encapsulated with metadata. The T0-SR in turn decapsulates the packet and uses the metadata to demultiplex the packet (e.g., to determine which of the T1-SRs 211-214 should the packet be sent based on VNI or application ID in the metadata).
In some embodiments, the cloud gateways send traffic to the managed service nodes (such as the managed service node 141) through overlay tunnels (such as Geneve®) to tunnel endpoints (TEPs) that correspond to the managed service nodes. In some embodiments, each managed service node is addressable by a unique TEP. In some embodiments, one managed service node is addressable by multiple TEPs, such as when one managed service node takes over for another managed service node that has failed in a high availability implementation.
The cloud gateway 111 uses the outer IP addresses to send packet traffic to the managed service node 341 and the managed service node 342. The packet traffic to the managed node 341 is encapsulated traffic in an overlay tunnel 301 destined for a TEP 351 (TEP X), and the packet traffic to the managed node 342 is encapsulated traffic in an overlay tunnel 302 destined for a TEP 352 (TEP Y). The managed service node 341 has a T0-SR 311 that decapsulates the incoming packet traffic to see if the traffic is tunneled towards the TEP 351. The tunnel traffic at the TEP 351 is further distributed to either T1-SR 321 for tenant segment A or T1-SR 322 for tenant segment B. Likewise, the tunnel traffic at the TEP 352 will then be further distributed to either T1-SR 323 for tenant segment A or T1-SR 324 for tenant segment B. The tunnel traffic is isolated based on different VNIs.
In some embodiments, cloud gateways maintain flows that are pinned to tunnel endpoints. Inner/user IP addresses (and flow 5-tuples in general) are unique within a VNI. In some embodiments, a cloud gateway is configured to have a list of tunnel endpoints it can direct traffic to for each tenant segment. In the example of
The process 400 starts when the cloud gateway receives (at 410) a packet from a tenant segment to have security service applied. The cloud gateway looks up (at 420) the VNI of the tenant segment and selects a viable tunnel endpoint for that tenant segment (if multiple endpoints are viable, the cloud gateway may load-balance among the endpoints, but packets of a same flow must remain pinned to the endpoint for the flow's duration.) The cloud gateway encapsulates (at 430) the packet with metadata that includes the VNI of the tenant segment. The cloud gateway then sends (at 440) the encapsulated packet to the selected tunnel endpoint. The packet may have a source MAC address unique to the cloud gateway and a destination MAC that is specified for the selected (viable) tunnel endpoint. In some embodiments, the operations 410-440 are performed by a transmit path of the cloud gateway.
The cloud gateway receives (at 450) an encapsulated packet from a viable tunnel endpoint. In some embodiments, the cloud gateway is configured to accept any packet coming from any tunnel endpoint to its destination port. The cloud gateway then decapsulates (at 460) the received packet to obtain its metadata. The cloud gateway maps (at 470) the VNI in the metadata to a tenant segment in the SD-WAN and forwards (at 480) the decapsulated packet to the tenant segment. The cloud gateway may verify whether the VNI is allowed to reach the tunnel endpoint from which the packet is received. The cloud gateway may also take further actions on the packet (e.g., forward, abort) based on the VNI/tunnel endpoint verification and/or the content in the packet, which includes the result of the security services. The process 400 then ends.
A cloud gateway is a stateful device that doesn't offer any security services but determines which packets belong to which flow as it stores contexts associated with that flow. The cloud gateway keeps a table of tuples for defining flows, so every subsequent packet of the flow would be sent out to the same tunnel endpoint and same T1-SR. The cloud gateway looks up which policy that it needs to apply and whether it involves network virtualization management. The cloud gateway also knows which T0-SRs and T1-SRs are available to process the traffic. This information is communicated from the orchestrator, so the cloud gateway has an idea as to the number of T0 and T1 entities, which entities are active, which entities are dead, and which entities are available for a specific VNI/tenant segment. In some embodiments, when the cloud gateway sees the first packet of a flow, the cloud gateway load balances across all the possible T1-SRs for that tenant segment. At that point, the cloud gateway generates the encapsulation and sends the packet to the T0-SR or managed service node.
In some embodiments, the process 500 starts when the orchestrator identifies (at 510) one or more cloud gateways to receive traffic belonging to a first tenant segment. The orchestrator also identifies (at 520) a first set of security policies for the first tenant segment.
The orchestrator (at 530) then configures a managed service node to implement a first set of T1-SRs (tenant-level service routers) to apply the first set of policies on packet traffic from the first tenant segment. Each T1-SR of the first set of T1-SRs is configured to process traffic having a first VNI that identifies to the first tenant segment, such that the first set of T1-SRs receive packet traffic from the first tenant segment and no other tenant segment.
The orchestrator also configures (at 540) the managed service node to implement a T0-SR (provider-level service router) to relay traffic tunneled by the cloud gateways to the first set of T1-SRs. The T0-SR is a tunnel endpoint for tunnel traffic from the cloud gateways. The T0-SR is also configured to tunnel a packet from a T1-SR back to a cloud gateway that earlier tunnel a corresponding packet to the managed service node. In some embodiments, the orchestrator configures the managed service node by communicating with a network virtualization manager to configure one or more host machines that host the managed service node (e.g., using API of the network virtualization manager.)
The orchestrator then configures (at 550) the identified cloud gateways to tunnel traffic of the first tenant segment to the first set of T1-SRs. The cloud gateways are configured to send packet traffic having the first VNI to a first tunnel endpoint. In some embodiments, each of the cloud gateways is configured to perform the process 400 of
The one or more cloud gateways may receive traffic belonging to a second tenant segment. The orchestrator may identify a second set of security policies for the second tenant segment, configure the managed service node to implement a second set of T1-SRs to apply the second set of security policies on packet traffic from the second tenant segment, and configure the identified cloud gateways to tunnel traffic of the second tenant segment to the second set of T1-SRs. The T0-SR may be configured to relay the traffic tunneled by the cloud gateways to the second set of T1-SRs. The cloud gateways are configured to receive packet traffic from both the first and second sets of T1-SRs.
In some embodiments, the SD-WAN orchestrator may determine the number of managed service nodes to be provisioned based on capacity required (e.g., 2-4 managed service nodes may be enough for small PoPs, while tens or hundreds of managed service nodes may be necessary for larger PoPs.) The number of managed service nodes may also depend on amount of traffic, number of customers, or complexity of policies being handled, or an input from a user interface.
In some embodiments, the cloud gateway encapsulates packet traffic to tunnel endpoints in the managed service nodes, and the encapsulation of such encapsulated packets includes metadata to indicate the VNI of the tenant segments. The metadata may also include other types of information, such as indicia for identifying which policy or security services to apply to the packet. The T0-SR implemented inside a managed service node decapsulates packet traffic from cloud gateways and encapsulates traffic to the cloud gateways. The T0-SR also demultiplexes packet traffic from cloud gateways to corresponding T1-SRs based on VNIs in the packets and multiplexes packet traffic from T1-SRs back to cloud gateways.
The cloud gateway 121 generates the L2 header 638 by sending out its own source MAC address that is unique among the cloud gateways connected to the managed service node 341. This source MAC address is later used to make sure packet traffic come back to the cloud gateway after service is applied. The destination MAC address belongs to the T1-SR that is targeted to process the packet with services. The cloud gateway also sets the destination outer IP and the destination MAC address based on a specified VNI.
The outer L2 header 638 is used to send the encapsulated packet 620 over L2 switching to the managed service node 341, and the outer L3 header 636 specifies the destination IP address to be 10.0.0.253, which is the address of the tunnel endpoint 351 (TEP X) at the managed service node 341. The T0-SR 311 of the managed service node 341 decapsulates the packet 620 to obtain the metadata 632, which indicates that the packet has a VNI=“1234” (which corresponds to tenant segment A.) The T0-SR 311 uses the VNI to select the T1-SR 321 to process the user packet 610 based on security policies implemented at the T1-SR 321. Another T1-SR 322 of the managed node 341 is associated with VNI=“5678”. Thus, had the encapsulated packet 620 had VNI=“5678” (to indicate tenant segment B), the T0-SR 311 would have selected the T1-SR 322 to process the packet. When the T1-SR 322 has finished processing the packet 610 according to its associated security policies (for tenant segment A), the managed service node 341 hairpins the resulting packet to where the original packet 610 came from, namely the cloud gateway 111.
A managed service node may receive packet traffic from multiple different cloud gateways. In the example of
The T1-SR 321 applies the security services for the VNI “1234” (tenant segment A) on the packet 710 and returns a resulting packet 712 back to the source of the packet 710. The T0-SR 311 receives the returning packet 712 at the tunnel port 701. Knowing the tunnel port 701 is associated with MAC address “:11” or IP address “10.0.0.1”, the T0-SR 311 tunnels the returning packet 712 back to the cloud gateway 111 using those addresses as destination addresses.
The T1-SR 321 applies the security services for the VNI “1234” (tenant segment A) on the packet 710 and returns a resulting packet 722 back to the source of the packet 720. The T0-SR 311 receives the returning packet 722 at the tunnel port 702. Knowing the tunnel port 702 is associated with MAC address “:22” or IP address “10.0.0.2”, the T0-SR 311 tunnels the returning packet 722 back to the cloud gateway 112 using those addresses as destination addresses.
In some embodiments, the managed service node 341 uses a firewall mechanism to recover the source of the packet for keeping the address mapping in a per-segment context (e.g., in T1-SRs), as different tenant segments may have overlap addressing. When an ingress packet (e.g., the packet 710 or 720) reach the T1-SR 321 initially, the firewall creates a stateful flow entry and stores the original inner L2 header. When the firewall sees an egress packet (e.g., the return packet 712 or 722), the firewall maps it to an existing flow. Since all traffic processed by the managed service node is initiated from the cloud gateway, if the managed service node has an egress packet of one flow, it can be assumed that there was a corresponding ingress packet for the same flow (e.g., the incoming packet 710 and the return packet 712 belong to a first flow; the incoming packet 720 and the return packet 722 belong to a second flow). Based on information of the individual flows, the T1-SR 321 sends the return packet to the same interface it came from and restores the original L2 header (with source and destination swapped around).
In some embodiments, the T0-SR has a trunk VNI port, which is an uplink of the T0-SR for reaching the remote cloud gateways. Since the managed service node receives packets using local IPs, the packets go to a CPU port which terminates local traffic. During decapsulation of the incoming packets, the T0-SR determines whether the packet came from IP address 10.0.0.1 or 10.0.0.2 (i.e., cloud gateway 111 or cloud gateway 112). That IP address is mapped into one of the two tunnel ports 701 and 702. Each of the tunnel ports 701 and 702 is in turn connected to logical switches for different VNIs.
In some embodiments, the process 800 starts when the managed service node receives (at 810) a packet belonging to a particular tenant segment from a source cloud gateway. The managed service node receives packets belonging to multiple different tenant segments from multiple different cloud gateways.
The managed service node receives (at 820) a VNI that identifies the particular tenant segment from a metadata encapsulated in the packet. In some embodiments, the packet is encapsulated to include the VNI for identifying the particular tenant segment, and the T0-SR of the managed service node is configured to decapsulate packets coming from cloud gateways and encapsulate packets to cloud gateways.
The managed service node relays (at 830) the packet to a particular T1-SR dedicated to the VNI through a tunnel port associated with the source cloud gateway. The service node includes multiple T1-SRs dedicated to multiple different VNIs and multiple tunnel ports that respectively correspond to the multiple cloud gateways. In some embodiments, a tunnel port that corresponds to a cloud gateway is associated with a MAC address of the cloud gateway.
The managed service node processes (at 840) the packet according to a set of policies (i.e., apply security services) associated with the VNI at the particular T1-SR. The managed service node sends (at 850) a return packet to the source cloud gateway through the tunnel port associated to the source cloud gateway. The cloud gateway then uses the VNI of the return packet to identify the tenant segment and to send the return packet to the corresponding cloud edge. The process 800 then ends. In some embodiments, the managed service node stores a set of flow identifiers for the ingress packet and sets a destination address of the egress packet based on the stored set of flow identifiers. The set of flow identifiers includes the L2 MAC address and/or L3 IP address of the source cloud gateway that is unique among the plurality of cloud gateways.
In some embodiments, the network virtualization management deployment provides stateful active-active (A/A) high availability services for SASE to protect against hardware failures in a PoP. Specifically, a pair of managed service nodes (or a grouping of two or more managed service nodes) in a same PoP are configured to jointly provide stateful network security services in A/A configuration. When one managed service node in the pair fails, the other managed service node takes over by assuming the tunnel endpoint and the service states of the failed managed service node.
In some embodiments, each cloud gateway sends packets to a pairing of two managed service nodes for applying security services. Each cloud gateway is aware that there are two managed service nodes and can address each managed service node individually.
A cloud gateway may establish tunnel communications with each of the two managed service nodes. In the example, the cloud gateway 111 (address 10.0.0.1) may establish one tunnel to the managed service node 341 and another tunnel to the managed service node 342. The cloud gateway 112 may do likewise and establish its own two tunnels to the pair of managed service nodes. The cloud gateway 111 (or the cloud gateway 112) may send packet traffic to either tunnel endpoint 10.0.0.253 or 10.0.0.254, as long as it does so statefully (e.g., consistently sending packet of a same flow to the same service node for stateful services.) For example, in the figure, the cloud gateway 111 sends packets of flow A1 to tunnel endpoint 10.0.0.253 and packets of flow A2 to tunnel endpoint 10.0.0.254. Each of the two managed service nodes has its own connections with the cloud gateways, so they are completely independent, and each managed service node has its own set of tunnel ports to support its hairpin return to source cloud gateways as described above by reference to
As mentioned, each cloud gateway may send different flows of a same tenant segment to different tunnel endpoints for processing. As illustrated, the cloud gateway 111 sends flows A1, B3, and B4 to be processed by tunnel endpoint 10.0.0.253 and flow A2 to be processed by tunnel endpoint 10.0.0.254. The cloud gateway 112 sends flows B6 and A8 to be processed by tunnel endpoint 10.0.0.253 and flows B5 and A7 to be processed by tunnel endpoint 10.0.0.254. The T1-SRs 321-324 of the managed edge nodes 341 and 342 in turn receive packets from flows of their respective VNI. Specifically, the T1-SR 321 processes tenant segment A traffic for flows A1 and A8, the T1-SR 322 processes tenant segment B traffic for flows B3, B4, and B6, the T1-SR 323 processes tenant segment A traffic for flows A2 and A7, and the T1-SR 324 processes tenant segment B traffic for flows B5.
In order to support stateful active-active operation, the managed edge nodes in the pair synchronizes or shares the states of their stateful services for the different flows, so when one managed edge node in the A/A pair fails, the counterpart T1-SRs of the remaining managed edge node can take over the stateful operations. In this case, T1-SR 321 shares the states of flows A1 and A8 with T1-SR 323, the T1-SR 322 shares the states of flows B3, B4, and B6 with T1-SR 324, T1-SR 323 shares the states of flows A2 and A7 with T1-SR 321, and T1-SR 324 shares the states of flow B5 with T1-SR 322.
Though the managed service node 342 has failed, the cloud gateways 111 and 112 can still send packets to the same two tunnel endpoints (10.0.0.253 and 10.0.0.254), which are now both implemented by the managed service node 341. The cloud gateways may continue to use the same tunnel as the outer encapsulation does not change. Also, in some embodiments, the corresponding T1-SRs (for the same VNI) in the two managed nodes share the same MAC address. (In the example of
The first service node implements a first plurality of T1-SRs that includes a first set of T1-SRs dedicated to a first tenant segment and a second set of T1-SRs dedicated to a second tenant segment. The second service node implements a second plurality of T1-SRs that includes a third set of T1-SRs dedicated to the first tenant segment and a fourth set of T1-SRs dedicated to the second tenant segment. In some embodiments, the first service node implements a first T0-SR for decapsulating and demultiplexing packets to the first plurality of T1-SRs, and the second service node implements a second T0-SR for decapsulating and demultiplexing packets to the second plurality of T1-SRs.
The process 1000 starts when the first service node or the second service node receives packet traffic from a cloud gateway of the SD-SWAN. The first service node receives (at 1010) packets from the cloud gateway to a first tunnel endpoint to be processed at the first plurality of T1-SRs. The second service node receives (at 1020) packets from the cloud gateway to a second tunnel endpoint to be processed at the second plurality of T1-SRs. Each T1-SR of the first and third sets of T1-SRs applies a set of security policies specific to the first tenant segment to packets from the first tenant segments. Each T1-SR of the second and fourth sets of T1-SRs applies a set of security policies specific to the second tenant segment to packets from the second tenant segments.
The first and second service nodes synchronize (at 1030) the states of the first plurality of T1-SRs with states of the second plurality of T1-SRs. Specifically, the states of individual flows processed by the T1-SRs of the first service node are shared with T1-SRs of the second service node and vice versa.
The process 1000 then determines (at 1040) whether the first service node or the second service node have failed. In some embodiments, whether one of the service nodes has failed is determined by the network virtualization management based on a status reported from the service nodes. The network virtualization management in turn configures the two service nodes accordingly (e.g., to have one service node take over the tunnel endpoint of the failed service node.) If the first service node fails, the process 1000 proceeds to 1050. If the second service node fails, the process proceeds to 1060. If neither service node fails, the process 1000 ends.
At 1050 (when the first service node fails), the second service node receives packets from the cloud gateway to both the first and second tunnel endpoints to be processed at the second plurality of T1-SRs. Packets from the first tenant segment to the first and second tunnel endpoints are processed by the third set of T1-SRs and packets from the second tenant segment to the first and second tunnel endpoints are processed by the fourth set of T1-SRs.
At 1060 (when the second service node fails), the first service node receives packets from the cloud gateway to both the first and second tunnel endpoints to be processed at the first plurality of T1-SRs. Packets from the first tenant segment to the first and second tunnel endpoints are processed by the first set of T1-SRs and packets from the second tenant segment to the first and second tunnel endpoints are processed by the second set of T1-SRs. The process 1000 then ends.
In some embodiments, the T1-SRs and T0-SRs as described above not only receive, process, and return packet traffic for local cloud gateways (e.g., of a same PoP), the T1-SRs and T0-SRs may also have uplink and downlink connections with an external network. The external network may refer to the Internet, or any remote site or PoP that requires an uplink to access from the local PoP. The uplink to the remote site can be part of a specific technology to bring together PoPs or datacenters in different locations to create a virtual network.
In some embodiments, a managed service node implementing a T0-SR and one or more T1-SRs performs two layers of address translation on packet traffic going to the external network. The two layers of address translation is for ensuring that the response traffic from the external network can successfully arrive back at the managed service node.
Since the packet 1110 is bound for a remote site external to the PoP, it will be sent into the Internet without being further processed by any cloud gateway of the SD-WAN. In some embodiments, in order to send the packet into the external network and be able to receive any corresponding return traffic at the correct T0-SR and T1-SR, the original source address of the packet goes through multiple source network address translation (SNAT) operations. Specifically, T1-SR 321 performs a first SNAT to translate the original source address “1.2.3.4” into 169.254.k.1″ (for an intermediate packet 1112), which is a private address of the T1-SR 321 used to distinguish among the multiple different T1-SRs within the managed service node 341. The T0-SR 311 then performs a second SNAT to translate the private address “169.254.k.1” into a public address “a.b.c.1” (for an outgoing packet 1114), which is a public facing IP of the T0-SR 311. The outgoing packet 1114 having “a.b.c.1” as the source address is sent through an uplink into the external network (e.g., Internet) to a destination IP “5.6.7.8”. Any corresponding response packet (of the same flow) will arrive at the T0-SR 311 using the IP “a.b.c.1”.
The process 1200 starts when the service node receives (at 1210) a packet from a cloud gateway. The cloud gateway is one of a plurality of cloud gateways of a SD-WAN configured to receive packet traffic from different datacenters or branch offices. The cloud gateway is configured by an orchestrator of the SD-WAN and the service node is managed by a network virtualization management software. The cloud gateway and the service node may be hosted by machines located in a same PoP.
The service node applies (at 1220) a security policy to the packet. For example, if the packet is from a first tenant segment, the T1-SR may apply a security policy associated with the first tenant segment to the packet. In some embodiments, if the packet is destined for a remote site, the service node may apply the security policy on a response packet from the external network.
The service node determines (at 1230) whether the packet is destined for a remote site or a local site. The local site may refer to a PoP in which both the service node and the cloud gateway are located, such that the packet traffic may stay in the PoP without going through an external network. The remote site may refer to a destination outside of the SD-WAN, or another PoP that is remote to the local site and can only be accessed through an uplink to an external network. If the packet is destined for a remote site, the process 1200 proceeds to 1240. If the packet is destined for the local site, the service node returns (at 1235) a packet based on a result of the security policy to the cloud gateway. The process 1200 then ends.
The service node translates (at 1240), at a particular T1-SR of the service node, a source address of the packet to a private address of the particular T1-SR. The private address of the T1-SR is used to identify the particular T1-SR among the plurality of T1-SRs behind the T0-SR. The service node translates (at 1250), at a T0-SR of the service node, the private address of the particular T1-SR into a public address of the T0-SR. The service node transmits (at 1260) the packet through an uplink to an external network using the public address of the T0-SR as a source address. The process 1200 ends. The service node may subsequently receive a response packet from the external network at the public address of the T0-SR.
A software defined wide area network (SD-WAN) is a virtual network. A virtual network can be for a corporation, non-profit organizations, educational entities, or other types of business entities. Also, as used in this document, data messages or packets refer to a collection of bits in a particular format sent across a network. One of ordinary skill in the art will recognize that the term data message or packet is used in this document to refer to various formatted collections of bits that are sent across a network. The formatting of these bits can be specified by standardized protocols or non-standardized protocols. Examples of data messages following standardized protocols include Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Also, as used in this document, references to L2, L3, L4, and L7 layers (or layer 2, layer 3, layer 4, and layer 7) are references respectively to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the OSI (Open System Interconnection) layer model.
As further described below, the group of components that form an MFN include in some embodiments (1) one or more VPN gateways for establishing VPN connections with an entity's compute nodes (e.g., offices, private datacenters, remote users, etc.) that are external machine locations outside of the public cloud datacenters, (2) one or more forwarding elements for forwarding encapsulated data messages between each other in order to define an overlay virtual network over the shared public cloud network fabric, (3) one or more service machines for performing middlebox service operations as well as L4-L7 optimizations, and (4) one or more measurement agents for obtaining measurements regarding the network connection quality between the public cloud datacenters in order to identify desired paths through the public cloud datacenters. In some embodiments, different MFNs can have different arrangements and different numbers of such components, and one MFN can have different numbers of such components for redundancy and scalability reasons.
Also, in some embodiments, each MFN's group of components execute on different computers in the MFN's public cloud datacenter. In some embodiments, several or all of an MFN's components can execute on one computer of a public cloud datacenter. The components of an MFN in some embodiments execute on host computers that also execute other machines of other tenants. These other machines can be other machines of other MFNs of other tenants, or they can be unrelated machines of other tenants (e.g., compute VMs or containers).
The virtual network 1300 in some embodiments is deployed by a virtual network provider (VNP) that deploys different virtual networks over the same or different public cloud datacenters for different entities (e.g., different corporate customers/tenants of the virtual network provider). The virtual network provider in some embodiments is the entity that deploys the MFNs and provides the controller cluster for configuring and managing these MFNs.
The virtual network 1300 connects the corporate compute endpoints (such as datacenters, branch offices and mobile users) to each other and to external services (e.g., public web services, or SaaS services such as Office365® or Salesforce®) that reside in the public cloud or reside in private datacenter accessible through the Internet. This virtual network 1300 leverages the different locations of the different public clouds to connect different corporate compute endpoints (e.g., different private networks and/or different mobile users of the corporation) to the public clouds in their vicinity. Corporate compute endpoints are also referred to as corporate compute nodes in the discussion below.
In some embodiments, the virtual network 1300 also leverages the high-speed networks that interconnect these public clouds to forward data messages through the public clouds to their destinations or to get as close to their destinations while reducing their traversal through the Internet. When the corporate compute endpoints are outside of public cloud datacenters over which the virtual network spans, these endpoints are referred to as external machine locations. This is the case for corporate branch offices, private datacenters and devices of remote users.
In the example illustrated in
In some embodiments, the branch offices 1330a and 1330b have their own private networks (e.g., local area networks) that connect computers at the branch locations and branch private datacenters that are outside of public clouds. Similarly, the corporate datacenter 1334 in some embodiments has its own private network and resides outside of any public cloud datacenter. In other embodiments, however, the corporate datacenter 1334 or the datacenter of the branch office 1330a and 1330b can be within a public cloud, but the virtual network 1300 does not span this public cloud, as the corporate datacenter 1334 or branch office datacenters 1330a and 1330b connect to the edge of the virtual network 1300.
As mentioned above, the virtual network 1300 is established by connecting different deployed managed forwarding nodes 1350 in different public clouds through overlay tunnels 1352. Each managed forwarding node 1350 includes several configurable components. As further described above and further described below, the MFN components include in some embodiments software-based measurement agents, software forwarding elements (e.g., software routers, switches, gateways, etc.), layer 4 proxies (e.g., TCP proxies) and middlebox service machines (e.g., VMs, containers, etc.). One or more of these components in some embodiments use standardized or commonly available solutions, such as Open vSwitch, OpenVPN, strongSwan, etc.
In some embodiments, each MFN (i.e., the group of components that conceptually forms an MFN) can be shared by different tenants of the virtual network provider that deploys and configures the MFNs in the public cloud datacenters. Conjunctively, or alternatively, the virtual network provider in some embodiments can deploy a unique set of MFNs in one or more public cloud datacenters for a particular tenant. For instance, a particular tenant might not wish to share MFN resources with another tenant for security reasons or quality of service reasons. For such a tenant, the virtual network provider can deploy its own set of MFNs across several public cloud datacenters.
In some embodiments, a logically centralized controller cluster 1360 (e.g., a set of one or more controller servers) operates inside or outside of one or more of the public clouds 1305 and 1310 and configures the public-cloud components of the managed forwarding nodes 1350 to implement the virtual network 1300 over the public clouds 1305 and 1310. In some embodiments, the controllers in this cluster 1360 are at various different locations (e.g., are in different public cloud datacenters) in order to improve redundancy and high availability. The controller cluster 1360 in some embodiments scales up or down the number of public cloud components that are used to establish the virtual network 1300, or the compute or network resources allocated to these components.
In some embodiments, the controller cluster 1360, or another controller cluster of the virtual network provider, establishes a different virtual network for another corporate tenant over the same public clouds 1305 and 1310, and/or over different public clouds of different public cloud providers. In addition to the controller cluster(s), the virtual network provider in other embodiments deploys forwarding elements and service machines in the public clouds that allow different tenants to deploy different virtual networks over the same or different public clouds.
Through the configured components of the MFNs, the virtual network 1300 of
In some embodiments, a managed service node may be implemented by a host machine that is running virtualization software, serving as a virtual network forwarding engine. Such a virtual network forwarding engine is also known as managed forwarding element (MFE), or hypervisors. Virtualization software allows a computing device to host a set of virtual machines (VMs) or data compute nodes (DCNs) as well as to perform packet-forwarding operations (including L2 switching and L3 routing operations). These computing devices are therefore also referred to as host machines. The packet forwarding operations of the virtualization software are managed and controlled by a set of central controllers, and therefore the virtualization software is also referred to as a managed software forwarding element (MSFE) in some embodiments. In some embodiments, the MSFE performs its packet forwarding operations for one or more logical forwarding elements as the virtualization software of the host machine operates local instantiations of the logical forwarding elements as physical forwarding elements. Some of these physical forwarding elements are managed physical routing elements (MPREs) for performing L3 routing operations for a logical routing element (LRE), some of these physical forwarding elements are managed physical switching elements (MPSEs) for performing L2 switching operations for a logical switching element (LSE).
As illustrated, the computing device 1400 has access to a physical network 1490 through a physical NIC (PNIC) 1495. The host machine 1400 also runs the virtualization software 1405 and hosts VMs 1411-1414. The virtualization software 1405 serves as the interface between the hosted VMs 1411-1414 and the physical MC 1495 (as well as other physical resources, such as processors and memory). Each of the VMs 1411-1414 includes a virtual MC (VNIC) for accessing the network through the virtualization software 1405. Each VNIC in a VM 1411-1414 is responsible for exchanging packets between the VM 1411-1414 and the virtualization software 1405. In some embodiments, the VNICs are software abstractions of physical NICs implemented by virtual NIC emulators.
The virtualization software 1405 manages the operations of the VMs 1411-1414, and includes several components for managing the access of the VMs 1411-1414 to the physical network 1490 (by implementing the logical networks to which the VMs connect, in some embodiments). As illustrated, the virtualization software 1405 includes several components, including a MPSE 1420, a set of MPREs 1430, a controller agent 1440, a network data storage 1445, a VTEP 1450, and a set of uplink pipelines 1470.
The VTEP (virtual tunnel endpoint) 1450 allows the host machine 1400 to serve as a tunnel endpoint for logical network traffic. An example of the logical network traffic is traffic for Virtual Extensible LAN (VXLAN), which is an overlay network encapsulation protocol. An overlay network created by VXLAN encapsulation is sometimes referred to as a VXLAN network, or simply VXLAN. When a VM 1411-1414 on the host machine 1400 sends a data packet (e.g., an Ethernet frame) to another VM in the same VXLAN network but on a different host (e.g., other machines 1480), the VTEP 1450 will encapsulate the data packet using the VXLAN network's VNI and network addresses of the VTEP 1450, before sending the packet to the physical network 1490. The packet is tunneled through the physical network 1490 (i.e., the encapsulation renders the underlying packet transparent to the intervening network elements) to the destination host. The VTEP at the destination host decapsulates the packet and forwards only the original inner data packet to the destination VM. In some embodiments, the VTEP module 1450 serves only as a controller interface for VXLAN encapsulation, while the encapsulation and decapsulation of VXLAN packets is accomplished at the uplink module 1470.
The controller agent 1440 receives control plane messages from a controller 1460 (e.g., a CCP node) or a cluster of controllers. In some embodiments, these control plane messages include configuration data for configuring the various components of the virtualization software 1405 (such as the MPSE 1420 and the MPREs 1430) and/or the virtual machines 1411-1414. In some embodiments, the configuration data includes those for configuring an edge node, specifically the tenant-level service routers (T1-SRs) and provider-level service routers (T0-SRs).
In the example illustrated in
The network data storage 1445 in some embodiments stores some of the data that are used and produced by the logical forwarding elements of the host machine 1400 (logical forwarding elements such as the MPSE 1420 and the MPRE 1430). Such stored data in some embodiments include forwarding tables and routing tables, connection mappings, as well as packet traffic statistics. These stored data are accessible by the controller agent 1440 in some embodiments and delivered to another computing device.
The MPSE 1420 delivers network data to and from the physical NIC 1495, which interfaces the physical network 1490. The MPSE 1420 also includes a number of virtual ports (vPorts) that communicatively interconnect the physical NIC 1495 with the VMs 1411-1414, the MPREs 1430, and the controller agent 1440. Each virtual port is associated with a unique L2 MAC address, in some embodiments. The MPSE 1420 performs L2 link layer packet forwarding between any two network elements that are connected to its virtual ports. The MPSE 1420 also performs L2 link layer packet forwarding between any network element connected to any one of its virtual ports and a reachable L2 network element on the physical network 1490 (e.g., another VM running on another host). In some embodiments, a MPSE is a local instantiation of a logical switching element (LSE) that operates across the different host machines and can perform L2 packet switching between VMs on a same host machine or on different host machines. In some embodiments, the MPSE performs the switching function of several LSEs according to the configuration of those logical switches.
The MPREs 1430 perform L3 routing on data packets received from a virtual port on the MPSE 1420. In some embodiments, this routing operation entails resolving a L3 IP address to a next-hop L2 MAC address and a next-hop VNI (i.e., the VNI of the next-hop's L2 segment). Each routed data packet is then sent back to the MPSE 1420 to be forwarded to its destination according to the resolved L2 MAC address. This destination can be another VM connected to a virtual port on the MPSE 1420, or a reachable L2 network element on the physical network 1490 (e.g., another VM running on another host, a physical non-virtualized machine, etc.).
As mentioned, in some embodiments, a MPRE is a local instantiation of a logical routing element (LRE) that operates across the different host machines and can perform L3 packet forwarding between VMs on a same host machine or on different host machines. In some embodiments, a host machine may have multiple MPREs connected to a single MPSE, where each MPRE in the host machine implements a different LRE. MPREs and MPSEs are referred to as “physical” routing/switching elements in order to distinguish from “logical” routing/switching elements, even though MPREs and MPSEs are implemented in software in some embodiments. In some embodiments, a MPRE is referred to as a “software router” and a MPSE is referred to as a “software switch”. In some embodiments, LREs and LSEs are collectively referred to as logical forwarding elements (LFEs), while MPREs and MPSEs are collectively referred to as managed physical forwarding elements (MPFEs). Some of the logical resources (LRs) mentioned throughout this document are LREs or LSEs that have corresponding local MPREs or a local MPSE running in each host machine.
In some embodiments, the MPRE 1430 includes one or more logical interfaces (LIFs) that each serve as an interface to a particular segment (L2 segment or VXLAN) of the network. In some embodiments, each LIF is addressable by its own IP address and serves as a default gateway or ARP proxy for network nodes (e.g., VMs) of its particular segment of the network. In some embodiments, all of the MPREs in the different host machines are addressable by a same “virtual” MAC address (or vMAC), while each MPRE is also assigned a “physical” MAC address (or pMAC) in order to indicate in which host machine the MPRE operates.
The uplink module 1470 relays data between the MPSE 1420 and the physical NIC 1495. The uplink module 1470 includes an egress chain and an ingress chain that each perform a number of operations. Some of these operations are pre-processing and/or post-processing operations for the MPRE 1430.
As illustrated by
The MPSE 1420 and the MPRE 1430 make it possible for data packets to be forwarded amongst VMs 1411-1414 without being sent through the external physical network 1490 (so long as the VMs connect to the same logical network, as different tenants' VMs will be isolated from each other). Specifically, the MPSE 1420 performs the functions of the local logical switches by using the VNIs of the various L2 segments (i.e., their corresponding L2 logical switches) of the various logical networks. Likewise, the MPREs 1430 perform the function of the logical routers by using the VNIs of those various L2 segments. Since each L2 segment/L2 switch has its own a unique VNI, the host machine 1400 (and its virtualization software 1405) is able to direct packets of different logical networks to their correct destinations and effectively segregate traffic of different logical networks from each other.
Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as computer-readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer-readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.
In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.
The bus 1505 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system 1500. For instance, the bus 1505 communicatively connects the processing unit(s) 1510 with the read-only memory 1530, the system memory 1520, and the permanent storage device 1535.
From these various memory units, the processing unit(s) 1510 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) 1510 may be a single processor or a multi-core processor in different embodiments. The read-only-memory (ROM) 1530 stores static data and instructions that are needed by the processing unit(s) 1510 and other modules of the computer system 1500. The permanent storage device 1535, on the other hand, is a read-and-write memory device. This device 1535 is a non-volatile memory unit that stores instructions and data even when the computer system 1500 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 1535.
Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device 1535. Like the permanent storage device 1535, the system memory 1520 is a read-and-write memory device. However, unlike storage device 1535, the system memory 1520 is a volatile read-and-write memory, such as random access memory. The system memory 1520 stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 1520, the permanent storage device 1535, and/or the read-only memory 1530. From these various memory units, the processing unit(s) 1510 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.
The bus 1505 also connects to the input and output devices 1540 and 1545. The input devices 1540 enable the user to communicate information and select commands to the computer system 1500. The input devices 1540 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 1545 display images generated by the computer system 1500. The output devices 1545 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices 1540 and 1545.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer-readable medium,” “computer-readable media,” and “machine-readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral or transitory signals.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. Several embodiments described above include various pieces of data in the overlay encapsulation headers. One of ordinary skill will realize that other embodiments might not use the encapsulation headers to relay all of this data.
Also, several figures conceptually illustrate processes of some embodiments of the invention. In other embodiments, the specific operations of these processes may not be performed in the exact order shown and described in these figures. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process could be implemented using several sub-processes, or as part of a larger macro process. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5652751 | Sharony | Jul 1997 | A |
| 5909553 | Campbell et al. | Jun 1999 | A |
| 6154465 | Pickett | Nov 2000 | A |
| 6157648 | Voit et al. | Dec 2000 | A |
| 6201810 | Masuda et al. | Mar 2001 | B1 |
| 6363378 | Conklin et al. | Mar 2002 | B1 |
| 6445682 | Weitz | Sep 2002 | B1 |
| 6744775 | Beshai et al. | Jun 2004 | B1 |
| 6976087 | Westfall et al. | Dec 2005 | B1 |
| 7003481 | Banka et al. | Feb 2006 | B2 |
| 7280476 | Anderson | Oct 2007 | B2 |
| 7313629 | Nucci et al. | Dec 2007 | B1 |
| 7320017 | Kurapati et al. | Jan 2008 | B1 |
| 7373660 | Guichard et al. | May 2008 | B1 |
| 7581022 | Griffin et al. | Aug 2009 | B1 |
| 7680925 | Sathyanarayana et al. | Mar 2010 | B2 |
| 7681236 | Tamura et al. | Mar 2010 | B2 |
| 7962458 | Holenstein et al. | Jun 2011 | B2 |
| 8094575 | Vadlakonda et al. | Jan 2012 | B1 |
| 8094659 | Arad | Jan 2012 | B1 |
| 8111692 | Ray | Feb 2012 | B2 |
| 8141156 | Mao et al. | Mar 2012 | B1 |
| 8224971 | Miller et al. | Jul 2012 | B1 |
| 8228928 | Parandekar et al. | Jul 2012 | B2 |
| 8243589 | Trost et al. | Aug 2012 | B1 |
| 8259566 | Chen et al. | Sep 2012 | B2 |
| 8274891 | Averi et al. | Sep 2012 | B2 |
| 8301749 | Finklestein et al. | Oct 2012 | B1 |
| 8385227 | Downey | Feb 2013 | B1 |
| 8566452 | Goodwin et al. | Oct 2013 | B1 |
| 8630291 | Shaffer et al. | Jan 2014 | B2 |
| 8661295 | Khanna et al. | Feb 2014 | B1 |
| 8724456 | Hong et al. | May 2014 | B1 |
| 8724503 | Johnsson et al. | May 2014 | B2 |
| 8745177 | Kazerani et al. | Jun 2014 | B1 |
| 8799504 | Capone et al. | Aug 2014 | B2 |
| 8804745 | Sinn | Aug 2014 | B1 |
| 8806482 | Nagargadde et al. | Aug 2014 | B1 |
| 8855071 | Sankaran | Oct 2014 | B1 |
| 8856339 | Mestery et al. | Oct 2014 | B2 |
| 8964548 | Keralapura et al. | Feb 2015 | B1 |
| 8989199 | Sella et al. | Mar 2015 | B1 |
| 9009217 | Nagargadde et al. | Apr 2015 | B1 |
| 9055000 | Ghosh et al. | Jun 2015 | B1 |
| 9060025 | Xu | Jun 2015 | B2 |
| 9071607 | Twitchell, Jr. | Jun 2015 | B2 |
| 9075771 | Gawali et al. | Jul 2015 | B1 |
| 9135037 | Petrescu-Prahova et al. | Sep 2015 | B1 |
| 9137334 | Zhou | Sep 2015 | B2 |
| 9154327 | Marino et al. | Oct 2015 | B1 |
| 9203764 | Shirazipour et al. | Dec 2015 | B2 |
| 9306949 | Richard et al. | Apr 2016 | B1 |
| 9323561 | Ayala et al. | Apr 2016 | B2 |
| 9336040 | Dong et al. | May 2016 | B2 |
| 9354983 | Yenamandra et al. | May 2016 | B1 |
| 9356943 | Lopilato et al. | May 2016 | B1 |
| 9379981 | Zhou et al. | Jun 2016 | B1 |
| 9413724 | Xu | Aug 2016 | B2 |
| 9419878 | Hsiao et al. | Aug 2016 | B2 |
| 9432245 | Sorenson et al. | Aug 2016 | B1 |
| 9438566 | Zhang et al. | Sep 2016 | B2 |
| 9450817 | Bahadur et al. | Sep 2016 | B1 |
| 9450852 | Chen et al. | Sep 2016 | B1 |
| 9462010 | Stevenson | Oct 2016 | B1 |
| 9467478 | Khan et al. | Oct 2016 | B1 |
| 9485163 | Fries et al. | Nov 2016 | B1 |
| 9521067 | Michael et al. | Dec 2016 | B2 |
| 9525564 | Lee | Dec 2016 | B2 |
| 9559951 | Sajassi et al. | Jan 2017 | B1 |
| 9563423 | Pittman | Feb 2017 | B1 |
| 9602389 | Maveli et al. | Mar 2017 | B1 |
| 9608917 | Anderson et al. | Mar 2017 | B1 |
| 9608962 | Chang | Mar 2017 | B1 |
| 9621460 | Mehta et al. | Apr 2017 | B2 |
| 9641551 | Kariyanahalli | May 2017 | B1 |
| 9665432 | Kruse et al. | May 2017 | B2 |
| 9686127 | Ramachandran et al. | Jun 2017 | B2 |
| 9715401 | Devine et al. | Jul 2017 | B2 |
| 9717021 | Hughes et al. | Jul 2017 | B2 |
| 9722815 | Mukundan et al. | Aug 2017 | B2 |
| 9747249 | Cherian et al. | Aug 2017 | B2 |
| 9755965 | Yadav et al. | Sep 2017 | B1 |
| 9787559 | Schroeder | Oct 2017 | B1 |
| 9807004 | Koley et al. | Oct 2017 | B2 |
| 9819540 | Bahadur et al. | Nov 2017 | B1 |
| 9819565 | Djukic et al. | Nov 2017 | B2 |
| 9825822 | Holland | Nov 2017 | B1 |
| 9825911 | Brandwine | Nov 2017 | B1 |
| 9825992 | Xu | Nov 2017 | B2 |
| 9832128 | Ashner et al. | Nov 2017 | B1 |
| 9832205 | Santhi et al. | Nov 2017 | B2 |
| 9875355 | Williams | Jan 2018 | B1 |
| 9906401 | Rao | Feb 2018 | B1 |
| 9930011 | Clemons, Jr. et al. | Mar 2018 | B1 |
| 9935829 | Miller et al. | Apr 2018 | B1 |
| 9942787 | Tillotson | Apr 2018 | B1 |
| 10038601 | Becker et al. | Jul 2018 | B1 |
| 10057183 | Salle et al. | Aug 2018 | B2 |
| 10057294 | Xu | Aug 2018 | B2 |
| 10135789 | Mayya et al. | Nov 2018 | B2 |
| 10142226 | Wu et al. | Nov 2018 | B1 |
| 10178032 | Freitas | Jan 2019 | B1 |
| 10187289 | Chen et al. | Jan 2019 | B1 |
| 10200264 | Menon et al. | Feb 2019 | B2 |
| 10229017 | Zou et al. | Mar 2019 | B1 |
| 10237123 | Dubey et al. | Mar 2019 | B2 |
| 10250498 | Bales et al. | Apr 2019 | B1 |
| 10263832 | Ghosh | Apr 2019 | B1 |
| 10320664 | Nainar et al. | Jun 2019 | B2 |
| 10320691 | Matthews et al. | Jun 2019 | B1 |
| 10326830 | Singh | Jun 2019 | B1 |
| 10348767 | Lee et al. | Jul 2019 | B1 |
| 10355989 | Panchai et al. | Jul 2019 | B1 |
| 10425382 | Mayya et al. | Sep 2019 | B2 |
| 10454708 | Mibu | Oct 2019 | B2 |
| 10454714 | Mayya et al. | Oct 2019 | B2 |
| 10461993 | Turabi et al. | Oct 2019 | B2 |
| 10498652 | Mayya et al. | Dec 2019 | B2 |
| 10511546 | Singarayan et al. | Dec 2019 | B2 |
| 10523539 | Mayya et al. | Dec 2019 | B2 |
| 10550093 | Ojima et al. | Feb 2020 | B2 |
| 10554538 | Spohn et al. | Feb 2020 | B2 |
| 10560431 | Chen et al. | Feb 2020 | B1 |
| 10565464 | Han et al. | Feb 2020 | B2 |
| 10567519 | Mukhopadhyaya et al. | Feb 2020 | B1 |
| 10574528 | Mayya et al. | Feb 2020 | B2 |
| 10594516 | Cidon et al. | Mar 2020 | B2 |
| 10594659 | El-Moussa et al. | Mar 2020 | B2 |
| 10608844 | Cidon et al. | Mar 2020 | B2 |
| 10637889 | Ermagan et al. | Apr 2020 | B2 |
| 10666460 | Cidon et al. | May 2020 | B2 |
| 10686625 | Cidon et al. | Jun 2020 | B2 |
| 10693739 | Naseri et al. | Jun 2020 | B1 |
| 10749711 | Mukundan et al. | Aug 2020 | B2 |
| 10778466 | Cidon et al. | Sep 2020 | B2 |
| 10778528 | Mayya et al. | Sep 2020 | B2 |
| 10805114 | Cidon et al. | Oct 2020 | B2 |
| 10805272 | Mayya et al. | Oct 2020 | B2 |
| 10819564 | Turabi et al. | Oct 2020 | B2 |
| 10826775 | Moreno et al. | Nov 2020 | B1 |
| 10841131 | Cidon et al. | Nov 2020 | B2 |
| 10911374 | Kumar et al. | Feb 2021 | B1 |
| 10938693 | Mayya et al. | Mar 2021 | B2 |
| 10951529 | Duan et al. | Mar 2021 | B2 |
| 10958479 | Cidon et al. | Mar 2021 | B2 |
| 10959098 | Cidon et al. | Mar 2021 | B2 |
| 10992558 | Silva et al. | Apr 2021 | B1 |
| 10992568 | Michael et al. | Apr 2021 | B2 |
| 10999100 | Cidon et al. | May 2021 | B2 |
| 10999137 | Cidon et al. | May 2021 | B2 |
| 10999165 | Cidon et al. | May 2021 | B2 |
| 11005684 | Cidon | May 2021 | B2 |
| 11018995 | Cidon et al. | May 2021 | B2 |
| 11044190 | Ramaswamy et al. | Jun 2021 | B2 |
| 11050588 | Mayya et al. | Jun 2021 | B2 |
| 11050644 | Hegde et al. | Jun 2021 | B2 |
| 11071005 | Shen et al. | Jul 2021 | B2 |
| 11089111 | Markuze et al. | Aug 2021 | B2 |
| 11095612 | Oswal et al. | Aug 2021 | B1 |
| 11102032 | Cidon et al. | Aug 2021 | B2 |
| 11108851 | Kurmala et al. | Aug 2021 | B1 |
| 11115347 | Gupta et al. | Sep 2021 | B2 |
| 11115426 | Pazhyannur et al. | Sep 2021 | B1 |
| 11115480 | Markuze et al. | Sep 2021 | B2 |
| 11121962 | Michael et al. | Sep 2021 | B2 |
| 11121985 | Cidon et al. | Sep 2021 | B2 |
| 11128492 | Sethi et al. | Sep 2021 | B2 |
| 11153230 | Cidon et al. | Oct 2021 | B2 |
| 11171885 | Cidon et al. | Nov 2021 | B2 |
| 11212140 | Mukundan et al. | Dec 2021 | B2 |
| 11212238 | Cidon | Dec 2021 | B2 |
| 11223514 | Mayya et al. | Jan 2022 | B2 |
| 11245641 | Ramaswamy et al. | Feb 2022 | B2 |
| 20020085488 | Kobayashi | Jul 2002 | A1 |
| 20020087716 | Mustafa | Jul 2002 | A1 |
| 20020198840 | Banka et al. | Dec 2002 | A1 |
| 20030061269 | Hathaway et al. | Mar 2003 | A1 |
| 20030088697 | Matsuhira | May 2003 | A1 |
| 20030112766 | Riedel et al. | Jun 2003 | A1 |
| 20030112808 | Solomon | Jun 2003 | A1 |
| 20030126468 | Markham | Jul 2003 | A1 |
| 20030161313 | Jinmei et al. | Aug 2003 | A1 |
| 20030189919 | Gupta et al. | Oct 2003 | A1 |
| 20030202506 | Perkins et al. | Oct 2003 | A1 |
| 20030219030 | Gubbi | Nov 2003 | A1 |
| 20040059831 | Chu et al. | Mar 2004 | A1 |
| 20040068668 | Lor et al. | Apr 2004 | A1 |
| 20040165601 | Liu et al. | Aug 2004 | A1 |
| 20040224771 | Chen et al. | Nov 2004 | A1 |
| 20050078690 | DeLangis | Apr 2005 | A1 |
| 20050154790 | Nagata et al. | Jul 2005 | A1 |
| 20050172161 | Cruz et al. | Aug 2005 | A1 |
| 20050195754 | Nosella | Sep 2005 | A1 |
| 20050265255 | Kodialam et al. | Dec 2005 | A1 |
| 20060002291 | Alicherry et al. | Jan 2006 | A1 |
| 20060114838 | Mandavilli et al. | Jun 2006 | A1 |
| 20060171365 | Borella | Aug 2006 | A1 |
| 20060182034 | Klinker et al. | Aug 2006 | A1 |
| 20060182035 | Masseur | Aug 2006 | A1 |
| 20060193247 | Naseh et al. | Aug 2006 | A1 |
| 20060193252 | Naseh et al. | Aug 2006 | A1 |
| 20070064604 | Chen et al. | Mar 2007 | A1 |
| 20070064702 | Bates et al. | Mar 2007 | A1 |
| 20070083727 | Johnston et al. | Apr 2007 | A1 |
| 20070091794 | Filsfils et al. | Apr 2007 | A1 |
| 20070103548 | Carter | May 2007 | A1 |
| 20070115812 | Hughes | May 2007 | A1 |
| 20070121486 | Guichard et al. | May 2007 | A1 |
| 20070130325 | Lesser | Jun 2007 | A1 |
| 20070162639 | Chu et al. | Jul 2007 | A1 |
| 20070177511 | Das et al. | Aug 2007 | A1 |
| 20070237081 | Kodialam et al. | Oct 2007 | A1 |
| 20070260746 | Mirtorabi et al. | Nov 2007 | A1 |
| 20070268882 | Breslau et al. | Nov 2007 | A1 |
| 20080002670 | Bugenhagen et al. | Jan 2008 | A1 |
| 20080049621 | McGuire et al. | Feb 2008 | A1 |
| 20080055241 | Goldenberg et al. | Mar 2008 | A1 |
| 20080080509 | Khanna et al. | Apr 2008 | A1 |
| 20080095187 | Jung et al. | Apr 2008 | A1 |
| 20080117930 | Chakareski et al. | May 2008 | A1 |
| 20080144532 | Chamarajanagar et al. | Jun 2008 | A1 |
| 20080181116 | Kavanaugh et al. | Jul 2008 | A1 |
| 20080219276 | Shah | Sep 2008 | A1 |
| 20080240121 | Xiong et al. | Oct 2008 | A1 |
| 20090013210 | McIntosh et al. | Jan 2009 | A1 |
| 20090125617 | Kiessig et al. | May 2009 | A1 |
| 20090141642 | Sun | Jun 2009 | A1 |
| 20090154463 | Hines et al. | Jun 2009 | A1 |
| 20090247204 | Sennett et al. | Oct 2009 | A1 |
| 20090274045 | Meier et al. | Nov 2009 | A1 |
| 20090276657 | Wetmore et al. | Nov 2009 | A1 |
| 20090303880 | Maltz et al. | Dec 2009 | A1 |
| 20100008361 | Guichard et al. | Jan 2010 | A1 |
| 20100017802 | Lojewski | Jan 2010 | A1 |
| 20100046532 | Okita | Feb 2010 | A1 |
| 20100061379 | Parandekar et al. | Mar 2010 | A1 |
| 20100080129 | Strahan et al. | Apr 2010 | A1 |
| 20100088440 | Banks et al. | Apr 2010 | A1 |
| 20100091823 | Retana et al. | Apr 2010 | A1 |
| 20100107162 | Edwards et al. | Apr 2010 | A1 |
| 20100118727 | Draves et al. | May 2010 | A1 |
| 20100118886 | Saavedra | May 2010 | A1 |
| 20100165985 | Sharma et al. | Jul 2010 | A1 |
| 20100191884 | Holenstein et al. | Jul 2010 | A1 |
| 20100223621 | Joshi et al. | Sep 2010 | A1 |
| 20100226246 | Proulx | Sep 2010 | A1 |
| 20100290422 | Haigh et al. | Nov 2010 | A1 |
| 20100309841 | Conte | Dec 2010 | A1 |
| 20100309912 | Mehta et al. | Dec 2010 | A1 |
| 20100322255 | Hao et al. | Dec 2010 | A1 |
| 20100332657 | Elyashev et al. | Dec 2010 | A1 |
| 20110007752 | Silva et al. | Jan 2011 | A1 |
| 20110032939 | Nozaki et al. | Feb 2011 | A1 |
| 20110040814 | Higgins | Feb 2011 | A1 |
| 20110075674 | Li et al. | Mar 2011 | A1 |
| 20110107139 | Middlecamp et al. | May 2011 | A1 |
| 20110110370 | Moreno et al. | May 2011 | A1 |
| 20110141877 | Xu et al. | Jun 2011 | A1 |
| 20110142041 | Imai | Jun 2011 | A1 |
| 20110153909 | Dong | Jun 2011 | A1 |
| 20110235509 | Szymanski | Sep 2011 | A1 |
| 20110255397 | Kadakia et al. | Oct 2011 | A1 |
| 20120008630 | Ould-Brahim | Jan 2012 | A1 |
| 20120027013 | Napierala | Feb 2012 | A1 |
| 20120136697 | Peles et al. | May 2012 | A1 |
| 20120157068 | Eichen et al. | Jun 2012 | A1 |
| 20120173694 | Yan et al. | Jul 2012 | A1 |
| 20120173919 | Patel et al. | Jul 2012 | A1 |
| 20120182940 | Taleb et al. | Jul 2012 | A1 |
| 20120221955 | Raleigh et al. | Aug 2012 | A1 |
| 20120227093 | Shalzkamer et al. | Sep 2012 | A1 |
| 20120250682 | Vincent et al. | Oct 2012 | A1 |
| 20120250686 | Vincent et al. | Oct 2012 | A1 |
| 20120281706 | Agarwal et al. | Nov 2012 | A1 |
| 20120300615 | Kempf et al. | Nov 2012 | A1 |
| 20120307659 | Yamada | Dec 2012 | A1 |
| 20120317291 | Wolfe | Dec 2012 | A1 |
| 20130019005 | Hui et al. | Jan 2013 | A1 |
| 20130021968 | Reznik et al. | Jan 2013 | A1 |
| 20130044764 | Casado et al. | Feb 2013 | A1 |
| 20130051237 | Ong | Feb 2013 | A1 |
| 20130051399 | Zhang et al. | Feb 2013 | A1 |
| 20130054763 | Merwe et al. | Feb 2013 | A1 |
| 20130086267 | Gelenbe et al. | Apr 2013 | A1 |
| 20130103834 | Dzerve et al. | Apr 2013 | A1 |
| 20130124718 | Griffith et al. | May 2013 | A1 |
| 20130124911 | Griffith et al. | May 2013 | A1 |
| 20130124912 | Griffith et al. | May 2013 | A1 |
| 20130128889 | Mathur et al. | May 2013 | A1 |
| 20130142201 | Kim et al. | Jun 2013 | A1 |
| 20130170354 | Takashima et al. | Jul 2013 | A1 |
| 20130173788 | Song | Jul 2013 | A1 |
| 20130182712 | Aguayo et al. | Jul 2013 | A1 |
| 20130191688 | Agarwal et al. | Jul 2013 | A1 |
| 20130238782 | Zhao et al. | Sep 2013 | A1 |
| 20130242718 | Zhang | Sep 2013 | A1 |
| 20130254599 | Katkar et al. | Sep 2013 | A1 |
| 20130258839 | Wang et al. | Oct 2013 | A1 |
| 20130266015 | Qu et al. | Oct 2013 | A1 |
| 20130266019 | Qu et al. | Oct 2013 | A1 |
| 20130283364 | Chang et al. | Oct 2013 | A1 |
| 20130286846 | Atlas et al. | Oct 2013 | A1 |
| 20130297611 | Moritz et al. | Nov 2013 | A1 |
| 20130297770 | Zhang | Nov 2013 | A1 |
| 20130301469 | Suga | Nov 2013 | A1 |
| 20130301642 | Radhakrishnan et al. | Nov 2013 | A1 |
| 20130308444 | Sem-Jacobsen et al. | Nov 2013 | A1 |
| 20130315242 | Wang et al. | Nov 2013 | A1 |
| 20130315243 | Huang et al. | Nov 2013 | A1 |
| 20130329548 | Nakil et al. | Dec 2013 | A1 |
| 20130329601 | Yin et al. | Dec 2013 | A1 |
| 20130329734 | Chesla et al. | Dec 2013 | A1 |
| 20130346470 | Obstfeld et al. | Dec 2013 | A1 |
| 20140019604 | Twitchell, Jr. | Jan 2014 | A1 |
| 20140019750 | Dodgson et al. | Jan 2014 | A1 |
| 20140040975 | Raleigh et al. | Feb 2014 | A1 |
| 20140064283 | Balus et al. | Mar 2014 | A1 |
| 20140092907 | Sridhar et al. | Apr 2014 | A1 |
| 20140108665 | Arora et al. | Apr 2014 | A1 |
| 20140112171 | Pasdar | Apr 2014 | A1 |
| 20140115584 | Mudigonda et al. | Apr 2014 | A1 |
| 20140123135 | Huang et al. | May 2014 | A1 |
| 20140126418 | Brendel et al. | May 2014 | A1 |
| 20140156818 | Hunt | Jun 2014 | A1 |
| 20140156823 | Liu et al. | Jun 2014 | A1 |
| 20140164560 | Ko et al. | Jun 2014 | A1 |
| 20140164617 | Jalan et al. | Jun 2014 | A1 |
| 20140173113 | Vemuri et al. | Jun 2014 | A1 |
| 20140173331 | Martin et al. | Jun 2014 | A1 |
| 20140181824 | Saund et al. | Jun 2014 | A1 |
| 20140208317 | Nakagawa | Jul 2014 | A1 |
| 20140219135 | Li et al. | Aug 2014 | A1 |
| 20140223507 | Xu | Aug 2014 | A1 |
| 20140229210 | Sharifian et al. | Aug 2014 | A1 |
| 20140244851 | Lee | Aug 2014 | A1 |
| 20140258535 | Zhang | Sep 2014 | A1 |
| 20140269690 | Tu | Sep 2014 | A1 |
| 20140279862 | Dietz et al. | Sep 2014 | A1 |
| 20140280499 | Basavaiah et al. | Sep 2014 | A1 |
| 20140317440 | Biermayr et al. | Oct 2014 | A1 |
| 20140321277 | Lynn, Jr. | Oct 2014 | A1 |
| 20140337500 | Lee | Nov 2014 | A1 |
| 20140341109 | Cartmell et al. | Nov 2014 | A1 |
| 20140372582 | Ghanwani et al. | Dec 2014 | A1 |
| 20150003240 | Drwiega et al. | Jan 2015 | A1 |
| 20150016249 | Mukundan et al. | Jan 2015 | A1 |
| 20150029864 | Raileanu et al. | Jan 2015 | A1 |
| 20150039744 | Niazi et al. | Feb 2015 | A1 |
| 20150046572 | Cheng et al. | Feb 2015 | A1 |
| 20150052247 | Threefoot et al. | Feb 2015 | A1 |
| 20150052517 | Raghu et al. | Feb 2015 | A1 |
| 20150056960 | Egner et al. | Feb 2015 | A1 |
| 20150058917 | Xu | Feb 2015 | A1 |
| 20150088942 | Shah | Mar 2015 | A1 |
| 20150089628 | Lang | Mar 2015 | A1 |
| 20150092603 | Aguayo et al. | Apr 2015 | A1 |
| 20150096011 | Watt | Apr 2015 | A1 |
| 20150124603 | Ketheesan et al. | May 2015 | A1 |
| 20150134777 | Onoue | May 2015 | A1 |
| 20150139238 | Pourzandi et al. | May 2015 | A1 |
| 20150146539 | Mehta | May 2015 | A1 |
| 20150163152 | Li | Jun 2015 | A1 |
| 20150169340 | Haddad et al. | Jun 2015 | A1 |
| 20150172121 | Farkas et al. | Jun 2015 | A1 |
| 20150172169 | DeCusatis et al. | Jun 2015 | A1 |
| 20150188823 | Williams et al. | Jul 2015 | A1 |
| 20150189009 | Bemmel | Jul 2015 | A1 |
| 20150195178 | Bhattacharya et al. | Jul 2015 | A1 |
| 20150201036 | Nishiki et al. | Jul 2015 | A1 |
| 20150222543 | Song | Aug 2015 | A1 |
| 20150222638 | Morley | Aug 2015 | A1 |
| 20150236945 | Michael et al. | Aug 2015 | A1 |
| 20150236962 | Weres et al. | Aug 2015 | A1 |
| 20150244617 | Nakil et al. | Aug 2015 | A1 |
| 20150249644 | Xu | Sep 2015 | A1 |
| 20150257081 | Ramanujan et al. | Sep 2015 | A1 |
| 20150271056 | Chunduri et al. | Sep 2015 | A1 |
| 20150271104 | Chikkamath et al. | Sep 2015 | A1 |
| 20150271303 | Neginhal et al. | Sep 2015 | A1 |
| 20150281004 | Kakadia et al. | Oct 2015 | A1 |
| 20150312142 | Barabash et al. | Oct 2015 | A1 |
| 20150312760 | O'Toole | Oct 2015 | A1 |
| 20150317169 | Sinha et al. | Nov 2015 | A1 |
| 20150334025 | Rader | Nov 2015 | A1 |
| 20150334696 | Gu et al. | Nov 2015 | A1 |
| 20150341271 | Gomez | Nov 2015 | A1 |
| 20150349978 | Wu et al. | Dec 2015 | A1 |
| 20150350907 | Timariu et al. | Dec 2015 | A1 |
| 20150363221 | Terayama et al. | Dec 2015 | A1 |
| 20150363733 | Brown | Dec 2015 | A1 |
| 20150365323 | Duminuco | Dec 2015 | A1 |
| 20150372943 | Hasan et al. | Dec 2015 | A1 |
| 20150372982 | Herle et al. | Dec 2015 | A1 |
| 20150381407 | Wang et al. | Dec 2015 | A1 |
| 20150381493 | Bansal et al. | Dec 2015 | A1 |
| 20160035183 | Buchholz et al. | Feb 2016 | A1 |
| 20160036924 | Koppolu et al. | Feb 2016 | A1 |
| 20160036938 | Aviles et al. | Feb 2016 | A1 |
| 20160037434 | Gopal et al. | Feb 2016 | A1 |
| 20160072669 | Saavedra | Mar 2016 | A1 |
| 20160072684 | Manuguri et al. | Mar 2016 | A1 |
| 20160080502 | Yadav et al. | Mar 2016 | A1 |
| 20160105353 | Cociglio | Apr 2016 | A1 |
| 20160105392 | Thakkar et al. | Apr 2016 | A1 |
| 20160105471 | Nunes et al. | Apr 2016 | A1 |
| 20160105488 | Thakkar et al. | Apr 2016 | A1 |
| 20160117185 | Fang et al. | Apr 2016 | A1 |
| 20160134461 | Sampath et al. | May 2016 | A1 |
| 20160134528 | Lin et al. | May 2016 | A1 |
| 20160134591 | Liao et al. | May 2016 | A1 |
| 20160142373 | Ossipov | May 2016 | A1 |
| 20160150055 | Choi | May 2016 | A1 |
| 20160164832 | Bellagamba et al. | Jun 2016 | A1 |
| 20160164914 | Madhav et al. | Jun 2016 | A1 |
| 20160173338 | Wolting | Jun 2016 | A1 |
| 20160191363 | Haraszti et al. | Jun 2016 | A1 |
| 20160191374 | Singh et al. | Jun 2016 | A1 |
| 20160192403 | Gupta et al. | Jun 2016 | A1 |
| 20160197834 | Luft | Jul 2016 | A1 |
| 20160197835 | Luft | Jul 2016 | A1 |
| 20160198003 | Luft | Jul 2016 | A1 |
| 20160210209 | Verkaik et al. | Jul 2016 | A1 |
| 20160212773 | Kanderholm et al. | Jul 2016 | A1 |
| 20160218947 | Hughes et al. | Jul 2016 | A1 |
| 20160218951 | Masseur et al. | Jul 2016 | A1 |
| 20160255169 | Kovvuri et al. | Sep 2016 | A1 |
| 20160261493 | Li | Sep 2016 | A1 |
| 20160261495 | Xia et al. | Sep 2016 | A1 |
| 20160261506 | Hegde et al. | Sep 2016 | A1 |
| 20160261639 | Xu | Sep 2016 | A1 |
| 20160269298 | Li et al. | Sep 2016 | A1 |
| 20160269926 | Sundaram | Sep 2016 | A1 |
| 20160285736 | Gu | Sep 2016 | A1 |
| 20160308762 | Teng et al. | Oct 2016 | A1 |
| 20160315912 | Mayya et al. | Oct 2016 | A1 |
| 20160323377 | Einkauf et al. | Nov 2016 | A1 |
| 20160328159 | Coddington et al. | Nov 2016 | A1 |
| 20160330111 | Manghirmalani et al. | Nov 2016 | A1 |
| 20160352588 | Subbarayan et al. | Dec 2016 | A1 |
| 20160353268 | Senarath et al. | Dec 2016 | A1 |
| 20160359738 | Sullenberger et al. | Dec 2016 | A1 |
| 20160366187 | Kamble | Dec 2016 | A1 |
| 20160371153 | Dornemann | Dec 2016 | A1 |
| 20160380886 | Blair et al. | Dec 2016 | A1 |
| 20160380906 | Hodique et al. | Dec 2016 | A1 |
| 20170005986 | Bansal et al. | Jan 2017 | A1 |
| 20170006499 | Hampel | Jan 2017 | A1 |
| 20170012870 | Blair et al. | Jan 2017 | A1 |
| 20170019428 | Cohn | Jan 2017 | A1 |
| 20170026283 | Williams et al. | Jan 2017 | A1 |
| 20170026355 | Mathaiyan et al. | Jan 2017 | A1 |
| 20170034046 | Cai et al. | Feb 2017 | A1 |
| 20170034052 | Chanda | Feb 2017 | A1 |
| 20170034129 | Sawant et al. | Feb 2017 | A1 |
| 20170048296 | Ramalho et al. | Feb 2017 | A1 |
| 20170053258 | Carney et al. | Feb 2017 | A1 |
| 20170055131 | Kong et al. | Feb 2017 | A1 |
| 20170063674 | Maskalik et al. | Mar 2017 | A1 |
| 20170063782 | Jain et al. | Mar 2017 | A1 |
| 20170063794 | Jain et al. | Mar 2017 | A1 |
| 20170064005 | Lee | Mar 2017 | A1 |
| 20170093625 | Pera et al. | Mar 2017 | A1 |
| 20170097841 | Chang et al. | Apr 2017 | A1 |
| 20170104653 | Badea et al. | Apr 2017 | A1 |
| 20170104755 | Arregoces et al. | Apr 2017 | A1 |
| 20170109212 | Gaurav et al. | Apr 2017 | A1 |
| 20170118173 | Arramreddy et al. | Apr 2017 | A1 |
| 20170123939 | Maheshwari et al. | May 2017 | A1 |
| 20170126516 | Tiagi et al. | May 2017 | A1 |
| 20170126564 | Mayya et al. | May 2017 | A1 |
| 20170134186 | Mukundan et al. | May 2017 | A1 |
| 20170134520 | Abbasi et al. | May 2017 | A1 |
| 20170139789 | Fries et al. | May 2017 | A1 |
| 20170155557 | Desai et al. | Jun 2017 | A1 |
| 20170163473 | Sadana et al. | Jun 2017 | A1 |
| 20170171310 | Gardner | Jun 2017 | A1 |
| 20170181210 | Nadella et al. | Jun 2017 | A1 |
| 20170195161 | Ruel et al. | Jul 2017 | A1 |
| 20170195169 | Mills et al. | Jul 2017 | A1 |
| 20170201585 | Doraiswamy et al. | Jul 2017 | A1 |
| 20170207976 | Rovner et al. | Jul 2017 | A1 |
| 20170214545 | Cheng et al. | Jul 2017 | A1 |
| 20170214701 | Hasan | Jul 2017 | A1 |
| 20170223117 | Messerli et al. | Aug 2017 | A1 |
| 20170237710 | Mayya et al. | Aug 2017 | A1 |
| 20170257260 | Govindan et al. | Sep 2017 | A1 |
| 20170257309 | Appanna | Sep 2017 | A1 |
| 20170264496 | Ao et al. | Sep 2017 | A1 |
| 20170279717 | Bethers et al. | Sep 2017 | A1 |
| 20170279803 | Desai et al. | Sep 2017 | A1 |
| 20170280474 | Vesterinen et al. | Sep 2017 | A1 |
| 20170288987 | Pasupathy et al. | Oct 2017 | A1 |
| 20170289002 | Ganguli et al. | Oct 2017 | A1 |
| 20170289027 | Ratnasingham | Oct 2017 | A1 |
| 20170295264 | Touitou et al. | Oct 2017 | A1 |
| 20170302565 | Ghobadi et al. | Oct 2017 | A1 |
| 20170310641 | Jiang et al. | Oct 2017 | A1 |
| 20170310691 | Vasseur et al. | Oct 2017 | A1 |
| 20170317954 | Masurekar | Nov 2017 | A1 |
| 20170317969 | Masurekar | Nov 2017 | A1 |
| 20170317974 | Masurekar et al. | Nov 2017 | A1 |
| 20170337086 | Zhu et al. | Nov 2017 | A1 |
| 20170339054 | Yadav et al. | Nov 2017 | A1 |
| 20170339070 | Chang et al. | Nov 2017 | A1 |
| 20170364419 | Lo | Dec 2017 | A1 |
| 20170366445 | Nemirovsky et al. | Dec 2017 | A1 |
| 20170366467 | Martin et al. | Dec 2017 | A1 |
| 20170373950 | Szilagyi et al. | Dec 2017 | A1 |
| 20170374174 | Evens et al. | Dec 2017 | A1 |
| 20180006995 | Bickhart et al. | Jan 2018 | A1 |
| 20180007005 | Chanda | Jan 2018 | A1 |
| 20180007123 | Cheng et al. | Jan 2018 | A1 |
| 20180013636 | Seetharamaiah et al. | Jan 2018 | A1 |
| 20180014051 | Phillips et al. | Jan 2018 | A1 |
| 20180020035 | Boggia et al. | Jan 2018 | A1 |
| 20180034668 | Mayya et al. | Feb 2018 | A1 |
| 20180041425 | Zhang | Feb 2018 | A1 |
| 20180062914 | Boutros et al. | Mar 2018 | A1 |
| 20180062917 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180063036 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180063193 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180063233 | Park | Mar 2018 | A1 |
| 20180069924 | Tumuluru et al. | Mar 2018 | A1 |
| 20180074909 | Bishop et al. | Mar 2018 | A1 |
| 20180077081 | Lauer et al. | Mar 2018 | A1 |
| 20180077202 | Xu | Mar 2018 | A1 |
| 20180084081 | Kuchibhotla et al. | Mar 2018 | A1 |
| 20180097725 | Wood et al. | Apr 2018 | A1 |
| 20180114569 | Strachan et al. | Apr 2018 | A1 |
| 20180123910 | Fitzgibbon | May 2018 | A1 |
| 20180131608 | Jiang et al. | May 2018 | A1 |
| 20180131615 | Zhang | May 2018 | A1 |
| 20180131720 | Hobson et al. | May 2018 | A1 |
| 20180145899 | Rao | May 2018 | A1 |
| 20180159796 | Wang et al. | Jun 2018 | A1 |
| 20180159856 | Gujarathi | Jun 2018 | A1 |
| 20180167378 | Kostyukov et al. | Jun 2018 | A1 |
| 20180176073 | Dubey et al. | Jun 2018 | A1 |
| 20180176082 | Katz et al. | Jun 2018 | A1 |
| 20180176130 | Banerjee et al. | Jun 2018 | A1 |
| 20180213472 | Ishii et al. | Jul 2018 | A1 |
| 20180219765 | Michael et al. | Aug 2018 | A1 |
| 20180219766 | Michael et al. | Aug 2018 | A1 |
| 20180234300 | Mayya et al. | Aug 2018 | A1 |
| 20180260125 | Botes et al. | Sep 2018 | A1 |
| 20180262468 | Kumar et al. | Sep 2018 | A1 |
| 20180270104 | Zheng et al. | Sep 2018 | A1 |
| 20180278541 | Wu et al. | Sep 2018 | A1 |
| 20180295101 | Gehrmann | Oct 2018 | A1 |
| 20180295529 | Jen et al. | Oct 2018 | A1 |
| 20180302286 | Mayya et al. | Oct 2018 | A1 |
| 20180302321 | Manthiramoorthy et al. | Oct 2018 | A1 |
| 20180307851 | Lewis | Oct 2018 | A1 |
| 20180316606 | Sung et al. | Nov 2018 | A1 |
| 20180351855 | Sood et al. | Dec 2018 | A1 |
| 20180351862 | Jeganathan et al. | Dec 2018 | A1 |
| 20180351863 | Vairavakkalai et al. | Dec 2018 | A1 |
| 20180351882 | Jeganathan et al. | Dec 2018 | A1 |
| 20180373558 | Chang et al. | Dec 2018 | A1 |
| 20180375744 | Mayya et al. | Dec 2018 | A1 |
| 20180375824 | Mayya et al. | Dec 2018 | A1 |
| 20180375967 | Pithawala et al. | Dec 2018 | A1 |
| 20190013883 | Vargas et al. | Jan 2019 | A1 |
| 20190014038 | Ritchie | Jan 2019 | A1 |
| 20190020588 | Twitchell, Jr. | Jan 2019 | A1 |
| 20190020627 | Yuan | Jan 2019 | A1 |
| 20190028552 | Johnson et al. | Jan 2019 | A1 |
| 20190036808 | Shenoy et al. | Jan 2019 | A1 |
| 20190036810 | Michael et al. | Jan 2019 | A1 |
| 20190036813 | Shenoy et al. | Jan 2019 | A1 |
| 20190046056 | Khachaturian et al. | Feb 2019 | A1 |
| 20190058657 | Chunduri et al. | Feb 2019 | A1 |
| 20190058709 | Kempf et al. | Feb 2019 | A1 |
| 20190068470 | Mirsky | Feb 2019 | A1 |
| 20190068493 | Ram et al. | Feb 2019 | A1 |
| 20190068500 | Hira | Feb 2019 | A1 |
| 20190075083 | Mayya et al. | Mar 2019 | A1 |
| 20190103990 | Cidon et al. | Apr 2019 | A1 |
| 20190103991 | Cidon et al. | Apr 2019 | A1 |
| 20190103992 | Cidon et al. | Apr 2019 | A1 |
| 20190103993 | Cidon et al. | Apr 2019 | A1 |
| 20190104035 | Cidon et al. | Apr 2019 | A1 |
| 20190104049 | Cidon et al. | Apr 2019 | A1 |
| 20190104050 | Cidon et al. | Apr 2019 | A1 |
| 20190104051 | Cidon et al. | Apr 2019 | A1 |
| 20190104052 | Cidon et al. | Apr 2019 | A1 |
| 20190104053 | Cidon et al. | Apr 2019 | A1 |
| 20190104063 | Cidon et al. | Apr 2019 | A1 |
| 20190104064 | Cidon et al. | Apr 2019 | A1 |
| 20190104109 | Cidon et al. | Apr 2019 | A1 |
| 20190104111 | Cidon et al. | Apr 2019 | A1 |
| 20190104413 | Cidon et al. | Apr 2019 | A1 |
| 20190109769 | Jain et al. | Apr 2019 | A1 |
| 20190140889 | Mayya et al. | May 2019 | A1 |
| 20190140890 | Mayya et al. | May 2019 | A1 |
| 20190158371 | Dillon et al. | May 2019 | A1 |
| 20190158605 | Markuze et al. | May 2019 | A1 |
| 20190199539 | Deng et al. | Jun 2019 | A1 |
| 20190220703 | Prakash et al. | Jul 2019 | A1 |
| 20190238364 | Boutros et al. | Aug 2019 | A1 |
| 20190238446 | Barzik et al. | Aug 2019 | A1 |
| 20190238449 | Michael et al. | Aug 2019 | A1 |
| 20190238450 | Michael et al. | Aug 2019 | A1 |
| 20190238483 | Marichetty et al. | Aug 2019 | A1 |
| 20190268421 | Markuze et al. | Aug 2019 | A1 |
| 20190268973 | Bull et al. | Aug 2019 | A1 |
| 20190280962 | Michael et al. | Sep 2019 | A1 |
| 20190280963 | Michael et al. | Sep 2019 | A1 |
| 20190280964 | Michael et al. | Sep 2019 | A1 |
| 20190306197 | Degioanni | Oct 2019 | A1 |
| 20190313907 | Khachaturian et al. | Oct 2019 | A1 |
| 20190319847 | Nahar et al. | Oct 2019 | A1 |
| 20190334813 | Raj et al. | Oct 2019 | A1 |
| 20190334820 | Zhao | Oct 2019 | A1 |
| 20190342219 | Liu et al. | Nov 2019 | A1 |
| 20190356736 | Narayanaswamy et al. | Nov 2019 | A1 |
| 20190364099 | Thakkar et al. | Nov 2019 | A1 |
| 20190372888 | Michael et al. | Dec 2019 | A1 |
| 20190372889 | Michael et al. | Dec 2019 | A1 |
| 20190372890 | Michael et al. | Dec 2019 | A1 |
| 20200014615 | Michael et al. | Jan 2020 | A1 |
| 20200014616 | Michael et al. | Jan 2020 | A1 |
| 20200014661 | Mayya et al. | Jan 2020 | A1 |
| 20200021514 | Michael et al. | Jan 2020 | A1 |
| 20200021515 | Michael et al. | Jan 2020 | A1 |
| 20200036624 | Michael et al. | Jan 2020 | A1 |
| 20200044943 | Bor-Yaliniz et al. | Feb 2020 | A1 |
| 20200059420 | Abraham | Feb 2020 | A1 |
| 20200059457 | Raza et al. | Feb 2020 | A1 |
| 20200059459 | Abraham et al. | Feb 2020 | A1 |
| 20200092207 | Sipra et al. | Mar 2020 | A1 |
| 20200097327 | Beyer et al. | Mar 2020 | A1 |
| 20200099659 | Cometto et al. | Mar 2020 | A1 |
| 20200106696 | Michael et al. | Apr 2020 | A1 |
| 20200106706 | Mayya et al. | Apr 2020 | A1 |
| 20200119952 | Mayya et al. | Apr 2020 | A1 |
| 20200127905 | Mayya et al. | Apr 2020 | A1 |
| 20200127911 | Gilson et al. | Apr 2020 | A1 |
| 20200153701 | Mohan et al. | May 2020 | A1 |
| 20200153736 | Liebherr et al. | May 2020 | A1 |
| 20200169473 | Rimar et al. | May 2020 | A1 |
| 20200177503 | Hooda et al. | Jun 2020 | A1 |
| 20200204460 | Schneider et al. | Jun 2020 | A1 |
| 20200213212 | Dillon et al. | Jul 2020 | A1 |
| 20200213224 | Cheng et al. | Jul 2020 | A1 |
| 20200218558 | Sreenath et al. | Jul 2020 | A1 |
| 20200235990 | Janakiraman et al. | Jul 2020 | A1 |
| 20200235999 | Mayya et al. | Jul 2020 | A1 |
| 20200236046 | Jain et al. | Jul 2020 | A1 |
| 20200244721 | S et al. | Jul 2020 | A1 |
| 20200252234 | Ramamoorthi et al. | Aug 2020 | A1 |
| 20200259700 | Bhalla et al. | Aug 2020 | A1 |
| 20200267184 | Vera-Schockner | Aug 2020 | A1 |
| 20200280587 | Janakiraman et al. | Sep 2020 | A1 |
| 20200287819 | Theogaraj et al. | Sep 2020 | A1 |
| 20200287976 | Theogaraj et al. | Sep 2020 | A1 |
| 20200296011 | Jain et al. | Sep 2020 | A1 |
| 20200296026 | Michael et al. | Sep 2020 | A1 |
| 20200314006 | Mackie et al. | Oct 2020 | A1 |
| 20200314614 | Moustafa et al. | Oct 2020 | A1 |
| 20200336336 | Sethi et al. | Oct 2020 | A1 |
| 20200344143 | Faseela et al. | Oct 2020 | A1 |
| 20200344163 | Gupta et al. | Oct 2020 | A1 |
| 20200351188 | Arora et al. | Nov 2020 | A1 |
| 20200366530 | Mukundan et al. | Nov 2020 | A1 |
| 20200366562 | Mayya et al. | Nov 2020 | A1 |
| 20200382345 | Zhao et al. | Dec 2020 | A1 |
| 20200382387 | Pasupathy et al. | Dec 2020 | A1 |
| 20200412576 | Kondapavuluru et al. | Dec 2020 | A1 |
| 20200413283 | Shen et al. | Dec 2020 | A1 |
| 20210006482 | Hwang et al. | Jan 2021 | A1 |
| 20210006490 | Michael et al. | Jan 2021 | A1 |
| 20210029019 | Kottapalli | Jan 2021 | A1 |
| 20210029088 | Mayya et al. | Jan 2021 | A1 |
| 20210036888 | Makkalla et al. | Feb 2021 | A1 |
| 20210036987 | Mishra | Feb 2021 | A1 |
| 20210067372 | Cidon et al. | Mar 2021 | A1 |
| 20210067373 | Cidon et al. | Mar 2021 | A1 |
| 20210067374 | Cidon et al. | Mar 2021 | A1 |
| 20210067375 | Cidon et al. | Mar 2021 | A1 |
| 20210067407 | Cidon et al. | Mar 2021 | A1 |
| 20210067427 | Cidon et al. | Mar 2021 | A1 |
| 20210067442 | Sundararajan et al. | Mar 2021 | A1 |
| 20210067461 | Cidon et al. | Mar 2021 | A1 |
| 20210067464 | Cidon et al. | Mar 2021 | A1 |
| 20210067467 | Cidon et al. | Mar 2021 | A1 |
| 20210067468 | Cidon et al. | Mar 2021 | A1 |
| 20210105199 | C H et al. | Apr 2021 | A1 |
| 20210112034 | Sundararajan et al. | Apr 2021 | A1 |
| 20210126853 | Ramaswamy et al. | Apr 2021 | A1 |
| 20210126854 | Guo et al. | Apr 2021 | A1 |
| 20210126860 | Ramaswamy et al. | Apr 2021 | A1 |
| 20210144091 | C H et al. | May 2021 | A1 |
| 20210160813 | Gupta et al. | May 2021 | A1 |
| 20210184952 | Mayya et al. | Jun 2021 | A1 |
| 20210184966 | Ramaswamy et al. | Jun 2021 | A1 |
| 20210184983 | Ramaswamy et al. | Jun 2021 | A1 |
| 20210194814 | Roux et al. | Jun 2021 | A1 |
| 20210226880 | Ramamoorthy et al. | Jul 2021 | A1 |
| 20210234728 | Cidon et al. | Jul 2021 | A1 |
| 20210234775 | Devadoss et al. | Jul 2021 | A1 |
| 20210234786 | Devadoss et al. | Jul 2021 | A1 |
| 20210234804 | Devadoss et al. | Jul 2021 | A1 |
| 20210234805 | Devadoss et al. | Jul 2021 | A1 |
| 20210235312 | Devadoss et al. | Jul 2021 | A1 |
| 20210235313 | Devadoss et al. | Jul 2021 | A1 |
| 20210266262 | Subramanian et al. | Aug 2021 | A1 |
| 20210279069 | Salgaonkar et al. | Sep 2021 | A1 |
| 20210314289 | Chandrashekhar | Oct 2021 | A1 |
| 20210328835 | Mayya et al. | Oct 2021 | A1 |
| 20210377109 | Shrivastava et al. | Dec 2021 | A1 |
| 20210377156 | Michael et al. | Dec 2021 | A1 |
| 20210392060 | Silva et al. | Dec 2021 | A1 |
| 20210392070 | Tootaghaj et al. | Dec 2021 | A1 |
| 20210399920 | Sundararajan | Dec 2021 | A1 |
| 20210399978 | Michael et al. | Dec 2021 | A9 |
| 20210400113 | Markuze | Dec 2021 | A1 |
| 20220006726 | Michael et al. | Jan 2022 | A1 |
| 20220006751 | Ramaswamy et al. | Jan 2022 | A1 |
| 20220006756 | Ramaswamy et al. | Jan 2022 | A1 |
| 20220035673 | Markuze et al. | Feb 2022 | A1 |
| 20220038370 | Vasseur et al. | Feb 2022 | A1 |
| 20220038557 | Markuze et al. | Feb 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 1926809 | Mar 2007 | CN |
| 102577270 | Jul 2012 | CN |
| 102811165 | Dec 2012 | CN |
| 104956329 | Sep 2015 | CN |
| 110447209 | Nov 2019 | CN |
| 1912381 | Apr 2008 | EP |
| 3041178 | Jul 2016 | EP |
| 3509256 | Jul 2019 | EP |
| 2010233126 | Oct 2010 | JP |
| 2017059991 | Mar 2017 | JP |
| 2574350 | Feb 2016 | RU |
| 03073701 | Sep 2003 | WO |
| 2007016834 | Feb 2007 | WO |
| 2012167184 | Dec 2012 | WO |
| 2016061546 | Apr 2016 | WO |
| 2017083975 | May 2017 | WO |
| 2019070611 | Apr 2019 | WO |
| 2019094522 | May 2019 | WO |
| 2020012491 | Jan 2020 | WO |
| 2020018704 | Jan 2020 | WO |
| 2020091777 | May 2020 | WO |
| 2020101922 | May 2020 | WO |
| 2020112345 | Jun 2020 | WO |
| 2021040934 | Mar 2021 | WO |
| 2021118717 | Jun 2021 | WO |
| 2021150465 | Jul 2021 | WO |
| 2022005607 | Jan 2022 | WO |
| Entry |
|---|
| Alsaeedi, Mohammed, et al., “Toward Adaptive and Scalable OpenFlow-SDN Flow Control: A Survey,” IEEE Access, Aug. 1, 2019, 34 pages, vol. 7, IEEE, retrieved from https://ieeexplore.ieee.org/document/8784036. |
| Long, Feng, “Research and Application of Cloud Storage Technology in University Information Service,” Chinese Excellent Masters' Theses Full-text Database, Mar. 2013, 72 pages, China Academic Journals Electronic Publishing House, China. |
| Mon-Published Commonly Owned International Patent Application PCT/US2021/065171, filed Dec. 24, 2021, 63 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/562,890, filed Dec. 27, 2021, 36 pages, Nicira, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/572,583, filed Jan. 10, 2022, 33 pages, Nicira, Inc. |
| Noormohammadpour, Mohammad, et al., “DCRoute: Speeding up Inter-Datacenter Traffic Allocation while Guaranteeing Deadlines,” 2016 IEEE 23rd International Conference on High Performance Computing (HiPC), Dec. 19-22, 2016, 9 pages, IEEE, Hyderabad, India. |
| Del Piccolo, Valentin, et al., “A Survey of Network Isolation Solutions for Multi-Tenant Data Centers,” IEEE Communications Society, Apr. 20, 2016, vol. 18, No. 4, 37 pages, IEEE. |
| Fortz, Bernard, et al., “Internet Traffic Engineering by Optimizing OSPF Weights,” Proceedings IEEE Infocom 2000, Conference on Computer Communications, Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies, Mar. 26-30, 2000, 11 pages, IEEE, Tel Aviv, Israel, Israel. |
| Francois, Frederic, et al., “Optimizing Secure SDN-enabled Inter-Data Centre Overlay Networks through Cognitive Routing,” 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), Sep. 19-21, 2016, 10 pages, IEEE, London, UK. |
| Huang, Cancan, et al., “Modification of Q.SD-WAN,” Rapporteur Group Meeting--Doc, Study Period 2017-2020, Q4/11-DOC1 (190410), Study Group 11, Apr. 10, 2019, 19 pages, International Telecommunication Union, Geneva, Switzerland. |
| Lasserre, Marc, et al., “Framework for Data Center (DC) Network Virtualization,” RFC 7365, Oct. 2014, 26 pages, IETF. |
| Lin, Weidong, et al., “Using Path Label Routing in Wide Area Software-Defined Networks with Open Flow,” 2016 International Conference on Networking and Network Applications, Jul. 2016, 6 pages, IEEE. |
| Michael, Nithin, et al., “HALO: Hop-by-Hop Adaptive Link-State Optimal Routing,” IEEE/ACM Transactions on Networking, Dec. 2015, 14 pages, vol. 23, No. 6, IEEE. |
| Mishra, Mayank, et al., “Managing Network Reservation for Tenants in Oversubscribed Clouds,” 2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems, Aug. 14-16, 2013, 10 pages, IEEE, San Francisco, CA, USA. |
| Mudigonda, Jayaram, et al., “NetLord: A Scalable Multi-Tenant Network Architecture for Virtualized Datacenters,” Proceedings of the ACM SIGCOMM 2011 Conference, Aug. 15-19, 2011, 12 pages, ACM, Toronto, Canada. |
| Non-Published Commonly Owned U.S. Appl. No. 17/072,764, filed Oct. 16, 2020, 33 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Patent Application 17/072,//4 (F919.02), filed Oct. 16, 2020, 34 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/085,893, filed Oct. 30, 2020, 34 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/085,916, filed Oct. 30, 2020, 35 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/103,614, filed Nov. 24, 2020, 38 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/143,092, filed Jan. 6, 2021, 42 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/143,094, filed Jan. 6, 2021, 42 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/194,038, filed Mar. 5, 2021, 35 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/227,016, filed Apr. 9, 2021, 37 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/227,044, filed Apr. 9, 2021, 37 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/240,890, filed Apr. 26, 2021, 325 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/240,906, filed Apr. 26, 2021, 18 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,327, filed Jun. 18, 2021, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,333, filed Jun. 18, 2021, 47 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,340, filed Jun. 18, 2021, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,342, filed Jun. 18, 2021, 47 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/351,345, filed Jun. 18, 2021, 48 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/361,292, filed Jun. 28, 2021, 35 pages, Micira, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/384,735 with similar specification, filed Jul. 24, 2021, 62 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/384,736 with similar specification, filed Jul. 24, 2021, 63 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/384,738 with similar specification, filed Jul. 24, 2021, 62 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/467,378, filed Sep. 6, 2021, 157 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/474,034, filed Sep. 13, 2021, 349 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 15/803,964, filed Nov. 6, 2017, 15 pages, The Mode Group. |
| Ray, Saikat, et al., “Always Acyclic Distributed Path Computation,” University of Pennsylvania Department of Electrical and Systems Engineering Technical Report, May 2008, 16 pages, University of Pennsylvania ScholarlyCommons. |
| Sarhan, Soliman Abd Elmonsef, et al., “Data Inspection in SDN Network,” 2018 13th International Conference on Computer Engineering and Systems (ICCES), Dec. 18-19, 2018, 6 pages, IEEE, Cairo, Egypt. |
| Webb, Kevin C., et al., “Blender: Upgrading Tenant-Based Data Center Networking,” 2014 ACM/IEEE Symposium an Architectures for Networking and Communications Systems (ANCS), Oct. 20-21, 2014, 11 pages, IEEE, Marina del Rey, CA, USA. |
| Xie, Junheng, et al., A Survey of Machine Learning Techniques Applied to Software Defined Networking (SDN): Research Issues and Challenges, IEEE Communications Surveys & Tutorials, Aug. 23, 2018, 38 pages, vol. 21, Issue 1, IEEE. |
| Yap, Kok-Kiong, et al., “Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering,” SIGCOMM '17: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, Aug. 21-25, 2017, 14 pages, Los Angeles, CA. |
| Guo, Xiangyi, et al., (U.S. Appl. No. 62/925,193), filed Oct. 23, 2019, 26 pages. |
| Non-Published Commonly Owned U.S. Appl. No. 17/542,413, filed Dec. 4, 2021, 173 pages, VMware, Inc. |
| Barozet, Jean-Marc, “Cisco SD-WAN as a Managed Service,” BRKRST-2558, Jan. 27-31, 2020, 98 pages, Cisco, Barcelona, Spain, retrieved from https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKRST-2558.pdf. |
| Barozet, Jean-Marc, “Cisco SDWAN,” Deep Dive, Dec. 2017, 185 pages, Cisco, Retreived from https://www.coursehero.com/file/71671376/Cisco-SDWAN-Deep-Divepdf/. |
| Cox, Jacob H., et al., “Advancing Software-Defined Networks: A Survey,” IEEE Access, Oct. 12, 2017, 40 pages, vol. 5, IEEE, retrieved from https://ieeexplore.ieee.org/document/8066287. |
| Ming, Gao, et al., “A Design of SD-WAN-Oriented Wide Area Network Access,” 2020 International Conference on Computer Communication and Network Security (CCNS), Aug. 21-23, 2020, 4 pages, IEEE, Xi'an, China. |
| PCT International Search Report and Written Opinion of Commonly Owned International Patent Application PCT/US2021/065171, dated Apr. 20, 2022, 15 pages, International Searching Authority (EPO). |
