Datacenters today use static, configuration intensive ways to distribute data messages between different application layers and to different service layers. A common approach today is to configure the virtual machines to send packets to virtual IP (VIP) addresses, and then configure the forwarding elements and load balancers in the datacenter with forwarding rules that direct them to forward VIP addressed packets to appropriate application and/or service layers. Another problem with existing message distribution schemes is that today's load balancers often are chokepoints for the distributed traffic. Accordingly, there is a need in the art for a new approach to seamlessly distribute data messages in the datacenter between different application and/or service layers. Ideally, this new approach would allow the distribution scheme to be easily modified without reconfiguring the servers that transmit the data messages.
Some embodiments provide novel methods for performing services for machines operating in one or more datacenters. For instance, for a group of related guest machines (e.g., a group of tenant machines), some embodiments define two different forwarding planes: (1) a guest forwarding plane and (2) a service forwarding plane. The guest forwarding plane connects to the machines in the group and performs L2 and/or L3 forwarding for these machines. The service forwarding plane (1) connects to the service nodes that perform services on data messages sent to and from these machines, and (2) forwards these data messages to the service nodes.
In some embodiments, the guest machines do not connect directly with the service forwarding plane. For instance, in some embodiments, each forwarding plane connects to a machine or service node through a port that receives data messages from, or supplies data messages to, the machine or service node. In such embodiments, the service forwarding plane does not have a port that directly receives data messages from, or supplies data messages to, any guest machine. Instead, in some such embodiments, data associated with a guest machine is routed to a port proxy module executing on the same host computer, and this port proxy module has a service plane port. This port proxy module in some embodiments indirectly can connect more than one guest machine on the same host to the service plane (i.e., can serve as the port proxy module for more than one guest machine on the same host).
In some embodiments, a guest machine is any machine that is not a service machine or node. A guest machine can be a tenant's machine in a multi-tenant datacenter, but it does not have to be. A guest machine in some embodiments is a guest virtual machine or guest container. A service node in some embodiments is a service virtual machine, a service container or a service appliance. In some embodiments, a service node performs a middlebox service operation, such as a firewall, an intrusion detection system, an intrusion prevention system, a load balancer, an encryptor, a message monitor, a message collector, or any number of other middlebox services. As such, a service as used in this document is any type of middlebox service operation in some embodiments.
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description, the Drawings and the Claims is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description and the Drawing.
The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Some embodiments provide novel methods for performing services for machines operating in one or more datacenters. For instance, for a group of related guest machines (e.g., a group of tenant machines), some embodiments define two different forwarding planes: (1) a guest forwarding plane and (2) a service forwarding plane. The guest forwarding plane connects to the machines in the group and performs L2 and/or L3 forwarding for these machines. The service forwarding plane (1) connects to the service nodes that perform services on data messages sent to and from these machines, and (2) forwards these data messages to the service nodes.
In some embodiments, the guest machines do not connect directly with the service forwarding plane. For instance, in some embodiments, each forwarding plane connects to a machine or service node through a port that receives data messages from, or supplies data messages to, the machine or service node. In such embodiments, the service forwarding plane does not have a port that directly receives data messages from, or supplies data messages to, any guest machine. Instead, in some such embodiments, data associated with a guest machine is routed to a port proxy module executing on the same host computer, and this other module has a service plane port. This port proxy module in some embodiments indirectly can connect more than one guest machine on the same host to the service plane (i.e., can serve as the port proxy module for more than one guest machine on the same host).
In some embodiments, a guest machine is any machine that is not a service machine or node. A guest machine can be a tenant's machine in a multi-tenant datacenter, but it does not have to be. A guest machine in some embodiments is a guest virtual machine or guest container. A service node in some embodiments is a service virtual machine, a service container or a service appliance. In some embodiments, a service node performs a middlebox service operation, such as a firewall, an intrusion detection system, an intrusion prevention system, a load balancer, an encryptor, a message monitor, a message collector, or any number of other middlebox services. As such, a service as used in this document is any type of middlebox service operation in some embodiments.
Also, as used in this document, data messages refer to a collection of bits in a particular format sent across a network. One of ordinary skill in the art will recognize that the term data message is used in this document to refer to various formatted collections of bits that are sent across a network. The formatting of these bits can be specified by standardized protocols or non-standardized protocols. Examples of data messages following standardized protocols include Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Also, as used in this document, references to L2, L3, L4, and L7 layers (or layer 2, layer 3, layer 4, and layer 7) are references respectively to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the OSI (Open System Interconnection) layer mod
Also, in this example, each logical forwarding element is a distributed forwarding element that is implemented by configuring multiple software forwarding elements (SFEs) on multiple host computers. To do this, each SFE or a module associated with the SFE in some embodiments is configured to encapsulate the data messages of the LFE with an overlay network header that contains a virtual network identifier (VNI) associated with the overlay network. As such, the LFEs are said to be overlay network constructs that span multiple host computers in the discussion below.
The LFEs also span in some embodiments configured hardware forwarding elements (e.g., top of rack switches). In some embodiments, each LFE is a logical switch that is implemented by configuring multiple software switches (called virtual switches or vswitches) or related modules on multiple host computers. In other embodiments, the LFEs can be other types of forwarding elements (e.g., logical routers), or any combination of forwarding elements (e.g., logical switches and/or logical routers) that form logical networks or portions thereof. Many examples of LFEs, logical switches, logical routers and logical networks exist today, including those provided by VMware's NSX network and service virtualization platform.
As shown, the LFE 130 defines the guest forwarding plane that connects the GVMs 102 and 104 in order to forward data messages between these GVMs. In some embodiments, this LFE is a logical switch that connects to a logical router, which connects the GVMs directly or through a logical gateway to networks outside of the logical switch's logical network. The LFE 130 is implemented in some embodiments by configuring software switches 120 and 122 and/or their related modules (e.g., related port/VNIC filter modules) on the host computers 112 and 114 to implement a first distributed logical switch.
The LFE 132 defines the service forwarding plane that connects the SVMs 106, 108 and 110 in order to forward data messages associated with the GVMs through service paths that include the SVMs. In some embodiments, the LFE 132 is also a logical switch that is implemented by configuring software switches 120, 122 and 124 and/or their related modules on the host computers 112, 114 and 116 to implement a second distributed logical switch. Instead of configuring the same set of SFEs to implement both the guest and service forwarding planes (i.e., the guest and service LFEs), other embodiments configure one set of SFEs on a set of host computers to implement the guest forwarding plane and another set of SFEs on the set of host computers to implement the service forwarding plane. For instance, in some embodiments, each host computer executes a guest software switch and a service software switch, and these two switches and/or their related modules can be configured to implement a guest logical switch and a service logical switch.
In some embodiments, the software switches 120, 122 and 124 and/or their related modules can be configured to implement multiple guest forwarding planes (e.g., guest LFEs) and multiple service forwarding planes (e.g., service LFEs) for multiple groups of machines. For instance, for a multi-tenant datacenter, some such embodiments define a guest LFE and a service LFE for each tenant for which at least one chain of services needs to be implemented. For each group of related machines (e.g., for each tenant's machines), some embodiments define two virtual network identifiers (VNIs) to configure a shared set of software forwarding elements (e.g., software switches) to implement the two different forwarding planes, i.e., the guest forwarding plane and the service forwarding plane. These two VNIs are referred to below as the guest VNI (GVNI) and the service VNI (SVNI). In
In some embodiments, the service plane 132 is also implemented by inserting modules in input/output (TO) chains of a GVM's egress and ingress datapaths to and from an SFE 120 or 122. In this implementation, the service plane 132 can identify a data message sent from the GVM or received for the GVM, forward the data message to a set of SVMs to perform a chain of services on the data message, and then to return the data message back to the GVM's datapath so that the data message can be proceed along its datapath to the software switch or to the GVM (i.e., so that the data message can be processed based on the destination network addresses specified by the source GVM). Such a GVM is referred to below as the source GVM as the data message being processed by the service nodes is a data message identified on the GVM's egress or ingress path. In some embodiments, a GVM's egress/ingress IO chain is implemented as a set of hooks (function calls) in the GVM's VNIC (virtual network interface card) 180 or the SFE port associated with the GVM's VNIC (e.g., the SFE port communicating with the GVM's VNIC).
Before providing an example of the IO chain components of some embodiments that implement the service plane,
The service LFE in some embodiments forwards the data message between hosts 112, 114 and 116 by using an overlay encapsulation header that stores the SVNI for the service LFE. Also, when the service LFE is a service logical switch, the service forwarding plane in some embodiments uses the MAC addresses associated with the SVMs (e.g., MAC addresses of SVM VNICs) to forward the data message between ports of the service logical switch. In some embodiments, the MAC forwarding also uses service plane MAC address associated with the source GVM, even though this GVM does not directly connect to the service plane but instead connects to the service plane through a port proxy, as further described below.
Once the data message 202 returns to the egress path of the GVM 102, the guest LFE 130 forwards the data message to its destination (e.g., as specified by the destination network address in the data message's header), which is GVM 104. The guest LFE 130 in some embodiments forwards the data message between hosts 112 and 114 by using an overlay encapsulation header that stores the GVNI for the guest LFE. Also, when the guest LFE is a logical switch, the guest forwarding plane in some embodiments uses the guest plane MAC addresses associated with the GVMs 102 and 104 to forward the data message (e.g., by using the guest plane MAC address of GVM 104 to forward the data message to the guest forwarding port 152 associated with this GVM). While the service plane of
In some embodiments, a chain of service operations is referred to as a service chain. A service chain in some embodiments can be implemented with one or more sets of service nodes (e.g., service machines or appliances), with each set of service nodes defining a service path. Hence, in some embodiments, a service chain can be implemented by each of one or more service paths. Each service path in some embodiments includes one or more service nodes for performing the set of one or more services of the service chain and a particular order through these nodes.
In some embodiments, each service chain 302 is defined by references to one or more service profiles 312, with each service profile associated with a service operation in the chain. As described below, a service node in some embodiments (1) receives, from a service manager, a mapping of a service chain identifier to a service profile that it has to implement, and (2) receives, with a data message, a service chain identifier that it maps to the service profile to determine the service operation that it has to perform. In some embodiments, the received mapping is not only based on the service chain identifier (SCI) but is also based on a service index value (that specifies the location of the service node in a service path) and a direction through a service chain (that specifies an order for performing the sequence of services specified by the service chain). The service profile in some embodiments describes the service operation that the service node has to perform. In some embodiments, a service profile can identify a set of rules for a service node to examine.
Also, in some embodiments, service insertion rules 314 are defined by reference to service chain identifies 306 for service insertion modules associated with GVMs. Such service insertion modules use these service insertion rules 314 to identify service chains to use to process data messages associated with a source GVM. As mentioned above, the data messages are referred to below as being from a source GVM as the data messages that are processed by the service chains are identified on the egress paths from or ingress paths to the GVMs.
As further described below, the service insertion (SI) rules associate flow identifiers with service chain identifiers. In other words, some embodiments try to match a data message's flow attributes to the flow identifiers (referred to below as rule identifiers of the SI rules) of the service insertion rules, in order to identify a matching service insertion rule (i.e., a rule with a set of flow identifiers that matches the data message's flow attributes) and to assign this matching rule's specified service chain as the service chain of the data message. A specific flow identifier (e.g., one defined by reference to a five-tuple identifier) could identify one specific data message flow, while a more general flow identifier (e.g., one defined by reference to less than the five tuples) can identify a set of several different data message flows that match the more general flow identifier. As such, a matching data message flow is any set of data messages that have a common set of attributes that matches a rule identifier of a service insertion rule.
As further described below, other embodiments use contextual attributes associated with a data message flow to associate the data message with a service insertion rule. Numerous techniques for capturing and using contextual attributes for performing forwarding and service operations are described in U.S. patent application Ser. No. 15/650,251, now issued as U.S. Pat. No. 10,802,857, which is incorporated herein. Any of these techniques can be used in conjunction with the embodiments described herein.
Next hop forwarding rules 316 in some embodiments are defined by reference to the SPI values 308 and service instance endpoints 310. Specifically, in some embodiments, a service path is selected for a service chain that has been identified for a data message. At each hop, these embodiments use the forwarding rules 314 to identify the next service instance endpoint based on the SPI value for this service path along with a current service index (SI) value, which identifies the location of the hop in the service path. In other words, each forwarding rule in some embodiments has a set of matching criteria defined in terms of the SPI/SI values, and specifies a network address of the next hop service instance endpoint that is associated with these SPI/SI values. To optimize the next hop lookup for the first hop, some embodiments provide to the source GVM's service insertion module the next hop network address with the SPI, as part of a service path selection process.
Since multiple SVMs in a cluster can provide a particular service, some embodiments define for a given service chain, multiple service paths through multiple different combinations of SVMs, with one SVM of each cluster being used in each combination. In the example of
Specifically, the first service path passes through first SVM 1,1 of the first service profile's cluster to implement the first service of the forward service chain 405, the first SVM 2,1 of the second service profile's cluster to implement the second service of the forward service chain 405, and third SVM M,3 of the Mth service profile's cluster to implement the Mth service of the forward service chain 405. The second service path passes through second SVM 1,2 of the first service profile's cluster to implement the first service of the forward service chain 405, the first SVM 2,1 of the second service profile's cluster to implement the second service of the forward service chain 405, and first SVM M,1 of the Mth service profile's cluster to implement the Mth service of the forward service chain 405.
The third service path passes through third SVM 1,3 of the first service profile's cluster to implement the first service of the forward service chain 405, the second SVM 2,2 of the second service profile's cluster to implement the second service of the forward service chain 405, and second SVM M,2 of the Mth service profile's cluster to implement the Mth service of the forward service chain 405. The Nth service path passes through third SVM 1,3 of the first service profile's cluster to implement the first service of the forward service chain 405, the second SVM 2,2 of the second service profile's cluster to implement the second service of the forward service chain 405, and fourth SVM M,4 of the Mth service profile's cluster to implement the Mth service of the forward service chain 405. As the example illustrates, different service paths may use the same SVM for a given service operation. However, regardless of the service path that a given data message traverses, the same set of service operations is performed in the same sequence, for paths that are associated with the same service chain and the same service direction.
In some embodiments, a service chain has to be performed in a forward direction for data messages from a first GVM to a second GVM, and then in the reverse direction for data messages from the second GVM to the first GVM. In some such embodiments, the service plane selects both the service path for the forward direction and the service path for the reverse direction when it processes the first data message in the flow from the first GVM to the second GVM. Also, in some of these embodiments, the forward and reverse service paths are implemented by the same sets of service nodes but in the reverse order.
Also, in the examples of
In some embodiments, the same service nodes are used for the forward and reverse paths because at least one of the service nodes (e.g., a firewall SVM) that implements one of the service profiles needs to see the data traffic in both directions between two data endpoints (e.g., two GYMS). In other embodiments, the same service nodes do not need to be used for both directions of data message flows between two data endpoints so long as the same set of service operations are performed in opposite orders.
In
The service transport layer 604, on the other hand, formulates the service overlay encapsulation header and encapsulates the data message with this header so that it can pass between service hops. In some embodiments, the service transport layer 604 modifies the SMD header to produce the service overlay encapsulation header. For instance, in some of these embodiments, the overlay encapsulation header is a Geneve header with the SMD attributes stored in a TLV (type, length, value) section of the Geneve header. In other embodiments, the service transport layer 604 adds the service overlay encapsulation header to an SMD header that is first used to encapsulate the data message. Also, when traversing between two hops (e.g., between two service nodes) executing on the same host computer, the service transport layer in several embodiments described below does not encapsulate the data message with an overlay encapsulation header in some embodiments. In other embodiments, even when traversing between two hops on the same host computer, the service transport layer encapsulates the data message with an overlay encapsulation header.
In some embodiments, the service insertion (SI) layer 602 includes an SI pre-processor 610 and an SI post-processor 612, in each the two IO chains 650 and 652 (i.e., the egress 10 chain 650 and the ingress IO chain 652) of a GVM for which one or more service chains are defined. The SI layer 602 also includes a service proxy 614 for each service node connected to the service plane (e.g., for each SVM with a VNIC paired with a service plane LFE port). The service transport (ST) layer 604 includes one STL port proxy 620 on each host computer that has one or more possible source GVMs for which one or more service chains are defined. The ST layer 604 also has (1) an STL caller 624 in each IO chain of each source GVM, (2) an STL module 626 in the IO chain of each SVM, and (3) one or more encap processors 628.
For a data message that passes through a GVM's ingress or egress datapath, the SI pre-processor 610 on this datapath performs several operations. It identifies the service chain for the data message and selects the service path for the identified service chain. The pre-processor also identifies the network address for a first hop service node in the selected service path and specifies the SMD attributes for the data message. The SMD attributes include in some embodiments the service chain identifier (SCI), the SPI and SI values, and the direction (e.g., forward or reverse) for processing the service operations of the service chain. In some embodiments, the SPI value identifies the service path while the SI value specifies the number of service nodes.
After the SI pre-processor completes its operation, the STL caller 624 in the same datapath calls the STL port proxy 620 to relay the SMD attributes and first hop's network address that the pre-processor identified, so that the port proxy can forward the SMD attributes through the service plane to the first hop. The port proxy formats the data message for forwarding to the first service node. In some embodiments, this formatting comprises replacing the original source and destination MAC addresses in the data message with a service plane MAC address that is associated with the source GVM 102 and the MAC address of the first hop service node. This formatting also stores a set of attributes for the data message that should be processed by other service transport layer modules (e.g., the other STL modules, etc.) on the same host computer. These data message attributes include the SMD attributes as well as the original source and destination MAC addresses.
The STL port proxy 620 passes the formatted data message along with its stored attributes to the software switch 120. Based on the destination MAC address (i.e., the first hop MAC address) of the formatted data message, the software switch delivers the data message to the switch port associated with the first hop SVM. When the first hop is on the same host computer as the port proxy 620, the data message is provided to the STL module 626 in the ingress IO chain of the first hop's service node on the same host computer. When the first hop is not on the same host computer, the data message is encapsulated with an encapsulating header and forwarded to the next hop, as further described below.
Each hop's STL module 626 re-formats the data message by replacing the service plane source MAC address and service plane destination MAC address (i.e., its service node's MAC address) with the original source and destination MAC addresses of the data message. It then passes this re-formatted data message with its accompanying SMD attributes to its hop's service proxy 614. This service proxy is in the IO chain of the ingress datapath of the GVM. For purposes of preventing the illustration in
The service proxy 614 encapsulates the received data message with an encapsulating NSH header that stores the data message's SMD attributes and provides this encapsulated data message to its service node when the service node can support NSH headers. When the service node is an SVM, the service proxy in some embodiments supplies the data messages and its NSH header to the SVM's VNIC through a VNIC injection process, as further described below. When the service node cannot process NSH headers, the service proxy 614 stores the SMD attributes into a legacy QinQ encapsulating header or a GRE encapsulating header, and then passes the encapsulated data message to the VNIC of the SVM. These headers will be further described below.
In some embodiments, the service proxy 614 of each service hop segregates the service node for that hop from the service transport layer. This segregation improves the security of both the SVM and the service transport layer. It also allows the service proxy to ensure that the data messages that are provided to its SVM are formatted properly, which is especially important for legacy SVMs that do not support the newer NSH format.
The service proxy 614 in some embodiments also performs liveness detection signaling with its service node to ensure that the service node is operational. In some embodiments, the service proxy sends a data message with a liveness value to its service node at least once in each recurring time period. To do this, the service proxy sets and resets a timer to ensure that it has sent a liveness signal for each time period to its service node. Each liveness value is accompanied with a liveness sequence number to allow the service proxy to keep track of liveness responses provided by the SVM. Each time the service node replies to a liveness signal, it provides to the service proxy the same liveness value in a responsive data message in some embodiments or its corresponding value in the responsive data message in other embodiments. Also, with each liveness responsive data message, the service node provides the same sequence number in some embodiments, or an incremented version of the sequence number provided by the service proxy in other embodiments.
As further described below, the service proxy of some embodiments piggybacks some of its liveness detection signaling on each data message that it passes to its service node from the service forwarding plane. Each time that the service proxy sends a liveness signal to its service node, it resets its liveness timer. Each time the service node processes the data message, it provides the processed data message back to the service node with the responsive liveness value and associated sequence number (incremented in some embodiments, or non-incremented in other embodiments, as mentioned above).
In some embodiments, the service proxy registers a liveness detection failure when the service node does not respond to its liveness signal within a particular time (e.g., within 0.3 seconds). After registering two successive liveness detection failures, the service proxy in some embodiments notifies a local control plane (LCP) module executing on its host the SVM has failed so that the LCP can notify a central control plane (CCP) server. In response to such a notification, the CCP removes the SVM and the service paths on which SVM resides from the forwarding and path selection rules in the data plane, and if needed, generates additional service paths for the failed SVM's associated service chain. Also, in some embodiments, the service proxy sends an in-band data message back to the source GVM to program its classifier to not select the service path on which the failed service node resides.
In some embodiments, the service proxy also performs flow programming at the behest of its service node. This flow programming in some embodiments involves modifying how the source GVM's IO chain selects service chains, service paths, and/or forwards data message flows along service paths. In other embodiments, this flow programming involves other modifications to how a data message flow is processed by the service plane. Flow programming will be further described below.
Upon receiving a data message and its SMD attributes (in an encapsulating NSH header or some other encapsulating header), the SVM performs its service operation. In some embodiments, the SVM uses mapping records that it receives from its service manager to map the SCI, SI and direction values in the SMD attributes to a service profile, and then maps this service profile to one of its rule sets, which it then examines to identify one or more service rules to process. In some embodiments, each service rule has a rule identifier that is defined in terms of data message attributes (e.g., five tuple attributes, which are the source and destination IP address, source and destination port addresses and the protocol). The SVM in some embodiments compares the rule's identifier with the attributes of the data message to identify a matching rule. Upon identifying one or more matching rules, the SVM in some embodiments performs an action specified by the highest priority matching rule. For instance, a firewall SVM might specify that the data message should be allowed to pass, should be dropped and/or should be redirected.
Once the SVM has completed its service operation, the SVM forwards the data message along its egress datapath. The service proxy in the egress datapath's IO chain then captures this data message and for this data message, identifies the network address of the next hop in the service path. To do this, the service proxy in some embodiments decrements the SI value, and then uses this decremented value along with the SPI value in the data message's stored attribute set to identify an exact match forwarding rule that identifies a next hop network address. In some embodiments, the SVM can decrement the SI value. For such cases, the service proxy in some embodiments can be configured not to decrement the SI value when its corresponding SVM decremented it.
In either configuration, the service proxy identifies the next hop network address by using the appropriate SPI/SI values to identify the next-hop forwarding rule applicable to the data message. When the proxy's service node is on multiple service paths, the proxy's forwarding rule storage stores multiple exact match forwarding rules that can specify different next hop network addresses for different SPI/SI values associated with different service paths. Assuming that the decremented SI value is not zero, the next hop in the service path is another service node. Hence, the proxy in some embodiments provides the next hop's MAC address to the proxy's associated STL module 626 in the SVM's egress datapath. This module then re-formats the data message, by specifying the SVM's MAC address and the next hop's MAC address as the source and destination MAC addresses and storing the original source and destination MAC addresses of the data message in the stored set of attributes stored for the data message. The STL module 626 then forward the data message along the egress path, where it reaches the software switch, which then has to forward the data message and its stored attributes to the next hop service node.
When the next hop is on the same host computer, the software switch passes the data message and its attributes to the port that connects to the STL module of the next hop's service node, as described above. On the other hand, when the next hop service node is on another host computer, the software switch provides data message to the uplink port that connects to the VTEP (VXLAN Tunnel Endpoint) that communicates through an overlay network tunnel with a VTEP on the other host computer. An encap processor 628 then captures this data message along the egress path of this port, defines an encapsulating overlay header for this data message and encapsulates the data message with this overlay header. In some embodiments, the overlay header is a single header that stores both SMD and STL attributes. For instance, in some embodiments, the overlay header is a Geneve header that stores the SMD and STL attributes in one or more TLVs.
As mentioned above, the SMD attributes in some embodiments include the SCI value, the SPI value, the SI value, and the service direction. Also, in some embodiments, the STL attributes includes the original L2 source MAC address, the original L2 destination MAC address, the data message direction, and the service-plane source MAC address of the source GVM. In some embodiments, the service direction and the service-plane source MAC address are already part of the SMD attributes. The service transport layer in some embodiments needs these attributes with each processed data message, in order to recreate the original data message and later at the end of the service-path, to return the data message to the original host to resume along its datapath.
When the encapsulated data message is received at the next hop's host computer, the data message is captured by the encap processor 628 of the software switch's downlink port that connects to the VTEP that received the data message from the prior hop's VTEP. This encap processor removes the encapsulation header from the data message and stores the STL and SMD attributes as the set of attributes of the data message. It then passes the decapsulated message to the downlink port, which then passes it to the software switch to forward to the next hop's switch port. From there the data message is processed by the STL module and service proxy before reaching the service node, as described above.
When the service proxy determines that the decremented SI value is zero, the service proxy matches the decremented SI value and the embedded SPI value with a rule that directs the service proxy to identify the next hop as the service plane MAC address of the source GVM. In some embodiments, this determination is not specified by a forwarding entry of a forwarding table, but rather is hard coded into the logic of the service proxy. Hence, when the SI value is zero, the proxy provides the source GVM's service plane MAC address to its associated STL module 626 to use to forward the data message back to the GVM's host computer. The STL module then defines the message's destination MAC (DMAC) address as the source GVM's service plane MAC address while defining the message's source MAC (SMAC) address as the service plane MAC address associated with its service node (e.g., the service plane MAC of the software switch's port associated with the service node). It also stores the original SMAC and DMAC of the data message in the attribute set of the data message.
The STL module then passes the formatted data message and its attributes along the egress path, where it reaches it associated software switch port. The software switch then passes this message to its uplink port. The encap processor 628 of this port then captures this data message, defines an encapsulating overlay header for this data message and encapsulates the data message with this overlay header. As mentioned above, this overlay header is a Geneve header that stores the SMD and STL attributes in one or more TLVs. This encapsulated data message then traverses the overlay network to reach the source GVM's host computer, where this data message is decapsulated by the downlink port's encap processor, and is then provided to the software switch, which then forwards it to the port proxy.
Once the port proxy 620 receives the decapsulated data message, it identifies the GVM associated with this data message from the original source MAC address that is now part of the decapsulated data message's stored attributes. In some embodiments, the port proxy has a record that maps the original source MAC address and service direction in the SMD attributes of a received data to a GVM on its host (e.g., to a software switch port associated with a guest forwarding plane and a GVM on its host). The port proxy then formats the data message to include its original SMAC and DMAC and provides the data message back to the source GVM's IO chain. The SI post-processor 612 in this IO chain then processes this data message, before returning this data message to the egress datapath of the GVM. The operations of this post-processor will be further described below.
One of ordinary skill will realize that the service insertion layer and service transport layer in other embodiments are implemented differently than the exemplary implementations described above. For instance, instead of using an L2 overlay (L2 transport layer) that relies on MAC addresses to traverse the different service hops, other embodiments use an L3 overlay (L3 transport layer) that uses L3 and/or L4 network addresses to identify successive service hops. Also, the above-described service insertion and/or transport modules can be configured to operate differently.
A more detailed example of the operations of the service insertion and service transport layers will now be described by reference to
As shown, the process 700 initially determines (at 705) whether the pre-processor 610 has previously selected a service chain and a service path for the data message's flow and stored the SMD attributes for the selected service chain and path. In some embodiments, the process 700 makes this determination by using the data message's attributes (e.g., its five tuple attributes) to try to identify a record for the message's flow in a connection tracker that stores records of message flows for which service chains and paths were previously selected, and SMD attributes were previously stored for these chains and paths in the connection tracker records.
When the process 700 determines (at 705) that the connection storage 804 has a connection record that matches the received data message, the process retrieves (at 710) the SMD attributes from this record, or from another record referenced by the matching connection record. The SMD attributes in some embodiments include the SCI, SPI, SI and direction values. From 710, the process transitions to 740, which will be described below.
On the other hand, when the process 700 determines (at 705) that the connection storage 804 does not have a connection record that matches the received data message, the process performs (at 715) a classification operation that tries to match the data message to a service insertion rule in a SI rule storage, which is illustrated in
At 720, the process determines whether the classification operation matches the data message's attributes to the rule identifier of a service insertion rule that requires a service chain to be performed on the data message. When the classification operation does not identify a service insertion rule that requires a service chain to be performed on the data message, the process 700 ends. In some embodiments, the SI rule storage 806 has a default low priority rule that matches any data message when the data message's attributes do not match any higher priority SI rule, and this default low priority rule specifies that no service chain has been defined for the data message's flow. No service chain is defined for a data message flow in some embodiments when no service operations needs to be performed on the data message flow.
On the other hand, when the classification operation matches the data message's attributes to the rule identifier of a service insertion rule that requires a service chain to be performed on the data message, the process 700 performs (725) a path selection operation to select a service path for the service chain specified by the service insertion rule identified at 715. As shown in
Each service path is specified in terms of its SPI value. When multiple service paths are specified for a service chain, the path storage 808 stores for each service chain a set of selection metrics 820 for selecting one SPI from the available SPIs. Different embodiments use different selection metrics. For instance, some embodiments use a selection metric that costs a service path based on the number of hosts on which the service nodes of the service path execute. In other embodiments, these selection metrics are weight values that allow the pre-processor to select SPIs for a service chain in a load balanced manner that is dictated by these weight values. For instance, in some embodiments, these weight values are generated by a central control plane based on the load on each of the service nodes in the service path and/or based on other costs (such as number of hosts traversed by the service path, etc.).
In some of these embodiments, the pre-processor maintains a record of previous selections that it has made for a particular service chain, and selects subsequent service paths based on these previous selections. For example, for four service paths, the weight values might be 1, 2, 2, 1, which specify that on six successive SPI selections for a service chain, the first SPI should be selected once, the second and third SPIs should then be selected twice each, and the fourth SPI should be selected one. The next SPI selection for this service chain will then select the first SPI, as the selection mechanism is round robin.
In other embodiments, the weight values are associated with a numerical range (e.g., a range of hash values) and a number is randomly or deterministically generated for each data message flow to map the data message flow to a numerical range and thereby to its associated SPI. In still other embodiments, the hosts LCP selects one service path for each service chain identifier from the pool of available service paths, and hence stores just one SPI for each SCI in the path table 808. The LCP in these embodiments selects the service path for each service chain based on costs (such as the number of hosts traversed by each service path and/or the load on the service nodes of the service paths).
After identifying a service path for the identified service chain, the process 700 next identifies (at 730) the network address for the first hop of the selected service path. In some embodiments, the MAC address for this hop is stored in the same record as the selected path's SPI. Hence, in these embodiments, this MAC address is retrieved from the path selection storage 808 with the selected SPI. In other embodiments, the pre-processor retrieves the first hop's MAC address from an exact match forwarding table 810 that stores next hop network addresses for associated pairs of SPI/SI values, as shown in
At 735, the process 700 specifies the SMD attributes for the data message, and associates these attributes with the data message. As mentioned above, the SMD attributes include in some embodiments the SCI, the SPI, SI and direction values. The service directions for service paths are stored with the SPI values in the path table 808 as the directions through the service chains are dependent on the service paths. Also, as mentioned below, a service chain in some embodiments has to be performed in a forward direction for data messages from a first GVM to a second GVM, and then in the reverse direction for data messages from the second GVM to the first GVM. For such service chains, the pre-processor 610 selects both the service path for the forward direction and the service path for the reverse direction when it processes the first data message in the flow from the first GVM to the second GVM.
After the SI pre-processor completes its operation, the STL caller 624 in the same datapath calls (at 740) the STL port proxy 620 to relay the SMD attributes and first hop's network address that the pre-processor identified, so that the port proxy can forward the SMD attributes through the service plane to the first hop. The operation of the port proxy 620 as well as other modules in the service insertion layers and service transport layers will be described by reference to
In these figures, each GVM is a compute machine of a tenant in a multi-tenant datacenter, and connects to the software switch through a switch port that is associated with a guest VNI (GVNI) of the tenant. Also, in these figures, each SVM is a service machine for processing the GVM message traffic, and connects to the software switch through a switch port that is associated with a service VNI (SVNI) of the tenant. As mentioned above and further described below, some embodiments use the GVNI for performing the guest logical forwarding operations (i.e., for establishing a guest logical forwarding element, e.g., a logical switch or router, or a guest logical network) for the tenant, while using the SVNI for performing the service logical forwarding operations for the tenant (i.e., for establishing a service logical forwarding element, e.g., a logical switch or router, or a service logical network).
Both of these logical network identifiers (i.e., the GVNI and SVNI) are generated for the tenant by the management or control plane in some embodiments. The management or control plane of some embodiments generates different GVNIs and SVNIs for different tenants such that no two tenants have the same GVNI or SVNI. In some embodiments, each SVM is dedicated to one tenant, while in other embodiments, an SVM can be used by multiple tenants. In the multi-tenant situation, each SVM can connect to different ports of different service planes (e.g., different logical switches) for different tenants.
As shown in
In this process 1000, the port proxy also adds (at 1010) the original source and destination MAC addresses of the data message to the set of attributes for the data message that should be processed by other service transport layer modules (e.g., the vswitch, other STL modules, the encap processor, etc.) on the same host computer. The reformatted data message 902 and the augmented attributed set 904 are depicted in
After reformatting the data message and augmenting its attribute set, the port proxy 620 passes (at 1015) the formatted data message along with its stored attribute set along its egress path where it reaches the software switch 120. Based on the destination MAC address (e.g, the first hop MAC address) of the formatted data message, the software switch determines (at 1020) whether the next hop's port is local. This is the case for the example illustrated in
This STL module 626 then re-formats (at 1030) the data message by replacing the GVM's service plane MAC address and the first hop MAC address (i.e., the MAC address of SVM 106) with the original source and destination MAC addresses of the data message, which it retrieves from the augmented attribute set 904. In retrieving the original SMAC and DMAC addresses, the STL module 626 modifies the data message's attribute set. The reformatted data message 906 and the modified attributed set 908 are depicted in
In these embodiments, the service proxy copies the data messages and performs the remaining operations 1110-1125 with respect to this copy, while passing the original data message to the next service hop or back to the source GVM. To forward the original data message to the next service hop or back to the GVM, the service proxy has to perform a next-hop lookup based on the SPI/SI values and then provide the next-hop address (e.g., the next service hop's address or the service plane MAC of the source GVM) to the STL module to forward. These look up and forwarding operations are similar to those described below by reference to
Next, at 1110, the service proxy sets a liveness attribute in the stored SMD attribute set of the data message (which, in some embodiments, might be the data message copy at this point). This liveness attribute is a value that directs the service node to provide a responsive liveness value (the same value or related value) with the data message once it has processed the data message. With this liveness attribute, the service proxy also provides a sequence number, which the service node has to return, or increment and then return, with the responsive liveness value, as described above.
At 1115, the service proxy formats the data message, if necessary, to put it in a form that can be processed by the service node. For instance, when the service node does not know the current next hop MAC that is set as the destination MAC of the data message, the service proxy changes the destination MAC of the message to a destination MAC associated with the service node.
After formatting the data message to sanitize it for forwarding to the service node, the service proxy 614 encapsulates (at 1120) the data message with one of three encapsulation headers that it can be configured to use, and passes (at 1125) the encapsulated message along the service node's ingress path so that it can be forwarded to the service node.
In addition to the three different types of encapsulating headers 1205, 1210 and 1215,
The poll mode driver passes a data message to the DPDK driver 1206, which then passes it to the message processing module in the user space when a flow is initially received. The interrupt-based driver 1204, on the other hand, provides the data message to the message processing module 1212 either in the kernel or in the user space. The message processing module of the SVM then decapsulates the encapsulated data message and performs the service operation of the SVM. In some embodiments, different SVMs perform different service operations based on the SCI, SI and service direction values that they receive with a data message.
In some embodiments, each service rule 1420 in the service rule table 145 has a rule identifier that is defined in terms of data message attributes (e.g., five tuple attributes). The SVM compares (at 1320) a rule's identifier to the attributes of the data message to identify a matching rule. Upon identifying one or more matching rules, the SVM in some embodiments performs (at 1325) an action specified by the highest priority matching rule. For instance, a firewall SVM might specify that the data message should be allowed to pass, should be dropped and/or should be redirected.
Once the SVM has completed its service operation, the SVM encapsulates (at 1330) the data message with an encapsulating header, assuming that the service operation does not result in the dropping of the data message. This encapsulating header has the same format (e.g., is an NSH header, GRE header, or QinQ header) as the data message that the SVM received. In this encapsulating header, the SVM in some embodiments sets (1) a liveness value to respond to the service proxy's liveness value and (2) the appropriate sequence number (e.g., unadjusted or incremented sequence number) for the liveness value.
In some embodiments, some service nodes are configured to decrement the SI values that they receive, while other service nodes are not configured to decrement the SI values. If the service node is configured to decrement the SI value, it decrements the SI value before inserting the decremented SI value in the encapsulating header at 1330. The SVM in some embodiments also sets the SMD attributes (SCI, SI and service direction) in the encapsulating header, while in other embodiments, the service proxy in the egress path retrieves these values from an earlier record that the service proxy created before passing the data message to the SVM.
In some embodiments, the SVM can also set flow programming attribute(s) in the encapsulating header to direct the service proxy to modify the service processing of the data message's flow. This flow programming will be further described below. After encapsulating the data message, the SVM forwards the data message along its egress path.
Next, 1610, the process resets the liveness timer (e.g., a timer that expires every 0.25 seconds) that it maintains to account for the liveness value that it has received from the service node, which signifies that this node is still operational. With this liveness value, the service proxy receives from the service node a sequence number, which the process validates to ensure that it is the next liveness value that needs to be received.
At 1615, the process determines whether the SVM specified any flow programming attribute(s), which require the service proxy to direct the SI post processor 612 for the source GVM to perform flow programming by sending to the post processor 612 in-band data messages. In some embodiments, the service proxy sends an in-band flow programming control signal with another data message that it generates to send back to the source GVM, where it will be intercepted by its post processor 612.
When the source GVM receives the data message with the flow programming control signal, its post processor can uniquely identify the data message flow to which it applies by using a flow identifier that is unique to this flow. As further described below, this flow identifier is derived partially based on a unique identifier of the source GVM. The unique flow identifier also allows other service plane modules, such as the service nodes, service proxies and STL modules, to uniquely identify each data message flow. This unique flow identifier in some embodiments is part of the SMD attributes that are passed between the service hops of a service path and passed back to the source GVM.
In some embodiments, however, the service proxy sends the in-band flow programming control signal with the current data message that it is processing. In some of these embodiments, the service proxy does this only when its associated service node is the last hop service node of the service path, while in other embodiments it does this even when its service node is not the last hop service node. When its service node is not the last hop service node of the service path, the service proxy embeds the flow programming in the SMD attributes of the data message, which in some embodiments eventually get forwarded to the source GVM's SI post processor as part of the data message encapsulation header when the last hop service is performed. Even in this situation, the service proxy of the last hop in other embodiments sends the flow programming signal as a separate message.
The flow programming signals will be further described below by reference to
At 1620, the process 1600 determines whether its service node specified that the data message should be dropped. If so, the process drops the data message and then ends. Otherwise, assuming the data message should not be dropped and should continue along its service path, the service proxy in some embodiments decrements (at 1625) the SI value in case the service node has not decremented the SI value, and then uses (at 1630) this decremented value along with the SPI value in the data message's stored attribute set to identify an exact match forwarding rule that identifies a next hop network address. When the proxy's service node is on multiple service paths, the proxy's forwarding rule storage stores multiple exact match forwarding rules that can specify different next hop network addresses for different SPI/SI values.
When the decremented SI value is zero, the service proxy in some embodiments that matches the decremented SI value and the embedded SPI value with a rule that directs the service proxy to identify the next hop as the service plane MAC address of the source GVM. This rule in some embodiments does not provide a MAC address, but rather refers to the service plane MAC address that is part of the SMD attribute set stored for the data message. In some embodiments, this instructions for returning the data message to the service plane MAC address of the source GVM when the SI value is zero is not specified by a forwarding entry of a forwarding table, but rather is hard coded into the logic of the service proxy.
At 1630, the service proxy stores the next hop network address (e.g., MAC address) in the attribute set that is stored for the data message.
Once the service proxy returns the data message to the service node's egress path, the STL module 626 receives this data message and commences the process 1000 of
At 1010, the STL module also adds the original source and destination MAC addresses of the data message to the set of attributes for the data message that should be processed by other service transport layer modules (e.g., the vswitch, the encap processor, etc.) on the same host computer. The reformatted data message 1508 and the augmented attributed set 1510 are depicted in
Based on the destination MAC address (i.e., the next hop MAC address) of the formatted data message, the software switch determines (at 1020) that the next hop's port is not local. Hence, the software switch provides (at 1035) the data message to the uplink port 1550 that connects to a VTEP1 that communicates through an overlay network tunnel with a VTEP2 on host 114, as illustrated in the example of
In some embodiments, the overlay header is a Geneve header that stores the SMD and STL attributes in one or more of its TLVs. As mentioned above, the SMD attributes in some embodiments include the SCI value, the SPI value, the SI value, and the service direction. Also, in some embodiments, the STL attributes includes the original L2 source MAC address and the original L2 destination MAC address.
When the encapsulated data message is received at the next hop's host computer 114, the data message is captured by the STL encap processor 628 of (e.g., defined as a hook for) a downlink port 1552 that connects to the VTEP connecting through the overlay network tunnel to the prior hop's VTEP.
As shown, this encap processor removes (at 1705) the encapsulation header from the data message, and stores (at 1705) the STL and SMD attributes as the associated set of attributes of the data message. It then passes (at 1710) the decapsulated message to the downlink port, which then passes it to the software switch to forward (at 1715) to its port that is connected to the next hop SVM (i.e., that is associated with the destination MAC address). This port then passes the data message 1508 and the attribute set 1510 to the ingress path of the next hop SVM, as shown in the example of
The STL module 626 on this ingress path then re-formats (at 1720) the data message by replacing the previous and current hop service plane MAC address (i.e., the hop1mac and hop2mac) with the original source and destination MAC addresses of the data message, which it retrieves from the data message attribute set. In retrieving the original SMAC and DMAC addresses, the STL module 626 modifies the data message's attribute set. The reformatted data message 1530 and the modified attributed set 1532 are depicted in
The operation of this service proxy is as described above by reference to
Once the SVM 108 performs its service operation on the data message (e.g., per the process 1300 of
The software switch then determines that the next hop is associated with its uplink port 1552, and hence passes the data message to this port. As shown in
The STL encap processor 628 in the ingress path of port 1554 then removes the encapsulation header from the data message and stores the STL and SMD attributes as the associated set of attributes of the data message. It then passes the decapsulated message to the port 1554, which then passes it to the software switch 124 to forward to its port connected to the next hop SVM 110 (i.e., to its port associated with the service plane DMAC). This port then passes the data message and attribute set to the ingress path of this SVM, as shown in
The STL module 626 in this ingress path replaces the previous and current hop service plane MAC address (i.e., the hop2mac and hop3mac) with the original source and destination MAC addresses of the data message, which it retrieves from the data message attribute set. The STL module 626 also modifies the data message's attribute set by removing the original SMAC and DMAC addresses, and then passes the re-formatted data message with its accompanying SMD attributes along the SVM's ingress path for this hop's ingress service proxy 614 to process. This service proxy passes to the SVM 110 an encapsulated data message with an encapsulating header supported by the SVM 110 and storing the SCI, SI, service direction and liveness values.
Once the SVM 110 performs its service operation on this data message (e.g., per the process 1300 of
The STL module next replaces the original SMAC and DMAC in the data message with the third hop MAC address and the source GVM's service plane MAC, stores the original SMAC and DMAC in the stored attribute set of the data message, and then passes the data message to its software switch 124. The software switch then determines that the next hop is associated with its port 1554, and hence passes the data message to this port. As shown in
The STL encap processor 628 in the ingress path of port 1550 then removes the encapsulation header from the data message and stores the STL and SMD attributes as the associated set of attributes of the data message. It then passes the decapsulated message to the port 1550, which then passes it to the software switch 120 to forward to its port connected to the port proxy 620. This port then passes the data message and attribute set to the port proxy 620, as shown in
The port proxy 620 then replaces the previous and current hop service plane MAC address (i.e., the hop3mac and spmac) with the original source and destination MAC addresses of the data message, which it retrieves from the data message attribute set. The port proxy 620 also modifies the data message's attribute set to remove the original SMAC and DMAC, and then passes this re-formatted data message with its accompanying SMD attributes back to the STL caller 624 that called it in the first place. In some embodiments, the port proxy uses a connection record that it created when the STL caller originally called it, to identify the STL caller to call back. In other embodiments, the port proxy uses a mapping table that maps each service plane MAC with a GVM's STL caller. The mapping table in some embodiments has records that associate service plane MACs and service directions with guest forwarding plane port identifiers associated with the GVMs.
Once called, the STL caller passes the data message along the egress path of GVM 102, where it will next be forwarded to the SI post-processor 612.
When the SI post processor 612 determines that it needs to process the data message, the process determines (at 2010) whether the SMD metadata associated with the data message specifies a flow programming tag that requires the post processor to perform a flow programming operation. In some embodiments, such a flow programming tag would be specified in the data message's SMD attributes by a service node to change the service path processing at the source GVM, or by a service proxy for the same reason when it detects failure of its service node. When the flow programming tag does not specify any flow programming, the process transitions to 2020, which will be described below.
Otherwise, when the flow programming tag specifies a flow programming operation, the process 2000 performs this operation, and then transitions to 2020. The flow programming operation entails in some embodiments modifying the connection record in the connection tracking storage 804 to specify the desired operation and/or SMD attributes (e.g., allow, drop, etc.) for the data message's flow. The post processor's writing to the connection tracker 804 is depicted in
In some embodiments, the flow programming tag can specify the following operations (1) NONE when no action is required (which causes no flow programming operation to be performed), (2) DROP when no further data messages of this flow should be forwarded along the service chain and instead should be dropped at the source GVM, (3) ACCEPT when no further data messages of this flow should be forwarded along the service chain and instead the flow should be accepted at the source GVM. In some embodiments, the flow programming tag can also specify DROP MESSAGE. The DROP MESSAGE is used when the service node needs to communicate with the proxy (e.g. to respond to a ping request) and wants the user data message (if any) to be dropped, even though no flow programming at the source is desired.
In some embodiments, an additional action is available for the service proxies to internally communicate failure of their SVMs. This action would direct the SI post processor in some embodiments to select another service path (e.g., another SPI) for the data message's flow. This action in some embodiments is carried in-band with a user data message by setting an appropriate metadata field in some embodiments. For instance, as further described below, the service proxies communicate with the post processor of the source GVM through OAM (Operation, Administration, and Maintenance) metadata of the NSH attributes through in-band data message traffic over the data plane. Given that by design flow programming actions are affected by signaling delays and are subject to loss, an SVM or service proxy might still see data messages belonging to a flow that was expected to be dropped, accepted or re-directed at the source for some time after communicating the flow programming action to the proxy. In this case, the service plane should continue set action to drop, allow or redirect at the source.
The process 2000 transitions to 2020 after completing the flow programming operation. It also transitions to 2020 when it determines (at 2005) that no SI post processing needs to be performed on the data message or determines that no flow programming needs to be performed for this data message. At 2020, the process 2000 lets the data message through the egress path of GVM 102, and then ends.
The examples described above by reference to
However, the service insertion and service transport operations for the ingress side processing are similar to the egress side processing of data messages to and from a particular GVM. In some cases, this GVM exchanges data messages with another GVM. As described above by reference to
The header formats used in some embodiments will now be described by reference to
As shown, all the fields of the first 8 bytes of the NSH header are used in compliance with RFC 8300. This header includes in some embodiments a fixed length metadata (MD) content header 2110. It also includes in some embodiments (1) a MD type, which is set to 1, (2) a next protocol value, which is 3 to signify Ethernet communications, and (3) a length value, which is 6 because the MD content header 2110 has a fixed length. Also, in some embodiments, the SPI and SI fields 2122 and 2124 are filled in with the service path identifier for the selected path and the current service index value, which is the initial SI value (i.e., the initial number of service hops) when the pre-processor 610 of the source GVM defines it.
In some embodiments, the service insertion modules do not store or cache metadata except in the NSH header carried along with data messages. In this model, service nodes preserve the metadata field that they do not intend to change. In some embodiments, certain metadata fields are used as a communication mechanism for data plane mediated signaling between the service proxies/nodes and the source GVM's service modules. In some embodiments, the data message metadata is encoded over the wire in NSH fixed length context header 2110. In some embodiments, this fixed-sized header provides sixteen bytes of information in total. In some embodiments, each service insertion deployment is free to define its own MD content format.
The source node identifier (ID) field 2206 unequivocally, for the service plane, identifies a data compute node (e.g., a GVM) that is the source or sink of the data message. In some embodiments, the source node ID includes the service plane MAC address of this source data compute node (DCN) for which the data message was inserted into the service plane. The MD content header also includes a sequence number 2208 that is an opaque 6-bit value that identifies the data message for the purpose of liveness detection. This value is typically zero unless a service proxy fills before forwarding the data message to its service node as part of its liveness detection.
The MD content header also includes a tenant ID 2212 that identifies a tenant uniquely for a multi-tenant datacenter. The tenant ID in some embodiments is a VNI associated with the tenant. The MD content header 2200 further includes flow ID 2215 and flow ID validity bit 2222. In some embodiments, the flow ID validity bit is set to 1 when the rest of the flow ID (also called flow tag) is present. The flow ID 2215 is a unique identifier per flow and source DCN (i.e., per flow and source node ID 2206). In some embodiments, the flow ID is set by the source DCN's classifier (e.g., the SI pre-processor 610 that performs the classification operation).
In some embodiments, the flow ID may be discarded when the data message traverses a service which is not in native mode (i.e., the service is not aware of the service plane). In this case, the flow ID is discarded when there are not enough bits to carry the flow ID in compatibility mode headers, which are described below. The flow ID may also be discarded when a native service (i.e., a service plane aware service node) modifies the data message in a way that makes the flow ID meaningless, for example when a service encrypts traffic from multiple flows into a single IPsec tunnel. In this case preserving the flow tag of the inner data message would be meaningless. In some embodiments, the service node sets the A bit to zero in this case.
The MD content header 2200 also includes an action field 2230 that is used for flow programming by the service proxies. In some embodiments, the action specifies the action that the source DCN's post-processor 612 should perform on a flow. For flow programming, the action fields have to be non-zero in some embodiments. In addition, for flow programming, the F bits 2202 are also set to 10 and the P bit 2204 is set to 0 at the proxy and ignored by the classifier, and the flow validity bit 2222 and flow tag 2215 have to be valid.
The following are one exemplary set of values for the action field 2230, but one of ordinary skill will realize that other values are specified in other embodiments. A value of 0 for the action bit specifies that no flow-programming action is specified. A value of 1 indicates that all messages of the data message's flow should be dropped at the source, and no further data message of this flow should be forwarded to the service plane. Instead, data messages should be dropped at the source after classification.
A value of 2 in the action field specifies that the data message should be accepted at the source, and that no further data messages of the same flow should be forwarded to the service function in some embodiments. Instead, the service function should be skipped and the next service in the chain invoked directly. A value of 3 in the action field specifies that only this data message should be dropped and does not indicate an action that should be taken on other data messages of the same flow. In some embodiments, this action is used when the service node communicates with the service proxy (e.g. to respond to a ping request) and wants a data message to be dropped, even though no flow programming should happen.
The MD content header 2200 also includes a direction field 2214 that specifies the direction of the data message from the source DCN to network perspective (e.g., from the DCN to the network is the egress direction and from the network to the DCN is the ingress direction). A value of 0 in the direction field indicates no direction or unknown direction when the direction is not unknown. In some embodiments, a value of 1 indicates that the data message is moving in the ingress direction (i.e., the data message is being processed for source DCN that is the destination of the data message), for example, the data message is on its way from a VTEP to its corresponding DCN. A value of 2 in some embodiments indicates an egress direction (e.g., the data message is being processed for source DCN that is the source of the data message).
In some embodiments, a value of 3 indicates the data message is merely in transit and applies to both ingress and egress. When used to define a rule, this indicates that the rule should match data messages in one direction or in any direction in some embodiments. From the service perspective, a value of 3 in the direction field indicates that this traffic was forwarded to the service plane by a transit device that is neither sourcing nor sinking this traffic in some embodiments. In some embodiments, the transit indication is used for traffic that is transiting through a router.
The MD content header 2200 further includes a service chain ID 2216 that specifies the service chain along which the data message should flow. Some embodiments do not embed the SCI in the NSH header, and instead just store the SPI value. However, other embodiments store the SCI in the filed 2216 because many SPIs can correspond to the same service chain and SPIs are also not persistent. In other words, some embodiments embed the service chain ID because the SCI provides a more stable identifier for the service nodes to use to identifying service rule that match the data messages that they process.
In some embodiments, other metadata content formats are used internally by the service plane without being exposed to service nodes, in order to perform data plane signaling between service proxies and service post-processor of the source DCN. In some of these embodiments, when the other metadata content formats are used, the OAM bit (the 0 bit 2170 in
In some embodiments, service plane unaware service nodes receive only a subset of the metadata, dependent on the type of non-NSH header used by the service proxies to communicate with the service nodes. As mentioned above, the service nodes in some embodiments can receive service metadata in GRE headers or in QinQ headers, when the service nodes cannot process NSH headers. The GRE and QinQ headers are referred to below as compatibility mode headers as they are headers that some existing service nodes support. Such compatibility mode encapsulation headers are needed in some embodiments in order to distinguish data message flows that are subject to different service processing and to isolate flows with conflicting L3 addresses (in case a single service node performs services on data messages of multiple networks, such as multiple tenant networks).
In some embodiments, a service node in a GRE compatibility mode connects to its service proxy through two VNICs and is configured in bump-in-the-wire mode. Also, in some embodiments, the VNICs are vmxnet3 devices, their MAC addresses do not change, and the MTU size that is used for them is set to a fixed size (e.g., 2048 bytes). One VNIC of the service node is defined as the unprotected side for receiving egress side traffic and supplying ingress side traffic of the source DCN, while the other VNIC is defined as the protected side for receiving ingress side traffic and supplying egress side traffic of the source DCN. In some embodiments, this information is communicated to a service manager or service node through OVF (Open Virtual Format) parameters, where OVF is a file format that supports exchange of virtual appliances across products and platforms.
Even though two VNICs are present to support bump-in-the-wire mode, some embodiments use only one service proxy instance per pair of compatibility-mode VNICs and use only one endpoint on the service plane to refer to the pair of interfaces.
Each protected virtual tunnel endpoint has a corresponding unprotected virtual tunnel endpoint. Each virtual tunnel endpoint is associated with a virtual IP address, a virtual MAC address and GRE parameters. The service proxy encapsulates data messages with GRE headers to traverse between corresponding pairs of endpoints through the service node, with this node operating in bump-in-wire mode that does not modify the GRE headers. As further described below, the service proxy embeds service metadata in the GRE headers to provide the service node with service metadata that it needs to process the data messages. Also, different tunnel endpoint pairs are used for different flows in some embodiments.
In some embodiments, the service insertion platform supports GRE encapsulation as defined in RFC 2784 with the key extension defined in RFC 2890. In some embodiments, GRE tunneling uses IPv4 addresses and the GRE protocol type is set to Transparent Ethernet Bridging as per RFC 1701. In the GRE compatibility mode, the service insertion layer (e.g., the service proxy) generates a tuple (e.g., source IP, destination IP, GRE key) per flow. In some embodiments, this process is deterministic and is based on the contents of the SMD header, which may then be stripped and replaced with the IP and GRE stack. In some embodiments, the IP addresses generated by this process are virtual and are not configured on any network entity other than the service proxy and its associated SVM, and as a result their scope is limited to the local link between a service proxy and its service node.
The IP address pair and the GRE key are generated in order to carry metadata along with the data message even when the service node does not support GRE. Both the service node and the service proxy in some embodiments consume that metadata. The service node, moreover, is expected to preserve the outer headers as-is without modifications in some embodiments. In some embodiments, each flow is consistently encapsulated in the same GRE tunnel and there can be no IP address conflicts inside a tunnel. Also, data messages differing only by their direction (ingress vs. egress) are encapsulated with the same GRE key with swapped source and destination IPs and traversing through the GRE tunnel endpoints in the proper (protected to unprotected, or unprotected to protected) direction.
In some embodiments, the IP source/destination addresses, and GRE key can be inspected by the service node as required to perform the proper data message processing.
In these figures, all fields are in network byte order. Path IDs are generated alongside service paths in some embodiments and have a global per-service value. As shown in
In some embodiments, the flow tag information along a service chain is discarded when entering the first GRE compatibility mode service and is not restored downstream. This can prevent subsequent services from being able to declare flow actions. As such, flow programming is not provided to service nodes in GRE compatibility mode of some embodiments. Liveness detection, moreover, is supported in some embodiments by passing BFD (bidirectional forwarding detection) messages between the trusted and untrusted interfaces. In some embodiments, these data messages are injected from the trusted and untrusted sides by the service proxy. The service node can recognize this traffic because it is not encapsulated in GRE. In some embodiments, the service node is expected to forward this traffic (and indeed any non-GRE encapsulated traffic) unmodified by bridging it to the other side of the virtual wire. Also, in some embodiments, the data messages can be hard-coded if a real instance of BFD is not available.
Due to space constrains in some embodiments, certain header fields are encoded in a summarized version. In some embodiments, the service chain tag, SPI and SI are summarized in a single 4-bit field. Each compatibility mode service node can therefore be present on at most 16 service chain hops in some embodiments. Each time a service is present inside a service chain, this consumes one service path ID. If the service is present on multiple chains, multiple service path IDs are consumed. In addition, each time a service is present on two directions of a service chain, two service path IDs are consumed.
In some embodiments, locally-generated traffic is supported in compatibility mode as long as a related outer header stack (up to and including GRE) is used. In some embodiments, no modification to the outer header stack is allowed, except (1) optionally replacing the outer Ethernet destination address with broadcast, (2) updating the IP total size field and IP checksum, and (3) the GRE checksum is ignored but the GRE key must be present.
For sake of efficiency, some embodiments combine these two headers into a single Geneve header 2805 of
As shown, in both implementations of
In some embodiments, a service manager object 2902 can be created before or after the creation of a service object 2904. An administrator or a service management system can invoke service manager APIs to create a service manager. A service manager 2902 can be associated with a service at any point of time. In some embodiments, the service manager 2902 includes service manager information, such as the vendor name, vendor identifier, restUrl (for callbacks) and authentication/certificate information.
As mentioned above, the service plane does not require the presence or use of a service manager as service nodes can operate in zero-awareness mode (i.e., have zero awareness of the service plane). In some embodiments, zero-awareness mode only allows basic operations (e.g., redirecting traffic towards the service's SVMs). In some such embodiments, no integration is provided to distribute object information (such as service chain information, service profiles, etc.) to the service manager servers. Instead, these servers can poll the network manager for objects of interest.
A service object 2904 represents a type of service that is provided by a service node. The service object has a transport type attribute, which specifies its mechanism (e.g., NSH, GRE, QinQ, etc.) for receiving service metadata. Each service object also has a state attribute (which can be enabled or disabled) as returned by service manager, and a reference to a service manager that may be used for exposing REST API endpoints to communicate events and perform API calls. It also includes a reference to an OVA/OVF attribute used to deploy instances of the service.
Vendor template objects 2907 include one or more service profile objects 2906. In some embodiments, service managers can register vendor templates, and the service profiles can be defined on a per service basis and based on a vendor template with potentially specialized parameters. A service chain can be defined by reference to one or more service profiles. In some embodiments, service profiles are not assigned tags and are not identified explicitly on the wire. In order to determine which function to apply to traffic, service nodes perform a look up (e.g., based on service chain identifier, service index and the service direction, as mentioned above) in order to identify the applicable service profile. The mapping for this lookup is provided by the management plane to service managers whenever a service chain is created of modified.
A service profile object 2906 in some embodiments includes (1) a vendor template attribute to identify its associated vendor template, (2) one or more custom attributes when the template exposes configurable values through the service profile, and (3) an action attribute, such as a forward action, or a copy-and-redirect, which respectively direct the service proxies to either forward the received data messages to their service nodes, or to forward a copy of the received data messages to their service nodes while forwarding the received data message to the next service hop or back to the original source GVM when their service node is the last hop.
The service attachment object 2908 represents the service plane (i.e., is a representation of the service plane of a perspective of a user, such as tenant's network administrator in a multi-tenant datacenter, or the network administrator in a private datacenter). This service attachment object is an abstraction that support any number of different implementations of the service plane (e.g., logical L2 overlay, logical L3 overlay, logical network overlay etc.). In some embodiments, each endpoint (on an SIR or a GVM) that communicates over the service plane specifies a service attachment. The service attachment is a communication domain. As such, services or GVMs outside a service attachment may not be able to communicate with one another.
In some embodiments, service attachments can be used to create multiple service planes with hard isolation between them. A service attachment has the following attributes (1) logical identifier (e.g., SVNI for a logical switch) that identifies a logical network or logical forwarding element that carries traffic for the service attachment, (2) a type of service attachment (e.g., L2 attachment, L3 attachment, etc.), and (3) an applied_To identifier that specifies a scope of the service attachment (e.g., Transport node 0 and Transport node 1 for north-south operations and a cluster or set of hosts for East-West operations). In some embodiments, the control plane (e.g., a central control plane) converts the service attachment representation that it receives from the management plane to a particular LFE or logical network deployment based on parameters specified by a network administrator (e.g., a datacenter administrator of a private or public cloud, or network virtualization provider in a public cloud).
A service instance object 2910 represents an actual deployed instance for a service. Hence, each such object is associated with one service object 2904 through a service deployment object 2913 that specifies the relationship between the service object 2904 and the service instance object 2910. The deployed service instance can be a standalone service node (e.g., standalone SVM) or it can be a high availability (HA) service node cluster. In some embodiments, the service deployment object 2913 describes the service instance type, e.g., standalone or HA. As described below, the service deployment object's API can be used in some embodiments to deploy several service instances for a service.
The service instance runtime (SIR) object 2912 represents an actual runtime service node that operates in a standalone mode, or an actual runtime service node of an HA cluster. The service instance object in some embodiments includes the following attributes (1) a deployment mode attribute that specifies whether the service instance is operating in a standalone mode, an active/standby mode, or an active/active model, (2) a state attribute that specifies whether the instance is enabled or disabled, and (3) a deployed_to attribute that in the case of north-south operations includes a reference to a service attachment identifier.
In some embodiments, SVMs provisioning is initiated manually. To this end, the management plane provides, in some embodiments, APIs for (1) creating a service instance of an existing service, (2) deleting a service instance, (3) growing a service instance that is already configured as a high availability cluster by adding additional SIRs, and (4) shrinking a service instance by removing one of its SIRs. When creating a service instance of an existing service, the service instance may be created in some embodiments on the basis of a template contained in the service. The caller can pick between a stand-alone instance or an HA cluster, in which case all the VMs in the HA cluster are provisioned. Again, in some embodiments, the API for the service instance deployment allows multiple service instances (e.g., for an HA cluster) to be deployed through just one API call.
In some embodiments, an API that creates one or more SVMs specifies one or more logical locations (e.g. clusters, host, resource pool) in which the SVMs should be placed. In some embodiments, the management plane tries to place SVMs belonging to the same service instance on different hosts whenever possible. Anti-affinity rules may also be configured as appropriate to maintain the distribution of SVMs across migration events (such as VMotion events supported by Dynamic Resource Scheduler of VMware, Inc.). Similarly, the management plane may configure affinity rules with specific hosts (or groups of hosts) when available or the user provisioning the service instance may explicitly pick a host or a cluster.
As mentioned above, a service instance runtime object 2912 represents an actual SVM running on a host to implement a service. An SIR is part of a service instance. Each SIR can have one or more traffic interfaces completely dedicated to service plane traffic. In some embodiments, at least one service proxy instance runs per SIR to handle data plane signaling and data message format conversion for the SIR as needed. When a service instance is deployed, the SIRs are created for every SVM associated with the service instance in some embodiments. The network manager also creates an instance endpoint for every service instance in an east-west service insertion. Each SIR object 2912 has the following attributes in some embodiments (1) a state attribute which is active for SVMs that can process traffic and inactive for all others, regardless of reason, and (2) a runtime state that specifies whether the data plane liveness detection detects that the SIR is up or down.
The instance runtime interface 2916 is the per-endpoint version of the service instance endpoint 2914. In some embodiments, the instance runtime interface 2916 is used to identify an interface for an SIR or GVM that can be the source or sink service plane traffic. In East-West service insertion, the lifecycle of an instance runtime interface in some embodiments is linked to the lifecycle of the service instance runtime. In some embodiments, no user action is required to configure an instance runtime interface.
In some embodiments, the instance runtime interface 2916 has the following attributes: an endpoint identifier, a type, a reference to a service attachment, and a location. The endpoint identifier is a data plane identifier for the SIR VNIC. The endpoint identifier is generated when the SIR or GVM is registered with the service transport layer, and may be a MAC address or part of a MAC address. The type attribute can be shared or dedicated. SIR VNICs are dedicated, meaning that only service plane traffic is able to reach them, while GVM VNICs are shared, meaning they will receive and transmit both service plane and regular traffic. The service-attachment reference is a reference to the service attachment that implements the service plane used to transmit and receive service plane traffic. This reference in some embodiments is to the SVNI of the service plane. The location attribute in some embodiments specifies the location of the instance runtime interface, which is the UUID of the host on which the instance runtime interface is currently located.
In some embodiments, a user defines a service chain object 2918 in terms of an ordered list of service profiles 2906. In some embodiments, each service chain conceptually provides separate paths for forward and reverse traffic directions, but if only one direction is provided at creation time, the other one is generated automatically by reversing service profile order. Either direction of the service chain (and even both directions) can be empty, meaning no services will process traffic in that direction. In some embodiments, the data plane will perform a lookup even for an empty service chain.
Service chains are abstract concepts. They do not point to a specific set of service nodes. Rather, the network controllers that are part of the service plane platform automatically generate service paths that point to sequences of service nodes for the service chain and direct messages/flows along the generated service paths. In some embodiments, a service chain is identified in the management plane or control plane by its UUID, a unique identifier of the service chain. Service nodes are provided with the meaning of service chain IDs through management plane APIs received through their service managers. One example of this was described above by reference to
A service chain tag in some embodiments may be used to identify a service chain in the dataplane because UUIDs are too long to be carried in encapsulating headers. A service chain ID in some embodiments is an unsigned integer like rule ID. Each data message redirected to a service carries the service chain tag for the service chain it is traversing. The management plane advertises UUID to service chain tag mappings when a service chain is created or modified. Service chain tags have a 1 to 1 mapping with service chain UUIDs, whereas a single service chain can have 0 to many service path indexes.
In addition to a service chain ID, a service chain in some embodiments has the following attributes: (1) references to all computed service paths, (2) failure policies, and (3) references to service profiles. References to computed service paths were described above. The failure policy is applied when a service path selected for a service chain cannot be traversed. In some embodiments, the failure policies may be PASS (forward traffic) and FAIL (drop traffic). The references to service profiles of the service chain may include an egress list of service profiles that egress traffic (e.g., data messages traveling from a GVM to a switch) must traverse, and an ingress list of service profiles that ingress traffic (e.g., data messages traveling from the switch to a GVM) must traverse. In some embodiments, the ingress list is initialized by default as the reverse of the egress list.
Different techniques can be used in some embodiments to define the service paths for the service chain. For instance, in some embodiments, a service chain can have an associated load balancing strategy, which can be one of the following strategies. The load balancing strategy is responsible for load balancing traffic across different service paths of a service chain. According to an ANY strategy, the service framework is free to redirect the traffic to any service path regardless of any load balancing consideration or flow pinning. Another strategy is a LOCAL strategy, which specifies that local service instances (e.g., SVMs executing on the same host computer as the source GVM) are to be preferred over remote service instances (e.g., SVMs executing on other host computers or external service appliances).
Some embodiments generate scores for service paths based on how many SIRs are local and the highest score is selected regardless of load. Another strategy is the cluster strategy, which specifies that service instances implemented by VMs that are co-located on the same host are preferred, whether that host is the local one or a different one. A ROUND ROBIN strategy directs that all active service paths are hit with equal probability or based on probabilities that are specified by a set of weight values.
An SI rule object 2920 associates a set of data message attributes with a service chain represented by the service chain object 2918. The service chain is implemented by one or more service paths, each of which is defined by a service path object 2922. Each service path has one or more service hops, which are represented by one or more service path hop objects 2924 with each hop being associated with one instance runtime interface 2916. Each service hop also refers to an associated service profile, an associated service path, and a next hop SIR endpoint identifier in some embodiments.
In some embodiments, a service path object has several attributes, some of which may be updated by the management or control plane when underlying conditions change. These properties include a service path index, a state (e.g., enabled or disabled), an administrative mode (e.g., enabled or disabled) used when a service path must be manually disabled (e.g., for debugging reasons), a host crossing count (indicating how many times a data message traversing the service path crosses hosts), a locality count (indicating how many of the SIRs along this path are located on the local host), a list of backup service paths, a length of the service path, a reverse path (listing the same set of SIRs in the reverse order), and a maintenance mode indicator (in some embodiments a bit indicating true if any hop in the service path is in maintenance mode).
The host crossing count is an integer and indicates how many times a data message going through the service path must be sent out of a PNIC. In some embodiments, a local or central control plane uses this metric to determine preferred paths when multiple available alternatives exist. This value is populated by the management plane or control plane and is the same for each host using the service path. The locality count in some embodiments is not initialized by the management plane or the control plane but rather computed by the local control plane when a service path is created or updated. Each LCP may potentially compute a different number. This value is used by the local control plane to identify preferred paths when multiple available alternatives exist. The service path length is one parameter that is used by the service plane to set the initial service index.
In some embodiments, the list of backup service paths is a pointer to a sorted list of all service paths for the same service chain. It lists all possible alternatives to be tried when a specific SIR along the path is down. This list may contain a service path for all possible permutations of SVMs in each HA cluster traversed by the service path. In some embodiments, the list will not contain SIRs belonging to different HA clusters.
In some embodiments a service path is disabled when at least one service hop is inactive. Such a condition is temporary and is triggered by service liveness detection failures. A service path can be disabled in this manner at any time. In some embodiments, a service path is also disabled when at least one service hop has no matching SIR. The service hop enters this condition when an SIR it is referring to disappears, but the service path still exists in the object model.
The service plane must be able to uniquely identify each SPI. In some embodiments, the control plane generated UUIDs are sent for each service path. Due to data message header limitations in the service plane, a large ID is not sent with each data message in some embodiments. In some embodiments, when the control plane generates a UUID for each service path, it also generates a small unique ID for it and this ID is sent with each data message in these embodiments.
Through a service partner interface 3002 (e.g., a set of APIs or a partner user interface (UI) portal), the service registrator 3004 receives vendor templates 3005 that specify services that different service partners perform. These templates define the partner services in terms of one or more service descriptors, including service profiles. The registrator 3004 stores the service profiles in a profile storage 3007 for the service chain creator 3006 to use to define service chains.
Specifically, through a user interface 3018 (e.g., a set of APIs or a UI portal), the service chain creator 3006 receives from a network administrator (e.g., a datacenter administrator, a tenant administrator, etc.) one or more service chain definitions. In some embodiments, each service chain definition associates a service chain identifier, which identified the service chain, with an ordered sequence of one or more service profiles. Each service profile in a defined service chain is associated with a service operation that needs to be performed by a service node. The service chain creator 3006 stores the definition of each service chain in the service chain storage 3020.
Through the user interface 3018 (e.g., a set of APIs or a UI portal), the service rule creator 3008 receives from a network administrator (e.g., a datacenter administrator, a tenant administrator, etc.) one or more service insertion rules. In some embodiments, each service insertion rule associates a set of data message flow attributes with a service chain identifier. The flow attributes in some embodiments are flow header attributes, like L2 attributes or L3/L4 attributes (e.g., five tuple attributes). In these or other embodiments, the flow attributes are contextual attributes (e.g., AppID, process ID, active directory ID, etc.). Numerous techniques for capturing and using contextual attributes for performing forwarding and service operations are described in U.S. patent application Ser. No. 15/650,251, which are incorporated herein. Any of these techniques can be used in conjunction with the embodiments described herein.
The service rule creator 3008 generates one or more service insertion rules and stores these rules in the SI rule storage 3022. In some embodiments, each service insertion rule has a rule identifier and a service chain identifier. The rule identifier in some embodiments can be defined in terms of flow identifiers (e.g., header attributes, contextual attributes, etc.) that identify data message flow(s) to which the SI rule is applicable. The service chain identifier of each SI rule, on the other hand, identifies the service chain that has to be performed by the service plane for any data message flow that matches the rule identifier of the SI rule.
For each service chain that is part of a service rule, the service path generator 3012 generates one or more service paths, with each path identifying one or more service instance endpoints for one or more service nodes to perform the service operations specified by the chain's sequence of service profiles. In some embodiments, the process that generates the service paths for a service chain accounts for one or more criteria, such as (1) the data message processing load on the service nodes (e.g., SVMs) that are candidate service nodes for the service paths, (2) the number of host computers crossed by the data messages of a flow as they traverse each candidate service path, etc.
The generation of these service paths is further described in U.S. patent application Ser. No. 16/282,802, now issued as U.S. Pat. No. 11,012,351, which is incorporated herein by reference. As described in this patent application, some embodiments identify the service paths to use for a particular GVM on a particular host based on one or more metrics, such as host crossing count (indicating how many times a data message traversing the service path crosses hosts), a locality count (indicating how many of the SIRs along this path are located on the local host), etc. Other embodiments identify service paths (i.e., select service nodes for service paths) based on other metrics, such as financial and licensing metrics.
The service path generator 3012 stores the identity of the generated service paths in the service path storage 3024. This storage in some embodiments associates each service chain identifier to one or more service path identifiers, and for each service path (i.e., each SPI) it provides a list of service instance endpoints that define the service path. Some embodiments store the service path definitions in one data storage, while storing the association between the service chain and its service paths in another data storage.
The service rule generator 3010 then generates rules for service insertion, next service hop forwarding, and service processing from the rules stored in storages 3020, 3022 and 3024, and stores these rules in rule storages 3026, 3028 and 3030, from where the rule distributor 3014 can retrieve these rules and distribute them to the SI pre-processors, service proxies and service nodes. The distributor 3014 also distributes in some embodiments the path definitions from the service path storage 3024. The path definitions in some embodiments includes the first hop network address (e.g., MAC address) of the first hop along each path. In some embodiments, the service rule generator 3010 and/or the rule distributor 3014 specify and distribute different sets of service paths for the same service chain to different host computers, as different sets of service paths are optimal or preferred for different host computers.
In some embodiments, the SI classification rules that are stored in the rule storage 3026 associate flow identifiers with service chain identifiers. Hence, in some embodiments, the rule generator 3010 retrieves these rules form the storage 3022 and stores them in the classification rule storage 3026. In some embodiments, the rule distributor 3014 directly retrieves the classification rules from the SI rule storage 3022. For these embodiments, the depiction of the SI classification rule storage 3026 is more of a conceptual illustration to highlight the three type of the distributed rules, along with the next-hop forwarding rules and the service node rules.
In some embodiments, the service rule generator 3010 generates the next hop forwarding rules for each hop service proxy of each service path for each service chain. As mentioned above, each service proxy's forwarding table in some embodiments has a forwarding rule that identifies the next hop network address for each service path on which the proxy's associated service node resides. Each such forwarding rule maps the current SPI/SI values to the next hop network address. The service rule generator 3010 generates these rules. For the embodiments in which the SI pre-processor has to look-up the first hop network address, the service rule generator also generates the first hop look-up rule for the SI pre-processor.
Also, in some embodiments, the service rule generator 3010 generates for the service nodes service rules that map service chain identifier, service index values and service directions to service profiles of the service nodes. To do this, the service rule generator uses the service chain and service path definitions from the storages 3020 and 3024, as well as the service profile definitions from the service profile storage 3007. In some embodiments, the rule distributor forwards the service node rules to a service node through a service manager of the service node when such a service manager exists. The service profile definitions are also distributed by the distributor 3014 to the host computers (e.g., to their LCPs) in some embodiments, so that these host computers (e.g., the LCPs) can use these service profiles to configure their service proxies, e.g., to configure the service proxies to forward received data messages to their service nodes, or to copy the received data messages and forward the copies to their service nodes, while forwarding the original received data messages to their next service node hops or back to their source GVMs when they are the last hops.
In some embodiments, the management and control plane dynamically modify the service paths for a service chain, based on the status of the service nodes of the service paths and the data message processing loads on these service nodes.
As shown, the CCP has a status updater 3102 that receives service node status data from status publishers 3103 on the host computers 3120. As mentioned above, each time that a service proxy determines that its associated service node has failed (e.g., each time a service node fails to respond to the service proxy's liveness signal twice in a row), the service proxy notifies the LCP 3110 of its host. The LCP then has its status publisher 3103 notify the CCP's status updater 3102 of the service node's failure.
The status updater 3102 relays any service node failures to the service path generator 3012, which in some embodiments is part of the CCP along with the SP rule generator 3010 and a statistic collector 3104. Each time a service node fails, the service path generator removes from the service path storage 3024 its previously defined service paths that use this service node. For each removed service path, the service path generator 3012 deletes or deactivates the removed path's SPI value for the service chain identifier of the corresponding service chain.
In some embodiments, each removed service path is removed (e.g., deleted or deactivated) from the records of all hosts that previously received forwarding rules or path definitions that were for this service path. In some embodiments, the CCP (e.g., the service path generator 3010 or the rule distributor 3014) directs these hosts to remove the service path from the forwarding and path definition rules of their forwarding rule storages 3128 and path definition storage 808. The LCP of the failed service node in some embodiments removes the service path from its forwarding and path definition rules, while in other embodiments even this LCP waits for instructions to do so from the CCP.
Each host 3120 also has a statistics publisher 3105 that publishes data message load statistics that the service proxies generate for their service nodes in some embodiments. Each time a service proxy receives a data message that has been processed by its service node, the service proxy in some embodiments increments statistics (e.g., data message count, byte count, etc.) that it maintains in a statistic storage 3107 for its service node. In some embodiments, the statistics publisher 3105 periodically or on-demand retrieves the collected statistics from the storage 3107 and forwards these statistics to a statistic collector 3104 of the CCP. In some embodiments, the statistics collector 3104 receives (through the management plane) statistics that the service managers of the service nodes receive from the service nodes.
The statistics collector 3104 relays the collected statistics to the service path generator 3012. As mentioned above, the service path generator in some embodiments defines the service paths through the service nodes based in part on the data message load on the service nodes. For instance, when the data message load on a service node exceeds a threshold value, the service path generator performs one or more actions in some embodiments to reduce the load on this service node. For instance, in some embodiments, it stops adding the service node to any new service paths that it might define. In these or other embodiments, it also directs the distributor 3014 to remove the service paths that use this service node from some or all of the hosts.
Conjunctively or alternatively, the service path generator directs a CCP module (e.g., the distributor 3014) to direct the LCPs of one or more host computers to adjust the selection criteria 820 used for selecting service paths that the LCPs generate in order to control how the SI pre-processor performs its path selections. In other embodiments, the service path generator or another CCP module aggregates the load statistics for each service node and distributes the aggregated load to host LCPs along with their associated SPI values so that the LCPs can analyze these statistics and adjust the path selection criteria that they generate. In some embodiments, each LCP uses or has a path evaluator 3115 to generate the path selection criteria to evaluate and select paths based on service node statistics, and/or based on other criteria, such as number of hosts traversed by each service path.
In some embodiments, the servers that implement the management plane, the control plane, the service managers are in the same datacenter as the host computers on which the guest and service machines and modules (e.g., GVMs, SVMs, service proxies, port proxies, STL modules, SFEs, etc.) execute. In these embodiments, the management plane servers, the control plane servers, the service managers and the host computer modules (e.g., the LCPs, SVMs, GVMs, hypervisor modules, etc.) communicate with each other through the shared network infrastructure (e.g., the switches, routers, wired and wireless links, etc.) of the datacenter.
In other embodiments, the management plane servers, the control plane servers, the service managers and/or the host computers operate in different datacenters (e.g., enterprise private datacenters and public cloud datacenters). In some such embodiments, management plane servers, the control plane servers, the service managers and/or the host computer modules (e.g., the LCPs, SVMs, GVMs, hypervisor modules, etc.) communicate with each other through network infrastructures outside of their respective datacenters. Also, some such embodiments implement the service transport layer as a distributed logical L3 routers and/or network that spans multiple datacenters (e.g., multiple private datacenters, multiple public datacenters, multiple private/public datacenters).
Service planes are used in some embodiments to segregate the service processing for the data traffic of one tenant from the service processing for the data traffic of other tenants. In these or other embodiments, different service planes are used to provide different QoS or SLA guarantees for different types of traffic. For example, some embodiments use different service planes to provide different QoS or SLA guarantees for traffic between different data compute endpoints of different tenants, or different QoS or SLA guarantees for different type of content carried by different data message flows of the same tenant or different tenants.
After creating the service attachment, the process creates (at 3210) service instances for the services that are to be provided by the service plane. For each deployed service instance, the process specifies whether the service instance should be provided by a high availability cluster or by a standalone service node. It also provides a service attachment identifier that identifies the service attachment associated with the service instance. It also provides the deployment specification and the instance deployment configuration.
Next, at 3215, the process deploys each service instance runtime for each service instance created at 3210. For each service instance runtime, an instance endpoint has to be created on the service attachment. When the service attachment is a logical switch, the created instance endpoint is a logical port of the logical switch. In some embodiments, the logical switch port is auto created when an SVM (that serves as the service instance runtime) gets attached to the logical switch. In some embodiments, the service instance endpoints are created by the management plane each time a service instance is deployed. Also, in some embodiments, the service instances and service instance runtimes for a service can be deployed by invoking one service deployment object API. As mentioned above, the use of this single API greatly alleviates the need to repeatedly invoke one API multiple times to deploy multiple service instances and service instance runtimes.
At 3220, the process creates one or more service chains. Each service chain is created as an ordered list of service profiles. Each service chain has a forward processing direction and a reverse processing direction. For each service chain, a failure policy is defined as described above. Also, as described above, the load balancing criteria in some embodiments is defined for each service chain as one of the following types: any, local, service cluster or round robin. Finally, at 3225, a section of service rules is defined for the tenant, and one or more service rules are defined in these sections. Each service rule correlates a set of data message flow attributes with a service chain identifier, in order to specify the service chain that has to be executed for data messages that match the specified flow attribute set.
Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.
In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.
The bus 3305 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system 3300. For instance, the bus 3305 communicatively connects the processing unit(s) 3310 with the read-only memory 3330, the system memory 3325, and the permanent storage device 3335.
From these various memory units, the processing unit(s) 3310 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only-memory (ROM) 3330 stores static data and instructions that are needed by the processing unit(s) 3310 and other modules of the computer system. The permanent storage device 3335, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 3300 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 3335.
Other embodiments use a removable storage device (such as a flash drive, etc.) as the permanent storage device. Like the permanent storage device 3335, the system memory 3325 is a read-and-write memory device. However, unlike storage device 3335, the system memory is a volatile read-and-write memory, such a random access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 3325, the permanent storage device 3335, and/or the read-only memory 3330. From these various memory units, the processing unit(s) 3310 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.
The bus 3305 also connects to the input and output devices 3340 and 3345. The input devices enable the user to communicate information and select commands to the computer system. The input devices 3340 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 3345 display images generated by the computer system. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, and any other optical or magnetic media. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer readable medium,” “computer readable media,” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral or transitory signals.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For instance, several figures conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process could be implemented using several sub-processes, or as part of a larger macro process.
Even though the service insertion rules in several of the above-described examples provide service chain identifiers, some of the inventions described herein can be implemented by having a service insertion rule provide the service identifiers (e.g., SPIs) of the different services specified by the service insertion rule. Similarly, several of the above-described embodiments perform distributed service routing that relies at each service hop identifying a next service hop by performing an exact match based on the SPI/SI values. However, some of the inventions described herein can be implemented by having the service insertion pre-processor embed all the service hop identifiers (e.g., service hop MAC addresses) as the data message's service attribute set and/or in the data message's encapsulating service header.
In addition, some embodiments decrement the SI value differently (e.g., at different times) than the approaches described above. Also, instead of performing the next hop lookup just based on the SPI and SI values, some embodiments perform this lookup based on the SPI, SI and service direction values as these embodiments use a common SPI value for both the forward and reverse directions of data messages flowing between two machines.
The above-described methodology is used in some embodiments to express path information in single tenant environments. Thus, one of ordinary skill will realize that some embodiments of the invention are equally applicable to single tenant datacenters. Conversely, in some embodiments, the above-described methodology is used to carry path information across different datacenters of different datacenter providers when one entity (e.g., one corporation) is a tenant in multiple different datacenters of different providers. In these embodiments, the tenant identifiers that are embedded in the tunnel headers have to be unique across the datacenters, or have to be translated when they traverse from one datacenter to the next. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.
Number | Date | Country | Kind |
---|---|---|---|
201941007860 | Feb 2019 | IN | national |
Number | Name | Date | Kind |
---|---|---|---|
6006264 | Colby et al. | Dec 1999 | A |
6104700 | Haddock et al. | Aug 2000 | A |
6154448 | Petersen et al. | Nov 2000 | A |
6772211 | Lu et al. | Aug 2004 | B2 |
6779030 | Dugan et al. | Aug 2004 | B1 |
6826694 | Dutta et al. | Nov 2004 | B1 |
6880089 | Bommareddy et al. | Apr 2005 | B1 |
6985956 | Luke et al. | Jan 2006 | B2 |
7013389 | Srivastava et al. | Mar 2006 | B1 |
7209977 | Acharya et al. | Apr 2007 | B2 |
7239639 | Cox et al. | Jul 2007 | B2 |
7379465 | Aysan et al. | May 2008 | B2 |
7406540 | Acharya et al. | Jul 2008 | B2 |
7447775 | Zhu et al. | Nov 2008 | B1 |
7480737 | Chauffour et al. | Jan 2009 | B2 |
7487250 | Siegel | Feb 2009 | B2 |
7649890 | Mizutani et al. | Jan 2010 | B2 |
7698458 | Liu et al. | Apr 2010 | B1 |
7818452 | Matthews et al. | Oct 2010 | B2 |
7898959 | Arad | Mar 2011 | B1 |
7948986 | Ghosh et al. | May 2011 | B1 |
8078903 | Parthasarathy et al. | Dec 2011 | B1 |
8094575 | Vadlakonda et al. | Jan 2012 | B1 |
8175863 | Ostermeyer et al. | May 2012 | B1 |
8190767 | Maufer et al. | May 2012 | B1 |
8201219 | Jones | Jun 2012 | B2 |
8223634 | Tanaka et al. | Jul 2012 | B2 |
8224885 | Doucette et al. | Jul 2012 | B1 |
8230493 | Davidson et al. | Jul 2012 | B2 |
8266261 | Akagi | Sep 2012 | B2 |
8339959 | Moisand et al. | Dec 2012 | B1 |
8451735 | Li | May 2013 | B2 |
8484348 | Subramanian et al. | Jul 2013 | B2 |
8488577 | Macpherson | Jul 2013 | B1 |
8521879 | Pena et al. | Aug 2013 | B1 |
8615009 | Ramamoorthi et al. | Dec 2013 | B1 |
8707383 | Bade et al. | Apr 2014 | B2 |
8743885 | Khan et al. | Jun 2014 | B2 |
8804720 | Rainovic et al. | Aug 2014 | B1 |
8804746 | Wu et al. | Aug 2014 | B2 |
8811412 | Shippy | Aug 2014 | B2 |
8830834 | Sharma et al. | Sep 2014 | B2 |
8832683 | Heim | Sep 2014 | B2 |
8849746 | Candea et al. | Sep 2014 | B2 |
8856518 | Sridharan et al. | Oct 2014 | B2 |
8862883 | Cherukur et al. | Oct 2014 | B2 |
8868711 | Skjolsvold et al. | Oct 2014 | B2 |
8873399 | Bothos et al. | Oct 2014 | B2 |
8874789 | Zhu | Oct 2014 | B1 |
8892706 | Dalal | Nov 2014 | B1 |
8913611 | Koponen et al. | Dec 2014 | B2 |
8914406 | Haugsnes et al. | Dec 2014 | B1 |
8966024 | Koponen et al. | Feb 2015 | B2 |
8966029 | Zhang et al. | Feb 2015 | B2 |
8971345 | McCanne et al. | Mar 2015 | B1 |
8989192 | Foo et al. | Mar 2015 | B2 |
8996610 | Sureshchandra et al. | Mar 2015 | B1 |
9009289 | Jacob | Apr 2015 | B1 |
9015823 | Koponen et al. | Apr 2015 | B2 |
9094464 | Scharber et al. | Jul 2015 | B1 |
9104497 | Mortazavi | Aug 2015 | B2 |
9148367 | Kandaswamy et al. | Sep 2015 | B2 |
9172603 | Padmanabhan et al. | Oct 2015 | B2 |
9178709 | Higashida et al. | Nov 2015 | B2 |
9191293 | Iovene et al. | Nov 2015 | B2 |
9195491 | Zhang et al. | Nov 2015 | B2 |
9203748 | Jiang et al. | Dec 2015 | B2 |
9225638 | Jain et al. | Dec 2015 | B2 |
9225659 | McCanne et al. | Dec 2015 | B2 |
9232342 | Seed et al. | Jan 2016 | B2 |
9258742 | Pianigiani et al. | Feb 2016 | B1 |
9264313 | Manuguri et al. | Feb 2016 | B1 |
9277412 | Freda et al. | Mar 2016 | B2 |
9397946 | Yadav | Jul 2016 | B1 |
9407540 | Kumar et al. | Aug 2016 | B2 |
9407599 | Koponen et al. | Aug 2016 | B2 |
9419897 | Cherian et al. | Aug 2016 | B2 |
9479358 | Klosowski et al. | Oct 2016 | B2 |
9503530 | Niedzielski | Nov 2016 | B1 |
9531590 | Jain et al. | Dec 2016 | B2 |
9577845 | Thakkar et al. | Feb 2017 | B2 |
9602380 | Strassner | Mar 2017 | B2 |
9660905 | Dunbar et al. | May 2017 | B2 |
9686192 | Sengupta et al. | Jun 2017 | B2 |
9686200 | Pettit et al. | Jun 2017 | B2 |
9705702 | Foo et al. | Jul 2017 | B2 |
9705775 | Zhang et al. | Jul 2017 | B2 |
9749229 | Previdi et al. | Aug 2017 | B2 |
9755898 | Jain et al. | Sep 2017 | B2 |
9755971 | Wang et al. | Sep 2017 | B2 |
9774537 | Jain et al. | Sep 2017 | B2 |
9787559 | Schroeder | Oct 2017 | B1 |
9787605 | Zhang et al. | Oct 2017 | B2 |
9804797 | Ng et al. | Oct 2017 | B1 |
9825810 | Jain et al. | Nov 2017 | B2 |
9860079 | Cohn et al. | Jan 2018 | B2 |
9900410 | Dalal | Feb 2018 | B2 |
9935827 | Jain et al. | Apr 2018 | B2 |
9979641 | Jain et al. | May 2018 | B2 |
9985896 | Koponen et al. | May 2018 | B2 |
10013276 | Fahs et al. | Jul 2018 | B2 |
10042722 | Chigurupati et al. | Aug 2018 | B1 |
10075470 | Vaidya et al. | Sep 2018 | B2 |
10079779 | Zhang et al. | Sep 2018 | B2 |
10084703 | Kumar et al. | Sep 2018 | B2 |
10089127 | Padmanabhan et al. | Oct 2018 | B2 |
10091276 | Bloomquist et al. | Oct 2018 | B2 |
10104169 | Moniz et al. | Oct 2018 | B1 |
10129077 | Jain et al. | Nov 2018 | B2 |
10129180 | Zhang et al. | Nov 2018 | B2 |
10135636 | Jiang et al. | Nov 2018 | B2 |
10135737 | Jain et al. | Nov 2018 | B2 |
10158573 | Lee et al. | Dec 2018 | B1 |
10187306 | Nainar et al. | Jan 2019 | B2 |
10200493 | Bendapudi et al. | Feb 2019 | B2 |
10212071 | Kancherla et al. | Feb 2019 | B2 |
10225137 | Jain et al. | Mar 2019 | B2 |
10237379 | Kumar et al. | Mar 2019 | B2 |
10250501 | Ni | Apr 2019 | B2 |
10257095 | Jain et al. | Apr 2019 | B2 |
10284390 | Kumar et al. | May 2019 | B2 |
10320679 | Jain et al. | Jun 2019 | B2 |
10333822 | Jeuk et al. | Jun 2019 | B1 |
10341233 | Jain et al. | Jul 2019 | B2 |
10341427 | Jalan et al. | Jul 2019 | B2 |
10375155 | Cai et al. | Aug 2019 | B1 |
10397275 | Jain et al. | Aug 2019 | B2 |
10514941 | Zhang et al. | Dec 2019 | B2 |
10516568 | Jain et al. | Dec 2019 | B2 |
10547508 | Kanakarajan | Jan 2020 | B1 |
10547692 | Salgueiro et al. | Jan 2020 | B2 |
10554484 | Chanda et al. | Feb 2020 | B2 |
10594743 | Hong et al. | Mar 2020 | B2 |
10609091 | Hong et al. | Mar 2020 | B2 |
10623309 | Gampel et al. | Apr 2020 | B1 |
10637750 | Bollineni et al. | Apr 2020 | B1 |
10645060 | Ao et al. | May 2020 | B2 |
10659252 | Boutros et al. | May 2020 | B2 |
10693782 | Jain et al. | Jun 2020 | B2 |
10708229 | Sevinc et al. | Jul 2020 | B2 |
10728174 | Boutros et al. | Jul 2020 | B2 |
10742544 | Roeland et al. | Aug 2020 | B2 |
10757077 | Rajahalme et al. | Aug 2020 | B2 |
10797910 | Boutros et al. | Oct 2020 | B2 |
10797966 | Boutros et al. | Oct 2020 | B2 |
10805181 | Boutros et al. | Oct 2020 | B2 |
10805192 | Boutros et al. | Oct 2020 | B2 |
10812378 | Nainar et al. | Oct 2020 | B2 |
10834004 | Yigit et al. | Nov 2020 | B2 |
10853111 | Gupta et al. | Dec 2020 | B1 |
10929171 | Gokhale et al. | Feb 2021 | B2 |
10938716 | Chin et al. | Mar 2021 | B1 |
10944673 | Naveen et al. | Mar 2021 | B2 |
10949244 | Naveen et al. | Mar 2021 | B2 |
11003482 | Rolando et al. | May 2021 | B2 |
11012420 | Sevinc et al. | May 2021 | B2 |
11036538 | Lecuyer et al. | Jun 2021 | B2 |
11038782 | Boutros et al. | Jun 2021 | B2 |
11042397 | Mishra et al. | Jun 2021 | B2 |
11074097 | Naveen et al. | Jul 2021 | B2 |
11075839 | Zhuang et al. | Jul 2021 | B2 |
11075842 | Jain et al. | Jul 2021 | B2 |
11086654 | Rolando et al. | Aug 2021 | B2 |
20020078370 | Tahan | Jun 2002 | A1 |
20020097724 | Halme et al. | Jul 2002 | A1 |
20020194350 | Lu et al. | Dec 2002 | A1 |
20030065711 | Acharya et al. | Apr 2003 | A1 |
20030093481 | Mitchell et al. | May 2003 | A1 |
20030097429 | Wu et al. | May 2003 | A1 |
20030105812 | Flowers et al. | Jun 2003 | A1 |
20030236813 | Abjanic | Dec 2003 | A1 |
20040066769 | Ahmavaara et al. | Apr 2004 | A1 |
20040210670 | Anerousis et al. | Oct 2004 | A1 |
20040215703 | Song et al. | Oct 2004 | A1 |
20050021713 | Dugan et al. | Jan 2005 | A1 |
20050089327 | Ovadia et al. | Apr 2005 | A1 |
20050091396 | Nilakantan et al. | Apr 2005 | A1 |
20050114429 | Caccavale | May 2005 | A1 |
20050114648 | Akundi et al. | May 2005 | A1 |
20050132030 | Hopen et al. | Jun 2005 | A1 |
20050198200 | Subramanian et al. | Sep 2005 | A1 |
20050249199 | Albert et al. | Nov 2005 | A1 |
20060069776 | Shim et al. | Mar 2006 | A1 |
20060112297 | Davidson | May 2006 | A1 |
20060130133 | Andreev et al. | Jun 2006 | A1 |
20060155862 | Kathi et al. | Jul 2006 | A1 |
20060195896 | Fulp et al. | Aug 2006 | A1 |
20060233155 | Srivastava | Oct 2006 | A1 |
20070061492 | Riel | Mar 2007 | A1 |
20070121615 | Weill et al. | May 2007 | A1 |
20070214282 | Sen | Sep 2007 | A1 |
20070248091 | Khalid et al. | Oct 2007 | A1 |
20070260750 | Feied et al. | Nov 2007 | A1 |
20070288615 | Keohane et al. | Dec 2007 | A1 |
20070291773 | Khan et al. | Dec 2007 | A1 |
20080005293 | Bhargava et al. | Jan 2008 | A1 |
20080031263 | Ervin et al. | Feb 2008 | A1 |
20080046400 | Shi et al. | Feb 2008 | A1 |
20080049614 | Briscoe et al. | Feb 2008 | A1 |
20080049619 | Twiss | Feb 2008 | A1 |
20080049786 | Ram et al. | Feb 2008 | A1 |
20080072305 | Casado et al. | Mar 2008 | A1 |
20080084819 | Parizhsky et al. | Apr 2008 | A1 |
20080095153 | Fukunaga et al. | Apr 2008 | A1 |
20080104608 | Hyser et al. | May 2008 | A1 |
20080195755 | Lu et al. | Aug 2008 | A1 |
20080225714 | Denis | Sep 2008 | A1 |
20080239991 | Applegate et al. | Oct 2008 | A1 |
20080247396 | Hazard | Oct 2008 | A1 |
20080276085 | Davidson et al. | Nov 2008 | A1 |
20080279196 | Friskney et al. | Nov 2008 | A1 |
20090003349 | Havemann et al. | Jan 2009 | A1 |
20090003364 | Fendick et al. | Jan 2009 | A1 |
20090003375 | Havemann et al. | Jan 2009 | A1 |
20090019135 | Eswaran et al. | Jan 2009 | A1 |
20090037713 | Khalid et al. | Feb 2009 | A1 |
20090063706 | Goldman et al. | Mar 2009 | A1 |
20090129271 | Ramankutty et al. | May 2009 | A1 |
20090172666 | Yahalom et al. | Jul 2009 | A1 |
20090199268 | Ahmavaara et al. | Aug 2009 | A1 |
20090235325 | Dimitrakos et al. | Sep 2009 | A1 |
20090238084 | Nadeau et al. | Sep 2009 | A1 |
20090249472 | Litvin et al. | Oct 2009 | A1 |
20090265467 | Peles et al. | Oct 2009 | A1 |
20090271586 | Shaath | Oct 2009 | A1 |
20090299791 | Blake et al. | Dec 2009 | A1 |
20090300210 | Ferris | Dec 2009 | A1 |
20090303880 | Maltz et al. | Dec 2009 | A1 |
20090307334 | Maltz et al. | Dec 2009 | A1 |
20090327464 | Archer et al. | Dec 2009 | A1 |
20100031360 | Seshadri et al. | Feb 2010 | A1 |
20100036903 | Ahmad et al. | Feb 2010 | A1 |
20100100616 | Bryson et al. | Apr 2010 | A1 |
20100131638 | Kondamuru | May 2010 | A1 |
20100165985 | Sharma et al. | Jul 2010 | A1 |
20100223364 | Wei | Sep 2010 | A1 |
20100223621 | Joshi et al. | Sep 2010 | A1 |
20100235915 | Memon et al. | Sep 2010 | A1 |
20100254385 | Sharma et al. | Oct 2010 | A1 |
20100265824 | Chao et al. | Oct 2010 | A1 |
20100281482 | Pike et al. | Nov 2010 | A1 |
20100332595 | Fullagar et al. | Dec 2010 | A1 |
20110010578 | Dominguez et al. | Jan 2011 | A1 |
20110016348 | Pace et al. | Jan 2011 | A1 |
20110022695 | Dalal et al. | Jan 2011 | A1 |
20110022812 | Van Der Linden et al. | Jan 2011 | A1 |
20110035494 | Pandey et al. | Feb 2011 | A1 |
20110040893 | Karaoguz et al. | Feb 2011 | A1 |
20110055845 | Nandagopal et al. | Mar 2011 | A1 |
20110058563 | Saraph et al. | Mar 2011 | A1 |
20110090912 | Shippy | Apr 2011 | A1 |
20110164504 | Bothos et al. | Jul 2011 | A1 |
20110194563 | Shen et al. | Aug 2011 | A1 |
20110211463 | Matityahu et al. | Sep 2011 | A1 |
20110225293 | Rathod | Sep 2011 | A1 |
20110235508 | Goel et al. | Sep 2011 | A1 |
20110261811 | Battestilli et al. | Oct 2011 | A1 |
20110268118 | Schlansker et al. | Nov 2011 | A1 |
20110271007 | Wang et al. | Nov 2011 | A1 |
20110276695 | Maldaner | Nov 2011 | A1 |
20110283013 | Grosser et al. | Nov 2011 | A1 |
20110295991 | Aida | Dec 2011 | A1 |
20110317708 | Clark | Dec 2011 | A1 |
20120005265 | Ushioda et al. | Jan 2012 | A1 |
20120014386 | Xiong et al. | Jan 2012 | A1 |
20120023231 | Ueno | Jan 2012 | A1 |
20120054266 | Kazerani et al. | Mar 2012 | A1 |
20120089664 | Igelka | Apr 2012 | A1 |
20120137004 | Smith | May 2012 | A1 |
20120140719 | Hui et al. | Jun 2012 | A1 |
20120144014 | Natham et al. | Jun 2012 | A1 |
20120147894 | Mulligan et al. | Jun 2012 | A1 |
20120155266 | Patel et al. | Jun 2012 | A1 |
20120176932 | Wu et al. | Jul 2012 | A1 |
20120185588 | Error | Jul 2012 | A1 |
20120195196 | Ghai et al. | Aug 2012 | A1 |
20120207174 | Shieh | Aug 2012 | A1 |
20120213074 | Goldfarb et al. | Aug 2012 | A1 |
20120230187 | Tremblay et al. | Sep 2012 | A1 |
20120239804 | Liu et al. | Sep 2012 | A1 |
20120246637 | Kreeger et al. | Sep 2012 | A1 |
20120281540 | Khan et al. | Nov 2012 | A1 |
20120287789 | Aybay et al. | Nov 2012 | A1 |
20120303784 | Zisapel et al. | Nov 2012 | A1 |
20120303809 | Patel et al. | Nov 2012 | A1 |
20120311568 | Jansen | Dec 2012 | A1 |
20120317260 | Husain et al. | Dec 2012 | A1 |
20120317570 | Dalcher et al. | Dec 2012 | A1 |
20120331188 | Riordan et al. | Dec 2012 | A1 |
20130003735 | Chao et al. | Jan 2013 | A1 |
20130021942 | Bacthu et al. | Jan 2013 | A1 |
20130031544 | Sridharan et al. | Jan 2013 | A1 |
20130039218 | Narasimhan et al. | Feb 2013 | A1 |
20130044636 | Koponen et al. | Feb 2013 | A1 |
20130058346 | Sridharan et al. | Mar 2013 | A1 |
20130073743 | Ramasamy et al. | Mar 2013 | A1 |
20130100851 | Bacthu et al. | Apr 2013 | A1 |
20130125120 | Zhang et al. | May 2013 | A1 |
20130136126 | Wang et al. | May 2013 | A1 |
20130142048 | Gross, IV et al. | Jun 2013 | A1 |
20130148505 | Koponen et al. | Jun 2013 | A1 |
20130151661 | Koponen et al. | Jun 2013 | A1 |
20130159487 | Patel et al. | Jun 2013 | A1 |
20130160024 | Shtilman et al. | Jun 2013 | A1 |
20130163594 | Sharma et al. | Jun 2013 | A1 |
20130166703 | Hammer et al. | Jun 2013 | A1 |
20130170501 | Egi et al. | Jul 2013 | A1 |
20130201989 | Hu et al. | Aug 2013 | A1 |
20130227097 | Yasuda et al. | Aug 2013 | A1 |
20130227550 | Weinstein et al. | Aug 2013 | A1 |
20130287026 | Davie | Oct 2013 | A1 |
20130291088 | Shieh et al. | Oct 2013 | A1 |
20130297798 | Arisoylu et al. | Nov 2013 | A1 |
20130301472 | Allan | Nov 2013 | A1 |
20130311637 | Kamath et al. | Nov 2013 | A1 |
20130318219 | Kancherla | Nov 2013 | A1 |
20130332983 | Koorevaar et al. | Dec 2013 | A1 |
20130336319 | Liu et al. | Dec 2013 | A1 |
20130343174 | Guichard et al. | Dec 2013 | A1 |
20130343378 | Veteikis et al. | Dec 2013 | A1 |
20140003232 | Guichard et al. | Jan 2014 | A1 |
20140003422 | Mogul et al. | Jan 2014 | A1 |
20140010085 | Kavunder et al. | Jan 2014 | A1 |
20140029447 | Schrum, Jr. | Jan 2014 | A1 |
20140046997 | Dain et al. | Feb 2014 | A1 |
20140046998 | Dain et al. | Feb 2014 | A1 |
20140050223 | Foo et al. | Feb 2014 | A1 |
20140052844 | Nayak et al. | Feb 2014 | A1 |
20140059204 | Nguyen et al. | Feb 2014 | A1 |
20140059544 | Koganty et al. | Feb 2014 | A1 |
20140068602 | Gember et al. | Mar 2014 | A1 |
20140092738 | Grandhi et al. | Apr 2014 | A1 |
20140092906 | Kandaswamy et al. | Apr 2014 | A1 |
20140092914 | Kondapalli | Apr 2014 | A1 |
20140096183 | Jain et al. | Apr 2014 | A1 |
20140101226 | Khandekar et al. | Apr 2014 | A1 |
20140101656 | Zhu et al. | Apr 2014 | A1 |
20140115578 | Cooper et al. | Apr 2014 | A1 |
20140129715 | Mortazavi | May 2014 | A1 |
20140149696 | Frenkel et al. | May 2014 | A1 |
20140164477 | Springer et al. | Jun 2014 | A1 |
20140169168 | Jalan et al. | Jun 2014 | A1 |
20140169375 | Khan et al. | Jun 2014 | A1 |
20140195666 | Dumitriu et al. | Jul 2014 | A1 |
20140207968 | Kumar et al. | Jul 2014 | A1 |
20140254374 | Janakiraman et al. | Sep 2014 | A1 |
20140254591 | Mahadevan et al. | Sep 2014 | A1 |
20140269487 | Kalkunte | Sep 2014 | A1 |
20140269717 | Thubert et al. | Sep 2014 | A1 |
20140269724 | Mehler et al. | Sep 2014 | A1 |
20140280896 | Papakostas et al. | Sep 2014 | A1 |
20140281029 | Danforth | Sep 2014 | A1 |
20140282526 | Basavaiah et al. | Sep 2014 | A1 |
20140301388 | Jagadish et al. | Oct 2014 | A1 |
20140304231 | Kamath et al. | Oct 2014 | A1 |
20140307744 | Dunbar et al. | Oct 2014 | A1 |
20140310391 | Sorenson et al. | Oct 2014 | A1 |
20140310418 | Sorenson et al. | Oct 2014 | A1 |
20140317677 | Vaidya et al. | Oct 2014 | A1 |
20140321459 | Kumar et al. | Oct 2014 | A1 |
20140330983 | Zisapel et al. | Nov 2014 | A1 |
20140334485 | Jain et al. | Nov 2014 | A1 |
20140334488 | Guichard | Nov 2014 | A1 |
20140341029 | Allan et al. | Nov 2014 | A1 |
20140351452 | Bosch et al. | Nov 2014 | A1 |
20140362682 | Guichard et al. | Dec 2014 | A1 |
20140362705 | Pan | Dec 2014 | A1 |
20140369204 | Anand et al. | Dec 2014 | A1 |
20140372567 | Ganesh et al. | Dec 2014 | A1 |
20140372616 | Arisoylu et al. | Dec 2014 | A1 |
20140372702 | Subramanyam et al. | Dec 2014 | A1 |
20150003453 | Sengupta et al. | Jan 2015 | A1 |
20150003455 | Haddad et al. | Jan 2015 | A1 |
20150009995 | Gross, IV et al. | Jan 2015 | A1 |
20150016279 | Zhang et al. | Jan 2015 | A1 |
20150023354 | Li et al. | Jan 2015 | A1 |
20150026345 | Ravinoothala et al. | Jan 2015 | A1 |
20150026362 | Guichard et al. | Jan 2015 | A1 |
20150030024 | Venkataswami et al. | Jan 2015 | A1 |
20150052262 | Chanda et al. | Feb 2015 | A1 |
20150052522 | Chanda et al. | Feb 2015 | A1 |
20150063102 | Mestery et al. | Mar 2015 | A1 |
20150063364 | Thakkar et al. | Mar 2015 | A1 |
20150071301 | Dalal | Mar 2015 | A1 |
20150073967 | Katsuyama et al. | Mar 2015 | A1 |
20150078384 | Jackson et al. | Mar 2015 | A1 |
20150092564 | Aldrin | Apr 2015 | A1 |
20150103645 | Shen et al. | Apr 2015 | A1 |
20150103679 | Tessmer et al. | Apr 2015 | A1 |
20150103827 | Quinn et al. | Apr 2015 | A1 |
20150109901 | Tan et al. | Apr 2015 | A1 |
20150124608 | Agarwal et al. | May 2015 | A1 |
20150124622 | Kovvali et al. | May 2015 | A1 |
20150124840 | Bergeron | May 2015 | A1 |
20150138973 | Kumar et al. | May 2015 | A1 |
20150139041 | Bosch et al. | May 2015 | A1 |
20150146539 | Mehta et al. | May 2015 | A1 |
20150156035 | Foo et al. | Jun 2015 | A1 |
20150188770 | Naiksatam et al. | Jul 2015 | A1 |
20150195197 | Yong et al. | Jul 2015 | A1 |
20150213087 | Sikri | Jul 2015 | A1 |
20150215819 | Bosch et al. | Jul 2015 | A1 |
20150222640 | Kumar | Aug 2015 | A1 |
20150236948 | Dunbar et al. | Aug 2015 | A1 |
20150237013 | Bansal et al. | Aug 2015 | A1 |
20150242197 | Alfonso et al. | Aug 2015 | A1 |
20150244617 | Nakil et al. | Aug 2015 | A1 |
20150263901 | Kumar | Sep 2015 | A1 |
20150263946 | Tubaltsev et al. | Sep 2015 | A1 |
20150271102 | Antich | Sep 2015 | A1 |
20150280959 | Vincent | Oct 2015 | A1 |
20150281089 | Marchetti | Oct 2015 | A1 |
20150281098 | Pettit et al. | Oct 2015 | A1 |
20150281125 | Koponen et al. | Oct 2015 | A1 |
20150281179 | Raman et al. | Oct 2015 | A1 |
20150281180 | Raman et al. | Oct 2015 | A1 |
20150288671 | Chan et al. | Oct 2015 | A1 |
20150288679 | Ben-Nun et al. | Oct 2015 | A1 |
20150295831 | Kumar | Oct 2015 | A1 |
20150319078 | Lee et al. | Nov 2015 | A1 |
20150319096 | Yip et al. | Nov 2015 | A1 |
20150358235 | Zhang | Dec 2015 | A1 |
20150365322 | Shalzkamer et al. | Dec 2015 | A1 |
20150370586 | Cooper et al. | Dec 2015 | A1 |
20150370596 | Fahs et al. | Dec 2015 | A1 |
20150372840 | Benny et al. | Dec 2015 | A1 |
20150372911 | Yabusaki et al. | Dec 2015 | A1 |
20150381493 | Bansal et al. | Dec 2015 | A1 |
20150381494 | Cherian et al. | Dec 2015 | A1 |
20150381495 | Cherian et al. | Dec 2015 | A1 |
20160006654 | Fernando et al. | Jan 2016 | A1 |
20160028640 | Zhang et al. | Jan 2016 | A1 |
20160043901 | Sankar et al. | Feb 2016 | A1 |
20160043952 | Zhang et al. | Feb 2016 | A1 |
20160057050 | Ostrom et al. | Feb 2016 | A1 |
20160057687 | Horn et al. | Feb 2016 | A1 |
20160065503 | Yohe et al. | Mar 2016 | A1 |
20160080253 | Wang et al. | Mar 2016 | A1 |
20160087888 | Jain et al. | Mar 2016 | A1 |
20160094384 | Jain et al. | Mar 2016 | A1 |
20160094389 | Jain et al. | Mar 2016 | A1 |
20160094451 | Jain et al. | Mar 2016 | A1 |
20160094452 | Jain et al. | Mar 2016 | A1 |
20160094453 | Jain et al. | Mar 2016 | A1 |
20160094454 | Jain et al. | Mar 2016 | A1 |
20160094455 | Jain et al. | Mar 2016 | A1 |
20160094456 | Jain et al. | Mar 2016 | A1 |
20160094457 | Jain et al. | Mar 2016 | A1 |
20160094631 | Jain et al. | Mar 2016 | A1 |
20160094632 | Jain et al. | Mar 2016 | A1 |
20160094633 | Jain et al. | Mar 2016 | A1 |
20160094642 | Jain et al. | Mar 2016 | A1 |
20160094643 | Jain et al. | Mar 2016 | A1 |
20160094661 | Jain et al. | Mar 2016 | A1 |
20160105333 | Lenglet et al. | Apr 2016 | A1 |
20160119226 | Guichard et al. | Apr 2016 | A1 |
20160127306 | Wang et al. | May 2016 | A1 |
20160127564 | Sharma et al. | May 2016 | A1 |
20160134528 | Lin et al. | May 2016 | A1 |
20160149816 | Roach et al. | May 2016 | A1 |
20160149828 | Vijayan et al. | May 2016 | A1 |
20160164776 | Biancaniello | Jun 2016 | A1 |
20160164787 | Roach et al. | Jun 2016 | A1 |
20160164826 | Riedel et al. | Jun 2016 | A1 |
20160173373 | Guichard et al. | Jun 2016 | A1 |
20160182684 | Connor et al. | Jun 2016 | A1 |
20160197831 | Foy et al. | Jul 2016 | A1 |
20160197839 | Li et al. | Jul 2016 | A1 |
20160205015 | Halligan et al. | Jul 2016 | A1 |
20160212048 | Kaempfer et al. | Jul 2016 | A1 |
20160212237 | Nishijima | Jul 2016 | A1 |
20160226700 | Zhang et al. | Aug 2016 | A1 |
20160226754 | Zhang et al. | Aug 2016 | A1 |
20160226762 | Zhang et al. | Aug 2016 | A1 |
20160248685 | Pignataro et al. | Aug 2016 | A1 |
20160277210 | Lin et al. | Sep 2016 | A1 |
20160277294 | Akiyoshi | Sep 2016 | A1 |
20160294612 | Ravinoothala et al. | Oct 2016 | A1 |
20160294933 | Hong et al. | Oct 2016 | A1 |
20160294935 | Hong et al. | Oct 2016 | A1 |
20160308758 | Li et al. | Oct 2016 | A1 |
20160308961 | Rao | Oct 2016 | A1 |
20160337189 | Liebhart et al. | Nov 2016 | A1 |
20160337249 | Zhang et al. | Nov 2016 | A1 |
20160344565 | Batz et al. | Nov 2016 | A1 |
20160344621 | Roeland et al. | Nov 2016 | A1 |
20160344803 | Batz | Nov 2016 | A1 |
20160352866 | Gupta et al. | Dec 2016 | A1 |
20160366046 | Anantharam et al. | Dec 2016 | A1 |
20160373364 | Yokota | Dec 2016 | A1 |
20160378537 | Zou | Dec 2016 | A1 |
20170005920 | Previdi et al. | Jan 2017 | A1 |
20170005923 | Babakian | Jan 2017 | A1 |
20170005988 | Bansal et al. | Jan 2017 | A1 |
20170019329 | Kozat et al. | Jan 2017 | A1 |
20170019341 | Huang | Jan 2017 | A1 |
20170026417 | Ermagan et al. | Jan 2017 | A1 |
20170033939 | Bragg et al. | Feb 2017 | A1 |
20170063683 | Li et al. | Mar 2017 | A1 |
20170063928 | Jain et al. | Mar 2017 | A1 |
20170064048 | Pettit et al. | Mar 2017 | A1 |
20170064749 | Jain et al. | Mar 2017 | A1 |
20170078176 | Lakshmikantha et al. | Mar 2017 | A1 |
20170078961 | Rabii et al. | Mar 2017 | A1 |
20170093698 | Farmanbar | Mar 2017 | A1 |
20170099194 | Wei | Apr 2017 | A1 |
20170126497 | Dubey et al. | May 2017 | A1 |
20170126522 | McCann et al. | May 2017 | A1 |
20170134538 | Mahkonen et al. | May 2017 | A1 |
20170142012 | Thakkar et al. | May 2017 | A1 |
20170147399 | Cropper et al. | May 2017 | A1 |
20170149582 | Cohn et al. | May 2017 | A1 |
20170149675 | Yang | May 2017 | A1 |
20170149680 | Liu et al. | May 2017 | A1 |
20170163531 | Kumar et al. | Jun 2017 | A1 |
20170163724 | Puri et al. | Jun 2017 | A1 |
20170195255 | Pham et al. | Jul 2017 | A1 |
20170208000 | Bosch et al. | Jul 2017 | A1 |
20170208011 | Bosch et al. | Jul 2017 | A1 |
20170208532 | Zhou | Jul 2017 | A1 |
20170214627 | Zhang et al. | Jul 2017 | A1 |
20170230333 | Glazemakers et al. | Aug 2017 | A1 |
20170230467 | Salgueiro et al. | Aug 2017 | A1 |
20170237656 | Gage | Aug 2017 | A1 |
20170250902 | Rasanen et al. | Aug 2017 | A1 |
20170250917 | Ruckstuhl et al. | Aug 2017 | A1 |
20170251065 | Furr | Aug 2017 | A1 |
20170257432 | Fu et al. | Sep 2017 | A1 |
20170264677 | Li | Sep 2017 | A1 |
20170273099 | Zhang et al. | Sep 2017 | A1 |
20170279938 | You et al. | Sep 2017 | A1 |
20170295021 | Gutiérrez et al. | Oct 2017 | A1 |
20170295100 | Hira et al. | Oct 2017 | A1 |
20170310588 | Zuo | Oct 2017 | A1 |
20170310611 | Kumar et al. | Oct 2017 | A1 |
20170317887 | Dwaraki et al. | Nov 2017 | A1 |
20170317926 | Penno et al. | Nov 2017 | A1 |
20170317936 | Swaminathan | Nov 2017 | A1 |
20170317954 | Masurekar et al. | Nov 2017 | A1 |
20170318097 | Drew et al. | Nov 2017 | A1 |
20170324651 | Penno et al. | Nov 2017 | A1 |
20170331672 | Fedyk et al. | Nov 2017 | A1 |
20170339110 | Ni | Nov 2017 | A1 |
20170339600 | Roeland et al. | Nov 2017 | A1 |
20170346764 | Tan et al. | Nov 2017 | A1 |
20170353387 | Kwak et al. | Dec 2017 | A1 |
20170359252 | Kumar et al. | Dec 2017 | A1 |
20170364794 | Mahkonen et al. | Dec 2017 | A1 |
20170366605 | Chang et al. | Dec 2017 | A1 |
20170373990 | Jeuk et al. | Dec 2017 | A1 |
20180006935 | Mutnuru et al. | Jan 2018 | A1 |
20180026911 | Anholt et al. | Jan 2018 | A1 |
20180027101 | Kumar et al. | Jan 2018 | A1 |
20180041524 | Reddy et al. | Feb 2018 | A1 |
20180063018 | Bosch et al. | Mar 2018 | A1 |
20180091420 | Drake et al. | Mar 2018 | A1 |
20180102919 | Hao et al. | Apr 2018 | A1 |
20180102965 | Hari et al. | Apr 2018 | A1 |
20180115471 | Curcio et al. | Apr 2018 | A1 |
20180123950 | Garg et al. | May 2018 | A1 |
20180124061 | Raman et al. | May 2018 | A1 |
20180139098 | Sunavala et al. | May 2018 | A1 |
20180145899 | Rao | May 2018 | A1 |
20180159733 | Poon et al. | Jun 2018 | A1 |
20180159801 | Rajan et al. | Jun 2018 | A1 |
20180159943 | Poon et al. | Jun 2018 | A1 |
20180176177 | Bichot et al. | Jun 2018 | A1 |
20180176294 | Vacaro et al. | Jun 2018 | A1 |
20180183764 | Gunda | Jun 2018 | A1 |
20180184281 | Tamagawa et al. | Jun 2018 | A1 |
20180191600 | Hecker et al. | Jul 2018 | A1 |
20180198692 | Ansari et al. | Jul 2018 | A1 |
20180198705 | Wang et al. | Jul 2018 | A1 |
20180198791 | Desai et al. | Jul 2018 | A1 |
20180205637 | Li | Jul 2018 | A1 |
20180213040 | Pak et al. | Jul 2018 | A1 |
20180219762 | Wang et al. | Aug 2018 | A1 |
20180227216 | Hughes | Aug 2018 | A1 |
20180234360 | Narayana et al. | Aug 2018 | A1 |
20180248713 | Zanier et al. | Aug 2018 | A1 |
20180248755 | Hecker et al. | Aug 2018 | A1 |
20180248986 | Dalal | Aug 2018 | A1 |
20180262427 | Jain et al. | Sep 2018 | A1 |
20180262434 | Koponen et al. | Sep 2018 | A1 |
20180278530 | Connor et al. | Sep 2018 | A1 |
20180295053 | Leung et al. | Oct 2018 | A1 |
20180302242 | Hao et al. | Oct 2018 | A1 |
20180337849 | Sharma et al. | Nov 2018 | A1 |
20180349212 | Liu et al. | Dec 2018 | A1 |
20180351874 | Abhigyan et al. | Dec 2018 | A1 |
20190020580 | Boutros et al. | Jan 2019 | A1 |
20190020600 | Zhang et al. | Jan 2019 | A1 |
20190020684 | Qian et al. | Jan 2019 | A1 |
20190028384 | Penno et al. | Jan 2019 | A1 |
20190036819 | Kancherla et al. | Jan 2019 | A1 |
20190068500 | Hira | Feb 2019 | A1 |
20190089679 | Kahalon et al. | Mar 2019 | A1 |
20190097838 | Sahoo et al. | Mar 2019 | A1 |
20190124096 | Ahuja et al. | Apr 2019 | A1 |
20190132220 | Boutros et al. | May 2019 | A1 |
20190132221 | Boutros et al. | May 2019 | A1 |
20190140863 | Nainar | May 2019 | A1 |
20190140947 | Zhuang et al. | May 2019 | A1 |
20190140950 | Zhuang et al. | May 2019 | A1 |
20190149512 | Sevinc et al. | May 2019 | A1 |
20190149516 | Rajahalme et al. | May 2019 | A1 |
20190149518 | Sevinc et al. | May 2019 | A1 |
20190166045 | Peng et al. | May 2019 | A1 |
20190173778 | Faseela et al. | Jun 2019 | A1 |
20190173850 | Jain et al. | Jun 2019 | A1 |
20190173851 | Jain et al. | Jun 2019 | A1 |
20190229937 | Nagarajan et al. | Jul 2019 | A1 |
20190230126 | Kumar et al. | Jul 2019 | A1 |
20190238363 | Boutros et al. | Aug 2019 | A1 |
20190238364 | Boutros et al. | Aug 2019 | A1 |
20190268384 | Hu | Aug 2019 | A1 |
20190288947 | Jain et al. | Sep 2019 | A1 |
20190306036 | Boutros et al. | Oct 2019 | A1 |
20190306086 | Boutros et al. | Oct 2019 | A1 |
20190342175 | Wan et al. | Nov 2019 | A1 |
20190379578 | Mishra et al. | Dec 2019 | A1 |
20190379579 | Mishra et al. | Dec 2019 | A1 |
20200007388 | Johnston et al. | Jan 2020 | A1 |
20200036629 | Roeland et al. | Jan 2020 | A1 |
20200059761 | Li et al. | Feb 2020 | A1 |
20200067828 | Liu et al. | Feb 2020 | A1 |
20200076684 | Naveen et al. | Mar 2020 | A1 |
20200076734 | Naveen et al. | Mar 2020 | A1 |
20200084141 | Bengough et al. | Mar 2020 | A1 |
20200136960 | Jeuk et al. | Apr 2020 | A1 |
20200145331 | Bhandari et al. | May 2020 | A1 |
20200162318 | Patil et al. | May 2020 | A1 |
20200195711 | Abhigyan et al. | Jun 2020 | A1 |
20200204492 | Sarva et al. | Jun 2020 | A1 |
20200213366 | Hong et al. | Jul 2020 | A1 |
20200220805 | Dhanabalan | Jul 2020 | A1 |
20200272493 | Lecuyer et al. | Aug 2020 | A1 |
20200272494 | Gokhale et al. | Aug 2020 | A1 |
20200272495 | Rolando et al. | Aug 2020 | A1 |
20200272496 | Mundaragi et al. | Aug 2020 | A1 |
20200272497 | Kavathia et al. | Aug 2020 | A1 |
20200272498 | Mishra et al. | Aug 2020 | A1 |
20200272500 | Feng et al. | Aug 2020 | A1 |
20200272501 | Chalvadi et al. | Aug 2020 | A1 |
20200274757 | Rolando et al. | Aug 2020 | A1 |
20200274769 | Naveen et al. | Aug 2020 | A1 |
20200274778 | Lecuyer et al. | Aug 2020 | A1 |
20200274779 | Rolando et al. | Aug 2020 | A1 |
20200274795 | Rolando et al. | Aug 2020 | A1 |
20200274801 | Feng et al. | Aug 2020 | A1 |
20200274808 | Mundaragi et al. | Aug 2020 | A1 |
20200274809 | Rolando et al. | Aug 2020 | A1 |
20200274810 | Gokhale et al. | Aug 2020 | A1 |
20200274826 | Mishra et al. | Aug 2020 | A1 |
20200274944 | Naveen et al. | Aug 2020 | A1 |
20200274945 | Rolando et al. | Aug 2020 | A1 |
20200322271 | Jain et al. | Oct 2020 | A1 |
20200344088 | Selvaraj et al. | Oct 2020 | A1 |
20200358696 | Hu et al. | Nov 2020 | A1 |
20200366526 | Boutros et al. | Nov 2020 | A1 |
20200366584 | Boutros et al. | Nov 2020 | A1 |
20200382412 | Chandrappa et al. | Dec 2020 | A1 |
20200382420 | Suryanarayana et al. | Dec 2020 | A1 |
20200389401 | Enguehard et al. | Dec 2020 | A1 |
20210029088 | Mayya et al. | Jan 2021 | A1 |
20210044502 | Boutros et al. | Feb 2021 | A1 |
20210120080 | Mishra et al. | Apr 2021 | A1 |
20210135992 | Tidemann et al. | May 2021 | A1 |
20210136140 | Tidemann et al. | May 2021 | A1 |
20210136141 | Tidemann et al. | May 2021 | A1 |
20210218587 | Mishra et al. | Jul 2021 | A1 |
20210227041 | Sawant et al. | Jul 2021 | A1 |
20210227042 | Sawant et al. | Jul 2021 | A1 |
Number | Date | Country |
---|---|---|
1689369 | Oct 2005 | CN |
101594358 | Dec 2009 | CN |
101729412 | Jun 2010 | CN |
103516807 | Jan 2014 | CN |
103795805 | May 2014 | CN |
104471899 | Mar 2015 | CN |
104521195 | Apr 2015 | CN |
107078950 | Aug 2017 | CN |
107204941 | Sep 2017 | CN |
2426956 | Mar 2012 | EP |
2466985 | Jun 2012 | EP |
3210345 | Aug 2017 | EP |
3300319 | Mar 2018 | EP |
2005311863 | Nov 2005 | JP |
9918534 | Apr 1999 | WO |
2008095010 | Aug 2008 | WO |
2014069978 | May 2014 | WO |
2014182529 | Nov 2014 | WO |
2016053373 | Apr 2016 | WO |
2016054272 | Apr 2016 | WO |
2019084066 | May 2019 | WO |
2019147316 | Aug 2019 | WO |
2020046686 | Mar 2020 | WO |
2020171937 | Aug 2020 | WO |
2021086462 | May 2021 | WO |
Entry |
---|
PCT International Search Report and Written Opinion of Commonly Owned International Patent Application PCT/US2020/016457, dated Jun. 23, 2020, 15 pages, International Searching Authority (EPO). |
Casado, Martin, et al., “Virtualizing the Network Forwarding Plane,” Dec. 2010, 6 pages. |
Karakus, Murat, et al., “Quality of Service (QoS) in Software Defined Networking (SDN): A Survey,” Journal of Network and Computer Applications, Dec. 9, 2016, 19 pages, vol. 80, Elsevier, Ltd. |
Non-Published Commonly Owned U.S. Appl. No. 16/905,909, filed Jun. 18, 2020, 36 pages, Nicira, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/945,675, filed Jul. 31, 2020, 51 pages, Nicira, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/945,868, filed Aug. 1, 2020, 48 pages, Nicira, Inc. |
Author Unknown, “Datagram,” Jun. 22, 2012, 2 pages, retrieved from https://web.archive.org/web/20120622031055/https://en.wikipedia.org/wiki/datagram. |
Author Unknown, “AppLogic Features,” Jul. 2007, 2 pages. 3TERA, Inc. |
Author Unknown, “Enabling Service Chaining on Cisco Nexus 1000V Series,” Month Unknown, 2012, 25 pages, Cisco. |
Dixon, Colin, et al., “An End to the Middle,” Proceedings of the 12th Conference on Hot Topics in Operating Systems, May 2009, 5 pages, USENIX Association, Berkeley, CA, USA. |
Dumitriu, Dan Mihai, et al., U.S. Appl. No. 61/514,990, filed Aug. 4, 2011. |
Greenberg, Albert, et al., “VL2: A Scalable and Flexible Data Center Network,” SIGCOMM '09, Aug. 17-21, 2009, 12 pages, ACM, Barcelona, Spain. |
Guichard, J., et al., “Network Service Chaining Problem Statement,” Network Working Group, Jun. 13, 2013, 14 pages, Cisco Systems, Inc. |
Halpern, J., et al., “Service Function Chaining (SFC) Architecture,” draft-ietf-sfc-architecture-02, Sep. 20, 2014, 26 pages, IETF. |
Joseph, Dilip Anthony, et al., “A Policy-aware Switching Layer for Data Centers,” Jun. 24, 2008, 26 pages, Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA. |
Kumar, S., et al., “Service Function Chaining Use Cases in Data Centers,” draft-ietf-sfc-dc-use-cases-01, Jul. 21, 2014, 23 pages, IETF. |
Liu, W., et al., “Service Function Chaining (SFC) Use Cases,” draft-liu-sfc-use-cases-02, Feb. 13, 2014, 17 pages, IETF. |
Non-Published Commonly Owned U.S. Appl. No. 15/937,615, filed Mar. 27, 2018, 42 pages, Nicira, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 15/937,621, filed Mar. 27, 2018, 42 pages, Nicira, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/005,628, filed Jun. 11, 2018, 44 pages, Nicira, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/005,636, filed Jun. 11, 2018, 45 pages, Nicira, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/120,281, filed Sep. 2, 2018, 5 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/120,283, filed Sep. 2, 2018, 46 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/427,294, filed May 30, 2019, 73 pages, Nicira, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,826, filed Jun. 18, 2019, 125 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,845, filed Jun. 18, 2019, 124 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,884, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,907, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,927, filed Jun. 18, 2019, 99 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,935, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,956, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,964, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,978, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/444,989, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,004, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,016, filed Jun. 18, 2019, 99 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,023, filed Jun. 18, 2019, 99 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,031, filed Jun. 18, 2019, 99 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,035, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,051, filed Jun. 18, 2019, 99 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,058, filed Jun. 18, 2019, 99 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,062, filed Jun. 18, 2019, 98 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 16/445,064, filed Jun. 18, 2019, 99 pages, VMware, Inc. |
Salsano, Stefano, et al., “Generalized Virtual Networking: An Enabler for Service Centric Networking and Network Function Virtualization,” 2014 16th International Telecommunications Network Strategy and Planning Symposium, Sep. 17-19, 2014, 7 pages, IEEE, Funchal, Portugal. |
Sekar, Vyas, et al., “Design and Implementation of a Consolidated Middlebox Architecture,” 9th USENIX Symposium on Networked Systems Design and Implementation, Apr. 25-27, 2012, 14 pages, USENIX, San Jose, CA, USA. |
Sherry, Justine, et al., “Making Middleboxes Someone Else's Problem: Network Processing as a Cloud Service,” In Proc. of SIGCOMM '12, Aug. 13-17, 2012, 12 pages, Helsinki, Finland. |
Non-Published Commonly Owned U.S. Appl. No. 16/816,067, filed Mar. 11, 2020, 55 pages, Nicira, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 17/346,255, filed Jun. 13, 2021, 49 pages, Nicira, Inc. |
Non-Published Commonly Owned Related U.S. Appl. No. 17/352,298 with similar specification, filed Jun. 19, 2021, 132 pages, VMware, Inc. |
Non-Published Commonly Owned U.S. Appl. No. 17/385,809, filed Jul. 26, 2021, 74 pages, Nicira, Inc. |
Lin, Po-Ching, et al., “Balanced Service Chaining in Software-Defined Networks with Network Function Virtualization,” Computer: Research Feature, Nov. 2016, 9 pages, vol. 49, No. 11, IEEE. |
Non-Published Commonly Owned U.S. Appl. No. 17/067,635, filed Oct. 9, 2020, 65 pages, Nicira, Inc. |
Siasi, N., et al., “Container-Based Service Function Chain Mapping,” 2019 SoutheastCon, Apr. 11-14, 2019, 6 pages, IEEE, Huntsville, AL, USA. |
Xiong, Gang, et al., “A Mechanism for Configurable Network Service Chaining and Its Implementation,” KSII Transactions on Internet and Information Systems, Aug. 2016, 27 pages, vol. 10, No. 8, KSII. |
Number | Date | Country | |
---|---|---|---|
20200272499 A1 | Aug 2020 | US |
Number | Date | Country | |
---|---|---|---|
62809464 | Feb 2019 | US |