Specifying and utilizing paths through a network

Description

BACKGROUND

Traffic engineering refers to a process by which a network administrative program defines specific paths through the network for a series of data message flows. The approaches used to date include MPLS (multiprotocol label switching) techniques that add path descriptive information between layers 2 and 3 headers. Because of this location of the path description, MPLS is commonly referred to as a layer 2.5 protocol. The MPLS techniques, and other previous traffic engineering techniques, however do not readily support encapsulating tenant identifiers. Tying these prior solutions to a tenant will require other policies and multiple encapsulations for the overlay and underlay.

BRIEF SUMMARY

For a multi-tenant environment, some embodiments of the invention provide a novel method for (1) embedding a specific path for a tenant's data message flow through a network in tunnel headers encapsulating the data message flow, and then (2) using the embedded path information to direct the data message flow through the network. In some embodiments, the encapsulating tunnel header is placed before (i.e., outside of) the layers 2-4 header of each data message in the flow.

In some embodiments, the method selects the specific path from two or more viable such paths through the network for the data message flow. To select the path from the set of viable paths between the source and destination of the data message flow, the method in some embodiments performs a classification operation on a data message flow. The classification operation in some embodiments can be based on the data message flow's layer 2-4 header values (e.g., five tuple identifier, i.e., protocol, source and destination ports, and source and destination IP addresses) and/or its associated contextual attributes, such as its traffic type (i.e., the type of content carried in the data message flow), QoS (Quality of Service) ratings, layer 7 parameters, process identifiers, user identifiers, group identifiers, etc.

In some of these embodiments, the contextual attributes are captured from the source machine (e.g., virtual machine, container, etc.) of the data message flow, and/or are captured on a host computer on which this source machine executes. When the set of contextual attributes include the flow's traffic type, the method provides a novel way for selecting a traffic-engineered path for a data message flow, embedding information about this path in the tunnel headers of the data message flow, and using this information to direct the data message flow through a network.

After selecting the path for the data message flow, the method in some embodiments embeds the identified path in tunnel headers that it uses to encapsulate the flow's data messages. Also, in some embodiments, the method embeds in the tunnel headers a tenant identifier (e.g., embeds a VNI, virtual network identifier) to specify that the data messages are associated with a particular tenant (e.g., are emanating from a machine of the particular tenant). The method sends the encapsulated messages along a tunnel to a first intervening forwarding element that serves as the first hop along the selected path.

Some embodiments use Generic Network Virtualization Encapsulation (Geneve) tunneling protocol, to carry the path information (e.g., the next hop index and the list of hops) and the tenant identifier. In some embodiments, the tenant identifier is embedded in the Geneve base header, while the path information is embedded in an option TLV (type, length, value) of the Geneve tunnel header. The embedding of the tenant identifier allows data message flows of multiple tenants to use the same tunnel between two hops.

The method in some embodiments identifies the selected path, and embeds this path in the tunnel header, by identifying and embedding a list of network addresses (e.g., destination IP addresses) for the hops (e.g., intervening forwarding elements) along the path. Also, the method in some embodiments embeds a hop index value in the tunnel header that identifies one of the embedded network addresses as the network address of the “next” hop that is used to identify the next hop along the path. As further described below, each hop along the path decrements the hop index value as the hops along the path are embedded in the tunnel header in a reverse order (with the first hop appearing last in the list of hops and the last hop appearing first on the list of hops).

In some embodiments, the hops along the selected path connect to each other through a set of tunnels. Some or all of the hops in some embodiments are software forwarding elements (e.g., software switches or routers) executing on host computers. In some embodiments, one or more of the hops are hardware forwarding elements, such as top-of-rack switches, hardware routers, etc.

Accordingly, after selecting the path for a data message and encapsulating the data message with a tunnel header that contains the network addresses of the hops along the path, the method in some embodiments passes the encapsulated data message along a tunnel to the first hop (e.g., first SFE) along the path after forwarding element connected to the source machine (e.g., after the software forwarding element executing on the same host computer with the source machine).

For each tunnel that connects two hops along the path, the encapsulating tunnel header of each message in some embodiments includes an outer header that includes source and destination network addresses. In some embodiments, the source address of the outer header being the network address (e.g., IP address) of the hop that sends the message along the tunnel and the destination address being the network address of the hop that receives the message along the tunnel. At each particular hop after the source machine's forwarding element, the particular hop inspects information embedded in the tunnel header to determine that the tunnel header identifies it as the next hop. The particular hop makes this determination in some embodiments by extracting the next hop index from the tunnel header of the received message, using this index to retrieve the network address of the next hop, and then determining that this network address is its address.

After determining that the received data message identifies it as the next hop, the particular hop determines whether it is specified as the last hop of the specified path. If so, the particular hop examines the destination network address embedded in the data message's header. This destination network address (e.g., IP address) specifies a machine (e.g., standalone computer, virtual machine or container) connected to the hop. Accordingly, the particular hop provides the decapsulated data message to this machine.

On the other hand, when the particular hop determines that it is not specified as the last hop along the path that was embedded in the tunnel header of the received data message, the particular hop forwards the encapsulated data message along another tunnel to another hop after adjusting the tunnel header parameters. Specifically, to forward the data message to the other hop, the particular hop in some embodiments (1) decrements the next hop index (extracted from the received data message's tunnel header) to identify the network address of the other hop as the network address of the next hop along the path, (2) resolves the next hop's network address to an underlay tunnel that starts at the particular hop and terminates at the next hop, and (3) embeds this decremented hop index value in the tunnel header along with the tenant identifier and the list of network addresses of the hops along the path. In some embodiments, the particular hop embeds in the outer portion of the tunnel header of the data message the network address of the particular hop as the source network address, and the network address of the other hop as the destination network address, of the encapsulated data message.

Instead of adjusting the tunnel header parameters, the particular hop in other embodiments decapsulates (i.e., removes) the tunnel header from the received data message, and re-encapsulates the data message with a new tunnel header. One of ordinary skill will realize that the above-described approach for processing the embedded path information is only one way for processing such information. Other embodiments process this information differently. For instance, other embodiments do not store a next hop index in the tunnel headers, as each intervening hop is configured to select the next hop address (e.g., the first hop address) and to remove its network address from the list of path-hop network addresses embedded in the tunnel before passing the encapsulated data message to the next hop along the path. Also, the above-described methodology is used in some embodiments to embed selected paths in single tenant environments. Thus, one of ordinary skill will realize that some embodiments of the invention are equally applicable to single tenant datacenters.

The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description, the Drawings and the Claims is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description and the Drawing.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.

FIG. 1 illustrates an example of specifying and using a specifically defined path according to some embodiments of the invention.

FIG. 2 illustrates a Geneve tunnel header of some embodiments.

FIG. 3 illustrates a Geneve base header of some embodiments.

FIG. 4 conceptually illustrates a process performed by an ingress hop in some embodiments.

FIG. 5 illustrates a process that a hop along a path performs when it receives a Geneve-encapsulated data message.

FIG. 6 illustrates a host computer that is used in some embodiments to execute the virtual machines, service machines and path processors of some embodiments.

FIG. 7 illustrates an example of how the path processors are managed in some embodiments.

FIG. 8 conceptually illustrates a computer system with which some embodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.

In some embodiments, the method selects the specific path from two or more viable such paths through the network for the data message flow. To select the path from the set of viable paths between the source and destination of the data message flow, the method in some embodiments performs a classification operation on a data message flow. The classification operation in some embodiments can be based on the data message flow's layer 2-4 header values (e.g., five tuple identifier, i.e., protocol, source and destination ports, and source and destination IP addresses) and/or its associated contextual attributes, such as its traffic type (i.e., the type of content carried in the data message flow), QoS ratings, layer 7 parameters, process identifiers, user identifiers, group identifiers, etc. When the set of contextual attributes include the flow's traffic type, the method provides a novel way for selecting a traffic-engineered path for a data message flow, embedding information about this path in the tunnel headers of the data message flow, and using this information to direct the data message flow through a network.

In some embodiments, the hops along the selected path are connected to each other through a set of tunnels. Some or all of the hops in some embodiments include software forwarding elements (SFEs) executing on host computers. Examples of such SFEs include software routers and switches in some embodiments. In some embodiments, one or more of the hops are hardware forwarding elements, such as top-of-rack switches, hardware routers, etc.

After selecting the path for a data message and encapsulating the data message with a tunnel header that contains the network addresses of the hops along the path, the method in some embodiments passes the encapsulated data message along a tunnel to the first hop (e.g., first SFE) along the path after forwarding element connected to the source machine (e.g., after the software forwarding element executing on the same host computer with the source machine). The outer portion of the tunnel header includes source and destination network addresses. At the ingress hop, the source address of the outer portion is the network address (e.g., IP address) of the ingress hop (or a virtual tunnel endpoint (VTEP) used by this hop) that initially sends the message along the tunnel and the destination address is the network address of the first hop (or its VTEP) that receives the message along the tunnel.

As used in this document, data messages refer to a collection of bits in a particular format sent across a network. One of ordinary skill in the art will recognize that the term data message is used in this document to refer to various formatted collections of bits that may be sent across a network, such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Also, as used in this document, references to L2, L3, L4, and L7 layers (or layer 2, layer 3, layer 4, layer 7) are references respectively to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the OSI (Open System Interconnection) layer model.

FIG. 1 illustrates an example of specifying and using a traffic-engineered path according to some embodiments of the invention. In this example, two paths 180 and 185 are specifically specified in a network between two guest virtual machines (GVMs) 102 and 106 executing on two hosts 104 and 108 in a multi-tenant datacenter. The GVMs belong to a tenant that is associated in the datacenter with a particular virtual network identifier VNI. Even though in the example illustrated in FIG. 1 the embedded path information allows the data message flow to take a particular path through one datacenter, one of ordinary skill will realize that in other embodiments this path information is used to specify a path between machines in two or more datacenters of the same enterprise, of the same public cloud provider, and/or of different public cloud providers. Similarly, in some embodiments, this path information can be used to specify a path between machines residing in one or more private datacenters or branch offices and one or more public cloud datacenters.

Each path 180 or 185 starts at an ingress SFE (e.g., software switch or router) 120 executing on the host 104, and terminates at an egress SFE 128 executing on the host 108. These SFEs 120 and 128 serve as the ingress and egress hops for each path. Each path also has one or more intervening hops, with path 180 having intervening hops 122 and 124, and the path 185 having intervening hop 126. Each path is formed by a series of tunnels 150-158 that are established between neighboring hops along the path. In this example, each hop includes a software forwarding element executing on a host computer, although this does not have to be the case. For instance, in some embodiments, one or more of the intervening hops can be top-of-rack switches, spine switches, hardware routers or any other kind of forwarding element.

In the example illustrated in FIG. 1, the path 180 is selected for a data message flow from the GVM 102 to the GVM 106. As shown, the ingress and intervening hops pass the data message along the path through the tunnels 150, 152, and 154 with an encapsulating tunnel header 140 that is placed before (i.e., outside of) the layers 2-4 header 130 of the data message 100. Also, as the encapsulated data message is sent along each tunnel, the tunnel header's outer portion (not shown) identifies the IP addresses of the source and destination endpoints of the tunnel as the source and destination IP addresses of the encapsulated data message.

In some embodiments, the source and destination tunnel endpoints are VTEPs associated with the two hop endpoints of the tunnel. For example, for the tunnel 150, the outer portion of the tunnel header 140 identifies the IP addresses of a VTEP of the ingress hop 120 as the encapsulated data message's source IP address, while identifying the IP addresses of a VTEP of the hop 122 as the encapsulated data message's destination IP address. In the encapsulated data message forwarded by the ingress hop 120 or either intervening hop 122 or 124, the tunnel header 140 includes the tenant's VNI and a path description that specifies the remaining portion of the path at that hop. As shown, the path description in some embodiments is expressed in terms of a list of IP addresses of the remaining hops.

In some embodiments, the ingress hop 120 includes the SFE that receives the data message from the source GVM 102, and a path processing filter that executes on the same host computer 104. As further described below, a hook (e.g., a callback function) is defined in a port of the SFE to intercept the data message sent by the source GVM 102 and to forward the data message to the path-processing filter. This filter then performs a classification operation on the data message, in order to select the path 180 for the data message to its destination GVM 106.

This classification operation selects the path 180 from the group of two candidate paths 180 and 185 that are viable candidate paths that have been specifically defined (e.g., by a controller or a forwarding element) to reach the destination GVM 106. In some embodiments, the classification operation is based on a set of attributes associated with the data message flow. This attribute set in some embodiments just includes the data message flow's layer 2-4 header values. In other embodiments, however, the attribute set includes contextual attributes related to the data message flow, such as the data message flow's traffic type (i.e., the type of content carried in the data message flow), QoS ratings, layer 7 parameters, process identifiers, user identifiers, group identifiers, etc.

When the set of contextual attributes include the flow's traffic type, the ingress hop selects a traffic-engineered path for a data message flow, embeds information about this path in the tunnel headers of the data message flow, and uses this information to direct the data message flow through a network. For instance, based on deep packet inspection performed on host computer 104, a first data message flow between the GVMs 102 and 106 might be identified as carrying video conference data for corporate executives, while a second data message flow between these two GVMs might be identified as carrying chat message data between two non-executives. Based on this contextual data, path 180 is selected for the first message flow as this path might be more optimal for delay or latency, while path 185 is selected for the second message flow as this path might be more optimal for cost.

After selecting the path for the intercepted data message, the ingress hop's path-processing filter in some embodiments embeds the selected path in the tunnel header 140 that it uses to encapsulate the data message 100. As shown, the path-processing filter in some embodiments identifies and embeds the selected path in terms of the IP addresses of the intervening hops 122 and 124 and the egress hop 128. In addition to the IP addresses of these hops, the ingress path-processing filter embeds in the tunnel header a tenant identifier (which in this case is the particular tenant's VNI) to specify that the data message 100 is associated with the particular tenant (e.g., are emanating from a GVM of the particular tenant).

The ingress path-processing filter in some embodiments embeds the network addresses of the endpoints of the first tunnel 150 in the outer portion of the tunnel header 140. This path-processing filter in some embodiments also embeds a next hop value in the tunnel header that identifies the embedded network address of the intervening hop 122 as the network address of the “next” hop. As further described below, subsequent hops along the path use this index in some embodiments to identify the next hop along the path.

After selecting the path for the data message 100 and encapsulating the data message with the tunnel header 140, the ingress hop 120 passes the encapsulated data message along the tunnel 150 to the intervening hop 122. As mentioned above, the outer portion of the tunnel header 140 in some embodiments identifies the IP addresses of a VTEP of the ingress hop 120 as the encapsulated data message's source IP address, while identifying the IP addresses of a VTEP of the first hop 122 as the encapsulated data message's destination IP address.

At each subsequent hop 122, 124 or 128 along the path 180, the hop inspects the tunnel header and determines that the received data message is addressed to it. Each particular hop makes this determination in some embodiments by extracting the hop index from the tunnel, using this index to retrieve the network address of the next hop from the tunnel header, and then determining that address is its network address. After determining that the received data message identifies it as the next hop, the particular hop determines whether it is specified as the last hop of the specified path.

If not, the particular hop forwards the encapsulated data message along another tunnel to another hop after adjusting the tunnel header parameters. Specifically, to forward the data message to the other hop, the particular hop in some embodiments (1) decrements the next hop index (extracted from the received data message's tunnel header) to identify the network address of the other hop as the network address of the next hop along the path, (2) resolves the next hop's network address to an underlay tunnel that starts at the particular hop and terminates at the next hop, and (3) embeds this decremented hop index value in the tunnel header along with the tenant identifier and the list of network addresses of the hops along the path.

FIG. 1 illustrates the adjustment to the next hop index by showing this index as (1) initially referencing the IP address of hop 122 when the encapsulated data message leaves the ingress hop, (2) referencing the IP address of hop 124 when the encapsulated data message leaves the hop 122, and (3) referencing the IP address of hop 128 when the encapsulated data message leave the hop 124. Each particular hop also adjusts the outer portion of the tunnel header so that the network address of the particular hop is the source network address, and the network address of the subsequent hop along the path 180 is the destination network address, of the encapsulated data message. For example, the hop 122 places its IP address as the source IP address of the outer header, and the IP address of hop 124 as the destination IP address in this header. Similarly, the hop 124 places its IP address as the source IP address of the outer header, and the IP address of egress hop 128 as the destination IP address in this header.

Instead of adjusting the tunnel header parameters, the particular hop in other embodiments decapsulates (i.e., removes) the tunnel header from the received data message, and re-encapsulates the data message with a new tunnel header. Also, other embodiments do not store a next hop index in the tunnel headers, as in these embodiments, each intervening hop is configured to select the next hop address (e.g., the first hop address) and to remove its network address from the list of path-hop network addresses embedded in the tunnel before passing the encapsulated data message to the next hop along the path.

When the encapsulated data message reaches the egress hop 128, this hop determines that it is specified as the last hop along the path that is embedded in the tunnel header of the data message (e.g., determines that the next hop index is its smallest value and it identifies this hop's address). Hence, it removes the tunnel header and examines the destination network address embedded in the data message's header 130. This destination network address (e.g., IP address) is the IP address of GVM 106 connected to it. Accordingly, egress hop provides the decapsulated data message to this machine.

In FIG. 1, the egress hop communicatively connects to the destination machine of the data message without having to go through intervening routing/switching fabric. In other examples, the egress hop connects to the message's destination (e.g., destination computer, virtual machine or container) through an intervening fabric (i.e., through intervening routers/switches). In these cases, the egress hop forwards the data message through the intervening fabric.

Some embodiments use Generic Network Virtualization Encapsulation (Geneve) tunneling protocol, to carry the path information (e.g., the next hop index and the list of hops along the path) and the tenant identifier. In some embodiments, the path information is embedded in a Geneve option TLV (type, length, value) that is examined by the hops along the path in order to identify the next hop, if any.

FIG. 2 illustrates the Geneve tunnel header 200 used in some embodiments. In some embodiments, this tunnel header 200 is a standard Geneve tunnel header that has an option TLV to store the path information and next hop index. Specifically, as shown, this tunnel header includes an outer header 205, a protocol field 210, a Geneve base header 215, and an option TLV 220. As described above, the outer portion 205 of the tunnel header 200 includes the network addresses (e.g., source and destination IP addresses of the two endpoints of the tunnel, with the source and destination designation depending on the direction of the message flow) in the underlay network that allow the encapsulated data message to traverse the underlay network and reach the tunnel destination endpoint. The protocol field 210 specifies a UDP protocol as well as attributes associated with this protocol.

FIG. 3 illustrates the Geneve base header 215 of some embodiments. As shown, this header in some embodiments is 64-bit wide and stores several tunnel parameters, including the tenant identifier and the option length. Its version field 305 is a 2-bit value that specifies the current Geneve tunnel version being used. Tunnel endpoints that receive messages with unknown versions drop the messages. Non-terminating devices processing Geneve packets with an unknown version number treat them as UDP packets with an unknown payload.

The option length field 310 specifies the length of the options fields, expressed in four byte multiples, not including the eight byte fixed tunnel header. This results in a minimum total Geneve header size of 8 bytes and a maximum of 260 bytes. The start of the payload headers can be found using this offset from the end of the base Geneve header. The O bit 315 specifies whether the data message is an OAM frame that contains a control message instead of a data payload. Endpoints do not forward the payload of this message and transit devices do not attempt to interpret or process it. Since control messages are not frequent, the endpoints typically direct these messages to a high priority control queue. The transit devices do not alter forwarding behavior on the basis of this bit, such as ECMP link selection.

The C bit 320 is a critical option field that when set, indicates that the options are present. One or more options can have the critical bit set. If this bit is set then tunnel endpoints parses the options list to interpret any critical options. When this bit is set, endpoints must drop the encapsulated message if they do not recognize this option. On devices where option parsing is not supported, the frame is dropped on the basis of the “C” bit in the base header. If the bit is not set tunnel endpoints may strip all options using “Opt Len” and forward the decapsulated frame. Transit devices do not drop or modify packets on the basis of this bit.

The first set of reserved bits 325 are 6 bits that must be zero on transmission and ignored on receipt. The protocol type bits 330 are 16 bits that express the type of the protocol data unit appearing after the Geneve header. The VNI bits 335 express a 24-bit identifier for a unique element of a virtual network. In many situations, this may represent an L2 segment, however, the control plane defines the forwarding semantics of decapsulated packets. The VNI may be used as part of ECMP forwarding decisions or may be used as a mechanism to distinguish between overlapping address spaces contained in the encapsulated packet when load balancing across CPUs. The second set of reserved bits 340 are 8 bits that must be zero on transmission and ignored on receipt.

The Geneve base header is followed by the option TLV 220. This TLV includes a four-byte option header and a variable amount of option data interpreted according to the option type. The option data in some embodiments includes the IP addresses of the hops along the path plus a next hop index that identifies one of these IP addresses as the IP address of the next hop along the path.

FIG. 4 conceptually illustrates a process 400 performed by an ingress hop (like hop 120) of some embodiments. This process selects a path for a data message sent by a tenant GVM on a host computer, and then embeds a path hop list, which defines this path, in a Geneve tunnel header that it uses to encapsulate the data message, before sending this encapsulated message along a tunnel to the first hop along the selected path. In some embodiments, the ingress hop includes an SFE (e.g., software switch) and a path processing filter that execute on the host computer (e.g., computer 104) along with the tenant GVM (e.g., GVM 102) that sends the data message. In some embodiments, the GVMs on the host computer communicatively connect with the SFE, which forwards data messages to and from the GVMs.

As shown, the process 400 initially receives (at 405) a data message from the tenant GVM (e.g., GVM 102) executing on its host (e.g., 104). In some embodiments, the process 400 receives this data message from a callback function that is defined in the port of the SFE or VNIC of the GVM. This callback function intercepts data messages sent by the GVM and forwards the data messages to the path-processing filter.

Based on a set of attributes associated with the data message, the process (e.g., the path-processing filter) performs (at 410) a classification operation on the data message, in order to select a defined path through the network to the data message's destination. For at least some of the data messages, this classification operation selects the path from a group of two or more candidate paths that are viable candidate paths that have been specifically defined (e.g., by a controller or a forwarding element) to reach the destination of the data message.

In some embodiments, the classification operation is based on a set of attributes associated with the data message. This attribute set in some embodiments just includes the data message's layer 2-4 header values. In other embodiments, however, the attribute set includes contextual attributes related to the data message, such as the data message's traffic type (i.e., the type of content carried in the data message's flow), QoS ratings, layer 7 parameters, process identifiers, user identifiers, group identifiers, etc. When the set of contextual attributes include the data message's traffic type, the process selects (at 410) a path that is defined for the data message's flow based on the traffic that is carried in this message flow. In some embodiments, the process 400 uses a deep packet inspector on host computer 104 to perform a deep packet inspection of the data message or its flow in order to identify the type of traffic carried in the data message or its flow.

Based on the data message's attributes, the process selects a path for the data message in some embodiments that is optimal from a perspective of one or more metrics, such as cost, delay, latency, loss etc. In some embodiments, the process performs the classification operation (at 410) by comparing one or more attributes of the data message (e.g., the data message's 5-tuple identifier and/or associated contextual attributes) with identifiers of several path-defining rules stored in a rule storage. Each rule specifies a path through a network, which in some embodiments are specified as IP addresses of hops along the path. Hence, after matching the data message's attributes to a set of identifiers of a rule, the process selects the path specified by the rule as the path for the data message.

It should be noted that the process 400 allows two different tenants of a multi-tenant network to optimize their traffic-engineered paths different. For instance, for two data message flows of two tenants that carry the same type of traffic (i.e., the same type of payload content) between the same source and destination computers (on which at least one source or destination machine of each tenant executes), the process 400 can select two different paths. This can occur because one tenant might wish to use a faster, lower-latency path for that type of content, while the other tenant might wish for a less expensive path for that type of content. To implement this, some embodiments define different path-selection rules for different tenants, while other embodiments use the tenant identifier as one of the attributes for performing the path-selection classification operation.

After identifying (at 410) the path for the data message, the process 400 embeds (at 415) the identified path in an option TLV 220 of a Geneve tunnel header 200 that it will use to encapsulate the received data message. As described above, the option TLV stores the path in terms of the IP addresses of the hops along the path. Also, as further described above, the option TLV stores these IP addresses in reverse order, with the first hop's address appearing last in the list while the last hop's address appears first in the list.

In the option TLV, the process also stores (at 415) the next hop value. The process sets this value to identify the last network address in the path hop list, which is the address of the first hop. The next hop index value is used to identify the embedded network address of the “next” hop, and subsequent hops use this index in some embodiments to identify the next hop along the path. At 415, the process 400 also embeds in the base header 215 of the Geneve tunnel header the VNI of the tenant associated with the source GVM that sent the received data message. The embedding of the VNI allows the same set of tunnels to be used to forward data messages for different tenants.

Next, at 420, the process configures other parts of the Geneve tunnel header. For instance, the process stores in the outer portion header 205 of the tunnel header 200 the L2 and L3 network addresses of its VTEP (e.g., a VTEP associated with its SFE) and the VTEP of the first hop as the source and destination network addresses. At 420, the process defines the UDP protocol in the Geneve tunnel header, and sets any other fields that need to be define, per the discussion above and the tunnel deployment configuration being used.

After finalizing the tunnel header configuration (at 420), the process passes (at 425) the encapsulated data message along a Geneve tunnel to the first hop (e.g., hop 122) along the selected path. To send this data message, the process in some embodiments resolves the IP address of the first hop to the tunnel that connects the ingress hop to the first hop. After 420, the process ends.

FIG. 5 illustrates a process 500 that a hop performs when it receives a Geneve-encapsulated data message. In some embodiments, this hop executes on a host computer along with one or more tenant machines (e.g., GVMs). As shown, the process inspects (at 505) the tunnel header and determines that the received data message is addressed to it. The hop makes this determination in some embodiments by extracting the next hop index from the tunnel, using this index to retrieve the network address of the next hop along the path, and then determining that this address is the IP address of its own.

Once the process 500 determines (at 505) that the received data message is addressed to it, the process determines whether it is the last hop in the path embedded in the tunnel header. In some embodiments, this would be the case when the next hop index is at is lowest value (e.g., it is 0 or 1, depending on which is defined as the lowest value), which is indicative that the last hop has been reached. When the process 500 determines (at 510) that it is the last hop in the embedded path, it removes the tunnel header (i.e., decapsulates the encapsulated tunnel header), and forwards the data message to the destination machine identified by the destination network address identified in the received data message's original header (e.g., header 130).

In some embodiments, the process 500 is performed by a path-processing filter that executes on the same host computer as the destination machine. A port of a SFE executing on the host computer in some embodiments passes the encapsulated data message to the path-processing filter. In these embodiments, the path-processing filter passes the decapsulated data message back to the SFE port (e.g., passes a handle to this message to this part) and the SFE port processes, or has the SFE process, this message so that it can be forwarded to the destination machine (e.g., the destination GVM or container) executing on the same host computer. In some embodiments, the egress hop connects to the data message's destination (e.g., destination computer, virtual machine or container) through an intervening fabric (i.e., through intervening routers/switches). In these cases, the egress hop forwards the data message through the intervening fabric to the data message's destination.

When the process 500 determines (at 510) that it is not the last hop in the embedded path, the process adjusts (at 520) the next hop index value in the tunnel header of the received encapsulated message, in order to identify the next hop in the path identified by the embedded path information. As mentioned above, the process adjusts the next hop index value in some embodiments by decrementing this value as the path is specified in the tunnel header by a reverse ordered list of hop IP addresses.

Next, at 525, the process adjusts the outer header values of the tunnel header to identify its own IP address as the source IP address of the encapsulated message, and the IP address of the next hop (now identified by the adjusted next hop index value) as the destination IP address of the encapsulated message. At 530, the process resolves the next hop's address to a tunnel between the current hop and the next hop, and sends the encapsulated data message along this tunnel to the next hop. Before sending the encapsulated data message along the resolved tunnel, the process 500 in some embodiments adjusts (at 530) one or more parameters in the data message's tunnel header (e.g., option, ID or other attributes of the next tunnel) to reflect new attributes and/or identity of the new tunnel along which the data message will get sent. After 530, the process ends.

FIG. 6 illustrates a host computer 600 that is used in some embodiments to execute the GVMs, hop-processing forwarding element and path-processing filter of some embodiments. This host computer also executes service engines that perform context-rich, attribute-based services in a datacenter. The host computer 600 includes several GVM 605, a software forwarding element 610, a context engine 650, service engines 630, context-based service rule storage 640, context-attribute storage 645, an attribute-mapping storage 623, a connection state cache storage 625, a MUX (multiplexer) 627, and a context-engine policy storage 643. The service engines 630 include a path processor 622, a deep packet inspector 624, a load balancer 626, a firewall engine 628, and one or more other service engines (not shown).

In FIG. 6, the GVMs 605 execute on a hypervisor. In some embodiments, the context engine 650, the software forwarding element 610, the service engines 630, the context-based service rule storages 640, the connection state cache storage 625, the context-engine policy storage 643, and the MUX 627 operate in the kernel space of the hypervisor, while the VMs 605 operate in the hypervisor's user space. In other embodiments, one or more service engines and/or the context engine are user space modules (e.g., are service VMs).

In some embodiments, the GVMs 605 are tenant machines that serve as data end points for some or all tenant data message flows in the datacenter. Examples of such machines include webservers, application servers, database servers, etc. In some cases, all the VMs belong to one entity, e.g., an enterprise that operates the host. In other cases, the host 600 operates in a multi-tenant environment (e.g., in a multi-tenant data center), and different VMs 605 may belong to one tenant or to multiple tenants.

Each VM 605 includes a GI agent 648 that communicates with the context engine 650 to provide context attribute sets to this engine, and to receive instructions and queries from this engine. The communications between the context engine 650 and the GI agents 648 are relayed through the MUX 627. One example of such a mux is the mux that is used by the Endpoint Security (EPSec) platform of ESX hypervisors of VMware, Inc. In some embodiments, the attributes collected by the context engine 650 from the GI agents 650 include a rich group of parameters (e.g., layer 7 parameters, process identifiers, user identifiers, group identifiers, etc.). U.S. patent application Ser. No. 15/650,251, filed Jul. 14, 2017, now issued as U.S. Pat. No. 10,802,857, further describes the capturing and use of these contextual attributes through the GI agent 648. The U.S. patent application Ser. No. 15/650,251, now issued as U.S. Pat. No. 10,802,857, is incorporated herein by reference.

As shown, each GVM 605 includes a virtual network interface card (VNIC) 655 in some embodiments. Each VNIC is responsible for exchanging messages between its VM and the SFE 610. Each VNIC connects to a particular port 660 of the SFE 610. The SFE 610 also connects to a physical network interface card (NIC) (not shown) of the host. In some embodiments, the VNICs are software abstractions created by the hypervisor of one or more physical NICs (PNICs) of the host.

In some embodiments, the SFE 610 maintains a single port 660 for each VNIC of each VM. The SFE 610 connects to the host PNIC (through a NIC driver (not shown)) to send outgoing messages and to receive incoming messages. In some embodiments, the SFE 610 is defined to include a port 665 that connects to the PNIC's driver to send and receive messages to and from the PNIC. The SFE 610 performs message-processing operations to forward messages that it receives on one of its ports to another one of its ports. For example, in some embodiments, the SFE tries to use data in the message (e.g., data in the message header) to match a message to flow based rules, and upon finding a match, to perform the action specified by the matching rule (e.g., to hand the message to one of its ports 660 or 665, which directs the message to be supplied to a destination VM or to the PNIC).

In some embodiments, the SFE 610 is a software switch, while in other embodiments it is a software router or a combined software switch/router. The SFE 610 in some embodiments implements one or more logical forwarding elements (e.g., logical switches or logical routers) with SFE executing on other hosts in a multi-host environment. A logical forwarding element in some embodiments can span multiple hosts to connect VMs that execute on different hosts but belong to one logical network.

Different logical forwarding elements can be defined to specify different logical networks for different users, and each logical forwarding element can be defined by multiple software forwarding elements on multiple hosts. Each logical forwarding element isolates the traffic of the VMs of one logical network from the VMs of another logical network that is serviced by another logical forwarding element. A logical forwarding element can connect VMs executing on the same host and/or different hosts. In some embodiments, the SFE extracts from a data message a logical network identifier (e.g., a VNI) and a MAC address. The SFE in these embodiments uses the extracted VNI to identify a logical port group, and then uses the MAC address to identify a port within the port group.

Software switches (e.g., software switches of hypervisors) are sometimes referred to as virtual switches because they operate in software and they provide the VMs with shared access to the PNIC(s) of the host. However, in this document, software switches are referred to as physical switches because they are items in the physical world. This terminology also differentiates software switches from logical switches, which are abstractions of the types of connections that are provided by the software switches. There are various mechanisms for creating logical switches from software switches. VXLAN provides one manner for creating such logical switches. The VXLAN standard is described in Mahalingam, Mallik; Dutt, Dinesh G.; et al. (2013 May 8), VXLAN: A Framework for Overlaying Virtualized Layer 6 Networks over Layer 3 Networks, IETF.

The ports of the SFE 610 in some embodiments include one or more function calls to one or more modules that implement special input/output (I/O) operations on incoming and outgoing messages that are received at the ports. Examples of I/O operations that are implemented by the ports 660 include ARP broadcast suppression operations and DHCP broadcast suppression operations, as described in U.S. Pat. No. 9,548,965. Other I/O operations (such as firewall operations, load-balancing operations, network address translation operations, etc.) can be so implemented in some embodiments of the invention. By implementing a stack of such function calls, the ports can implement a chain of I/O operations on incoming and/or outgoing messages in some embodiments. Also, in some embodiments, other modules in the data path (such as the VNICs 655, port 665, etc.) implement the I/O function call operations instead of, or in conjunction with, the ports 660.

In some embodiments, one or more of function calls of the SFE ports 660 can be to one or more service engines 630 that process service rules on the data message flows sent to and received for the GVMs. Some of these service rules are context-based service rules in the context-based service rule storages 640. Each service engine 630 in some embodiments has its own context-based service rule storage 640, attribute-mapping storage 623, and connection state cache storage 625. FIG. 6 presents just one context-based service rule storage 640, attribute-mapping storage 623, and connection state cache storage 625 for all the service engines in order not to obscure the presentation in this figure with unnecessary detail. Also, in some embodiments, each VM can have its own instance of a service engine (e.g., its own instance of path processor 622, deep packet inspector 624, load balancer 626, and firewall engine 628). In other embodiments, one service engine can service data message flows for multiple VMs on a host (e.g., VMs for the same logical network).

To perform its service operation for a data message flow, a service engine 630 in some embodiments tries to match the flow identifier (e.g., the five-tuple identifier) and/or the flow's associated context attribute set to the rule identifiers of its service rules in its context-based service rule storage 640. Specifically, for a service engine 630 to perform its service check operation for a data message flow, the SFE port 660 that calls the service engine supplies a set of attributes of a message that the port receives. In some embodiments, the set of attributes are message identifiers, such as traditional five-tuple identifiers. In some embodiments, one or more of the identifier values can be logical values that are defined for a logical network (e.g., can be IP addresses defined in a logical address space). In other embodiments, all of the identifier values are defined in the physical domains. In still other embodiments, some of the identifier values are defined in the logical domain, while other identifier values are defined in the physical domain.

The service engine in some embodiments then uses the received message's attribute set (e.g., the message's five-tuple identifier) to identify the context attribute set that the service engine has stored for this flow in the attribute-mapping storage 623. The context engine 650 in some embodiments supplies the context attributes for new flows (i.e., new network connection events) to the service engines 630, along with a flow identifier (e.g., a five-tuple identifier). The context-engine policy storage 643 contains the rules that control the operation of the context engine 650. In some embodiments, these policies direct the context engine to generate rules for the service engines or to direct the service engines to generate rules. The service engines 630 in these embodiments store the context attributes that they receive from the context engine in the attribute-mapping storage 623.

In some embodiments, a service engine 630 stores the context attribute set for each new flow with that flow's identifier (e.g., five-tuple identifier) in the attribute-mapping storage. In this manner, the service engine can identify the context attribute set for each new flow that it receives from the SFE ports 660 by searching its attribute-mapping storage 623 for a context record that has a matching flow identifier. The context record with the matching flow identifier includes the context attribute set for this flow.

Some or all of the service engines in some embodiments pull the context attribute sets for a new flow from the context engine. For instance, in some embodiments, a service engine supplies a new flow's five-tuple identifier that it receives from the SFE port 660, to the context engine 650. This engine 650 then examines its attribute storage 645 to identify a set of attributes that is stored for this five-tuple identifier, and then supplies this attribute set (or a subset of it that it obtains by filtering the identified attribute set for the service engine) to the service engine.

Some embodiments implement the pull model by using a service token to encode the attribute set for a new message flow. When notified of a new network connection event, the context engine 650 in some embodiments (1) collects the context attribute set for the new event, (2) filters this set to discard the attributes that are not relevant for performing one or more services on the flow, (3) stores the remaining filtering attribute subset in the attribute storage 645 along with a service token, and (4) provides the service token to the GI agent 648. The GI agent 648 then causes this token to be passed to the service engine(s) in-band (e.g., in a header of the data message that the agent's VM sends to a destination) or out-of-band (i.e., separately from the data messages that the agent's VM sends to a destination).

When the service engine gets the new flow through the SFE port 660, it supplies this flow's service token to the context engine, which uses this service token to identify in its attribute storage 645 the context attributes to supply to the service engine. In the embodiments that the SFE port does not provide this service token to the service engine, the service engine first has to identify the service token by searching its data stores using the flow's identifier before supplying the service token to the context engine.

After identifying the contextual attribute set for a data message flow, the service engine 630 in some embodiments performs its service operation based on service rules that are stored in the context-based service rule storage 640. To perform its service operation, the service engine 630 matches the received attribute subset with corresponding attribute sets that are stored for the service rules. In some embodiments, each service rule in the context-based service rule storage 640 has a rule identifier and an action parameter set.

As mentioned above, the rule identifier of a service rule in some embodiments can be defined in terms of one or more contextual attributes that are not L2-L4 header parameters (e.g., are L7 parameters, user identifiers, group identifiers, process name, loaded module identifiers, consumption parameters, etc.). In some embodiments, a rule identifier can also include L2-L4 header parameters. Also, in some embodiments, one or more parameters in a rule identifier can be specified in terms of an individual value or a wildcard value. Also, in some embodiments, a rule identifier can include a set of individual values or a group identifier, such as a security group identifier, a compute construct identifier, a network construct identifier, etc.

To match a received attribute set with the rules, the service engine compares the received attribute set with the associated identifiers of the service rules stored in the context-based service rule storage 640. Upon identifying a matching rule, the service engine 630 performs a service operation (e.g., a path processing operation, a firewall operation, a load balancing operation, an encryption operation, other middlebox operation, etc.), based on the action parameter set (e.g., based on the path hop list, Allow/Drop parameters, the load balancing criteria, encryption parameters, etc.) of the matching rule.

In some embodiments, the context-based service rule storage 640 is defined in a hierarchical manner to ensure that a message rule check will match a higher priority rule before matching a lower priority rule, when the message's attribute subset matches multiple rules. Also, in some embodiments, the context-based service rule storage 640 contains a default rule that specifies a default action for any message rule check that cannot identify any other service rules; this default rule will be a match for all possible attribute subsets in some embodiments, and ensures that the service rule engine will return an action for all received attribute subsets. In some embodiments, the default rule will specify no service.

Multiple messages can have the same message identifier attribute sets, e.g., when the messages are part of one flow that is associated with one communication session between two machines. Accordingly, after matching a data message with a service rule in the context-based service rule storage 640 based on the message's identified context attribute set, the service engine of some embodiments stores the service rule (or a reference to the service rule) in the connection state cache storage 625, so that it can later use this service rule for subsequent data messages of the same flow.

In some embodiments, the connection state cache storage 625 stores the service rule, or a reference to the service rule, that the service engine 630 identifies for different message identifier sets (e.g., for different five-tuple identifiers that identify different data message flows). In some embodiments, the connection state cache storage 625 stores each service rule, or reference to the service rule, with an identifier (e.g., a flow's five-tuple identifier) that is generated from the matching message identifier set.

Before checking with the context-based service rule storage 640 for a particular message, a service engine 630 of some embodiments checks the connection state cache storage 625 to determine whether this storage has previously identified a service rule for this message's flow. If not, the service engine 630 identifies the contextual attribute set for the message flow, and then checks the context-based service rule storage 640 for a service rule that matches the message's identified attribute set and/or its five-tuple identifier. When the connection state data storage has an entry for the particular message, the service engine performs its service operation (e.g., its path selection operation) based on this service rule's action parameter set (e.g., based on hop list identified by this service rule in the connection cache 625).

At each hop along a path, a hop path processor in some embodiments processes the path information embedded in the data message's tunnel header. On the host computer, the software forwarding element 610 is associated with the VTEP that serves as a Geneve tunnel endpoint in some embodiments. For an ingress hop (e.g., hop 120), the path processor performs the operations of the process 400 of FIG. 4 in some embodiments. For a data message sent by a GVM 605 on its host computer, the path processor 622 selects an engineered path through the network for the data message to its destination based on the data message's attributes.

As mentioned above, these attributes can just include the message's flow identifier (e.g., five tuple identifier) in some cases, just include this message's captured contextual metadata in other cases, or can include both the message's flow identifier and captured contextual metadata. To select a path, the path processor in some embodiments matches the message's flow identifier and/or contextual metadata attributes with the rule identifier attributes of path-specifying rules of the path processor that are stored in the rule storage 640. In some embodiments, each path-specifying rule in the storage 640 specifies a path in terms of a list of IP addresses for the hops along the path.

In some embodiments, the captured context for a data message includes the traffic type for the data message's flow (i.e., the type of content carried in that flow). In some embodiments, the DPI engine 624 identifies this traffic type when an SFE port 660 calls the DPI engine for the data message and/or other data messages in the same flow. The DPI engine in some embodiments provides this traffic type to the context engine 650 to store in the attribute storage 645, so that the context engine 650 can subsequently provide the traffic type to the path processor 622 when this processor asks for this information for the data message (e.g., by providing the message's flow identifier attributes, or service token). In other embodiments, the context engine 650 pushes the traffic type for a flow to the attribute storage 623 (through the path processor 622 or another service engine) so that the path processor can later retrieve the flow's traffic type when processing a data message of the flow.

When the captured contextual attributes include the data message's traffic type, the ingress hop can select between two or more traffic-engineered paths to the data message's destination, when two or more such paths are previously defined and stored in the rule storage 640. For instance, as described above by reference to FIG. 1, path 180 might be more optimal based on delay and/or latency considerations, while path 185 might be more optimal from a cost perspective. Hence, for a first data message flow between the GVMs 102 and 106 that carries video conference data for corporate executives, the path processor matches the data message flow's attributes to a path-specifying rule that identifies path 180 as the flow's path to the GVM 106. On the other hand, for a second data message flow between these two GVMs that carries chat message data between two non-executives, the path processor matches the data message flow's attributes to a path-specifying rule that identifies path 185 as the flow's path to the GVM 106.

After matching the message's attribute set with one of path-specifying rules, the path processor 622 of the ingress node embeds the path hop list of the of path-specifying rule in a Geneve tunnel header, along with other tunnel attributes described above (e.g., outer header parameters, next hop index, etc.). In some embodiments, the path processor 622 has another one of the service engines 630 perform the tunnel encapsulation. After encapsulating the data message with a Geneve tunnel header 200, the path processor returns the data message to the SFE (e.g., provides a handle for the location in memory at which the encapsulated data message is stored) so that the SFE can forward the data message along the Geneve tunnel to the hop associated with the first service machine in the specified path hop list.

When the path processor 622 is performing the operations of a non-ingress hop, the path processor performs the operations of the process 500 of FIG. 5 in some embodiments. For instance, it determines that a tunnel encapsulated data message has a path hop list that identifies the IP address of its SFE VTEP as the address of the next hop.

Once the path processor 622 determines that the received data message is addressed to it, the processor determines whether it is the last hop in the path embedded in the tunnel header. In some embodiments, this would be the case when the next hop index identifies the current hop's address as the first address in the path hop address list (i.e., when the next hop index has its lowest value). When the path processor 622 determines that it is the last hop in the embedded path, it removes the tunnel header (i.e., decapsulates the encapsulated tunnel header), and passes the decapsulated data message back to the SFE port (e.g., passes a handle to this message to this part) and the SFE port processes, or has the SFE process, this message so that it can be forwarded to the destination machine (e.g., the destination GVM or container) executing on the same host computer. In some embodiments, the egress hop connects to the data message's destination (e.g., destination computer, virtual machine or container) through an intervening fabric (i.e., through intervening routers/switches). In these cases, the egress hop forwards the data message through the intervening fabric to the data message's destination.

When the path processor is not the last hop in the embedded path, the path processor adjusts the next hop index value in the tunnel header of the received encapsulated message, in order to identify the next hop in the path identified by the embedded path information. As mentioned above, the path processor adjusts the next hop index value in some embodiments by decrementing this value as the path is specified in the tunnel header by an reverse ordered list of hop IP addresses.

The path processor 622 adjusts the outer header values of the tunnel header to identify its hop's IP address as the source IP address of the encapsulated message, and the IP address of the next hop (now identified by the adjusted next hop index value) as the destination IP address of the encapsulated message. The path processor resolves the next hop's address to a tunnel between the current hop and the next hop. Before sending the encapsulated data message along the resolved tunnel, the path processor in some embodiments adjusts one or more parameters in the data message's tunnel header to reflect the attributes and/or identity of the new tunnel along which the data message will get sent. It then provides the encapsulated data message to its SFE to forward long a tunnel to the next hop.

FIG. 7 illustrates an example of how the path-processing filters 622 are managed in some embodiments. This figure illustrates multiple hosts 600 in a datacenter. As shown, each host includes several service engines 630, a context engine 650, a path-processing filter 622, several GVMs 605, and an SFE 610. It also illustrates a set of controllers 705 for managing the path-processing filters 622 and the service engines 630, GVMs 605, and SFEs 610. The hosts and controllers communicatively connect to each other through a network 710, which can be a local area network, a wide area network, a network of networks (such as the Internet), etc.

The controller set provides a user interface for the administrators to define context-based service rules in terms of contextual attributes, and communicates with the hosts through the network 710 to provide these policies. In some embodiments, the context engines 650 collect contextual attributes that are passed to management servers in the controller set through a network 710 so that these contextual attributes can be used to define policies. The management servers in some embodiments interact with discovery engines executing on the host computers 600 in the datacenter to obtain and refresh inventory of all processes and services that are running on the GVMs on the hosts. The management plane in some embodiments then provides a rule creation interface for allowing administrators to create context-based service rules for the service engines and the path processors 622.

Once the high-level service policies are defined in the management plane, the management plane directly supplies some or all of these policies to the management proxies (not shown) on the hosts 600, and/or indirectly supplies some or all of these policies to these proxies through a set of configuring controllers. In some embodiments, the management proxies publish the received policies as rules to the context-based service rule storages 640. In some embodiments, the proxies transform these policies before publishing them to the context-based service rule storages 640. Also, the context engines 650 on the hosts 600 in some embodiments resolve the policies based on collected contextual attributes, in order to generate rules for the service engines.

In some embodiments, different policies are specified for different data message flows from a source GVM based on different traffic content carried by these flows. For instance, one policy might define a path hop list that use low-latency path for a data message flow between two GVMs that is for a video conference involving the executive staff of a corporation, while another policy might define a path hop list that uses low cost path for a data message flow from the same two GVMs when this message flow pertains to a chat between two non-executives.

Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

FIG. 8 conceptually illustrates a computer system 800 with which some embodiments of the invention are implemented. The computer system 800 can be used to implement any of the above-described hosts, controllers, and managers. As such, it can be used to execute any of the above described processes. This computer system includes various types of non-transitory machine readable media and interfaces for various other types of machine readable media. Computer system 800 includes a bus 805, processing unit(s) 810, a system memory 825, a read-only memory 830, a permanent storage device 835, input devices 840, and output devices 845.

The bus 805 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system 800. For instance, the bus 805 communicatively connects the processing unit(s) 810 with the read-only memory 830, the system memory 825, and the permanent storage device 835.

From these various memory units, the processing unit(s) 810 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only-memory (ROM) 830 stores static data and instructions that are needed by the processing unit(s) 810 and other modules of the computer system. The permanent storage device 835, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 800 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 835.

Other embodiments use a removable storage device (such as a flash drive, etc.) as the permanent storage device. Like the permanent storage device 835, the system memory 825 is a read-and-write memory device. However, unlike storage device 835, the system memory is a volatile read-and-write memory, such a random access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 825, the permanent storage device 835, and/or the read-only memory 830. From these various memory units, the processing unit(s) 810 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.

The bus 805 also connects to the input and output devices 840 and 845. The input devices enable the user to communicate information and select commands to the computer system. The input devices 840 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 845 display images generated by the computer system. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.

Finally, as shown in FIG. 8, bus 805 also couples computer system 800 to a network 865 through a network adapter (not shown). In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of computer system 800 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, and any other optical or magnetic media. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer readable medium,” “computer readable media,” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral or transitory signals.

While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For instance, several figures conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process could be implemented using several sub-processes, or as part of a larger macro process.

The above-described methodology is used in some embodiments to express path information in single tenant environments. Thus, one of ordinary skill will realize that some embodiments of the invention are equally applicable to single tenant datacenters. Conversely, in some embodiments, the above-described methodology is used to carry path information across different datacenters of different datacenter providers when one entity (e.g., one corporation) is a tenant in multiple different datacenters of different providers. In these embodiments, the tenant identifiers that are embedded in the tunnel headers have to be unique across the datacenters, or have to be translated when they traverse from one datacenter to the next. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.

Claims

1. A method of specifying a path for a data message flow through a network to reach a destination, the method comprising: at an ingress forwarding element connected to a source machine for the data message flow: based on a type of content carried by the data message flow, selecting a particular path from a plurality of candidate paths that are viable paths through the network to a destination of the data message flow;encapsulating each data message in the flow with a tunnel header that stores a set of hop identifiers identifying a set of hops along the network that define the selected path through the network; andforwarding the encapsulated data messages of the flow along the network, said set of next hop identifiers processed by forwarding elements in the network to direct the data messages to the destination.
2. The method of claim 1, wherein selecting the particular path comprises: identifying the type of content carried by the data message flow; andfrom the plurality of paths to the destination, selecting the path to the destination by using the identified content type of the data message flow to identify a record that identifies the particular path.
3. The method of claim 1, wherein the set of next hop identifiers comprises a set of forwarding-element identifiers that identify a set of forwarding elements along the selected particular path to the destination.
4. The method of claim 1, wherein the network is a multi-tenant network, the method further comprising storing a tenant identifier in the encapsulating tunnel header to identify a tenant associated with the data message flow.
5. The method of claim 4, wherein the stored tenant identifier allowing multiple different tenants to use a same set of tunnels between a set of forwarding elements through which a set of paths are defined.
6. The method of claim 5, wherein the tunnel header comprises a base tunnel header to store the tenant identifier, and an option header to store the set of hop identifiers.
7. The method of claim 1, wherein the tunnel header encapsulates each data messages by being placed outside of layers 2 and 3 headers of the data message.
8. The method of claim 1, wherein the tunnel header is based on a tunneling protocol that allows the tunnel header to have a variable size by having a variable-sized option field, in which the set of next hop identifiers are stored.
9. The method of claim 1, wherein the selected path traverses through a set of hops based on a particular sequence;an outer portion of each tunnel header comprises a destination network address that is a network address of a forwarding element that is first hop in the particular sequence.
10. The method of claim 9, wherein at each subsequent hop in the selected path that is not the last hop in the selected path, a forwarding element processes a received encapsulated data message by: identifying the next hop identifier in the hop identifier set to identify the next forwarding element along the selected path;identifying a tunnel to the next hop; andadjusting the encapsulating header of the data message for forwarding along the identified tunnel, said adjusting comprising adjusting the outer portion of the tunnel header to include a network address of the next hop as the destination network address.
11. The method of claim 10, wherein the tunnel headers include a next hop index value that identifies the next hop in the selected path, and each subsequent hop that is not the last forwarding element uses the next hop index value to identify the next hop identifier in the selected path.
12. The method of claim 1, wherein the hope identifiers are the network addresses in the network.
13. A non-transitory machine readable medium storing a program which when executed by at least one processing unit specifies a path for a data message flow through a network to reach a destination, the program implementing an ingress forwarding element connected to a source machine for the data message flow, the program comprising sets of instructions for: based on a type of content carried by the data message flow, selecting a particular path from a plurality of candidate paths that are viable paths through the network to a destination of the data message flow;encapsulating each data message in the flow with a tunnel header that stores a set of hop identifiers identifying a set of hops along the network that define the selected path through the network; andforwarding the encapsulated data messages of the flow along the network, said set of next hop identifiers processed by forwarding elements in the network to direct the data messages to the destination.
14. The non-transitory machine readable medium of claim 13, wherein the set of instructions for selecting the particular path comprises sets of instructions for: identifying the type of content carried by the data message flow; andfrom the plurality of paths to the destination, selecting the path to the destination by using the identified content type of the data message flow to identify a record that identifies the particular path.
15. The non-transitory machine readable medium of claim 13, wherein the set of next hop identifiers comprises a set of forwarding-element identifiers that identify a set of forwarding elements along the selected particular path to the destination.
16. The non-transitory machine readable medium of claim 13, wherein the network is a multi-tenant network, the program further comprises a set of instructions for storing a tenant identifier in the encapsulating tunnel header to identify a tenant associated with the data message flow.
17. The non-transitory machine readable medium of claim 16, wherein the stored tenant identifier allows multiple different tenants to use a same set of tunnels between a set of forwarding elements through which a set of paths are defined.
18. The non-transitory machine readable medium of claim 17, wherein the tunnel header comprises a base tunnel header to store the tenant identifier, and an option header to store the set of hop identifiers.
19. The non-transitory machine readable medium of claim 13, wherein the tunnel header encapsulates each data messages by being placed outside of layers 2 and 3 headers of the data message.
20. The non-transitory machine readable medium of claim 13, wherein the tunnel header is based on a tunneling protocol that allows the tunnel header to have a variable size by having a variable-sized option field, in which the set of next hop identifiers are stored.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

This application is a continuation application of U.S. patent application Ser. No. 15/881,663, filed Jan. 26, 2018, now published as U.S. Patent Publication 2019/0238364. U.S. patent application Ser. No. 15/881,663, now published as U.S. Patent Publication 2019/0238364, is incorporated herein by reference.

US Referenced Citations (667)

Number	Name	Date	Kind
6006264	Colby et al.	Dec 1999	A
6104700	Haddock et al.	Aug 2000	A
6154448	Petersen et al.	Nov 2000	A
6772211	Lu et al.	Aug 2004	B2
6779030	Dugan et al.	Aug 2004	B1
6826694	Dutta et al.	Nov 2004	B1
6880089	Bommareddy et al.	Apr 2005	B1
6985956	Luke et al.	Jan 2006	B2
7013389	Srivastava et al.	Mar 2006	B1
7209977	Acharya et al.	Apr 2007	B2
7239639	Cox et al.	Jul 2007	B2
7379465	Aysan et al.	May 2008	B2
7406540	Acharya et al.	Jul 2008	B2
7447775	Zhu et al.	Nov 2008	B1
7480737	Chauffour et al.	Jan 2009	B2
7487250	Siegel	Feb 2009	B2
7649890	Mizutani et al.	Jan 2010	B2
7698458	Liu et al.	Apr 2010	B1
7818452	Matthews et al.	Oct 2010	B2
7898959	Arad	Mar 2011	B1
7948986	Ghosh et al.	May 2011	B1
8078903	Parthasarathy et al.	Dec 2011	B1
8094575	Vadlakonda et al.	Jan 2012	B1
8175863	Ostermeyer et al.	May 2012	B1
8190767	Maufer et al.	May 2012	B1
8201219	Jones	Jun 2012	B2
8223634	Tanaka et al.	Jul 2012	B2
8224885	Doucette et al.	Jul 2012	B1
8230493	Davidson et al.	Jul 2012	B2
8266261	Akagi	Sep 2012	B2
8339959	Moisand et al.	Dec 2012	B1
8451735	Li	May 2013	B2
8484348	Subramanian et al.	Jul 2013	B2
8488577	Macpherson	Jul 2013	B1
8521879	Pena et al.	Aug 2013	B1
8615009	Ramamoorthi et al.	Dec 2013	B1
8707383	Bade et al.	Apr 2014	B2
8743885	Khan et al.	Jun 2014	B2
8804720	Rainovic et al.	Aug 2014	B1
8804746	Wu et al.	Aug 2014	B2
8811412	Shippy	Aug 2014	B2
8830834	Sharma et al.	Sep 2014	B2
8832683	Heim	Sep 2014	B2
8849746	Candea et al.	Sep 2014	B2
8856518	Sridharan et al.	Oct 2014	B2
8862883	Cherukuri et al.	Oct 2014	B2
8868711	Skjolsvold et al.	Oct 2014	B2
8873399	Bothos et al.	Oct 2014	B2
8874789	Zhu	Oct 2014	B1
8892706	Dalal	Nov 2014	B1
8913611	Koponen et al.	Dec 2014	B2
8914406	Haugsnes et al.	Dec 2014	B1
8966024	Koponen et al.	Feb 2015	B2
8966029	Zhang et al.	Feb 2015	B2
8971345	McCanne et al.	Mar 2015	B1
8989192	Foo et al.	Mar 2015	B2
8996610	Sureshchandra et al.	Mar 2015	B1
9009289	Jacob	Apr 2015	B1
9015823	Koponen et al.	Apr 2015	B2
9094464	Scharber et al.	Jul 2015	B1
9104497	Mortazavi	Aug 2015	B2
9148367	Kandaswamy et al.	Sep 2015	B2
9172603	Padmanabhan et al.	Oct 2015	B2
9178709	Higashida et al.	Nov 2015	B2
9191293	Iovene et al.	Nov 2015	B2
9195491	Zhang et al.	Nov 2015	B2
9203748	Jiang et al.	Dec 2015	B2
9225638	Jain et al.	Dec 2015	B2
9225659	McCanne et al.	Dec 2015	B2
9232342	Seed et al.	Jan 2016	B2
9258742	Pianigiani et al.	Feb 2016	B1
9264313	Manuguri et al.	Feb 2016	B1
9277412	Freda et al.	Mar 2016	B2
9397946	Yadav	Jul 2016	B1
9407540	Kumar et al.	Aug 2016	B2
9407599	Koponen et al.	Aug 2016	B2
9419897	Cherian et al.	Aug 2016	B2
9479358	Klosowski et al.	Oct 2016	B2
9503530	Niedzielski	Nov 2016	B1
9531590	Jain et al.	Dec 2016	B2
9577845	Thakkar et al.	Feb 2017	B2
9602380	Strassner	Mar 2017	B2
9660905	Dunbar et al.	May 2017	B2
9686192	Sengupta et al.	Jun 2017	B2
9686200	Pettit et al.	Jun 2017	B2
9705702	Foo et al.	Jul 2017	B2
9705775	Zhang et al.	Jul 2017	B2
9749229	Previdi et al.	Aug 2017	B2
9755898	Jain et al.	Sep 2017	B2
9755971	Wang et al.	Sep 2017	B2
9774537	Jain et al.	Sep 2017	B2
9787605	Zhang et al.	Oct 2017	B2
9804797	Ng et al.	Oct 2017	B1
9825810	Jain et al.	Nov 2017	B2
9860079	Cohn et al.	Jan 2018	B2
9900410	Dalal	Feb 2018	B2
9935827	Jain et al.	Apr 2018	B2
9979641	Jain et al.	May 2018	B2
9985896	Koponen et al.	May 2018	B2
10013276	Fahs et al.	Jul 2018	B2
10042722	Chigurupati et al.	Aug 2018	B1
10075470	Vaidya et al.	Sep 2018	B2
10079779	Zhang et al.	Sep 2018	B2
10084703	Kumar et al.	Sep 2018	B2
10089127	Padmanabhan et al.	Oct 2018	B2
10091276	Bloomquist et al.	Oct 2018	B2
10104169	Moniz et al.	Oct 2018	B1
10129077	Jain et al.	Nov 2018	B2
10129180	Zhang et al.	Nov 2018	B2
10135636	Jiang et al.	Nov 2018	B2
10135737	Jain et al.	Nov 2018	B2
10158573	Lee et al.	Dec 2018	B1
10187306	Nainar et al.	Jan 2019	B2
10200493	Bendapudi et al.	Feb 2019	B2
10212071	Kancherla et al.	Feb 2019	B2
10225137	Jain et al.	Mar 2019	B2
10237379	Kumar et al.	Mar 2019	B2
10250501	Ni	Apr 2019	B2
10257095	Jain et al.	Apr 2019	B2
10284390	Kumar et al.	May 2019	B2
10320679	Jain et al.	Jun 2019	B2
10333822	Jeuk et al.	Jun 2019	B1
10341233	Jain et al.	Jul 2019	B2
10341427	Jalan et al.	Jul 2019	B2
10375155	Cai et al.	Aug 2019	B1
10397275	Jain et al.	Aug 2019	B2
10514941	Zhang et al.	Dec 2019	B2
10516568	Jain et al.	Dec 2019	B2
10547692	Salgueiro et al.	Jan 2020	B2
10554484	Chanda et al.	Feb 2020	B2
10594743	Hong et al.	Mar 2020	B2
10609091	Hong et al.	Mar 2020	B2
10623309	Gampel et al.	Apr 2020	B1
10637750	Bollineni et al.	Apr 2020	B1
10645060	Ao et al.	May 2020	B2
10659252	Boutros et al.	May 2020	B2
10693782	Jain et al.	Jun 2020	B2
10708229	Sevinc et al.	Jul 2020	B2
10728174	Boutros et al.	Jul 2020	B2
10742544	Roeland et al.	Aug 2020	B2
10757077	Rajahalme et al.	Aug 2020	B2
10797910	Boutros et al.	Oct 2020	B2
10797966	Boutros et al.	Oct 2020	B2
10805181	Boutros et al.	Oct 2020	B2
10805192	Boutros et al.	Oct 2020	B2
10812378	Nainar et al.	Oct 2020	B2
10853111	Gupta et al.	Dec 2020	B1
10929171	Gokhale et al.	Feb 2021	B2
10938716	Chin et al.	Mar 2021	B1
10944673	Naveen et al.	Mar 2021	B2
10949244	Naveen et al.	Mar 2021	B2
11003482	Rolando et al.	May 2021	B2
11012420	Sevinc et al.	May 2021	B2
11036538	Lecuyer et al.	Jun 2021	B2
11038782	Boutros et al.	Jun 2021	B2
11042397	Mishra et al.	Jun 2021	B2
11074097	Naveen et al.	Jul 2021	B2
11075839	Zhuang et al.	Jul 2021	B2
11075842	Jain et al.	Jul 2021	B2
11086654	Rolando et al.	Aug 2021	B2
11119804	Gokhale et al.	Sep 2021	B2
20020078370	Tahan	Jun 2002	A1
20020097724	Halme et al.	Jul 2002	A1
20020194350	Lu et al.	Dec 2002	A1
20030065711	Acharya et al.	Apr 2003	A1
20030093481	Mitchell et al.	May 2003	A1
20030097429	Wu et al.	May 2003	A1
20030105812	Flowers et al.	Jun 2003	A1
20030188026	Denton	Oct 2003	A1
20030236813	Abjanic	Dec 2003	A1
20040066769	Ahmavaara et al.	Apr 2004	A1
20040210670	Anerousis et al.	Oct 2004	A1
20040215703	Song et al.	Oct 2004	A1
20050021713	Dugan et al.	Jan 2005	A1
20050089327	Ovadia et al.	Apr 2005	A1
20050091396	Nilakantan et al.	Apr 2005	A1
20050114429	Caccavale	May 2005	A1
20050114648	Akundi et al.	May 2005	A1
20050132030	Hopen et al.	Jun 2005	A1
20050198200	Subramanian et al.	Sep 2005	A1
20050249199	Albert et al.	Nov 2005	A1
20060069776	Shim et al.	Mar 2006	A1
20060112297	Davidson	May 2006	A1
20060130133	Andreev et al.	Jun 2006	A1
20060155862	Kathi et al.	Jul 2006	A1
20060195896	Fulp et al.	Aug 2006	A1
20060233155	Srivastava	Oct 2006	A1
20070061492	Riel	Mar 2007	A1
20070121615	Weill et al.	May 2007	A1
20070214282	Sen	Sep 2007	A1
20070248091	Khalid et al.	Oct 2007	A1
20070260750	Feied et al.	Nov 2007	A1
20070288615	Keohane et al.	Dec 2007	A1
20070291773	Khan et al.	Dec 2007	A1
20080005293	Bhargava et al.	Jan 2008	A1
20080031263	Ervin et al.	Feb 2008	A1
20080046400	Shi et al.	Feb 2008	A1
20080049614	Briscoe et al.	Feb 2008	A1
20080049619	Twiss	Feb 2008	A1
20080049786	Ram et al.	Feb 2008	A1
20080072305	Casado et al.	Mar 2008	A1
20080084819	Parizhsky et al.	Apr 2008	A1
20080095153	Fukunaga et al.	Apr 2008	A1
20080104608	Hyser et al.	May 2008	A1
20080195755	Lu et al.	Aug 2008	A1
20080225714	Denis	Sep 2008	A1
20080239991	Applegate et al.	Oct 2008	A1
20080247396	Hazard	Oct 2008	A1
20080276085	Davidson et al.	Nov 2008	A1
20080279196	Friskney et al.	Nov 2008	A1
20090003349	Havemann et al.	Jan 2009	A1
20090003364	Fendick et al.	Jan 2009	A1
20090003375	Havemann et al.	Jan 2009	A1
20090019135	Eswaran et al.	Jan 2009	A1
20090037713	Khalid et al.	Feb 2009	A1
20090063706	Goldman et al.	Mar 2009	A1
20090129271	Ramankutty et al.	May 2009	A1
20090172666	Yahalom et al.	Jul 2009	A1
20090199268	Ahmavaara et al.	Aug 2009	A1
20090235325	Dimitrakos et al.	Sep 2009	A1
20090238084	Nadeau et al.	Sep 2009	A1
20090249472	Litvin et al.	Oct 2009	A1
20090265467	Peles et al.	Oct 2009	A1
20090271586	Shaath	Oct 2009	A1
20090299791	Blake et al.	Dec 2009	A1
20090300210	Ferris	Dec 2009	A1
20090303880	Maltz et al.	Dec 2009	A1
20090307334	Maltz et al.	Dec 2009	A1
20090327464	Archer et al.	Dec 2009	A1
20100031360	Seshadri et al.	Feb 2010	A1
20100036903	Ahmad et al.	Feb 2010	A1
20100100616	Bryson et al.	Apr 2010	A1
20100131638	Kondamuru	May 2010	A1
20100165985	Sharma et al.	Jul 2010	A1
20100223364	Wei	Sep 2010	A1
20100223621	Joshi et al.	Sep 2010	A1
20100235915	Memon et al.	Sep 2010	A1
20100265824	Chao et al.	Oct 2010	A1
20100281482	Pike et al.	Nov 2010	A1
20100332595	Fullagar et al.	Dec 2010	A1
20110010578	Dominguez et al.	Jan 2011	A1
20110016348	Pace et al.	Jan 2011	A1
20110022695	Dalal et al.	Jan 2011	A1
20110022812	Van Der et al.	Jan 2011	A1
20110035494	Pandey et al.	Feb 2011	A1
20110040893	Karaoguz et al.	Feb 2011	A1
20110055845	Nandagopal et al.	Mar 2011	A1
20110058563	Saraph et al.	Mar 2011	A1
20110090912	Shippy	Apr 2011	A1
20110164504	Bothos et al.	Jul 2011	A1
20110194563	Shen et al.	Aug 2011	A1
20110211463	Matityahu et al.	Sep 2011	A1
20110225293	Rathod	Sep 2011	A1
20110235508	Goel et al.	Sep 2011	A1
20110261811	Battestilli et al.	Oct 2011	A1
20110268118	Schlansker et al.	Nov 2011	A1
20110271007	Wang et al.	Nov 2011	A1
20110276695	Maldaner	Nov 2011	A1
20110283013	Grosser et al.	Nov 2011	A1
20110295991	Aida	Dec 2011	A1
20110317708	Clark	Dec 2011	A1
20120005265	Ushioda et al.	Jan 2012	A1
20120014386	Xiong et al.	Jan 2012	A1
20120023231	Ueno	Jan 2012	A1
20120054266	Kazerani et al.	Mar 2012	A1
20120089664	Igelka	Apr 2012	A1
20120137004	Smith	May 2012	A1
20120140719	Hui et al.	Jun 2012	A1
20120144014	Natham et al.	Jun 2012	A1
20120147894	Mulligan et al.	Jun 2012	A1
20120155266	Patel et al.	Jun 2012	A1
20120176932	Wu et al.	Jul 2012	A1
20120185588	Error	Jul 2012	A1
20120195196	Ghai et al.	Aug 2012	A1
20120207174	Shieh	Aug 2012	A1
20120213074	Goldfarb et al.	Aug 2012	A1
20120230187	Tremblay et al.	Sep 2012	A1
20120239804	Liu et al.	Sep 2012	A1
20120246637	Kreeger et al.	Sep 2012	A1
20120281540	Khan et al.	Nov 2012	A1
20120287789	Aybay et al.	Nov 2012	A1
20120303784	Zisapel et al.	Nov 2012	A1
20120303809	Patel et al.	Nov 2012	A1
20120311568	Jansen	Dec 2012	A1
20120317260	Husain et al.	Dec 2012	A1
20120317570	Dalcher et al.	Dec 2012	A1
20120331188	Riordan et al.	Dec 2012	A1
20130003735	Chao et al.	Jan 2013	A1
20130021942	Bacthu et al.	Jan 2013	A1
20130031544	Sridharan et al.	Jan 2013	A1
20130039218	Narasimhan et al.	Feb 2013	A1
20130044636	Koponen et al.	Feb 2013	A1
20130058346	Sridharan et al.	Mar 2013	A1
20130073743	Ramasamy et al.	Mar 2013	A1
20130100851	Bacthu et al.	Apr 2013	A1
20130125120	Zhang et al.	May 2013	A1
20130136126	Wang et al.	May 2013	A1
20130142048	Gross, IV et al.	Jun 2013	A1
20130148505	Koponen et al.	Jun 2013	A1
20130151661	Koponen et al.	Jun 2013	A1
20130159487	Patel et al.	Jun 2013	A1
20130160024	Shtilman et al.	Jun 2013	A1
20130163594	Sharma et al.	Jun 2013	A1
20130166703	Hammer et al.	Jun 2013	A1
20130170501	Egi et al.	Jul 2013	A1
20130201989	Hu et al.	Aug 2013	A1
20130227097	Yasuda et al.	Aug 2013	A1
20130227550	Weinstein et al.	Aug 2013	A1
20130287026	Davie	Oct 2013	A1
20130291088	Shieh et al.	Oct 2013	A1
20130297798	Arisoylu et al.	Nov 2013	A1
20130301472	Allan	Nov 2013	A1
20130311637	Kamath et al.	Nov 2013	A1
20130318219	Kancherla	Nov 2013	A1
20130332983	Koorevaar et al.	Dec 2013	A1
20130336319	Liu et al.	Dec 2013	A1
20130343174	Guichard et al.	Dec 2013	A1
20130343378	Veteikis et al.	Dec 2013	A1
20140003232	Guichard et al.	Jan 2014	A1
20140003422	Mogul et al.	Jan 2014	A1
20140010085	Kavunder et al.	Jan 2014	A1
20140029447	Schrum, Jr.	Jan 2014	A1
20140046997	Dain et al.	Feb 2014	A1
20140046998	Dain et al.	Feb 2014	A1
20140050223	Foo et al.	Feb 2014	A1
20140052844	Nayak et al.	Feb 2014	A1
20140059204	Nguyen et al.	Feb 2014	A1
20140059544	Koganty et al.	Feb 2014	A1
20140068602	Gember et al.	Mar 2014	A1
20140092738	Grandhi et al.	Apr 2014	A1
20140092906	Kandaswamy et al.	Apr 2014	A1
20140092914	Kondapalli	Apr 2014	A1
20140096183	Jain et al.	Apr 2014	A1
20140101226	Khandekar et al.	Apr 2014	A1
20140101656	Zhu et al.	Apr 2014	A1
20140115578	Cooper et al.	Apr 2014	A1
20140129715	Mortazavi	May 2014	A1
20140149696	Frenkel et al.	May 2014	A1
20140164477	Springer et al.	Jun 2014	A1
20140169168	Jalan et al.	Jun 2014	A1
20140169375	Khan et al.	Jun 2014	A1
20140195666	Dumitriu et al.	Jul 2014	A1
20140207968	Kumar et al.	Jul 2014	A1
20140254374	Janakiraman et al.	Sep 2014	A1
20140254591	Mahadevan et al.	Sep 2014	A1
20140269487	Kalkunte	Sep 2014	A1
20140269717	Thubert et al.	Sep 2014	A1
20140269724	Mehler et al.	Sep 2014	A1
20140280896	Papakostas et al.	Sep 2014	A1
20140281029	Danforth	Sep 2014	A1
20140282526	Basavaiah et al.	Sep 2014	A1
20140301388	Jagadish et al.	Oct 2014	A1
20140304231	Kamath et al.	Oct 2014	A1
20140307744	Dunbar et al.	Oct 2014	A1
20140310391	Sorenson et al.	Oct 2014	A1
20140310418	Sorenson et al.	Oct 2014	A1
20140317677	Vaidya et al.	Oct 2014	A1
20140321459	Kumar et al.	Oct 2014	A1
20140330983	Zisapel et al.	Nov 2014	A1
20140334485	Jain et al.	Nov 2014	A1
20140334488	Guichard et al.	Nov 2014	A1
20140351452	Bosch et al.	Nov 2014	A1
20140362682	Guichard et al.	Dec 2014	A1
20140362705	Pan	Dec 2014	A1
20140369204	Anand et al.	Dec 2014	A1
20140372567	Ganesh et al.	Dec 2014	A1
20140372616	Arisoylu et al.	Dec 2014	A1
20140372702	Subramanyam et al.	Dec 2014	A1
20150003453	Sengupta et al.	Jan 2015	A1
20150003455	Haddad et al.	Jan 2015	A1
20150009995	Gross, IV et al.	Jan 2015	A1
20150016279	Zhang et al.	Jan 2015	A1
20150023354	Li et al.	Jan 2015	A1
20150026345	Ravinoothala et al.	Jan 2015	A1
20150026362	Guichard et al.	Jan 2015	A1
20150030024	Venkataswami et al.	Jan 2015	A1
20150052262	Chanda et al.	Feb 2015	A1
20150052522	Chanda et al.	Feb 2015	A1
20150063102	Mestery et al.	Mar 2015	A1
20150063364	Thakkar et al.	Mar 2015	A1
20150071285	Kumar et al.	Mar 2015	A1
20150071301	Dalal	Mar 2015	A1
20150073967	Katsuyama et al.	Mar 2015	A1
20150078384	Jackson et al.	Mar 2015	A1
20150092564	Aldrin	Apr 2015	A1
20150103645	Shen et al.	Apr 2015	A1
20150103679	Tessmer et al.	Apr 2015	A1
20150103827	Quinn et al.	Apr 2015	A1
20150109901	Tan et al.	Apr 2015	A1
20150124608	Agarwal et al.	May 2015	A1
20150124622	Kovvali et al.	May 2015	A1
20150124840	Bergeron	May 2015	A1
20150138973	Kumar et al.	May 2015	A1
20150139041	Bosch et al.	May 2015	A1
20150146539	Mehta et al.	May 2015	A1
20150156035	Foo et al.	Jun 2015	A1
20150188770	Naiksatam et al.	Jul 2015	A1
20150195197	Yong et al.	Jul 2015	A1
20150213087	Sikri	Jul 2015	A1
20150215819	Bosch et al.	Jul 2015	A1
20150222640	Kumar et al.	Aug 2015	A1
20150236948	Dunbar et al.	Aug 2015	A1
20150237013	Bansal et al.	Aug 2015	A1
20150242197	Alfonso et al.	Aug 2015	A1
20150244617	Nakil et al.	Aug 2015	A1
20150263901	Kumar et al.	Sep 2015	A1
20150263946	Tubaltsev et al.	Sep 2015	A1
20150271102	Antich	Sep 2015	A1
20150280959	Vincent	Oct 2015	A1
20150281089	Marchetti	Oct 2015	A1
20150281098	Pettit et al.	Oct 2015	A1
20150281125	Koponen et al.	Oct 2015	A1
20150281179	Raman et al.	Oct 2015	A1
20150281180	Raman et al.	Oct 2015	A1
20150288671	Chan et al.	Oct 2015	A1
20150288679	Ben-Nun et al.	Oct 2015	A1
20150295831	Kumar et al.	Oct 2015	A1
20150319078	Lee et al.	Nov 2015	A1
20150319096	Yip et al.	Nov 2015	A1
20150358235	Zhang et al.	Dec 2015	A1
20150365322	Shalzkamer et al.	Dec 2015	A1
20150370596	Fahs et al.	Dec 2015	A1
20150372840	Benny et al.	Dec 2015	A1
20150372911	Yabusaki et al.	Dec 2015	A1
20150381493	Bansal et al.	Dec 2015	A1
20150381494	Cherian et al.	Dec 2015	A1
20150381495	Cherian et al.	Dec 2015	A1
20160006654	Fernando et al.	Jan 2016	A1
20160028640	Zhang et al.	Jan 2016	A1
20160043901	Sankar et al.	Feb 2016	A1
20160057050	Ostrom et al.	Feb 2016	A1
20160057687	Horn et al.	Feb 2016	A1
20160065503	Yohe et al.	Mar 2016	A1
20160080253	Wang et al.	Mar 2016	A1
20160087888	Jain et al.	Mar 2016	A1
20160094384	Jain et al.	Mar 2016	A1
20160094389	Jain et al.	Mar 2016	A1
20160094451	Jain et al.	Mar 2016	A1
20160094452	Jain et al.	Mar 2016	A1
20160094453	Jain et al.	Mar 2016	A1
20160094454	Jain et al.	Mar 2016	A1
20160094455	Jain et al.	Mar 2016	A1
20160094456	Jain et al.	Mar 2016	A1
20160094457	Jain et al.	Mar 2016	A1
20160094631	Jain et al.	Mar 2016	A1
20160094632	Jain et al.	Mar 2016	A1
20160094633	Jain et al.	Mar 2016	A1
20160094642	Jain et al.	Mar 2016	A1
20160094643	Jain et al.	Mar 2016	A1
20160094661	Jain et al.	Mar 2016	A1
20160105333	Lenglet et al.	Apr 2016	A1
20160119226	Guichard et al.	Apr 2016	A1
20160127306	Wang et al.	May 2016	A1
20160127564	Sharma et al.	May 2016	A1
20160134528	Lin et al.	May 2016	A1
20160149816	Roach et al.	May 2016	A1
20160149828	Vijayan et al.	May 2016	A1
20160164776	Biancaniello	Jun 2016	A1
20160164787	Roach et al.	Jun 2016	A1
20160164826	Riedel et al.	Jun 2016	A1
20160173373	Guichard et al.	Jun 2016	A1
20160182684	Connor et al.	Jun 2016	A1
20160197831	Foy et al.	Jul 2016	A1
20160197839	Li et al.	Jul 2016	A1
20160205015	Halligan et al.	Jul 2016	A1
20160212048	Kaempfer et al.	Jul 2016	A1
20160212237	Nishijima	Jul 2016	A1
20160226700	Zhang et al.	Aug 2016	A1
20160226754	Zhang et al.	Aug 2016	A1
20160226762	Zhang et al.	Aug 2016	A1
20160248685	Pignataro et al.	Aug 2016	A1
20160277210	Lin et al.	Sep 2016	A1
20160277294	Akiyoshi	Sep 2016	A1
20160294612	Ravinoothala et al.	Oct 2016	A1
20160294933	Hong et al.	Oct 2016	A1
20160294935	Hong et al.	Oct 2016	A1
20160308758	Li	Oct 2016	A1
20160308961	Rao	Oct 2016	A1
20160337189	Liebhart et al.	Nov 2016	A1
20160337249	Zhang et al.	Nov 2016	A1
20160344565	Batz et al.	Nov 2016	A1
20160344621	Roeland et al.	Nov 2016	A1
20160352866	Gupta et al.	Dec 2016	A1
20160366046	Anantharam et al.	Dec 2016	A1
20160373364	Yokota	Dec 2016	A1
20160378537	Zou	Dec 2016	A1
20170005920	Previdi et al.	Jan 2017	A1
20170005923	Babakian	Jan 2017	A1
20170005988	Bansal et al.	Jan 2017	A1
20170019329	Kozat et al.	Jan 2017	A1
20170019331	Yong	Jan 2017	A1
20170019341	Huang et al.	Jan 2017	A1
20170026417	Ermagan et al.	Jan 2017	A1
20170033939	Bragg et al.	Feb 2017	A1
20170063683	Li et al.	Mar 2017	A1
20170063928	Jain et al.	Mar 2017	A1
20170064048	Pettit et al.	Mar 2017	A1
20170064749	Jain et al.	Mar 2017	A1
20170078176	Lakshmikantha et al.	Mar 2017	A1
20170078961	Rabii et al.	Mar 2017	A1
20170093698	Farmanbar	Mar 2017	A1
20170099194	Wei	Apr 2017	A1
20170126497	Dubey et al.	May 2017	A1
20170126522	McCann et al.	May 2017	A1
20170134538	Mahkonen et al.	May 2017	A1
20170142012	Thakkar et al.	May 2017	A1
20170147399	Cropper et al.	May 2017	A1
20170149582	Cohn	May 2017	A1
20170149675	Yang	May 2017	A1
20170149680	Liu et al.	May 2017	A1
20170163531	Kumar et al.	Jun 2017	A1
20170163724	Puri et al.	Jun 2017	A1
20170195255	Pham et al.	Jul 2017	A1
20170208000	Bosch et al.	Jul 2017	A1
20170208011	Bosch et al.	Jul 2017	A1
20170208532	Zhou	Jul 2017	A1
20170214627	Zhang et al.	Jul 2017	A1
20170230333	Glazemakers et al.	Aug 2017	A1
20170230467	Salgueiro et al.	Aug 2017	A1
20170237656	Gage	Aug 2017	A1
20170250902	Rasanen et al.	Aug 2017	A1
20170250917	Ruckstuhl et al.	Aug 2017	A1
20170251065	Furr et al.	Aug 2017	A1
20170257432	Fu et al.	Sep 2017	A1
20170264677	Li	Sep 2017	A1
20170273099	Zhang et al.	Sep 2017	A1
20170279938	You et al.	Sep 2017	A1
20170295021	Gutiérrez et al.	Oct 2017	A1
20170295100	Hira et al.	Oct 2017	A1
20170310588	Zuo	Oct 2017	A1
20170310611	Kumar et al.	Oct 2017	A1
20170317887	Dwaraki et al.	Nov 2017	A1
20170317926	Penno et al.	Nov 2017	A1
20170317936	Swaminathan et al.	Nov 2017	A1
20170317954	Masurekar et al.	Nov 2017	A1
20170318081	Hopen et al.	Nov 2017	A1
20170318097	Drew et al.	Nov 2017	A1
20170324651	Penno et al.	Nov 2017	A1
20170331672	Fedyk et al.	Nov 2017	A1
20170339110	Ni	Nov 2017	A1
20170339600	Roeland et al.	Nov 2017	A1
20170346764	Tan et al.	Nov 2017	A1
20170353387	Kwak et al.	Dec 2017	A1
20170359252	Kumar et al.	Dec 2017	A1
20170364794	Mahkonen	Dec 2017	A1
20170373990	Jeuk et al.	Dec 2017	A1
20180006935	Mutnuru et al.	Jan 2018	A1
20180027101	Kumar et al.	Jan 2018	A1
20180041524	Reddy et al.	Feb 2018	A1
20180063018	Bosch et al.	Mar 2018	A1
20180091420	Drake et al.	Mar 2018	A1
20180102919	Hao et al.	Apr 2018	A1
20180102965	Hari et al.	Apr 2018	A1
20180115471	Curcio et al.	Apr 2018	A1
20180123950	Garg et al.	May 2018	A1
20180124061	Raman et al.	May 2018	A1
20180139098	Sunavala et al.	May 2018	A1
20180145899	Rao	May 2018	A1
20180159733	Poon et al.	Jun 2018	A1
20180159801	Rajan et al.	Jun 2018	A1
20180159943	Poon et al.	Jun 2018	A1
20180176177	Bichot et al.	Jun 2018	A1
20180176294	Vacaro et al.	Jun 2018	A1
20180183764	Gunda	Jun 2018	A1
20180184281	Tamagawa et al.	Jun 2018	A1
20180191600	Hecker et al.	Jul 2018	A1
20180198692	Ansari et al.	Jul 2018	A1
20180198705	Wang et al.	Jul 2018	A1
20180198791	Desai et al.	Jul 2018	A1
20180205637	Li	Jul 2018	A1
20180213040	Pak et al.	Jul 2018	A1
20180219762	Wang et al.	Aug 2018	A1
20180227216	Hughes	Aug 2018	A1
20180234360	Narayana	Aug 2018	A1
20180248713	Zanier et al.	Aug 2018	A1
20180248755	Hecker et al.	Aug 2018	A1
20180248986	Dalal	Aug 2018	A1
20180262427	Jain et al.	Sep 2018	A1
20180262434	Koponen et al.	Sep 2018	A1
20180278530	Connor et al.	Sep 2018	A1
20180295053	Leung et al.	Oct 2018	A1
20180302242	Hao et al.	Oct 2018	A1
20180337849	Sharma et al.	Nov 2018	A1
20180349212	Liu et al.	Dec 2018	A1
20180351874	Abhigyan et al.	Dec 2018	A1
20190020580	Boutros et al.	Jan 2019	A1
20190020600	Zhang et al.	Jan 2019	A1
20190020684	Qian et al.	Jan 2019	A1
20190028384	Penno et al.	Jan 2019	A1
20190036819	Kancherla et al.	Jan 2019	A1
20190068500	Hira	Feb 2019	A1
20190089679	Kahalon et al.	Mar 2019	A1
20190097838	Sahoo et al.	Mar 2019	A1
20190124096	Ahuja et al.	Apr 2019	A1
20190132220	Boutros et al.	May 2019	A1
20190132221	Boutros et al.	May 2019	A1
20190140863	Nainar et al.	May 2019	A1
20190140947	Zhuang et al.	May 2019	A1
20190140950	Zhuang et al.	May 2019	A1
20190149512	Sevinc et al.	May 2019	A1
20190149516	Rajahalme et al.	May 2019	A1
20190149518	Sevinc et al.	May 2019	A1
20190166045	Peng et al.	May 2019	A1
20190173778	Faseela et al.	Jun 2019	A1
20190173850	Jain et al.	Jun 2019	A1
20190173851	Jain et al.	Jun 2019	A1
20190229937	Nagarajan et al.	Jul 2019	A1
20190230126	Kumar et al.	Jul 2019	A1
20190238363	Boutros et al.	Aug 2019	A1
20190238364	Boutros et al.	Aug 2019	A1
20190268384	Hu et al.	Aug 2019	A1
20190288947	Jain et al.	Sep 2019	A1
20190306036	Boutros et al.	Oct 2019	A1
20190306086	Boutros et al.	Oct 2019	A1
20190342175	Wan et al.	Nov 2019	A1
20190379578	Mishra et al.	Dec 2019	A1
20190379579	Mishra et al.	Dec 2019	A1
20200007388	Johnston et al.	Jan 2020	A1
20200036629	Roeland et al.	Jan 2020	A1
20200059761	Li et al.	Feb 2020	A1
20200067828	Liu et al.	Feb 2020	A1
20200076684	Naveen et al.	Mar 2020	A1
20200076734	Naveen et al.	Mar 2020	A1
20200084141	Bengough et al.	Mar 2020	A1
20200136960	Jeuk et al.	Apr 2020	A1
20200145331	Bhandari et al.	May 2020	A1
20200162318	Patil et al.	May 2020	A1
20200204492	Sarva et al.	Jun 2020	A1
20200213366	Hong et al.	Jul 2020	A1
20200220805	Dhanabalan	Jul 2020	A1
20200272493	Lecuyer et al.	Aug 2020	A1
20200272494	Gokhale et al.	Aug 2020	A1
20200272495	Rolando et al.	Aug 2020	A1
20200272496	Mundaragi et al.	Aug 2020	A1
20200272497	Kavathia et al.	Aug 2020	A1
20200272498	Mishra et al.	Aug 2020	A1
20200272499	Feng et al.	Aug 2020	A1
20200272500	Feng et al.	Aug 2020	A1
20200272501	Chalvadi et al.	Aug 2020	A1
20200274757	Rolando et al.	Aug 2020	A1
20200274769	Naveen et al.	Aug 2020	A1
20200274778	Lecuyer et al.	Aug 2020	A1
20200274779	Rolando et al.	Aug 2020	A1
20200274795	Rolando et al.	Aug 2020	A1
20200274801	Feng et al.	Aug 2020	A1
20200274808	Mundaragi et al.	Aug 2020	A1
20200274809	Rolando et al.	Aug 2020	A1
20200274810	Gokhale et al.	Aug 2020	A1
20200274826	Mishra et al.	Aug 2020	A1
20200274944	Naveen et al.	Aug 2020	A1
20200274945	Rolando et al.	Aug 2020	A1
20200322271	Jain et al.	Oct 2020	A1
20200344088	Selvaraj et al.	Oct 2020	A1
20200358696	Hu et al.	Nov 2020	A1
20200366584	Boutros et al.	Nov 2020	A1
20200382412	Chandrappa et al.	Dec 2020	A1
20200382420	Suryanarayana et al.	Dec 2020	A1
20210029088	Mayya et al.	Jan 2021	A1
20210044502	Boutros et al.	Feb 2021	A1
20210120080	Mishra et al.	Apr 2021	A1
20210135992	Tidemann et al.	May 2021	A1
20210136140	Tidemann et al.	May 2021	A1
20210136141	Tidemann et al.	May 2021	A1
20210218587	Mishra et al.	Jul 2021	A1
20210227041	Sawant et al.	Jul 2021	A1
20210227042	Sawant et al.	Jul 2021	A1
20210306240	Boutros et al.	Sep 2021	A1

Foreign Referenced Citations (26)

Number	Date	Country
1689369	Oct 2005	CN
101594358	Dec 2009	CN
101729412	Jun 2010	CN
103516807	Jan 2014	CN
103795805	May 2014	CN
104471899	Mar 2015	CN
104521195	Apr 2015	CN
107078950	Aug 2017	CN
107204941	Sep 2017	CN
2426956	Mar 2012	EP
2466985	Jun 2012	EP
3210345	Aug 2017	EP
3300319	Mar 2018	EP
3673628	Jul 2020	EP
2005311863	Nov 2005	JP
9918534	Apr 1999	WO
2008095010	Aug 2008	WO
2014069978	May 2014	WO
2014182529	Nov 2014	WO
2016053373	Apr 2016	WO
2016054272	Apr 2016	WO
2019084066	May 2019	WO
2019147316	Aug 2019	WO
2020046686	Mar 2020	WO
2020171937	Aug 2020	WO
2021086462	May 2021	WO

Non-Patent Literature Citations (57)

Entry
Author Unknown, “Datagram,” Jun. 22, 2012, 2 pages, retrieved from https://web.archive.org/web/20120622031055/https://en.wikipedia.org/wiki/datagram.
Author Unknown, “AppLogic Features,” Jul. 2007, 2 pages. 3TERA, Inc.
Author Unknown, “Enabling Service Chaining on Cisco Nexus 1000V Series,” Month Unknown, 2012, 25 pages, Cisco.
Casado, Martin, et al., “Virtualizing the Network Forwarding Plane,” Dec. 2010, 6 pages.
Dixon, Colin, et al., “An End to the Middle,” Proceedings of the 12th Conference on Hot Topics in Operating Systems, May 2009, 5 pages, USENIX Association, Berkeley, CA, USA.
Dumitriu, Dan Mihai, et al., (U.S. Appl. No. 61/514,990), filed Aug. 4, 2011, 31 pages.
Greenberg, Albert, et al., “VL2: A Scalable and Flexible Data Center Network,” SIGCOMM '09, Aug. 17-21, 2009, 12 pages, ACM, Barcelona, Spain.
Guichard, J., et al., “Network Service Chaining Problem Statement,” Network Working Group, Jun. 13, 2013, 14 pages, Cisco Systems, Inc.
Halpern, J., et al., “Service Function Chaining (SFC) Architecture,” draft-ietf-sfc-architecture-02, Sep. 20, 2014, 26 pages, IETF.
International Search Report and Written Opinion of Commonly Owned International Patent Application PCT/US2018/057181, dated Jan. 24, 2019, 12 pages, International Searching Authority (EPO).
Joseph, Dilip Anthony, et al., “A Policy-aware Switching Layer for Data Centers,” Jun. 24, 2008, 26 pages, Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA.
Karakus, Murat, et al., “Quality of Service (QoS) in Software Defined Networking (SDN): A Survey,” Journal of Network and Computer Applications, Dec. 9, 2016, 19 pages, vol. 80, Elsevier, Ltd.
Kumar, S., et al., “Service Function Chaining Use Cases in Data Centers,” draft-ietf-sfc-dc-use-cases-01, Jul. 21, 2014, 23 pages, IETF.
Liu, W., et al., “Service Function Chaining (SFC) Use Cases,” draft-liu-sfc-use-cases-02, Feb. 13, 2014, 17 pages, IETF.
Non-Published Commonly Owned U.S. Appl. No. 16/444,826, filed Jun. 18, 2019, 125 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,845, filed Jun. 18, 2019, 124 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,884, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,907, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,927, filed Jun. 18, 2019, 99 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,935, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,956, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,964, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,978, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/444,989, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,004, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,016, filed Jun. 18, 2019, 99 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,023, filed Jun. 18, 2019, 99 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,031, filed Jun. 18, 2019, 99 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,035, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,044, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,051, filed Jun. 18, 2019, 99 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,058, filed Jun. 18, 2019, 99 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,062, filed Jun. 18, 2019, 98 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/445,064, filed Jun. 18, 2019, 99 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/668,477, filed Oct. 30, 2019, 31 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/668,485, filed Oct. 30, 2019, 55 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/668,505, filed Oct. 30, 2019, 39 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/741,544, filed Jan. 13, 2020, 31 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/785,674, filed Feb. 10, 2020, 29 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/843,913, filed Apr. 9, 2020, 119 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/843,919, filed Apr. 9, 2020, 123 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/904,377, filed Jun. 17, 2020, 120 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/904,390, filed Jun. 17, 2020, 121 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/904,399, filed Jun. 17, 2020, 121 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/904,430, filed Jun. 17, 2020, 120 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/904,437, filed Jun. 17, 2020, 121 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/904,442, filed Jun. 17, 2020, 121 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/904,446, filed Jun. 17, 2020, 121 pages, VMware, Inc.
Non-Published Commonly Owned U.S. Appl. No. 16/905,909, filed Jun. 18, 2020, 36 pages, Nicira, Inc.
Salsano, Stefano, et al., “Generalized Virtual Networking: An Enabler for Service Centric Networking and Network Function Virtualization,” 2014 16th International Telecommunications Network Strategy and Planning Symposium, Sep. 17-19, 2014, 7 pages, IEEE, Funchal, Portugal.
Non-Published Commonly Owned U.S. Appl. No. 17/067,635, filed Oct. 9, 2020, 65 pages, Nicira, Inc.
Siasi, N., et al., “Container-Based Service Function Chain Mapping,” 2019 SoutheastCon, Apr. 11-14, 2019, 6 pages, IEEE, Huntsville, AL, USA.
Sekar, Vyas, et al., “Design and Implementation of a Consolidated Middlebox Architecture,” 9th USENIX Symposium on Networked Systems Design and Implementation, Apr. 25-27, 2012, 14 pages, USENIX, San Jose, CA, USA.
Sherry, Justine, et al., “Making Middleboxes Someone Else's Problem: Network Processing as a Cloud Service,” In Proc. of SIGCOMM '12, Aug. 13-17, 2012, 12 pages, Helsinki, Finland.
Lin, Po-Ching, et al., “Balanced Service Chaining in Software-Defined Networks with Network Function Virtualization,” Computer: Research Feature, Nov. 2016, 9 pages, vol. 49, No. 11, IEEE.
Non-Published Commonly Owned U.S. Appl. No. 17/385,809, filed Jul. 26, 2021, 74 pages, Nicira, Inc.
Xiong, Gang, et al., “A Mechanism for Configurable Network Service Chaining and Its Implementation,” KSII Transactions on Internet and Information Systems, Aug. 2016, 27 pages, vol. 10, No. 8, KSII.

Related Publications (1)

	Number	Date	Country
	20200366526 A1	Nov 2020	US

Continuations (1)

	Number	Date	Country
Parent	15881663	Jan 2018	US
Child	16945675		US

Specifying and utilizing paths through a network

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

CPC

International Classifications

Disclaimer

Abstract