Patent Grant
11593148
Many current enterprises have large and sophisticated networks comprising switches, hubs, routers, middleboxes (e.g., firewalls), servers, workstations and other networked devices, which support a variety of connections, applications and systems. The increased sophistication of computer networking, including virtual machine migration, dynamic workloads, multi-tenancy, and customer specific quality of service and security configurations require a better paradigm for network control. Networks have traditionally been managed through low-level configuration of individual network components. Network configurations often depend on the underlying network: for example, blocking a user's access with an access control list (“ACL”) entry requires knowing the user's current IP address. More complicated tasks require more extensive network knowledge: forcing guest users' port 80 traffic to traverse an HTTP proxy requires knowing the current network topology and the location of each guest. This process is of increased difficulty where the network switching elements are shared across multiple users.
In response, there is a growing movement towards a new network control paradigm called Software-Defined Networking (SDN). In the SDN paradigm, a network controller, running on one or more servers in a network, controls, maintains, and implements control logic that governs the forwarding behavior of shared network switching elements on a per user basis. Making network management decisions often requires knowledge of the network state. To facilitate management decision-making, the network controller creates and maintains a view of the network state and provides an application programming interface upon which management applications may access a view of the network state.
Some of the primary goals of maintaining large networks (including both datacenters and enterprise networks) are scalability, mobility, and multi-tenancy. Many approaches taken to address one of these goals results in hampering at least one of the others. For instance, one can easily provide network mobility for virtual machines within an L2 domain, but L2 domains cannot scale to large sizes. Furthermore, retaining user isolation greatly complicates mobility. As such, improved solutions that can satisfy the scalability, mobility, and multi-tenancy goals are needed.
Some embodiments provide a network control system that allows a user to specify a logical network that includes one or more middleboxes as well as logical data path sets. The user specifies (1) a network topology including logical forwarding elements (e.g., logical routers, logical switches) and middlebox locations within the network, (2) routing policies for forwarding traffic to the middleboxes, and (3) configurations for the different middleboxes. The network control system of some embodiments uses a set of network controllers to distribute both flow entries that implement the network topology and the middlebox configurations to host machines on which managed switching elements and distributed middleboxes operate, as well as to centralized middlebox appliances operating outside of the host machines.
The network controllers, in some embodiments, are arranged in a hierarchical manner. A user enters the topology and configuration information into a logical controller, or an input translation controller that passes the information to a logical controller as a set of records. The logical controller communicatively couples to a set of physical controllers, with each physical controller in charge of distributing the configuration data to one or more host machines. That is, each host machine is assigned to a particular physical controller that acts as the master for that host machine. The logical controller identifies which host machines need to receive the configuration, then passes the appropriate information to the physical controllers that manage the identified host machines. Before exporting the records to the physical controllers, the logical controller translates the flow entry data. In some embodiments, the middlebox configuration data is not translated, however.
The physical controllers receive the information, perform additional translation for at least some of the data, and pass the translated data to the host machines (i.e., to the managed switching elements and middleboxes on the host machines). As with the logical controllers, in some embodiments the physical controllers translate the flow entries destined for the managed switches but do not perform any translation on the middlebox configuration data. The physical controllers of some embodiments do, however, generate additional data for the middleboxes. Specifically, because the distributed middlebox applications elements operating on the host machine (e.g., as daemons, or applications) may perform several separate middlebox processes for different tenant networks, the physical controllers assign a slicing identifier to the particular configuration for the middlebox. This slicing identifier is also communicated to the managed switching element, which adds the identifier to packets destined for the middlebox in some embodiments.
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description and the Drawings is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description and the Drawing, but rather are to be defined by the appended claims, because the claimed subject matters can be embodied in other specific forms without departing from the spirit of the subject matters.
The novel features of the invention are set forth in the appended claims. However, for purpose of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Some embodiments provide a network control system that allows a user to specify a logical network that includes one or more middleboxes (e.g., firewalls, load balancers, network address translators, intrusion detection systems (IDS), wide area network (WAN) optimizers, etc.) as well as logical data path sets. The user specifies (1) a network topology including logical forwarding elements (e.g., logical routers, logical switches) and middlebox locations within the network, (2) routing policies for forwarding traffic to the middleboxes, and (3) configurations for the different middleboxes. The network control system of some embodiments uses a set of network controllers to distribute both flow entries that implement the network topology and the middlebox configurations to host machines on which managed switching elements and distributed middleboxes operate, as well as to centralized middlebox appliances operating outside of the host machines.
The network controllers, in some embodiments, are arranged in a hierarchical manner. A user enters the topology and configuration information into a logical controller, or an input translation controller that passes the information to a logical controller as a set of records. The logical controller communicatively couples to a set of physical controllers, with each physical controller in charge of distributing the configuration data to one or more host machines. That is, each host machine is assigned to a particular physical controller that acts as the master for that host machine. The logical controller identifies which host machines need to receive the configuration, then passes the appropriate information to the physical controllers that manage the identified host machines. Before exporting the records to the physical controllers, the logical controller translates the flow entry data. In some embodiments, the middlebox configuration data is not translated, however.
The physical controllers receive the information, perform additional translation for at least some of the data, and pass the translated data to the host machines (i.e., to the managed switching elements and middleboxes on the host machines). As with the logical controllers, in some embodiments the physical controllers translate the flow entries destined for the managed switches but do not perform any translation on the middlebox configuration data. The physical controllers of some embodiments do, however, generate additional data for the middleboxes. Specifically, because the distributed middlebox applications elements operating on the host machine (e.g., as daemons, or applications) may perform several separate middlebox processes for different tenant networks, the physical controllers assign a slicing identifier to the particular configuration for the middlebox. This slicing identifier is also communicated to the managed switching element, which adds the identifier to packets destined for the middlebox in some embodiments.
In addition, a middlebox 140 attaches to the logical router 115. One of ordinary skill in the art will recognize that the network topology 100 represents just one particular logical network topology into which a middlebox may be incorporated. In various embodiments, the middlebox may be located directly between two other components (e.g.,), directly between the external network and logical router (e.g., in order to monitor and process all traffic entering or exiting the logical network), or in other locations in a more complex network. In the architecture shown in
The logical network topology entered by a user (e.g., a network administrator) is distributed, through the network control system, to various physical machines in order to implement the logical network. The second stage of
In addition, each of the host machines includes a managed switching element (“MSE”). The managed switching elements of some embodiments are software forwarding elements that implement logical forwarding elements for one or more logical networks. For instance, the MSEs in the hosts 155-165 include flow entries in forwarding tables that implement the logical forwarding elements of network 100. Specifically, the MSEs on the host machines implement the logical switches 105 and 110, as well as the logical router 115. On the other hand, some embodiments only implement logical switches at a particular node when at least one virtual machine connected to the logical switch is located at the node (i.e., only implementing logical switch 105 and logical router 115 in the MSE at host 155).
The implementation 300 of some embodiments also includes a pool node 340 that connects to the host machines. In some embodiments, the MSEs residing on the host perform first-hop processing. That is, these MSEs are the first forwarding elements a packet reaches after being sent from a virtual machine, and attempt to perform all of the logical switching and routing at this first hop. However, in some cases a particular MSE may not store flow entries containing all of the logical forwarding information for a network, and therefore may not know what to do with a particular packet. In some such embodiments, the MSE sends the packet to a pool node 340 for further processing. These pool nodes are interior managed switching elements which, in some embodiments, store flow entries that encompass a larger portion of the logical network than the edge software switching elements.
Similar to the distribution of the logical switching elements across the hosts on which the virtual machines of network 100 reside, the middlebox 140 is distributed across middlebox elements on these hosts 155-165. In some embodiments, a middlebox module (or set of modules) resides on the host machines (e.g., operating in the hypervisor of the host). As stated, the network control system of some embodiments is used to configure the distributed forwarding elements (MSEs) and middleboxes. Each of the three hosts 155-165 is assigned to a particular physical controller that receives flow entries for the MSEs and configuration information for the middleboxes, performs any necessary translations on the data, and passes the data to the elements on the hosts.
While
In addition, when the MSE on the host sends packets to the middlebox, some embodiments append (e.g., prepend) a slice identifier (or tag) on the packet to identify to which of the several virtual middleboxes the packet is being sent. When multiple middleboxes are implemented on the same middlebox element for a single logical network (e.g., two different load balancers), the slice identifier will need to identify the particular middlebox slice rather than just the logical network to which the packet belongs. Different embodiments may use different slice identifiers for the middleboxes.
In some embodiments, these slice identifiers are assigned by the network control system. Because, for distributed middleboxes such as middlebox 140, the middlebox element will generally only receive packets from the MSE on that host, the slice identifiers can be assigned separately for each middlebox element by the physical controller that manages the particular middlebox element (and MSE). In the case of a centralized middlebox (i.e., a separate physical appliance to which all of the MSEs send packets), a single slice identifier will be used for the virtual middlebox operating on that appliance. In some embodiments, a physical controller that manages the appliance assigns this identifier and then distributes this through the network control system to the other physical controllers, in order for the other physical controllers to pass the information to the MSEs.
The above illustrates examples of the implementation of logical middleboxes in a network of some embodiments. Several more detailed embodiments are described below. Section I describes the network control system of some embodiments for configuring a network in order to implement a logical network that includes a firewall. Section II describes the architecture of a network controller of some embodiments. Next, Section III describes packet processing between two virtual machines when the packet passes through a middlebox. Finally, Section IV describes an electronic system with which some embodiments of the invention are implemented.
I. Network Control System
As described above, some embodiments use a network control system in order to provision middleboxes and managed switching elements for a managed network. In some embodiments, the network control system is a hierarchical set of network controllers, with each level in the hierarchy performing different functions in the provisioning of the managed switching elements and middleboxes.
In some embodiments, the middlebox configuration interface 207 is actually a part of the input translation controller 205; in this figure, the two are shown separately as they receive different inputs and include different APIs for communicating with the user. In some embodiments, each of the controllers in the network control system has the capability to function as an input translation controller, logical controller, and/or physical controller. That is, each controller machine includes the necessary application stack for performing the functions of any of the different controller types, but only one of those application stacks is used at any time. Alternatively, in some embodiments a given controller may only have the functionality to operate as a particular one of the types of controller (e.g., as a physical controller). In addition, different combinations of controllers may run in the same physical machine. For instance, the input translation controller 205, middlebox configuration interface 207 and the logical controller 210 may run in the same computing device, with which a user interacts.
Furthermore, each of the controllers illustrated in
The input translation controller 205 of some embodiments includes an input translation application that translates network configuration information received from a user. For example, a user may specify a network topology such as that shown in
For example, a user might enter a network topology such as that shown in
The middlebox configuration interface 207 receives middlebox configuration input from a user. In some embodiments, each different middlebox (e.g., middleboxes from different providers, different types of middleboxes) may have a different API particular to the middlebox implementation. That is, the different middlebox implementations have different interfaces presented to the user (i.e., the user will have to enter information in different formats for different particular middleboxes). As shown in the middlebox data generation column of
In some embodiments, the middlebox configuration data, as translated by the configuration interface 207, is also a set of records, with each record specifying a particular rule. These records, in some embodiments, are in a similar format to the flow entries propagated to the managed switching elements. In fact, some embodiments use the same applications on the controllers to propagate the firewall configuration records as for the flow entries, and the same table mapping language (e.g., nLog) for the records.
While this figure illustrates the middlebox configuration data being sent to the logical controller, some centralized middleboxes of some embodiments are only accessible through a direct interface with the middlebox device. That is, rather than entering a configuration that is sent to the logical controller and distributed through the network control system, the user enters a configuration directly into the middlebox device. In such a case, the user will still need to enter routing policies to send packets to the middlebox as part of the network topology configuration. In some such embodiments, the network control system will still generate slicing data (i.e., virtualization identifiers) for the middlebox as described below. On the other hand, in some embodiments the user configures the slicing data for the middlebox and either the middlebox or the user provides this information to the network control system. In some embodiments, each logical network is governed by a particular logical controller (e.g., logical controller 210). With respect to the flow generation for the managed switching elements, the logical controller 210 of some embodiments translates the logical control plane received from the input translation controller 205 data into logical forwarding plane data, and the logical forwarding plane data into universal control plane data. In some embodiments, the logical controller application stack includes a control application for performing the first translation and a virtualization application for performing the second translation. Both of these applications, in some embodiments, use a rules engine for mapping a first set of tables into a second set of tables. That is, the different data planes are represented as tables (e.g., nLog tables), and the controller applications use a table mapping engine to translate between the data planes. In some embodiments, both the control application and virtualization application use the same rules engine to perform their translations.
Logical forwarding plane data, in some embodiments, consists of flow entries described at a logical level. For the MAC address B at logical port X, logical forwarding plane data might include a flow entry specifying that if the destination of a packet matches MAC B, forward the packet to port X.
The translation from logical forwarding plane to physical control plane, in some embodiments, adds a layer to the flow entries that enables a managed switching element provisioned with the flow entries to convert packets received at a physical layer port (e.g., a virtual interface) into the logical domain and perform forwarding in this logical domain. That is, while traffic packets are sent and received within the network at the physical layer, the forwarding decisions are made according to the logical network topology entered by the user. The conversion from the logical forwarding plane to the physical control plane enables this aspect of the network in some embodiments.
As shown, the logical controller converts the logical forwarding plane data to a universal physical control plane, while the physical controllers convert the universal physical control plane data to a customized physical control plane. The universal physical control plane data of some embodiments is a data plane that enables the control system of some embodiments to scale even when it contains a large number of managed switching elements (e.g., thousands) to implement a logical data path set. The universal physical control plane abstracts common characteristics of different managed switching elements in order to express physical control plane data without considering differences in the managed switching elements and/or location specifics of the managed switching elements.
For the example noted above (attachment of MAC B to logical port X), the universal physical control plane would involve several flow entries. The first entry states that if a packet matches the particular logical data path set (e.g., based on the packet being received at a particular logical ingress port), and the destination address matches MAC B, then forward the packet to logical port X. This adds the match over the logical data path set (the conversion from a physical port to a logical port) to the forwarding entry that performs its analysis in the logical domain. This flow entry will be the same in the universal and customized physical control planes, in some embodiments.
Additional flows are generated to match a physical ingress port (e.g., a virtual interface of the host machine) to the logical ingress port X (for packets received from MAC A), as well as to match logical port X to the particular egress port of the physical managed switch (for packets sent to MAC A). However, these physical ingress and egress ports are specific to the host machine containing the managed switching element. As such, the universal physical control plane entries include abstract physical ports (i.e., a generic abstraction of a port not specific to any particular physical host machine) to logical ingress ports as well as for mapping logical egress ports to generic physical egress ports.
The middlebox configuration data, on the other hand, is not converted by the logical controller in some embodiments, while in other embodiments the logical controller performs at least a minimal translation of the middlebox configuration data records. As many middlebox packet processing, modification, and analysis rules operate on the IP address (or TCP connection state) of the packets, and the packets sent to the middlebox will have this information exposed (i.e., not encapsulated within the logical port information), the middlebox configuration does not require translation from logical to physical data planes. Thus, the same middlebox configuration data is passed from the middlebox configuration interface 207 to the logical controller 210, and then to the physical controllers 215 and 220.
In order to distribute the physical control plane data, as well as the middlebox configuration data, the logical controller has to identify which of the host machines (and thus which of the physical controllers) need to receive which flow entries and which middlebox configuration information. In some embodiments, the logical controller 210 stores a description of the logical network and of the physical implementation of that physical network. The logical controller receives the one or more middlebox configuration records for a distributed middlebox, and identifies which of the various nodes will need to receive the configuration information.
In some embodiments, the entire middlebox configuration is distributed to middlebox elements at all of the host machines, so the logical controller identifies all of the machines on which at least one virtual machine resides whose packets require use of the firewall. In general, the identified machines are the hosts for all of the virtual machines in a network (e.g., as for the middlebox shown in
Similarly, the logical controller identifies which nodes should receive each flow entry in the physical control plane. For instance, the flow entries implementing the logical switch 105 are distributed to the hosts 155 and 160, but not the host 165 in
Once the logical controller identifies the particular nodes to receive the records, the logical controller identifies the particular physical controllers that manage these particular nodes. In some embodiments, each host machine has an assigned master physical controller. Thus, if the logical controller identifies only first and second hosts as destinations for the configuration data, the physical controllers for these hosts will be identified to receive the data from the logical controller (and other physical controllers will not receive this data). For a centralized middlebox, the logical controller needs only to identify the (single) physical controller that manages the appliance implementing the middlebox. When the centralized middlebox is implemented as a cluster (e.g., as a set of resources, a master-backup cluster, etc.), each of the middlebox appliances in the cluster will receive the configuration data. The middleboxes in the cluster are all managed by a single physical controller in some embodiments, while in other embodiments different physical controllers manage different middleboxes within a cluster.
In order to supply the middlebox configuration data to the hosts, the logical controller of some embodiments pushes the data (using an export module that accesses the output of the table mapping engine in the logical controller) to the physical controllers. In other embodiments, the physical controllers request configuration data (e.g., in response to a signal that the configuration data is available) from the export module of the logical controller.
As stated, each of the physical controllers 215 and 220 is a master of one or more managed switching elements (e.g., located within host machines). In this example, the first physical controller 215 is a master of the managed switching elements at host machines 225 and 230, while the second physical controller 220 is a master of the managed switching element at host machine 235. In some embodiments, a physical controller receives the universal physical control plane data for a logical network and translates this data into customized physical control plane data for the particular managed switches that the physical controller manages that need to receive the data (as the physical controller may also manage additional managed switching elements that do not receive the data for a particular logical network). In other embodiments, the physical controller passes the appropriate universal physical control plane data to the managed switching elements, which includes the ability (e.g., in the form of a chassis controller running on the host machine) to perform the conversion itself.
The universal physical control plane to customized physical control plane translation involves a customization of various data in the flow entries. While the universal physical control plane entries are applicable to any managed switching element because the entries include generic abstractions for any data that is different for different switching elements, the customized physical control plane entries include substituted data specific to the particular managed switching element to which the entry will be sent. For instance, the physical controller customizes the physical layer ports in the universal physical control plane ingress and egress port integration entries to include the actual physical layer ports (e.g., virtual interfaces) of the specific host machines.
As shown in
The customized physical control plane data passed to the managed switching element includes attachment and slicing information to enable the managed switching element to send packets to the middleboxes in some embodiments. This slicing data, as shown by the middlebox slice information generation column of
Essentially, the slicing information is a tag for the managed switching element to add to packets that it sends to the middlebox. The tag indicates to which of the (potentially) several processes being run by the middlebox the packet should be sent. Thus, when the middlebox receives the packet, the tag enables the middlebox to use the appropriate set of packet processing, analysis, modification, etc. rules in order to perform its operations on the packet. Some embodiments, rather than adding slicing information to the packet, either define different ports of the managed switching element for each middlebox instance, and essentially use the ports to slice the traffic destined for the firewall (in the distributed case), or connect to different ports of the centralized appliance to differentiate between the instances (in the centralized case).
In order to send the slicing data to the managed switching element as part of the customized physical control plane data, in some embodiments the physical controller adds flow entries specifying slicing information particular to the middlebox. Specifically, for a particular managed switching element, the flow entry may specify to add the slicing tag for a particular middlebox (which may be, e.g., a VLAN tag or similar tag) to a packet before sending the packet to the particular middlebox based on a match of the port connecting to the middlebox.
The attachment information, in some embodiments, includes flow entries that enable the managed switching element to send packets to the middlebox. In the distributed middlebox case, with the middlebox in the same physical machine as the managed switching element, the middlebox and the managed switching element negotiate a software port abstraction through which packets will be transferred in some embodiments. In some embodiments, the managed switching element (or the middlebox element) pass this information up to the physical controller, enabling the physical controller to use the information in the customized physical control plane data (i.e., using the specific software port for the customized physical control plane entries).
For centralized middleboxes, some embodiments provide tunneling attachment data to both the managed switching element and the middlebox. The middlebox, in some embodiments, will need to know the type of tunnel encapsulation various host machines will use to send packets to the middlebox. In some embodiments, the middlebox has a list of accepted tunneling protocols (e.g., STT, GRE, etc.), and the chosen protocol is coordinated between the managed switching element(s) and the middlebox. The tunneling protocol may be entered by the user as part of the middlebox configuration, or may be automatically determined by the network control system in different embodiments. The physical controller will also add the tunnel encapsulation information to the customized physical control plane flow entries in order for the managed switching element to encapsulate packets properly for sending to the middlebox.
Upon receiving the customized physical control plane data from the physical controller, a managed switching element performs a translation of the customized physical control plane data into physical forwarding plane data. The physical forwarding plane data, in some embodiments, are the flow entries stored within a forwarding table of a switching element (either a physical router or switch or a software switching element) against which the switching element actually matches received packets and performs actions on the packets based on those matches.
The middlebox receives its configuration data from the physical controller, and in some embodiments translates this configuration data. The middlebox configuration data will be received in a particular language to express the packet processing, analysis, modification, etc. rules, through a control plane API of the middlebox. The middlebox (distributed and/or centralized) of some embodiments compiles these rules into more optimized packet classification rules. In some embodiments, this transformation is similar to the physical control plane to physical forwarding plane data translation. When a packet is received by the middlebox, it applies the compiled optimized rules in order to efficiently and quickly perform its operations on the packet.
As shown in
The routing policy 310 is entered by the user in order to indicate which packets the logical router should send to the middlebox. When the middlebox is located on a logical wire between two logical forwarding elements (e.g., between a logical router and a logical switch), then all packets sent over that logical wire will automatically be forwarded to the middlebox. However, for an out-of-band middlebox such as that in network topology 305, the logical router will only send packets to the middlebox when particular policies are specified by the user.
Whereas routers and switches will normally forward packets according to the destination address (e.g., MAC address or IP address) of the packet, policy routing allows forwarding decisions to be made based on other information stored by the packet (e.g., source addresses, a combination of source and destination addresses, etc.). For example, the user might specify that all packets with source IP addresses in a particular subnet, or that have destination IP addresses not matching a particular set of subnets, should be forwarded to the middlebox. In this specific case, the routing policy 310 specified by the user routes traffic with a source IP in subnet A and an ingress context of logical Port L (i.e., coming from the logical switch A) to the middlebox. While the source IP address would be enough to route packets to the middlebox D, the ingress port prevents packets coming back from the middlebox from being sent to the middlebox again (i.e., a never-ending loop). The packets from the middlebox will have a different ingress port and therefore will not be routed by the policy 310.
As shown, the routing policy is sent, as logical control plane data 315, to the logical controller 320 (e.g., from an input translation controller). In this example, the control plane entry states “Send packets with source IP address in subnet A received at ingress Port L to middlebox D at Port K”. This L3 (logical routing) control plane entry combines the routing policy 310 with the network topology of the middlebox being located at Port K. As shown, the logical controller 320 first converts the logical control plane entry 315 to logical forwarding plane entry 325. As described, in some embodiments a table mapping rules engine at the logical controller 320 performs this conversion. That is, the entry 315 is a first database table record, which is mapped via the rules engine to the entry 325. The logical forwarding plane entry 325 is a flow entry in the format of match action, stating “If source IP matches {A} and Ingress match Port L forward to Port K”. Because the network performs forwarding in the logical plane, the flow entry sends packets to a logical port of the logical router.
Next, the logical controller 320 translates the logical forwarding plane entry 320 into a set of universal control plane entries 330-340. The first entry 330 is the forwarding entry in the universal physical control plane, stating “If match L3 C and source IP match {A} and Ingress match Port L→forward to Port K”. This entry adds a match over the L3 router to the forwarding entry, ensuring that a packet acted upon by the flow entry is not part of a different logical network.
In addition, the logical controller adds ingress and egress port integration entries 335 and 340. Because these entries are part of the universal physical control plane, the physical port information in these entries is generic. These entries include an ingress port integration entry 335 that maps packets received via a software port connected to the middlebox in the host machine to the logical ingress Port K. Similarly, the egress port integration entry 340 maps packets forwarded to Port K to the software port connected to the middlebox. As this port may have different specific identifiers in different host machines, at the universal control plane level it is represented using a generic abstraction of such a port. The universal physical control plane will include various other entries, such as ingress mapping to map packets received from a (generic) virtual interface at which VM1 is located to an ingress port on the L2 A logical switch, and a L2 forwarding entry that maps packets with a destination not on the L2 A logical switch to the egress port connected logically to the L3 C logical router.
As with the logical control plane to logical forwarding plane, the logical controller 320 performs the second conversion using a table mapping rules engine. In some embodiments, the first conversion is performed by a control application within the logical controller while the second conversion is performed by a virtualization application. In some such embodiments, these two applications use the same rules engine.
Next, the logical controller 320 identifies which physical controllers in the network control system should receive the universal physical control plane entries. For instance, the ingress and egress port integration entries at the L2 level for a particular virtual machine will only need to be sent to the physical controller that is the master of the node on which the particular virtual machine is hosted. The entries 330 for the L3 router are sent to all of the machines in some embodiments, and therefore the logical controller 320 identifies all of the physical controllers that receive any universal control plane data for the logical network to receive the entries 330-340. On the other hand, in some cases the middlebox may only be implemented at a subset of the nodes (e.g., because the middlebox will only need to process traffic at those nodes), and therefore these entries are only exported to the physical controllers managing nodes in the subset.
Upon receiving the entries 330-340, the physical controller 345 (one of several physical controllers to receive the entries) performs the universal physical control plane to customized physical control plane conversion. As shown, the entry 330 stays the same, as there is no information in the flow entry that requires specification. On the other hand, the generic port abstraction in the ingress and egress port integration entries 335 and 340 are converted into the specific software port of the MSE 350 to which the middlebox connects for entries 355 and 360.
As shown, in some embodiments after negotiating the software port connection with the middlebox, the MSE passes the middlebox attachment port information 365 up to the physical controller 345. The physical controller then uses this information as an input to its table mapping engine that converts the universal physical control plane records into customized physical control plane records. The physical controller passes this information to the MSE, which converts the customized physical control plane entries into physical forwarding plane entries in its forwarding tables 370. These forwarding tables are used by the MSE to match against received packets and perform actions (e.g., forwarding, encapsulation, etc.) on the packets. While not shown here, additional flow entries are generated by the physical controller to add the slicing tag to the packet before sending the packet over the software port to the middlebox.
As shown, the logical controller 320 receives the rule 415 (e.g., as a database table record). The logical controller identifies the particular nodes that should receive the rule 415. In some embodiments, this is all of the nodes that implement Middlebox D, which may be all of the nodes implementing the logical forwarding elements of the network. In some embodiments, however, the logical controller parses the routing policies for the middlebox to determine that only a subset of the nodes implementing the logical forwarding elements need to implement the middlebox (i.e., based on its placement and function in the logical network). Based on the identified nodes, the logical controller identifies a set of physical controllers to which it distributes the rule 415.
This set of physical controllers includes the illustrated physical controller 345, to which it exports the rule 415. As at the logical controller 320, the physical controller 345 does not perform any transformation (or at least only minimal transformation) on the rule 415. However, as the new configuration entails starting up a new virtual middlebox on the middlebox application running on the host machine (e.g., a middlebox daemon), the physical controller assigns a slicing identifier 420 for the middlebox instance.
The physical controller 345 distributes the rule 415 and the slicing identifier 420 to the middlebox 425 on the same host machine as the MSE 350. The middlebox generates a new middlebox instance, and converts the rule 415 (along with other configuration rules received from the physical controller 345) into a compiled set of data plane rules 430 for the middlebox instance. In some embodiments, these data plane rules 430 act effectively as a hardcoded set of forwarding tables for packet processing in the middlebox. In addition, the middlebox 425 adds an entry to its slice binding table 435. For each instance, the middlebox 425 creates its own internal identifier, and stores a binding of this internal ID to the packed slicing ID assigned by the physical controller in the slice binding table 435. Furthermore, as shown, the middlebox would have contracted with the MSE 350 in the host machine to create the software port for transferring packets between the two modules.
As shown, a second logical controller 515 receives the rule 520 (e.g., as a database table record). The logical controller 515 identifies the particular nodes that should receive the rule 520, as in the previous example. In this case, the node with middlebox 425 and MSE 350 hosts virtual machines for both the first and second networks 305 and 505, so the same physical controller 345 that manages that node receives the rule 415.
Because the Middlebox H requires the creation of a new virtual middlebox on the middlebox application 425, the physical controller 345 assigns a new, different slicing identifier 525 for the middlebox instance. The physical controller 345 then distributes the rule 520 and the slicing identifier 525 to the middlebox 425 on the host machine. The middlebox 425 creates a new middlebox instance, and converts the rule 520 (along with other configuration rules received from the physical controller 345) into a new set of data plane rules 530 for the newly created middlebox instance.
In addition, the middlebox 425 adds another entry to its slice binding table 435. As stated above, the middlebox creates its own internal identifier, and stores the binding of this internal ID to the assigned slicing ID from the physical controller 345 in its slice binding table 435. This way, when a packet is received from the MSE 350, the middlebox 425 can remove the slicing identifier and match that identifier with its internal ID in order to use the appropriate set of middlebox rules to process the packet. In addition, when the middlebox creates new states (e.g., for new TCP connections), it uses the internal identifier to associate the state with a particular instance.
The above example relates to a distributed middlebox. For a centralized middlebox, the user enters the configuration in the same manner (through an interface designed for the middlebox). The logical controller identifies only a single physical controller that manages the middlebox (or the host machine at which the middlebox is located, if implemented as a single virtual machine), and exports the configuration rules to this physical controller. The physical controller assigns a slice identifier for the configuration, as the centralized middleboxes may also be virtualized between several networks.
Whereas in the distributed case, only one managed switching element sends packets to a particular distributed middlebox element, in the centralized case numerous different MSEs will need to receive the slicing identifier. As such, in some embodiments, the physical controller managing the centralized middlebox sends the slicing identifier to the logical controller that manages the particular network, which distributes this slicing identifier (e.g., in the form of flow entries that add the identifier to packets destined for the middlebox) to all of the nodes that may send packets to the middlebox (via their managing physical controllers). In other embodiments, the logical controller assigns the slicing identifier for the middlebox and distributes this information to all of the nodes via the managing physical controllers.
In addition, the network control system sets up tunnels between the centralized middlebox and the various managed switching elements that send packets to the middlebox. The tunneling information may be entered by a user at the input translation controller interface, or automatically generated by the logical controller with knowledge of the different tunneling protocols supported by the middlebox. In the conversion to the physical control plane, the logical controller adds tunnel encapsulation flow entries that add or remove tunnel encapsulation from the packet. These entries can then be customized at the physical controller level to account for the particular ports and encapsulation used at each different managed switching element.
II. Network Controller Architecture
The above section describes a network control system that includes several different types of network controllers.
In some embodiments, the input tables 615 include tables with different types of data depending on the role of the controller 600 in the network control system. For instance, when the controller 600 functions as a logical controller for a user's logical forwarding elements, the input tables 615 include LCP data and LFP data for the logical forwarding elements. When the controller 600 functions as a physical controller, the input tables 615 include LFP data. The input tables 615 also include middlebox configuration data received from the user or another controller. The middlebox configuration data is associated with a logical datapath set parameter that identifies the logical switching elements to which the middlebox to be is integrated.
In addition to the input tables 615, the control application 600 includes other miscellaneous tables (not shown) that the rules engine 610 uses to gather inputs for its table mapping operations. These miscellaneous tables include constant tables that store defined values for constants that the rules engine 610 needs to perform its table mapping operations (e.g., the value 0, a dispatch port number for resubmits, etc.). The miscellaneous tables further include function tables that store functions that the rules engine 610 uses to calculate values to populate the output tables 625.
The rules engine 610 performs table mapping operations that specifies one manner for converting input data to output data. Whenever one of the input tables is modified (referred to as an input table event), the rules engine performs a set of table mapping operations that may result in the modification of one or more data tuples in one or more output tables.
In some embodiments, the rules engine 610 includes an event processor (not shown), several query plans (not shown), and a table processor (not shown). Each query plan is a set of rules that specifies a set of join operations that are to be performed upon the occurrence of an input table event. The event processor of the rules engine 610 detects the occurrence of each such event. In some embodiments, the event processor registers for callbacks with the input tables for notification of changes to the records in the input tables 615, and detects an input table event by receiving a notification from an input table when one of its records has changed.
In response to a detected input table event, the event processor (1) selects an appropriate query plan for the detected table event, and (2) directs the table processor to execute the query plan. To execute the query plan, the table processor, in some embodiments, performs the join operations specified by the query plan to produce one or more records that represent one or more sets of data values from one or more input and miscellaneous tables. The table processor of some embodiments then (1) performs a select operation to select a subset of the data values from the record(s) produced by the join operations, and (2) writes the selected subset of data values in one or more output tables 620.
Some embodiments use a variation of the datalog database language to allow application developers to create the rules engine for the controller, and thereby to specify the manner by which the controller maps logical datapath sets to the controlled physical switching infrastructure. This variation of the datalog database language is referred to herein as nLog. Like datalog, nLog provides a few declaratory rules and operators that allow a developer to specify different operations that are to be performed upon the occurrence of different events. In some embodiments, nLog provides a limited subset of the operators that are provided by datalog in order to increase the operational speed of nLog. For instance, in some embodiments, nLog only allows the AND operator to be used in any of the declaratory rules.
The declaratory rules and operations that are specified through nLog are then compiled into a much larger set of rules by an nLog compiler. In some embodiments, this compiler translates each rule that is meant to address an event into several sets of database join operations. Collectively the larger set of rules forms the table mapping rules engine that is referred to as the nLog engine.
Some embodiments designate the first join operation that is performed by the rules engine for an input event to be based on the logical datapath set parameter. This designation ensures that the rules engine's join operations fail and terminate immediately when the rules engine has started a set of join operations that relate to a logical datapath set (i.e., to a logical network) that is not managed by the controller.
Like the input tables 615, the output tables 620 include tables with different types of data depending on the role of the controller 600. When the controller 600 functions as a logical controller, the output tables 615 include LFP data and UPCP data for the logical switching elements. When the controller 600 functions as a physical controller, the output tables 620 include CPCP data. Like the input tables, the output tables 615 may also include the middlebox configuration data. Furthermore, the output tables 615 may include a slice identifier when the controller 600 functions as a physical controller.
In some embodiments, the output tables 620 can be grouped into several different categories. For instance, in some embodiments, the output tables 620 can be rules engine (RE) input tables and/or RE output tables. An output table is a RE input table when a change in the output table causes the rules engine to detect an input event that requires the execution of a query plan. An output table can also be an RE input table that generates an event that causes the rules engine to perform another query plan. An output table is a RE output table when a change in the output table causes the exporter 625 to export the change to another controller or a MSE. An output table can be an RE input table, a RE output table, or both an RE input table and a RE output table.
The exporter 625 detects changes to the RE output tables of the output tables 620. In some embodiments, the exporter registers for callbacks with the RE output tables for notification of changes to the records of the RE output tables. In such embodiments, the exporter 625 detects an output table event when it receives notification from a RE output table that one of its records has changed.
In response to a detected output table event, the exporter 625 takes each modified data tuple in the modified RE output tables and propagates this modified data tuple to one or more other controllers or to one or more MSEs. When sending the output table records to another controller, the exporter in some embodiments uses a single channel of communication (e.g., a RPC channel) to send the data contained in the records. When sending the RE output table records to MSEs, the exporter in some embodiments uses two channels. One channel is established using a switch control protocol (e.g., OpenFlow) for writing flow entries in the control plane of the MSE. The other channel is established using a database communication protocol (e.g., JSON) to send configuration data (e.g., port configuration, tunnel information).
In some embodiments, the controller 600 does not keep in the output tables 620 the data for logical datapath sets that the controller is not responsible for managing (i.e., for logical networks managed by other logical controllers). However, such data is translated by the translator 635 into a format that can be stored in the PTD 640 and is then stored in the PTD. The PTD 640 propagates this data to PTDs of one or more other controllers so that those other controllers that are responsible for managing the logical datapath sets can process the data.
In some embodiments, the controller also brings the data stored in the output tables 620 to the PTD for resiliency of the data. Therefore, in these embodiments, a PTD of a controller has all the configuration data for all logical datapath sets managed by the network control system. That is, each PTD contains the global view of the configuration of the logical networks of all users.
The importer 630 interfaces with a number of different sources of input data and uses the input data to modify or create the input tables 610. The importer 620 of some embodiments receives the input data from another controller. The importer 620 also interfaces with the PTD 640 so that data received through the PTD from other controller instances can be translated and used as input data to modify or create the input tables 610. Moreover, the importer 620 also detects changes with the RE input tables in the output tables 630.
III. Packet Processing
The above sections describe in detail the creation of flow entries and middlebox configurations for a logical network using a network control system. This data is then used to process and forward traffic within the physical implementation of the network (e.g., by matching packets to flow entries in the managed switching elements, applying the middlebox rules to packets, etc.).
The VM 705 sends a packet to the MSE 715 (e.g., through a virtual interface within the host machine 710). The packet has a source MAC and IP address corresponding to VM 705, and a destination IP address (and, if available, destination MAC address) corresponding to VM 725. The MSE 715 begins by executing the L2 flow for logical switch A (the logical switch to which VM 725 attaches), which includes mapping the physical ingress port (the virtual interface) to a logical ingress port, then performing a logical L2 forwarding decision to send the packet to the logical router. This logical router is also implemented by the MSE 715, so sending the packet only involves a resubmit of the packet.
The MSE 715 then executes the L3 flow for the logical router. In some embodiments, the forwarding tables include a flow entry to forward the packet to the logical switch B to which the VM 725 attaches based on the destination IP address. However, the L3 forwarding tables also include a higher-priority entry to forward the packet to the middlebox based on a user-entered routing policy (e.g., based on the source IP address and/or the logical ingress port, or other data). Thus, the L3 forwarding decision is to send the packet to the middlebox 720. Before sending the packet over the software port negotiated between the two software elements, the MSE 715 adds a slice tag to the packet to identify the correct middlebox instance. In some embodiments, the MSE prepends this tag into a particular field of the packet header.
The middlebox 720 receives the packet, and removes the slice tag in order to identify the correct middlebox instance for processing the packet. With the correct instance identified (through the slice binding table), the middlebox performs its processing. This may involve modifying the source IP (for S-NAT), modifying the destination IP (for a load balancer), determining whether to drop or allow the packet (for a firewall), or other process. After performing its processing on the packet, the middlebox sends the packet back to the managed switching element.
In some embodiments, the middlebox sends a new packet back to the MSE. Because the MSE may receive packets from multiple middlebox instances over the same port, in some embodiments the middlebox adds the slice tag to the packet before sending it over the software port. At the MSE 715, the flow entries map the packet back to the logical L3 router (i.e., to the ingress port of the L3 router connected to the middlebox), and then execute the L3 forwarding. Because the logical ingress port is now the port connecting to the middlebox, the routing policy to send to the middlebox is not executed, and the destination IP address results in a forwarding decision to the logical switch B. The logical forwarding decision at this switch is based on the destination MAC address, which results in a forwarding decision to the VM 725. The forwarding decision is used to encapsulate the packet, and also maps to a particular physical port of the host machine, over which the packet is sent (after adding tunneling encapsulation).
The packet traverses the network via the tunnel in order to arrive at the MSE 735 of the second host 730. After removing the tunneling, the MSE 735 reads the egress context, removes it, and delivers the packet to the VM 725. This maps to a virtual interface within the host machine 730 such that the packet arrives at the VM 725.
One of ordinary skill will recognize that this example is only one of numerous possible packet processing examples involving a distributed middlebox. If the destination VM 725 was located in the host 710 along with the source VM 705, the packet would go to the MSE 715, to the middlebox 720, back to the MSE 715, and then to the destination VM 725 without ever leaving the physical machine. As another example, if the middlebox only receives duplicate packets, then the MSE 715 would send the received packet to the second host 730 and a duplicate packet to the middlebox 720, which would not send out a packet after its processing was finished.
While the example of
The centralized middlebox 805 maps the slice identifier to one of its virtual middleboxes (i.e., using its slice binding table), then performs the middlebox processing. Examples of centralized middleboxes include, in some embodiments, firewalls (which may also be distributed), intrusion detection systems (which are passive middleboxes that receive duplicate packets), and WAN optimizers (for performing various data compression techniques before sending packets out over a WAN), as well as other middleboxes. After performing its middlebox processing, the middlebox 805 sends a new packet to a pool node.
In some embodiments, each centralized middlebox sends all of its packets to a particular pool node (or different pool nodes for different middlebox instances), because the middlebox may not have the functionality to perform the logical forwarding required to send the packet to its destination. Thus, the middlebox 805 encapsulates the packet for the tunnel (and, in some embodiments, adds the slice tag to the packet), then sends it to the pool node 810. The pool node 810 maps the packet to the correct logical router, and uses the destination IP address of the packet to make a forwarding decision to the logical switch to which the destination VM 725 connects. The logical L2 flows in the pool node forward the packet to the destination machine 725, and this egress context is used to encapsulate the packet. The pool node 810 then adds the tunnel encapsulation for transport over the physical network, and sends the packet to the host. 730. The MSE 735 reads this egress context, removes it, and delivers the packet to the VM 725 (in the same manner as in
IV. Electronic System
Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more computational or processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard drives, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.
In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.
The bus 1005 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 1000. For instance, the bus 1005 communicatively connects the processing unit(s) 1010 with the read-only memory 1030, the system memory 1025, and the permanent storage device 1035.
From these various memory units, the processing unit(s) 1010 retrieves instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments.
The read-only-memory (ROM) 1030 stores static data and instructions that are needed by the processing unit(s) 1010 and other modules of the electronic system. The permanent storage device 1035, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 1000 is off Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 1035.
Other embodiments use a removable storage device (such as a floppy disk, flash memory device, etc., and its corresponding drive) as the permanent storage device. Like the permanent storage device 1035, the system memory 1025 is a read-and-write memory device. However, unlike storage device 1035, the system memory 1025 is a volatile read-and-write memory, such a random access memory. The system memory 1025 stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 1025, the permanent storage device 1035, and/or the read-only memory 1030. From these various memory units, the processing unit(s) 1010 retrieves instructions to execute and data to process in order to execute the processes of some embodiments.
The bus 1005 also connects to the input and output devices 1040 and 1045. The input devices 1040 enable the user to communicate information and select commands to the electronic system. The input devices 1040 include alphanumeric keyboards and pointing devices (also called “cursor control devices”), cameras (e.g., webcams), microphones or similar devices for receiving voice commands, etc. The output devices 1045 display images generated by the electronic system or otherwise output data. The output devices 1045 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD), as well as speakers or similar audio output devices. Some embodiments include devices such as a touchscreen that function as both input and output devices.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.
As used in this specification and any claims of this application, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this application, the terms “computer readable medium,” “computer readable media,” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.
This application is a continuation application of U.S. patent application Ser. No. 16/403,487, filed May 3, 2019, now published as U.S. Patent Publication 2019/0258507. U.S. patent application Ser. No. 16/403,487 is a continuation application of U.S. patent application Ser. No. 15/398,709, filed Jan. 4, 2017, now issued as U.S. Pat. No. 10,310,886. U.S. patent application Ser. No. 15/398,709 is a continuation application of U.S. patent application Ser. No. 14/595,195, filed Jan. 12, 2015, now issued as U.S. Pat. No. 9,558,027. U.S. patent application Ser. No. 14/595,195 is a continuation application of U.S. patent application Ser. No. 13/678,485, filed Nov. 15, 2012, now issued as U.S. Pat. No. 8,966,029. U.S. patent application Ser. No. 13/678,485 claims the benefit of U.S. Provisional Patent Application 61/560,279, entitled “Virtual Middleboxes Services”, filed Nov. 15, 2011. U.S. patent application Ser. No. 16/403,487, now published as U.S. Patent Publication 2019/0258507; U.S. patent application Ser. No. 15/398,709, now issued as U.S. Pat. No. 10,310,886; U.S. patent application Ser. No. 14/595,195, now issued as U.S. Pat. No. 9,558,027; U.S. patent application Ser. No. 13/678,485, now issued as U.S. Pat. No. 8,966,029; and U.S. Provisional Patent Application 61/560,279 are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5473771 | Burd et al. | Dec 1995 | A |
| 5504921 | Dev et al. | Apr 1996 | A |
| 5550816 | Hardwick et al. | Aug 1996 | A |
| 5729685 | Chatwani et al. | Mar 1998 | A |
| 5751967 | Raab et al. | May 1998 | A |
| 5796936 | Watabe et al. | Aug 1998 | A |
| 6092121 | Bennett et al. | Jul 2000 | A |
| 6104699 | Holender et al. | Aug 2000 | A |
| 6219699 | McCloghrie et al. | Apr 2001 | B1 |
| 6353614 | Borella et al. | Mar 2002 | B1 |
| 6505192 | Godwin et al. | Jan 2003 | B1 |
| 6512745 | Abe et al. | Jan 2003 | B1 |
| 6539432 | Taguchi et al. | Mar 2003 | B1 |
| 6678274 | Walia et al. | Jan 2004 | B1 |
| 6680934 | Cain | Jan 2004 | B1 |
| 6781990 | Puri et al. | Aug 2004 | B1 |
| 6785843 | McRae et al. | Aug 2004 | B1 |
| 6862264 | Moura et al. | Mar 2005 | B1 |
| 6880089 | Bommareddy et al. | Apr 2005 | B1 |
| 6907042 | Oguchi | Jun 2005 | B1 |
| 6963585 | Pennec et al. | Nov 2005 | B1 |
| 7042912 | Smith et al. | May 2006 | B2 |
| 7046630 | Abe et al. | May 2006 | B2 |
| 7055027 | Gunter et al. | May 2006 | B1 |
| 7055173 | Chaganty et al. | May 2006 | B1 |
| 7126923 | Yang et al. | Oct 2006 | B1 |
| 7197572 | Matters et al. | Mar 2007 | B2 |
| 7206861 | Callon | Apr 2007 | B1 |
| 7209439 | Rawlins et al. | Apr 2007 | B2 |
| 7283465 | Zelig et al. | Oct 2007 | B2 |
| 7283473 | Arndt et al. | Oct 2007 | B2 |
| 7286490 | Saleh et al. | Oct 2007 | B2 |
| 7342916 | Das et al. | Mar 2008 | B2 |
| 7343410 | Mercier et al. | Mar 2008 | B2 |
| 7447775 | Zhu et al. | Nov 2008 | B1 |
| 7450598 | Chen et al. | Nov 2008 | B2 |
| 7467198 | Goodman et al. | Dec 2008 | B2 |
| 7478173 | Delco | Jan 2009 | B1 |
| 7512744 | Banga et al. | Mar 2009 | B2 |
| 7554995 | Short et al. | Jun 2009 | B2 |
| 7555002 | Arndt et al. | Jun 2009 | B2 |
| 7606229 | Foschiano et al. | Oct 2009 | B1 |
| 7606260 | Oguchi et al. | Oct 2009 | B2 |
| 7627692 | Pessi | Dec 2009 | B2 |
| 7647426 | Patel et al. | Jan 2010 | B2 |
| 7649851 | Takashige et al. | Jan 2010 | B2 |
| 7706266 | Plamondon | Apr 2010 | B2 |
| 7706325 | Fodor et al. | Apr 2010 | B2 |
| 7710872 | Vasseur | May 2010 | B2 |
| 7710874 | Balakrishnan et al. | May 2010 | B2 |
| 7725602 | Liu et al. | May 2010 | B2 |
| 7730486 | Herington | Jun 2010 | B2 |
| 7742398 | Tene et al. | Jun 2010 | B1 |
| 7761259 | Seymour | Jul 2010 | B1 |
| 7764599 | Doi et al. | Jul 2010 | B2 |
| 7792987 | Vohra et al. | Sep 2010 | B1 |
| 7802000 | Huang et al. | Sep 2010 | B1 |
| 7808929 | Wong et al. | Oct 2010 | B2 |
| 7818452 | Matthews et al. | Oct 2010 | B2 |
| 7826390 | Noel et al. | Nov 2010 | B2 |
| 7826482 | Minei et al. | Nov 2010 | B1 |
| 7839847 | Nadeau et al. | Nov 2010 | B2 |
| 7855950 | Zwiebel et al. | Dec 2010 | B2 |
| 7856549 | Wheeler | Dec 2010 | B2 |
| 7885276 | Lin | Feb 2011 | B1 |
| 7925661 | Broussard et al. | Apr 2011 | B2 |
| 7925850 | Waldspurger et al. | Apr 2011 | B1 |
| 7933198 | Pan | Apr 2011 | B1 |
| 7936770 | Frattura et al. | May 2011 | B1 |
| 7937438 | Miller et al. | May 2011 | B1 |
| 7941837 | Jiang et al. | May 2011 | B1 |
| 7945658 | Nucci et al. | May 2011 | B1 |
| 7948986 | Ghosh et al. | May 2011 | B1 |
| 7953865 | Miller et al. | May 2011 | B1 |
| 7991859 | Miller et al. | Aug 2011 | B1 |
| 7995483 | Bayar et al. | Aug 2011 | B1 |
| 8005015 | Belqasmi et al. | Aug 2011 | B2 |
| 8018866 | Kasturi et al. | Sep 2011 | B1 |
| 8027260 | Venugopal et al. | Sep 2011 | B2 |
| 8027354 | Portolani et al. | Sep 2011 | B1 |
| 8031606 | Memon et al. | Oct 2011 | B2 |
| 8031633 | Bueno et al. | Oct 2011 | B2 |
| 8046456 | Miller et al. | Oct 2011 | B1 |
| 8051180 | Mazzaferri et al. | Nov 2011 | B2 |
| 8054832 | Shukla et al. | Nov 2011 | B1 |
| 8055789 | Richardson et al. | Nov 2011 | B2 |
| 8060779 | Beardsley et al. | Nov 2011 | B2 |
| 8060875 | Lambeth | Nov 2011 | B1 |
| 8064362 | Mekkattuparamban et al. | Nov 2011 | B2 |
| 8131852 | Miller et al. | Mar 2012 | B1 |
| 8149737 | Metke et al. | Apr 2012 | B2 |
| 8155028 | Abu-Hamdeh et al. | Apr 2012 | B2 |
| 8166201 | Richardson et al. | Apr 2012 | B2 |
| 8190767 | Maufer et al. | May 2012 | B1 |
| 8194674 | Pagel et al. | Jun 2012 | B1 |
| 8196144 | Kagan et al. | Jun 2012 | B2 |
| 8199750 | Schultz et al. | Jun 2012 | B1 |
| 8204982 | Casado et al. | Jun 2012 | B2 |
| 8224931 | Brandwine et al. | Jul 2012 | B1 |
| 8224971 | Miller et al. | Jul 2012 | B1 |
| 8265071 | Sindhu et al. | Sep 2012 | B2 |
| 8265075 | Pandey | Sep 2012 | B2 |
| 8312129 | Miller et al. | Nov 2012 | B1 |
| 8321936 | Green et al. | Nov 2012 | B1 |
| 8339959 | Moisand et al. | Dec 2012 | B1 |
| 8339994 | Gnanasekaran et al. | Dec 2012 | B2 |
| 8456984 | Ranganathan et al. | Jun 2013 | B2 |
| 8463904 | Casado et al. | Jun 2013 | B2 |
| 8468548 | Kulkarni et al. | Jun 2013 | B2 |
| 8473557 | Ramakrishnan et al. | Jun 2013 | B2 |
| 8516158 | Wu et al. | Aug 2013 | B1 |
| 8543808 | Ahmed | Sep 2013 | B2 |
| 8571031 | Davies et al. | Oct 2013 | B2 |
| 8611351 | Gooch et al. | Dec 2013 | B2 |
| 8612627 | Brandwine | Dec 2013 | B1 |
| 8615579 | Vincent et al. | Dec 2013 | B1 |
| 8621058 | Eswaran et al. | Dec 2013 | B2 |
| 8625594 | Safrai et al. | Jan 2014 | B2 |
| 8644188 | Brandwine et al. | Feb 2014 | B1 |
| 8650299 | Huang et al. | Feb 2014 | B1 |
| 8650618 | Asati et al. | Feb 2014 | B2 |
| 8705527 | Addepalli et al. | Apr 2014 | B1 |
| 8743885 | Khan et al. | Jun 2014 | B2 |
| 8762501 | Kempf et al. | Jun 2014 | B2 |
| 8813209 | Bhattacharya et al. | Aug 2014 | B2 |
| 8819678 | Tsirkin | Aug 2014 | B2 |
| 8874757 | Souza | Oct 2014 | B2 |
| 8913611 | Koponen et al. | Dec 2014 | B2 |
| 8913661 | Bivolarsky et al. | Dec 2014 | B2 |
| 8966024 | Koponen et al. | Feb 2015 | B2 |
| 8966029 | Zhang et al. | Feb 2015 | B2 |
| 8966035 | Casado et al. | Feb 2015 | B2 |
| 8996683 | Maltz et al. | Mar 2015 | B2 |
| 9015823 | Koponen et al. | Apr 2015 | B2 |
| 9104458 | Brandwine et al. | Aug 2015 | B1 |
| 9172603 | Padmanabhan et al. | Oct 2015 | B2 |
| 9195491 | Zhang et al. | Nov 2015 | B2 |
| 9203747 | Brandwine | Dec 2015 | B1 |
| 9237132 | Mihelich et al. | Jan 2016 | B2 |
| 9306843 | Koponen et al. | Apr 2016 | B2 |
| 9306909 | Koponen et al. | Apr 2016 | B2 |
| 9329886 | Vincent | May 2016 | B2 |
| 9424144 | Sridharan et al. | Aug 2016 | B2 |
| 9448821 | Wang | Sep 2016 | B2 |
| 9552219 | Zhang et al. | Jan 2017 | B2 |
| 9558027 | Zhang et al. | Jan 2017 | B2 |
| 9697030 | Koponen et al. | Jun 2017 | B2 |
| 9697033 | Koponen et al. | Jul 2017 | B2 |
| 10089127 | Padmanabhan et al. | Oct 2018 | B2 |
| 10191763 | Koponen et al. | Jan 2019 | B2 |
| 10235199 | Zhang et al. | Mar 2019 | B2 |
| 10310886 | Zhang et al. | Jun 2019 | B2 |
| 10514941 | Zhang et al. | Dec 2019 | B2 |
| 10884780 | Koponen et al. | Jan 2021 | B2 |
| 10922124 | Zhang et al. | Feb 2021 | B2 |
| 10949248 | Zhang et al. | Mar 2021 | B2 |
| 10977067 | Padmanabhan et al. | Apr 2021 | B2 |
| 11372671 | Koponen et al. | Jun 2022 | B2 |
| 20010043614 | Viswanadham et al. | Nov 2001 | A1 |
| 20020034189 | Haddock et al. | Mar 2002 | A1 |
| 20020093952 | Gonda | Jul 2002 | A1 |
| 20020161867 | Cochran et al. | Oct 2002 | A1 |
| 20020194369 | Rawlins et al. | Dec 2002 | A1 |
| 20030009559 | Ikeda | Jan 2003 | A1 |
| 20030058850 | Rangarajan et al. | Mar 2003 | A1 |
| 20030069972 | Yoshimura et al. | Apr 2003 | A1 |
| 20030079000 | Chamberlain | Apr 2003 | A1 |
| 20030093481 | Mitchell et al. | May 2003 | A1 |
| 20030097454 | Yamakawa et al. | May 2003 | A1 |
| 20030131116 | Jain et al. | Jul 2003 | A1 |
| 20040049701 | Pennec et al. | Mar 2004 | A1 |
| 20040054793 | Coleman | Mar 2004 | A1 |
| 20040073659 | Rajsic et al. | Apr 2004 | A1 |
| 20040098505 | Clemmensen | May 2004 | A1 |
| 20040131059 | Ayyakad et al. | Jul 2004 | A1 |
| 20050013280 | Buddhikot et al. | Jan 2005 | A1 |
| 20050018669 | Arndt et al. | Jan 2005 | A1 |
| 20050021683 | Newton et al. | Jan 2005 | A1 |
| 20050027881 | Figueira et al. | Feb 2005 | A1 |
| 20050050377 | Chan et al. | Mar 2005 | A1 |
| 20050060365 | Robinson et al. | Mar 2005 | A1 |
| 20050083953 | May | Apr 2005 | A1 |
| 20050132030 | Hopen et al. | Jun 2005 | A1 |
| 20050240654 | Wolber et al. | Oct 2005 | A1 |
| 20050249199 | Albert et al. | Nov 2005 | A1 |
| 20060026225 | Canali et al. | Feb 2006 | A1 |
| 20060092976 | Lakshman et al. | May 2006 | A1 |
| 20060215684 | Capone | Sep 2006 | A1 |
| 20060221961 | Basso et al. | Oct 2006 | A1 |
| 20070101323 | Foley et al. | May 2007 | A1 |
| 20070101421 | Wesinger, Jr. et al. | May 2007 | A1 |
| 20070140128 | Klinker et al. | Jun 2007 | A1 |
| 20070233838 | Takamoto et al. | Oct 2007 | A1 |
| 20070239987 | Hoole et al. | Oct 2007 | A1 |
| 20070266433 | Moore | Nov 2007 | A1 |
| 20070283348 | White | Dec 2007 | A1 |
| 20070286185 | Eriksson et al. | Dec 2007 | A1 |
| 20080002579 | Lindholm et al. | Jan 2008 | A1 |
| 20080005293 | Bhargava et al. | Jan 2008 | A1 |
| 20080049621 | McGuire et al. | Feb 2008 | A1 |
| 20080071900 | Hecker et al. | Mar 2008 | A1 |
| 20080072305 | Casado et al. | Mar 2008 | A1 |
| 20080151893 | Nordmark et al. | Jun 2008 | A1 |
| 20080163207 | Reumann et al. | Jul 2008 | A1 |
| 20080186990 | Abali et al. | Aug 2008 | A1 |
| 20080189769 | Casado et al. | Aug 2008 | A1 |
| 20080196100 | Madhavan et al. | Aug 2008 | A1 |
| 20080205377 | Chao et al. | Aug 2008 | A1 |
| 20080225853 | Melman et al. | Sep 2008 | A1 |
| 20080232250 | Park | Sep 2008 | A1 |
| 20080240122 | Richardson et al. | Oct 2008 | A1 |
| 20080281908 | McCanne et al. | Nov 2008 | A1 |
| 20090025077 | Trojanowski | Jan 2009 | A1 |
| 20090031041 | Clemmensen | Jan 2009 | A1 |
| 20090063750 | Dow | Mar 2009 | A1 |
| 20090064305 | Stiekes et al. | Mar 2009 | A1 |
| 20090070877 | Davids et al. | Mar 2009 | A1 |
| 20090083445 | Ganga | Mar 2009 | A1 |
| 20090092137 | Haigh et al. | Apr 2009 | A1 |
| 20090122710 | Bar-Tor et al. | May 2009 | A1 |
| 20090129271 | Ramankutty et al. | May 2009 | A1 |
| 20090150527 | Tripathi et al. | Jun 2009 | A1 |
| 20090161547 | Riddle et al. | Jun 2009 | A1 |
| 20090199177 | Edwards et al. | Aug 2009 | A1 |
| 20090240924 | Yasaki et al. | Sep 2009 | A1 |
| 20090249470 | Litvin et al. | Oct 2009 | A1 |
| 20090249472 | Litvin et al. | Oct 2009 | A1 |
| 20090249473 | Cohn | Oct 2009 | A1 |
| 20090279536 | Unbehagen et al. | Nov 2009 | A1 |
| 20090292858 | Lambeth et al. | Nov 2009 | A1 |
| 20090300210 | Ferris | Dec 2009 | A1 |
| 20090303880 | Maltz et al. | Dec 2009 | A1 |
| 20090327464 | Archer et al. | Dec 2009 | A1 |
| 20090327781 | Tripathi | Dec 2009 | A1 |
| 20100046530 | Hautakorpi et al. | Feb 2010 | A1 |
| 20100046531 | Louati et al. | Feb 2010 | A1 |
| 20100098092 | Luo et al. | Apr 2010 | A1 |
| 20100103837 | Jungck et al. | Apr 2010 | A1 |
| 20100115080 | Kageyama | May 2010 | A1 |
| 20100115101 | Lain et al. | May 2010 | A1 |
| 20100125667 | Soundararajan | May 2010 | A1 |
| 20100128623 | Dunn et al. | May 2010 | A1 |
| 20100131636 | Suri et al. | May 2010 | A1 |
| 20100131638 | Kondamuru | May 2010 | A1 |
| 20100138830 | Astete et al. | Jun 2010 | A1 |
| 20100146074 | Srinivasan | Jun 2010 | A1 |
| 20100153554 | Anschutz et al. | Jun 2010 | A1 |
| 20100162036 | Linden et al. | Jun 2010 | A1 |
| 20100165876 | Shukla et al. | Jul 2010 | A1 |
| 20100165877 | Shukla et al. | Jul 2010 | A1 |
| 20100169467 | Shukla et al. | Jul 2010 | A1 |
| 20100214949 | Smith et al. | Aug 2010 | A1 |
| 20100254385 | Sharma et al. | Oct 2010 | A1 |
| 20100257263 | Casado et al. | Oct 2010 | A1 |
| 20100275199 | Smith et al. | Oct 2010 | A1 |
| 20100284402 | Narayanan | Nov 2010 | A1 |
| 20100287548 | Zhou et al. | Nov 2010 | A1 |
| 20100306408 | Greenberg et al. | Dec 2010 | A1 |
| 20100318665 | Demmer et al. | Dec 2010 | A1 |
| 20100322255 | Hao et al. | Dec 2010 | A1 |
| 20100333165 | Basak et al. | Dec 2010 | A1 |
| 20110002346 | Wu | Jan 2011 | A1 |
| 20110004698 | Wu | Jan 2011 | A1 |
| 20110004876 | Wu et al. | Jan 2011 | A1 |
| 20110004877 | Wu | Jan 2011 | A1 |
| 20110022695 | Dalal et al. | Jan 2011 | A1 |
| 20110026521 | Gamage et al. | Feb 2011 | A1 |
| 20110075674 | Li et al. | Mar 2011 | A1 |
| 20110085557 | Gnanasekaran et al. | Apr 2011 | A1 |
| 20110085559 | Chung et al. | Apr 2011 | A1 |
| 20110085563 | Kotha et al. | Apr 2011 | A1 |
| 20110113483 | Rangegowda et al. | May 2011 | A1 |
| 20110119748 | Edwards et al. | May 2011 | A1 |
| 20110173692 | Liu et al. | Jul 2011 | A1 |
| 20110197039 | Green et al. | Aug 2011 | A1 |
| 20110225594 | Iyengar et al. | Sep 2011 | A1 |
| 20110255538 | Srinivasan et al. | Oct 2011 | A1 |
| 20110261825 | Ichino | Oct 2011 | A1 |
| 20110289230 | Ueno | Nov 2011 | A1 |
| 20110299534 | Koganti et al. | Dec 2011 | A1 |
| 20110299538 | Maruta | Dec 2011 | A1 |
| 20110310899 | Alkhatib et al. | Dec 2011 | A1 |
| 20120011264 | Izawa | Jan 2012 | A1 |
| 20120014386 | Xiong et al. | Jan 2012 | A1 |
| 20120144014 | Natham et al. | Jun 2012 | A1 |
| 20120147894 | Mulligan et al. | Jun 2012 | A1 |
| 20120155266 | Patel et al. | Jun 2012 | A1 |
| 20120185577 | Giaretta et al. | Jul 2012 | A1 |
| 20120210416 | Mihelich et al. | Aug 2012 | A1 |
| 20120210417 | Shieh | Aug 2012 | A1 |
| 20120236734 | Sampath et al. | Sep 2012 | A1 |
| 20120240182 | Narayanaswamy et al. | Sep 2012 | A1 |
| 20120246637 | Kreeger et al. | Sep 2012 | A1 |
| 20120281540 | Khan et al. | Nov 2012 | A1 |
| 20130003735 | Chao et al. | Jan 2013 | A1 |
| 20130019015 | Devarakonda et al. | Jan 2013 | A1 |
| 20130036416 | Raju et al. | Feb 2013 | A1 |
| 20130041987 | Warno | Feb 2013 | A1 |
| 20130054761 | Kempf et al. | Feb 2013 | A1 |
| 20130058250 | Casado et al. | Mar 2013 | A1 |
| 20130058350 | Fulton | Mar 2013 | A1 |
| 20130061047 | Sridharan et al. | Mar 2013 | A1 |
| 20130073743 | Ramasamy et al. | Mar 2013 | A1 |
| 20130074066 | Sanzgiri et al. | Mar 2013 | A1 |
| 20130074181 | Singh | Mar 2013 | A1 |
| 20130097356 | Dang et al. | Apr 2013 | A1 |
| 20130103834 | Dzerve et al. | Apr 2013 | A1 |
| 20130121209 | Padmanabhan et al. | May 2013 | A1 |
| 20130125120 | Zhang et al. | May 2013 | A1 |
| 20130125230 | Koponen et al. | May 2013 | A1 |
| 20130128891 | Koponen et al. | May 2013 | A1 |
| 20130132531 | Koponen et al. | May 2013 | A1 |
| 20130132532 | Zhang et al. | May 2013 | A1 |
| 20130132533 | Padmanabhan et al. | May 2013 | A1 |
| 20130132536 | Zhang et al. | May 2013 | A1 |
| 20130163427 | Beliveau et al. | Jun 2013 | A1 |
| 20130163475 | Beliveau et al. | Jun 2013 | A1 |
| 20130163594 | Sharma et al. | Jun 2013 | A1 |
| 20130227097 | Yasuda et al. | Aug 2013 | A1 |
| 20130332619 | Xie et al. | Dec 2013 | A1 |
| 20130332983 | Koorevaar et al. | Dec 2013 | A1 |
| 20140016501 | Kamath et al. | Jan 2014 | A1 |
| 20140019639 | Ueno | Jan 2014 | A1 |
| 20140068602 | Gember et al. | Mar 2014 | A1 |
| 20140153577 | Janakiraman et al. | Jun 2014 | A1 |
| 20140169375 | Khan et al. | Jun 2014 | A1 |
| 20140195666 | Dumitriu et al. | Jul 2014 | A1 |
| 20140359620 | Kerkwyk et al. | Dec 2014 | A1 |
| 20150081861 | Koponen et al. | Mar 2015 | A1 |
| 20150098360 | Koponen et al. | Apr 2015 | A1 |
| 20150124651 | Zhang et al. | May 2015 | A1 |
| 20150142938 | Koponen et al. | May 2015 | A1 |
| 20150222598 | Koponen et al. | Aug 2015 | A1 |
| 20160070588 | Zhang et al. | Mar 2016 | A1 |
| 20160241491 | Tripathi et al. | Aug 2016 | A1 |
| 20170116023 | Zhang et al. | Apr 2017 | A1 |
| 20170126493 | Zhang et al. | May 2017 | A1 |
| 20170277557 | Koponen et al. | Sep 2017 | A1 |
| 20190034220 | Padmanabhan et al. | Jan 2019 | A1 |
| 20190138343 | Koponen et al. | May 2019 | A1 |
| 20190258507 | Zhang et al. | Aug 2019 | A1 |
| 20200081732 | Zhang et al. | Mar 2020 | A1 |
| 20210149708 | Koponen et al. | May 2021 | A1 |
| 20210200572 | Zhang et al. | Jul 2021 | A1 |
| 20220326980 | Koponen et al. | Oct 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 2012340387 | Apr 2014 | AU |
| 2015258160 | Dec 2015 | AU |
| 2017204765 | Jul 2017 | AU |
| 1886962 | Dec 2006 | CN |
| 101904155 | Dec 2010 | CN |
| 102113274 | Jun 2011 | CN |
| 1653688 | May 2006 | EP |
| 2395712 | Dec 2011 | EP |
| 2748716 | Jul 2014 | EP |
| 3373560 | Sep 2018 | EP |
| 2000332817 | Nov 2000 | JP |
| 2003124976 | Apr 2003 | JP |
| 2003318949 | Nov 2003 | JP |
| 2005260299 | Sep 2005 | JP |
| 2011188433 | Sep 2011 | JP |
| 2011211502 | Oct 2011 | JP |
| 2012525017 | Oct 2012 | JP |
| 9918534 | Apr 1999 | WO |
| 2005112390 | Nov 2005 | WO |
| 2008095010 | Aug 2008 | WO |
| 2010090182 | Aug 2010 | WO |
| 2010116606 | Oct 2010 | WO |
| 2012051884 | Apr 2012 | WO |
| 2013074827 | May 2013 | WO |
| 2013074828 | May 2013 | WO |
| 2013074831 | May 2013 | WO |
| 2013074842 | May 2013 | WO |
| 2013074844 | May 2013 | WO |
| 2013074847 | May 2013 | WO |
| 2013074855 | May 2013 | WO |
| Entry |
|---|
| Author Unknown, “Enabling Service Chaining on Cisco Nexus 1000V Series,” Month Unknown, 2012, 25 pages, Cisco. |
| Casado, Martin, et al. “Ethane: Taking Control of the Enterprise,” SIGCOMM'07, Aug. 27-31, 2007, 12 pages, ACM, Kyoto, Japan. |
| Casado, Martin, et al., “Scaling Out: Network Virtualization Revisited,” Month Unknown 2010, 8 pages. |
| Casado, Martin, et al., “Virtualizing the Network Forwarding Plane,” Dec. 2010, 6 pages. |
| Dixon, Colin, et al., “An End to the Middle,” Proceedings of the 12th Conference on Hot Topics in Operating Systems, May 2009, 5 pages, USENIX Association, Berkeley, CA, USA. |
| Dumitriu, Dan Mihai, et al., (U.S. Appl. No. 61/514,990), filed Aug. 4, 2011, 31 pages. |
| Foster, Nate, et al. “Frenetic: A Network Programming Language,” ICFP'11, Sep. 19-21, 2011, 13 pages, Tokyo, Japan. |
| Gude, Natasha, et al., “NOX: Towards an Operating System for Networks,” ACM SIGCOMM Computer Communication Review, Jul. 2008, 6 pages, vol. 38, No. 3, ACM. |
| Hinrichs, Timothy L., et al., “Practical Declarative Network Management,” WREN'09, Aug. 21, 2009, 10 pages, Barcelona, Spain. |
| Ioannidis, Sotiris, et al., “Implementing a Distributed Firewall,” CCS '00, Month Unknown 2000, 10 pages, ACM, Athens, Greece. |
| Joseph, Dilip Anthony, et al., “A Policy-aware Switching Layer for Data Centers,” Jun. 24, 2008, 26 pages, Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA. |
| Keller, Eric, et al., “The ‘Platform as a Service’ Model for Networking,” Month Unknown 2010, 6 pages. |
| Kent, Stephen, “IP Encapsulating Security Payload (ESP),” RFC 4303, Dec. 2005, 44 pages, The Internet Society. |
| Koponen, Teemu, et al., “Onix: A Distributed Control Platform for Large-scale Production Networks,” In Proc. OSDI, Oct. 2010, 14 pages. |
| Luo, Jianying, et al., “Prototyping Fast, Simple, Secure Switches for Ethane,” Month Unknown 2007, 6 pages. |
| Mahalingham, Mallik, et al., “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks,” draft-mahalingham-dutt-dcops-vxlan-00.txt Internet Draft, Aug. 26, 2011, 20 pages, Internet Engineering Task Force. |
| Mckeown, Nick, et al., “OpenFlow: Enabling Innovation in Campus Networks,” ACM SIGCOMM Computer Communication Review, Apr. 2008, 6 pages, vol. 38, No. 2, ACM. |
| Non-published commonly owned U.S. Appl. No. 17/140,792, filed Jan. 4, 2021, 67 pages, Nicira, Inc. |
| PCT International Search Report and Written Opinion of Commonly Owned International Patent Application PCT/US2012/065345, dated Feb. 1, 2013, 7 pages, International Searching Authority (U.S.). |
| Pettit, Justin, et al., “Virtual Switching in an Era of Advanced Edges,” In Proc. 2nd Workshop on Data Center-Converged and Virtual Ethernet Switching (DCCAVES), Sep. 2010, 7 pages, vol. 22. ITC. |
| Pfaff, Ben., et al., “Extending Networking into the Virtualization Layer,” Proc. Of HotNets, Oct. 2009, 6 pages. |
| Popa, Lucian, et al., “Building Extensible Networks with Rule-Based Forwarding,” In USENIX OSDI, Month Unknown 2010, 14 pages. |
| Sekar, Vyas, et al., “Design and Implementation of a Consolidated Middlebox Architecture,” 9th USENIX Symposium on Networked Systems Design and Implementation, Apr. 25-27, 2012, 14 pages, USENIX, San Jose, CA, USA. |
| Sherry, Justine, et al., “Making Middleboxes Someone Else's Problem: Network Processing as a Cloud Service,” In Proc. of SIGCOMM '12, Aug. 13-17, 2012, 12 pages, Helsinki, Finland. |
| Stiemerling, M., et al., “Middlebox Communication (MIDCOM) Protocol Semantics,” Mar. 2008, 70 pages, Internet Engineering Task Force. |
| Number | Date | Country | |
|---|---|---|---|
| 20210191750 A1 | Jun 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61560279 | Nov 2011 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16403487 | May 2019 | US |
| Child | 17174330 | US | |
| Parent | 15398709 | Jan 2017 | US |
| Child | 16403487 | US | |
| Parent | 14595195 | Jan 2015 | US |
| Child | 15398709 | US | |
| Parent | 13678485 | Nov 2012 | US |
| Child | 14595195 | US |
