Patent Application
20180054475
The present disclosure relates to cloud-based network appliances, and more particularly to availability and load balancing of cloud-based appliances.
The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent the work is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
A cloud-based network (referred to herein as “a cloud”) can include network appliances and application servers. Examples of network appliances are firewalls, proxy servers, World Wide Web (or Web) servers, wide area network (WAN) accelerators, intrusion detection system (IDS) devices, and intrusion prevention system (IPS) devices. The network appliances provide intermediary services between the application servers and client stations, which are outside the cloud. The application servers terminate connections with the user stations and host particular subscriber applications. The network appliances and application servers may be implemented as one or more virtual machines (VMs). Cloud-based networks allow computer processing and storing needs to be moved from traditional privately-owned networks to publically shared networks while satisfying data security access requirements.
A load balancing system is provided and includes: one or more virtual machines implemented in a cloud-based network and comprising a processor; and a load balancing application implemented in the one or more virtual machines and executed by the processor. The load balancing application is configured such that the processor: receives one or more health messages indicating states of health of multiple network appliances, where the network appliances are implemented in an appliance layer of the cloud-based network; receives a forwarding packet from a network device for an application server; and based on the one or more health messages, determines whether to perform at least one of a failover process or select a network appliance. The load balancing application is further configured such that the processor: performs a first iteration of a symmetric conversion to route the forwarding packet to the application server via the selected network appliance; receives a return packet from the application server based on the forwarding packet; and performs a second iteration of the symmetric conversion to route the return packet to the network device via the network appliance. During the failover process, the load balancing application is configured such that the processor switches routing of traffic between the network device and the application server through a first instance of the network appliance to routing the traffic between the network device and the application server through a second instance of the network appliance.
In other features, a load balancing system is provided and includes: one or more virtual machines implemented in a public cloud-based network and including a first processor and a second processor; a probing application implemented in the one or more virtual machines and configured such that the first processor (i) transmits probe request messages to network appliances, (ii) based on the probe request messages, receives response messages from the network appliances, and (iii) based on the response messages, generates a health report message indicating states of health of the network appliances, where the network appliances are implemented in an appliance layer of the public cloud-based network; and a first load balancing application implemented in the one or more virtual machines. The first load balancing application is configured such that the second processor: receives a forwarding packet from a network device for a first application server; based on the health report, determines whether to perform at least one of a failover process or select a first network appliance of the network appliances; and performs a first iteration of a symmetric conversion to route the forwarding packet to the first application server via the selected first network appliance; receives a return packet from the first application server based on the forwarding packet; and performs a second iteration of the symmetric conversion to route the return packet to the network device via the first network appliance. During the failover process, the first load balancing application via the second processor switches routing of traffic between the network device and the first application server through a first instance of the first network appliance to routing the traffic between the network device and the first application server through a second instance of the first network appliance.
In other features, a load balancing method for operating a load balancing system implemented in one or more virtual machines of a cloud-based network is provided. The one or more virtual machines includes a first processor and a second processor. The method includes: executing a probing application on the first processor to transmit probe request messages to network appliances, where the probing application is implemented in the one or more virtual machines; based on the probe request messages, receiving response messages from the network appliances at the first processor; and based on the response messages and via the first processor, generating a health report message indicating states of health of the network appliances, where the network appliances are implemented in an appliance layer of the cloud-based network. The method further includes: receiving a forwarding packet from a network device for a first application server at the second processor; executing a first load balancing application and based on the health report, determining via the second processor whether to perform at least one of a failover process or select a first network appliance of the network appliances. The first load balancing application is implemented in the one or more virtual machines. The method further includes: performing via the second processor a first iteration of a symmetric conversion to route the forwarding packet to the first application server via the selected first network appliance; receiving a return packet at the second processor from the first application server based on the forwarding packet; performing via the second processor a second iteration of the symmetric conversion to route the return packet to the network device via the first network appliance; and while performing the failover process and via the second processor, switching routing of traffic between the network device and the first application server through a first instance of the first network appliance to routing the traffic between the network device and the first application server through a second instance of the first network appliance.
Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.
In the drawings, reference numbers may be reused to identify similar and/or identical elements.
A cloud may include network appliances including one or more load balancers. The load balancer balances flow of traffic from client stations to the network appliances. Packets are routed from the client stations to the network appliances for intermediary processing prior to being received at application servers. A large number of client stations may access the network appliances during the same period. For at least this reason, high availability of the network appliances is needed. High availability includes providing access to the network appliances for the user stations during the same period while providing a high level of data throughput at a high data rate through the network appliances.
Traditionally, layer 2 (L2) protocols and custom network setups have been utilized to assure high availability of network appliances. As a first L2 example, a cloud may include network appliances, such as two load balancers that communicate with each other and share a medium access control (MAC) address. The sharing of the MAC address between two devices is referred to as “MAC address masquerading”. The load balancers transmit to each other state signals, referred to as “heartbeat signals”. The heartbeat signals indicate states of health of the load balancers. A load balancer is healthy when the load balancer is able to reliably transmit and receive packets. In operation, one of the two load balancers that is healthy transmits a network protocol signal to, for example, a router in the cloud. The router then routes packets from a user station to the healthy load balancer. Examples of the network protocol signal include a gratuitous address resolution protocol (ARP) signal and an Ethernet frame for Internet control message protocol (ICMP) or ECMP. MAC address masquerading and IP address announcement may be performed by any network appliance, such as a firewall or a load balancer to achieve high availability.
As another L2 example, two load balancers may share the same Internet protocol (IP) address. The load balancer that is healthy or, if both of the load balancers are healthy, a primary one of the load balancers transmits a routing protocol signal indicating the healthy state of the corresponding load balancer. An example of the routing protocol is an optimum selective decoding scheme (OSDS) protocol. Traffic is then routed to the load balancer that was announced as being healthy.
In certain applications, such as in a public cloud application (e.g., the cloud computing platform Microsoft® Azure®), the L2 protocols and custom network setups are not available, difficult to implement and/or are not scalable. For example, MAC address masquerading is difficult to implement in a public cloud environment due to an inability to announce MAC addresses of healthy network appliances, inability to change IP addresses of network appliances, and virtualization of load balancing. As a result, availability of network appliances can be limited. Also, network appliances are typically implemented in corresponding VMs. VMs can fail, which can further reduce availability of the network appliances. A single VM is also functionally limited, such that the VM may not be able to provide enough throughput needed for a current load of one or more network appliances.
The examples set forth below include load balancing systems and methods that provide high availability of network appliances in a public cloud. The load balancing systems perform remote health monitoring of network appliances, load balancing of packets to the network appliances, and failover processing for the network appliances. The load balancing assures that reverse flow traffic returns through the same network appliance as corresponding forward traffic. Forward traffic (or forward packets) refers to packets transmitted from a user (or client) station to an application server. Reverse flow traffic (or return packets) refers to packets transmitted from an application server to the user station based on the forward packets.
The public cloud includes one or more virtual machines (VMs) 14A-D, 14E11-m, and 14F1-p (collectively referred to as VMs 14) that are implemented on one or more servers 16A-D, 16E1-m, and 16F1-p (collectively referred to as servers 16), where m, n and p are integers greater than or equal to 1. Each of the servers 16 includes and/or has access to memory. Although a single client station is shown, any number of client stations may be included and may access the public cloud as described herein with respect to the client station 12. Also, a distributed communication system 20, which may include a network (e.g., the Internet) may be connected between the client stations and the public cloud.
The load balancing system 10 is implemented in the public cloud and includes a controller 30, a probing application 32, a load balancing application 34, a router 36, network appliances 38, and server applications 40. Although the controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and application applications 40 are shown as being implemented in corresponding VMs and servers, the controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and server applications 40 may collectively be implemented in one or more VMs and/or in one or more servers. In one embodiment, instances of the controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and server applications 40 are grouped in any combination, where each of the combinations is implemented in a corresponding VM and one or more servers. An “instance” refers to a copy and/or another implementation of a same item. For example, the probing application 32 may be implemented multiple times in the public cloud, where each implementation is referred to as an instance of the probing application. By providing multiple instances the same services may be implemented by different VMs and/or by different servers. In another embodiment, the instances of the controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and server applications 40 are grouped in any combination, where each of the combinations is implemented in a corresponding server. Other example implementations are shown in
Also, although a certain number of instances of each of the controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and server applications 40 are shown in
In the following description, tasks stated as being performed by the controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and server applications 40 are performed by processors associated with and/or executing the controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and server applications 40. The controller 30 may be implemented as an application and/or a module and communicates with the client station 12, the probing application 32 and the load balancing application 34 and operates to setup an account for the client station 12 and/or user of the client station 12. The controller 30 controls setup of requested services for the client station 12. The services are executed by the probing application, 32, the load balancing application 34, the network appliances 38 and the server applications 40.
The probing application 32 monitors and tracks states of health of the network appliances 38 and reports the states of health to the load balancing application 34. The states of health indicate whether the network appliances 38 are operable to reliably receive, process and/or transmit packets. In one embodiment, reliable operation refers to receiving, processing and transmitting packets without more than a predetermined number of errors. The probing application 32 may transmit probe requests and the network appliances may respond with probe responses indicating the states of health, as is further described below with respect to the method of
The load balancing application 34 performs load balancing based on the states of health, performs symmetric conversions and controls routing of forward traffic and reverse traffic through the network appliances. A symmetric conversion includes converting a received packet based on one or more fields of the received packet to a resultant packet by adding a header to the received packet. The added header may be a service chain header (SCH). The added header includes one or more IP addresses of one or more of the network appliances. The number of IP addresses depends on the configuration of the network appliances and how many of the network appliances the received packet is to be passed through. As an example, the added header may include an IP address for each of the network appliances that the received packets are to be passed through prior to being received at one of the server applications 40.
The load balancing application 34 performs symmetric conversions to forwarding packets of forward traffic and to return packets of reverse traffic to assure that the return packets are sent through the same network appliance as the forwarding packets. If reverse flow traffic does not pass through the same network appliance as the corresponding forward traffic, the network appliance may falsely indicate that the network appliance is not healthy and/or reliable. The load balancing application 34 prevents this false indication and/or other related potential issues from occurring by routing the reverse flow traffic through the same network appliance. Reverse flow through the same network appliance becomes important when two or more network appliance instances are active and transferring packets.
The load balancing performed by the load balancing application 34 may be referred to as internal load balancing, which provides an internal load balancing virtual IP (VIP) address. The VIP address (referred to below as the IP address of a network appliance instance) is not a publically accessible IP address and is created and used by the load balancing application 34 to route packets to the network appliances 38.
The router 36 routes packets from the load balancing application 34 to the network appliances 38 based on the IP addresses in the added headers of the packets. The router 36 also routes return packets received from the server applications 40 via the load balancing application 34 to the client station 12.
The network appliances 38 are intermediary service modules and/or applications that are implemented in an appliance service layer of the public cloud and perform intermediary services which may be requested by the client station 12. The network appliances 38 may be implemented as and/or include software and/or hardware. Examples of the network appliances 38 are load balancers, firewalls, proxy servers, World Wide Web (or Web) servers, wide area network (WAN) accelerators, IDS devices, and IPS devices. One or more types of the network appliances 38 may be requested by a user (or subscriber) and associated with the client station 12. The controller 30, when setting up the account for the client station, may associate multiple instances of each of the types of network appliances requested. As an example, the network appliances 38 are shown in rows and columns, where each row may include instances of a same type of network appliance and/or implementation of a same network appliance. For example, if the client station requests application load balancing services and firewall services, the network appliances11-1n may be instances of a load balancer and the network appliances21-2, may be instances of a firewall module, where n is an integer greater than or equal to 2. The firewall module instances may include and/or be implemented as firewalls. Multiple client stations may share the same instances of any one or more of the network appliances 38.
The server applications 40 include particular subscriber applications. The server applications 40 are implemented in an application service layer of the public cloud and are implemented in a device, VM and/or server that terminates one or more connections with the client station. Termination of a connection refers to a device, VM and/or server that is implemented as an end device and that performs an end process for a client station. No additional processing and/or subscriber applications are located downstream from the server applications 40. The server applications 40 process forwarding packets received from the client station 12 via the network appliances 38 and generate return packets, which are sent back to the client station 12. The servers in which the server applications 40 are implemented may be referred to as application servers.
In
The processor 104 of the client station 100 executes an operating system 114 and one or more client applications 118. The client station 100 further includes a wired interface (such as an Ethernet interface) and/or wireless interface (such as a Wi-Fi, Bluetooth, near field communication (NFC) or other wireless interface (collectively identified at 120)) that establishes a communication channel over the distributed communication system and/or network 20. The distributed system and/or network may include the Internet. The client station 100 further includes a display subsystem 124 including a display 126.
In
The processor 152 executes a server operating system 158 and one or more server applications 160 and VM applications. An example of a server application is a virtual server service application 162, which is implemented in a virtualization layer and is executed along with the server operating system (OS) 158. The virtual server service application 162 creates a virtual environment in which VM (or guest) OSs (e.g., VM1 OS and VM2 OS) run. Example VM applications App 1A, App 1B, App 3, App 4 are shown as being implemented in VM memories 164, 166 of VMs 168, 170. The VM applications may include instances of the probing application 32, the load balancing application 34, the server applications 40 and/or applications of the controller 30 and the network appliances 38 of
In the example shown in
Any of the instances of the VMs 1801-N and 184, 185 and any other instances of the server memories 176, 177 may be implemented as instances of the client station 12, controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and server applications 40 of
Processors of the servers associated with the server memories 176, 178 may be in communication with each other and/or transfer data between each other via corresponding interfaces. Examples of the processors and the interfaces are shown by the processor 152 and the interface 172 of
Operations of the client station 12, controller 30, probing application 32, load balancing application 34, router 36, network appliances 38, and server applications 40 of
Although the following tasks are primarily described with respect to the implementations of
The method may begin at 200. At 202, services, such as probing services, load balancing services for network appliances (e.g., some of the network appliances 38), network appliance services, virtual machine services, server application services, etc. are setup for the client station 12. The controller 30 is configured to cause a processor executing the controller 30 to setup the services. The client station 12 requests certain services from a service provider and the service provider via a public cloud and the controller 30 sets up the requested services and/or other corresponding services. The setup of the services includes associating selected and/or predetermined numbers of selected types of instances of network appliances with the client station 12. Setup of the services is further described below with respect to the method of
At 204, if health monitoring services are enabled for the client station at 202, the probing application 32 is configured to cause a processor executing the probing application 32 to monitor health statuses of the associated network appliances and report the health statuses to the load balancing application 34. In addition or as an alternative, the associated network appliances may determine respectively the health statuses and report the health statuses to the load balancing application 34. A health monitoring method that may be performed at 204 is described below with respect to
At 206, a forwarding process is performed. The forwarding process includes (i) receiving and forwarding a packet from the client station 12 to one of the server applications 40, (ii) load balancing instances of network appliances, and (iii) performing a failover process for one of the instances. The load balancing of network appliances includes determining which network appliances to send forwarding packets. As described above, each of the forwarding packets includes a first header. The load balancing may include encapsulating and/or adding a second header to each of the forwarding packets, as described above. The failover process includes determining whether to change traffic flow of forwarding packets and return packets from a currently used instance of a network appliance to another instance of the currently used network appliance. This is based on a change in health status of the currently used instance. If the health status of the currently used instance degrades to a level no longer permitted for providing services, the load balancing application 34 is configured to cause a corresponding processor to change instances. The forwarding process is further described below with respect to steps 300-310 of the load balancing method shown in
Although the following tasks are primarily described with respect to processing of a single forwarding packet and a single return packet, multiple forwarding and return packets may be processed and handled by assigned instances of the associated network appliances, which are healthy. The forwarding and return packets may be handled in series or in parallel. At 208, a healthy instance of one of the network appliance that received the forwarding packet performs a corresponding network appliance service. Prior to performing task 210, the healthy instance may remove the second header and encapsulation provided during task 206.
At 210, the forwarding packet is sent from the healthy instance of the network appliance to one of the server applications 40 to perform a server application service. This may be based on a destination IP address and/or a destination IP port number in the first header of the healthy instance.
At 212, the server application that received the forwarding packet is configured to cause a processor of the server application to perform server application servicing including generation of a return packet associated with the forwarding packet.
At 214, a reverse process is performed. The reverse process includes (i) generating and routing a reverse packet from the server application, which received the forwarding packet, to the client station 12, (ii) load balancing instances of network appliances, and (iii) performing a failover process for one of the instances. The load balancing of network appliances includes determining which network appliances to send return packets. As described above, each of the return packets includes a first header. The load balancing may include encapsulating and/or adding a second header to each of the return packets, as described above. In one embodiment, the second header is a SCH. The failover process includes determining whether to change traffic flow of forwarding packets and return packets from a currently used instance of a network appliance to another instance of the currently being used network appliance. This is based on a change in health status of the currently used instance. If the health status of the currently used instance degrades to a level no longer permitted for providing services, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to change instances. The reverse process is further described below with respect to steps 312-326 of the load balancing method shown in
Although the following tasks are primarily described with respect to the implementations of
The method may begin at 230. At 232, the processor of the controller 30 in the public cloud receives a service request message from the client station 12 to request setup of services. The client station 12 accesses the public cloud and sends a service request message to the controller 30. The service request message indicates services requested by a user. The services may include probing (or health monitoring) services of network appliances, load balancing services of network appliances, load balancing services of server applications, failover services for one or more network appliances, and/or other services associated with different types of network appliances. The other services may include failover services of one or more devices, appliances, modules, virtual machines, and/or servers in the public cloud and associated with the client station 12. The other services may also include firewall services, proxy server services, World Wide Web (or Web) server services, WAN accelerator services, IDS device services, and/or IPS device services. A customer associated with the client station 12 may move processing associated with a workload to the public cloud and request certain capabilities. The capabilities include the selected services.
At 234, the controller 30 sets up services for the client station 12. The controller 30, if probing services and load balancing services of network appliances are selected, informs the one or more processors corresponding probing application 32 and the load balancing application 34 to perform the respective services for the client station 12. This may include informing one or more processors of the probing application 32 and the load balancing application 34 a selected and/or predetermined number of instances of each network appliance and/or module in the public cloud to provide services for the client station 12. The number of instances may be indicated via (i) a first setup signal transmitted from the controller 30 to the probing application 32, as represented by task 236, and (ii) a second setup signal transmitted from the controller 30 to the load balancing application 34, as represented by task 238. The numbers of instances may be selected by the client station 12 and indicated in the service request signal. In one embodiment, the numbers of instances are not provided and predetermined numbers of instances are created and/or associated with the client station 12. The numbers of instances of the network appliances indicates to the one or more processors of the applications 32, 34 the number of network appliances to probe for health status checks and provide packets to during load balancing. The method may end at 240.
The following tasks of the methods of
The method may begin at 260. At 262, the probing application 32 is configured to cause a corresponding processor of the probing application 32 to perform health monitoring. This includes determining which instances of the network appliances 38 to monitor based on the first setup signal generated by the controller 30. In the described example, the probing application 32 is operating in the active setup mode.
At 264, the probing application 32 is configured to cause the processor of the probing application 32 to transmit probe request messages to the instances of the network appliances (e.g., ones of the network appliances 38 associated with the client station 12). The probe request messages request health statuses of the network appliances.
At 266, the probing application 32 is configured to cause the processor of the probing application 32 to receive probe response messages from the instances of the network appliances 38 in response to the probe request messages. Each of the probe response messages indicates a health status of the corresponding network appliance. The health status may include a value indicating whether the network appliance is able to reliably receive, process, and/or transmit packets. The health status may also indicate whether return packets have been received for corresponding forwarding packets. A network appliance, based on the forwarding packet, may determine that a return pack is expected to be provided from a server application based on the forwarding packet. If the network appliance does not receive the return packet, the network appliance may indicate non-receipt of the return packet and/or may reduce a health status ranking of the network appliance. In one embodiment, the probe response messages are provided to the processor of probing application 32. In another embodiment, the network appliances 38 send the probe response messages and/or states of health directly to the processor of the load balancing application 34. The processor of the load balancing application 34 may be the same or a different processor than the processor of the probing application 32.
At 268, the probing application 32 is configured to cause the processor of the probing application 32 to store the health statuses of the network appliances being monitored in a memory (e.g., a memory of a server and/or a VM memory) associated with the probing application 32. The health statuses may be stored for example as a health status table relating health statuses and/or heath status rankings to the instances of the monitored network appliances. The health statuses and/or health status rankings are for more instances than the number of instances through which packets are being passed. This allows for failover processes disclosed herein.
At 270, the probing application 32 is configured to cause the processor of the probing application 32 to generate a health report message, which includes health statuses of the network appliances 38 and/or the network appliances associated with the client station 12. The health report message is generated based on content in the probe response messages. The health statuses may be provided in tabular form including health status rankings for the network appliances relative to IP addresses and/or other identifiers (e.g., port numbers, device numbers, etc.) of the network appliances 38.
At 272, the health report message is transmitted to the processor of the load balancing module 34. At 274, the probing application 32 determines whether to update health statuses of one or more of the instances of the network appliances 38. If a health status is to be updated task 262 may be performed, otherwise the method may end at 276.
The method may begin at 300. At 302, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to receive a forwarding packet from the client station 12. In one embodiment, the forwarding packet includes a first header and a data payload. The first header includes header information, such as a source IP address of the forwarding packet, a destination IP address of the forwarding packet, a source port number, a destination port number, a protocol field, etc. The source port number and destination port number may be transmission control protocol (TCP) port numbers, user datagram protocol (UDP) port numbers, or layer four (L4) port numbers.
At 304, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to determine whether the forwarding packet is a first packet received from the client station 12, which is to be transmitted to the processor of a server application. If the forwarding packet is a first forwarding packet received, task 310 is performed, otherwise task 306 is performed.
At 306, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to determine whether a state of health of an instance of a network appliance has changed and/or whether to perform a failover process for the network appliance. If the health status ranking of the instance of the network appliance has degraded to a predetermined level, the load balancing application 34 initiates a failover process by performing task 308 to switch from the degraded instance of the network appliance to a healthy instance of the network appliance. If a health status ranking has not degraded, task 310 may be performed. In one embodiment, the instance is healthy when it has a health status ranking greater than the predetermined level.
At 308, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to perform the failover process to change instances of the network appliance having the degraded health status ranking. As an example, if the instances are load balancers (or load balancing modules), then failover of a load balancer is performed. Both future forward traffic from the client station 12 and reverse traffic to the client station 12 are changed from the degraded instance to the healthy instance. In one embodiment, this is done by incorporating the IP address of the healthy instance in a second header added to forwarding and return packets. As examples, the second headers may be added below during tasks 310a and 320a.
The failover process that is performed for the network appliance is different than a failover process performed for a server and/or a virtual machine. The instances of the network appliance may be on different or the same servers and/or virtual machines. As an example, the instances may be implemented in the same server and/or in the same virtual machine. Thus, when a failover process of a network appliance is performed, the load balancing application 34 may not switch between servers and/or virtual machines. As another example, two or more instances of a network appliance may be on a same server and/or virtual machine while one or more other instances of the network appliance may be implemented on other servers and/or virtual machines.
At 310, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to perform load balancing of network appliances. If the network appliances are load balancers, load balancing is performed for the load balancers. At 310A, based on the health report message, the probe response messages, and/or the states of health of the network appliances, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to perform a symmetric conversion (or first symmetric conversion) of the forwarding packet including determining to which network appliance (or network appliance instance) to send the forwarding packet. In one embodiment, a second header is added to the forwarding packet that includes an IP address for the selected network appliance instance. The load balancing application 34 may be configured to cause the processor of the load balancing application 34 to encapsulate the forwarding packet when adding the second header to generate a first resultant packet.
In one embodiment, a hashing function is evaluated to determine the network appliance instance and/or IP address of the network appliance instance. The hash function may be implemented as and/or may be based on a symmetric function. An example, symmetric function is exclusive-OR (XOR). In one embodiment, an XOR is taken of two fields of the first header of the forwarding packet. The XOR function is symmetric because XOR of fields (a, b) is the same as XOR of fields (b, a). As an example, the fields a, b are the source IP address and destination IP address of the forwarding packet. As another example, the fields a, b are the source port number and destination port number of the forwarding packet. As an example, equation 1 may be used to determine the IP address of the network appliance instance, where H(X, XOR) is the hash function of a key X and the XOR value, n is an integer greater than or equal to 2 and represents a total number of instances of the network appliance that are associated with the client station and/or are healthy, IP is value representing the IP address of the network appliance instance selected, and mod is the modulo operation. The key X may be an integer between 0 and n. The hash function may be based on i) the XOR value, or ii) X and the XOR value. The hash function is used to map the key value X and/or the XOR value to an IP address representative value.
IP=H(X,XOR)mod n (1)
In an embodiment, the hash function is performed on a 2-tuple, 3-tuple, 4-tuple, or 5-tuple first header. The hash function may be based on one or more XOR values generated by performing the XOR operation on respective pairs of fields of the first header and/or forwarding packet. The first header may include any number of fields. A 2-tuple first header may include a source IP address and a destination IP address, where the XOR function is performed on both fields. A 3-tuple first header may include a source IP address, a destination IP address, and a protocol field, where the XOR function is performed one or more times; each time including two of the fields. A 4-tuple first header may include a source IP address, a destination IP address, a source port number, and a destination port number, where the XOR function is performed one or more times; each time including two of the fields. A 5-tuple first header may include a source IP address, a destination IP address, a source port number, a destination port number, and a protocol field, where the XOR function is performed one or more times; each time including two of the fields.
Alternative symmetric hash functions and/or symmetric conversion may be performed. As an example, a symmetric table may be used to look up the IP addressed based on one or more fields of the forwarding packet. The symmetric conversion assures that the same IP address is provided for both the forwarding packet and a return packet associated with the forwarding packet. For example, XOR of a source IP address and a destination IP address is the same as XOR of the destination IP address and the source IP address.
The load balancing application 34 may also be configured to cause the processor of the load balancing application 34 to determine the network appliance instance based on other parameters, such as health status rankings of the network appliance instances. A selection may be made between the network appliance instances with the highest health status ranking and/or with health status rankings above a predetermined health status ranking.
As another example, the selection of the network appliance instance may also be based on an amount of traffic being provided to each of the instances of the network appliance. In one embodiment, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to balance the amount of traffic flow across the healthy ones of the network appliances. In an embodiment, forwarding packets and/or return packets that are associated with each other are sent through the same network appliance. At 310B, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to send the forwarding packet (or first resultant packet) to the IP address of the selected network appliance instance. In one embodiment, tasks 208-212 of
At 312, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to receive a return packet from the server application. The return packet is generated based on the forwarding packet. The server application is configured to cause the processor of the server application to send the return packet to the load balancing application 34.
At 314, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to determine whether the return packet is a first return packet received from the processor of the server application, which is to be transmitted to the client station. If the return packet is a first return packet received, task 320 is performed, otherwise task 316 is performed.
At 316, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to determine (i) whether the state of health of the instance of the network appliance that received the forwarding packet has changed, and/or (ii) whether to perform a failover process for the network appliance. If the health status ranking of the instance of the network appliance has degraded to a predetermined level, the load balancing application 34 initiates a failover process by performing task 318 to switch from the degraded instance of the network appliance to a healthy instance of the network appliance. If a health status ranking has not degraded, task 320 may be performed. In one embodiment, the instance is healthy when it has a health status ranking greater than the predetermined level.
At 318, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to perform the failover process to change instances of the network appliance having the degraded health status ranking. As an example, if the instances are load balancers (or load balancing modules), then failover of a load balancer is performed. Both future forward traffic from the client station 12 and reverse traffic to the client station 12 are changed from the degraded instance to the healthy instance. In one embodiment, this is done by incorporating the IP address of the healthy instance in a second header added to forwarding and return packets. As examples, the second headers may be added below during 320A.
The failover process that is performed for the network appliance is different than a failover process performed for a server and/or a virtual machine. The instances of the network appliance may be on different or the same servers and/or virtual machines. As an example, the instances may be implemented in the same server and/or in the same virtual machine. Thus, when a failover process of a network appliance is performed, the load balancing application 34 may not switch between servers and/or virtual machines. As another example, two or more instances of a network appliance may be on a same server and/or virtual machine while one or more other instances of the network appliance may be implemented on other servers and/or virtual machines.
At 320, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to perform load balancing of the network appliances similarly to the load balancing described above with respect to task 310 except load balancing is performed for a return packet instead of for a forwarding packet. At 320A, based on the health report message, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to perform a symmetric conversion (or second symmetric conversion) of the return packet including determining to which network appliance (or network appliance instance) to send the return packet. The conversion process used for the return packet is the same as the conversion process used for the forwarding packet to assure that the return packet is sent back through the same network appliance as the forwarding packet. In one embodiment, a second header is added to the return packet that includes an IP address for the selected network appliance instance. The load balancing application 34 may be configured to cause the processor of the load balancing application 34 to encapsulate the return packet when adding the second header to generate a second resultant packet.
In one embodiment, the hashing function used at 310 is evaluated to determine the network appliance instance and/or IP address of the network appliance instance. The hash function may be implemented as and/or may be based on the symmetric function and be used to provide a same IP address as determined at 210. The XOR function may be used. In one embodiment, an XOR is taken of two fields of the first header of the return packet. As an example the fields are the source IP address and destination IP address of the return packet. As another example, the fields are the source port number and destination port number of the return packet. As an example, equation 1 may be used to determine the IP address of the network appliance instance.
In an embodiment, the hash function is performed on a 2-tuple, 3-tuple, 4-tuple, or 5-tuple first header of the return packet. The hash function may be based on one or more XOR values generated by performing the XOR operation on respective pairs of fields of the return packet and/or the first header of the return packet. The first header may include any number of fields. A 2-tuple first header may include a source IP address and a destination IP address, where the XOR function is performed on both fields. A 3-tuple first header may include a source IP address, a destination IP address, and a protocol field, where the XOR function is performed one or more times; each time including two of the fields. A 4-tuple first header may include a source IP address, a destination IP address, a source port number, and a destination port number, where the XOR function is performed one or more times; each time including two of the fields. A 5-tuple first header may include a source IP address, a destination IP address, a source port number, a destination port number, and a protocol field, where the XOR function is performed one or more times; each time including two of the fields.
Alternative symmetric hash functions and/or symmetric conversion may be performed during task 320A. As an example, a symmetric table may be used to look up the IP addressed based on one or more fields of the return packet.
The load balancing application 34 may be configured to cause the processor of the load balancing application 34 to determine the network appliance instance based on other parameters, such as health status rankings of the network appliance instances. A selection may be made between the network appliance instances with the highest health status ranking and/or with health status rankings above a predetermined health status ranking. As another example, the selection of the network appliance instance may also be based on an amount of traffic being provided to each of the instances of the network appliance. In one embodiment, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to balance the amount of traffic flow across the healthy ones of the network appliances. At 320B, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to send the return packet (or second resultant packet) to the IP address of the selected network appliance instance.
At 322, the network appliance instance sends the return packet to the client station via the router 36. Prior to sending the return packet to the router 36, the network appliance instance may remove the second header and encapsulation provided during task 320A.
At 324, the load balancing application 34 is configured to cause the processor of the load balancing application 34 to determine whether another packet has been received by the client station 12. If another packet has been received, task 304 is performed; otherwise the method may end at 326. Tasks 302-324 may be performed in an overlapping manner and/or multiple versions of tasks 302-324 may be performed in parallel. For example, in one embodiment, tasks 302-324 are performed for subsequent packets prior to completion of tasks 302-324 for previous packets. This increases transfer rates.
The above-described tasks of
The above-described method is network service header (NSH) compatible. The symmetric conversions described may include performing a symmetric conversion of one or more fields of a NSH of a packet. The described failover processes, load balancing of network appliances, and health status monitoring and reporting are compatible with packets having NSHs.
The above-described method provides a load balancing system and method that provides high availability of network appliances and ensures reverse traffic flow passes through a same network appliance instance as corresponding forwarding packets. A load balancer is provided that operates in an standby setup mode or an active setup mode and while in the active setup mode provides a n-active failover solution, where n is an integer greater than or equal to 2 and represents a number of instances of a network appliance. If n instances are provided, as many as n−1 backup instances of the network appliance are provided. In one embodiment, a symmetric has function including an XOR of a 5-tuple is performed to ensure reverse traffic is affinitized to a same network appliance instance as corresponding forward traffic.
The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims. It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure. Further, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the disclosure can be implemented in and/or combined with features of any of the other embodiments, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with one another remain within the scope of this disclosure.
Spatial and functional relationships between elements (for example, between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements. As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”
In the figures, the direction of an arrow, as indicated by the arrowhead, generally demonstrates the flow of information (such as data or instructions) that is of interest to the illustration. For example, when element A and element B exchange a variety of information but information transmitted from element A to element B is relevant to the illustration, the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is transmitted from element B to element A. Further, for information sent from element A to element B, element B may send requests for, or receipt acknowledgements of, the information to element A.
In this application, including the definitions below, the terms “module”, “processor” and/or “controller” may be replaced with the term “circuit.” The term terms “module”, “processor” and/or “controller” may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.
A module, processor and/or controller may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces that are connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof. The functionality of any given module, processor and/or controller of the present disclosure may be distributed among multiple module, processor and/or controller that are connected via interface circuits. For example, multiple modules may provide load balancing. In a further example, a server (also known as remote, or cloud) module may accomplish some functionality on behalf of a client module and/or client station.
The term code, as used above, may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules. The term group processor circuit encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules. The term group memory circuit encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more modules.
The term memory circuit is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory. Non-limiting examples of a non-transitory, tangible computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).
In this application, apparatus elements described as having particular attributes or performing particular operations are specifically configured to have those particular attributes and perform those particular operations. Specifically, a description of an element to perform an action means that the element is configured to perform the action. The configuration of an element may include programming of the element, such as by encoding instructions on a non-transitory, tangible computer-readable medium associated with the element.
The apparatuses and methods described in this application may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to execute one or more particular functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.
The computer programs include processor-executable instructions that are stored on at least one non-transitory, tangible computer-readable medium. The computer programs may also include or rely on stored data. The computer programs may encompass a basic input/output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.
The computer programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language) or XML (extensible markup language), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. As examples only, source code may be written using syntax from languages including C, C++, C#, Objective C, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5, Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, and Python®.
None of the elements recited in the claims are intended to be a means-plus-function element within the meaning of 35 U.S.C. §112(f) unless an element is expressly recited using the phrase “means for,” or in the case of a method claim using the phrases “operation for” or “step for.”
