Patent Grant
8218454
The present disclosure relates generally to communication systems and, more particularly, to methods and apparatus to implement scalable routing in network communication systems.
Network service providers enable data communication services using networks interconnected via switches and routing devices including provider edge (PE) routers and customer edge (CE) routers. Customer edge routers communicatively couple customer equipment (e.g., computers and other processor systems, local routers, and local switches) to wide area networks (WANs) via PE routers. Provider edge routers are communicatively coupled to other PE routers across a WAN to enable communication between local CE routers (e.g., CE routers coupled to a particular PE router) and remote CE routers (e.g., CE routers coupled to different PE routers). To deliver data, each PE router is provided with routes (e.g., network paths) that can be used to forward data packets based on destination addresses stored therein. A PE router stores each route in a routing table, and retrieves that route from the routing table each time the PE router receives a data packet having a destination address matching the destination address associated with that route.
As service providers expand their networks, additional routes are brought on line to enable data communications with the new portions of the network. Each time an additional route is created, PE routers to which that route is relevant must store the route in their routing tables. As the quantity of routes increase, so do the requirements for memory capacity to store those routes in each router.
Enterprise customers having multiple locations or sites are increasingly adopting multi-protocol label switching (MPLS)-based virtual private network (VPN) services to implement communication networks among their respective sites via a service provider's network. Such MPLS-based VPNs provide direct any-to-any reachability among sites of the enterprise.
The example methods and apparatus disclosed herein can be used to implement scalable multi-protocol label switching (MPLS) based networks having multiple virtual private networks (VPNs). Enterprise networks are increasingly adopting VPNs, including MPLS VPNs, to connect geographically separated locations of the enterprise. A VPN architecture, known as a Layer 3 MPLS VPN, typically offers direct any-to-any reachability among different customer sites of an enterprise. However, implementing such an any-to-any reachability model requires a relatively large memory footprint among provider routers to store possible routes, causing routing tables of those provider routers to grow very large (e.g., some VPNs can contain more than 10,000 routes). As a result, having sufficient memory is a significant hurdle for providing any-to-any reachability when provisioning customers on a Provider Edge (PE) router.
Relaying is a technique that can be used to reduce the PE memory footprint of VPNs by having select hub PE routers in each VPN maintain full reachability information (e.g., all routes) for that VPN, and enabling non-hub PEs (spoke PEs) to maintain only partial reachability information (e.g., some routes), while being able to reach other routers by relaying communications through hub PE routers. Relaying techniques are disclosed in U.S. application Ser. No. 12/130,329, filed May 30, 2008, and titled “Scalable Multiprotocol Label Switching Based Virtual Private networks and Methods to Implement the Same,” which is hereby incorporated herein by reference in its entirety.
The example methods and apparatus disclosed herein can be used to implement relaying techniques in a multi-VPN environment by selecting hub PE routers and spoke-to-hub assignments of spoke PEs based on how such selections and assignments affect the performance of all or at least some of the VPNs in the multi-VPN environment. For instance, while routing entries in routing tables are specific to each VPN, a provider network and individual PEs are shared resources serving many different VPNs. Thus, the techniques disclosed herein can be used to find optimal or near-optimal spoke-to-hub assignments associating spoke PEs to hub PEs based on network characteristics (e.g., traffic volumes, bandwidth usages, costs, etc.) and constraints (e.g., bandwidth constraints).
Different VPNs can have very different characteristics in terms of size, traffic patterns, and routing table sizes. Decisions made for one VPN, such as hub PE router selections, can affect the performance of other VPNs in different ways including, for example, the amount of available free PE router memory or the available free capacity on the ingress and egress links of a PE router available to all of the VPNs served by that PE router.
The example methods and apparatus disclosed herein involve a constraint optimization approach that may consider several factors including memory utilization and bandwidth usage constraints at each PE router, the cost of transporting traffic across the network, and the needs for maintaining fault-tolerant resilience for each VPN to withstand network failures. The constraint optimization approach is implemented using a cost model analysis based on several constraints to observe the potential impacts on different VPNs in a multi-VPN environment that would result from selecting different PE routers as hubs and creating certain spoke-to-hub assignments for spoke PEs. In the example implementations described herein, the cost model reflects: (a) the fixed cost associated with the procurement, deployment, and maintenance of each physical PE router; (b) the utilization cost indicative of the capacities of the physical PE routers used by the VPNs; and (c) the transport cost indicative of the distance traversed by traffic that is relayed. Fixed costs associated with a PE router include costs to cover the hardware purchase, the deployment costs, and the maintenance costs regardless of the usage (e.g., bandwidth usage of memory utilization) of the PE router. Utilization costs are a function of multiple router resources including the memory utilization that is driven by the number of routes installed (i.e., ratio of memory used on a PE router to the total memory available), the central processing unit (CPU) utilization that is driven by border gateway protocol (BGP) sessions, customer features and traffic load, and the number of spare ports available on a particular router. The transport cost is a function of communication latency because the added distance traversed in a backbone is approximately proportional to the added latency. The transport cost is a direct cost for a provider because it involves additional resource consumption on a provider network backbone due to the increase in the distances traversed by traffic that is relayed through hub PE routers.
As disclosed herein, some example methods involve selecting a plurality of virtual private networks in a communication network and selecting a candidate hub router from a plurality of routers in the communication network. The example methods also involve determining a spoke-to-hub assignment solution having a least memory utilization cost relative to a plurality of other spoke-to-hub assignment solutions. Each of the spoke-to-hub assignments solution and the plurality of other spoke-to-hub assignment solutions has a different proposed combination of spoke routers assigned to a candidate hub router. In addition, the example methods involve determining a spoke router quantity associated with a least memory utilization relative to a plurality of other spoke router quantities. Each of the spoke router quantity and the plurality of other spoke router quantities are representative of different quantities of spoke routers proposed to be assigned to the candidate hub router. Also, the example methods involve determining whether to configure the candidate hub router to operate as a hub router for one of the proposed combination of spoke routers based on the least memory utilization cost and the spoke router quantity associated with the least memory utilization.
In accordance with other example methods disclosed herein, a plurality of virtual private networks in a communication network are selected and a candidate hub router from a plurality of routers in the communication network is also selected. The example methods also involve determining a plurality of least memory utilization costs associated with assigning different quantities of spoke routers to the candidate hub router. Each of the least memory utilization costs and corresponding quantity of spoke routers is associated with a corresponding one of the plurality of virtual private networks. For each of the least memory utilization costs, a bandwidth usage of a corresponding one of the quantities of spoke routers is identified. The candidate hub router is selected to operate as a hub router for one of the quantities of spoke routers associated with a corresponding one of the bandwidth usages that does not exceed a bandwidth capacity of the candidate hub router.
Throughout the following disclosure references are made to the example MPLS-based multi-VPN network 400 of
Turning now to
For each VPN implemented by the service-provider network 105 (e.g., a VPNx communicatively coupling the CEx routers), each of the example PE routers PE1, PE2 and PE3 of
Each of the example PE routers PE1-PE5 of
To identify a particular CE router CE1-CE5, each of the example entries 210 of
To identify how a packet addressed to a particular CE router CE1-CE5 is to be routed, each of the example entries 210 of
When a packet is received at a PE router PE1-PE5 from a CE router CE1-CE5, the PE router PE1-PE5 uses the final destination of the packet to perform a query of its VRF table 205. The PE router PE1-PE5 performs the query by comparing the final destination address or label contained in a header of the packet with prefixes in the prefix fields 215. Based on the query, the PE router PE1-PE5 determines how the packet is to be routed or forwarded. If the reference field 220 associated with the destination prefix contains a reference to another PE router PE1-PE5, the PE router PE1-PE5 prepends an MPLS label corresponding to the identified PE router PE1-PE5 to the packet, and forwards the prepended packet to the identified PE router PE1-PE5. If the reference field 220 contains a reference to a CE router CE1-CE5 communicatively coupled to the VPN, the PE router PE1-PE5 routes the packet to the CE router CE1-CE5 without prepending an MPLS label.
When a packet is received at a PE router PE1-PE5 from another PE router PE1-PE5, the PE router PE1-PE5 removes the prepended MPLS label, performs a query of its VRF table 205 to identify whether the CE router CE1-CE5 to which the packet is addressed is coupled to the VPN via the PE router PE1-PE5. If the identified CE router CE1-CE5 is coupled to the VPN via the PE router PE1-PE5, the PE router PE1-PE5 delivers the packet to the identified CE router CE1-CE5.
Because the example spoke routers PE2, PE4 and PE5 of
Each of the PE routers PE1-PE5 can have installed on it one or more of the VPNs VPN1-VPN4. For example, one of the PE routers PE1-PE5 can carry traffic for one or more of the VPNs VPN1-VPN4, while another one of the PE routers PE1-PE5 can carry traffic for one or more of the same or different ones of the VPNs VPN1-VPN4. When a VPN is installed on a PE router (i.e., a physical PE router), a virtual PE (VPE) router is created. A virtual PE router is a logical representation of the portion of a PE router that serves a particular VPN. When two or more VPNs are installed on a single PE router, the same number of VPE routers are created on that PE router. Each VEP router corresponds to a respective logical portion of the PE router that serves a corresponding one of the installed VPNs. Turning briefly to
Returning to
To select which of the PE routers PE1-PE5 are to operate as hub routers and which are to operate as spoke routers, the example network 408 is provided with an example multi-VPN hub selector 404 and an example multi-VPN traffic monitor 406. The multi-VPN traffic monitor 406 monitors the links connecting each of the PE routers PE1-PE5 to its adjacent one or more of the PE routers PE1-PE5 using, for example, the Cisco Systems® NetFlow Services Export protocol. Records collected by the multi-VPN traffic monitor 406 allow the source, destination and volume of flows to be measured and/or recorded. Traffic data may be collected by the multi-VPN traffic monitor 406 over any interval(s) of time, such as weeks.
Using the traffic data collected by the multi-VPN traffic monitor 406, the multi-VPN hub selector 404 selects which of the PE routers PE1-PE5 are to serve as hub routers. In the illustrated example of
In equation 1 above, the coefficients (α), (β), and (γ) (used as multipliers) represent monetary costs, (|H|) is the number of hub PE routers, (ui) is the memory utilization of PE router (i), (νijk) is the traffic volume exchanged between PE router (i) and PE router (j) for VPN (k) during a particular instance in time or duration (i.e., traffic volume per time (Mbps), (lijk) is the additional latency due to the additional distance d(i, hub(i,k))+d(hub(i,k), j)−d(i,j) where hub(i,k) is the hub assigned to PE router (i) for VPN (k) and d(i,j) is the distance between PE router (i) and PE router (j). In equation 1, the coefficient multipliers (α), (β), and (γ) enable converting the PE router and VPN operating characteristics of memory utilization (ui), traffic volume per time (νijk) (i.e., bandwidth usage), and latency (lijk) to a monetary cost. In this manner, the operating characteristics associated with a multi-VPN network (e.g., the multi-VPN network 400 of
In the cost model of equation 1, a fixed equipment cost coefficient (α) is used to determine the costs associated with the procurement, deployment, and maintenance of each physical PE router (e.g., the PE routers PE1-PE5); the utilization cost coefficient (β) is used to determine the costs associated with the capacities of the physical PE routers utilized by the VPNs (e.g., the VPNs VPN1-VPN4 of
The multi-VPN hub selector 404 can re-evaluate the cost model of equation 1 on a periodic or aperiodic basis as network needs or characteristics change. During each evaluation of the cost model, the multi-VPN hub selector 404 can compare the costs of selecting different hub routers and spoke-to-hub assignment solutions. Each spoke-to-hub assignment solution has one of the PE routers PE1-PE5 selected to operate as a hub router and one or more others of the PE routers PE1-PE5 assigned as spokes to the hub router. Based on the cost comparisons, the multi-VPN hub selector 404 can determine which spoke-to-hub assignment solution results in the minimum or near-minimum cost. For example, the multi-VPN hub selector 404 can evaluate the cost model for a spoke-to-hub assignment solution and compare that cost to another cost associated with another spoke-to-hub assignment solution. In similar manner, the multi-VPN hub selector 404 can compare other costs associated with different spoke-to-hub assignments. The multi-VPN hub selector 404 can then select one or more of the spoke-to-hub assignment solutions for each VPN that result(s) in a minimum or near-minimum cost.
In some example implementations, constraints can also be used as additional factors in selecting spoke-to-hub assignments. That is, spoke-to-hub assignments can be selected while minimizing the cost represented by the cost model of equation 1 so long as certain constraints are not violated. An example constraint is a reliability constraint, which requires that communication paths remain reliable even during the failure of a particular number of PE routers. To ensure such reliability, each spoke PE router can be assigned to a certain quantity of hubs equal to a reliability parameter (ρ), for a constant integer (ρ>0), to provide a topology that is survivable when up to (ρ−1) PE router failures occur. Thus, each VPE router (e.g., the VPE routers VPE1-VPE3 of
Another example constraint is a latency constraint, which can require that no packet incurs a latency increase of more than a maximum latency parameter (θ) so that the experience of the end user is not impacted by the relaying architecture. To avoid such an increased latency, a spoke router should not be assigned (e.g., logically connected) to a hub router if doing so would increase the latency of the spoke router to be more than a value of the latency parameter (θ) by forcing traffic from the spoke router to pass through the (intermediate) hub router.
Yet another example constraint is a bandwidth usage limit constraint (bi), which can require that the total traffic volume during a given duration (i.e., bandwidth usage) through any single PE router (i) should not exceed a bandwidth limit value. Another example constraint is a memory utilization limit constraint, which defines the maximum number of routes that can be stored in a VRF table of any single PE router. In this manner, the memory utilization of any single PE router is bounded by a configurable memory utilization upper limit (i.e., a memory limit value).
In addition to minimizing the cost represented by the cost model of equation 1, hub routers and spoke-to-hub assignment solutions can be selected in a manner that balances the routing table size across the PE routers PE1-PE5. In this manner, network provisioning teams can more easily provision new customers on any PE router. Such balancing of routing table size can be achieved by changing the memory utilization constraint to lower the memory utilization allowed.
In some example implementations, certain approximations and assumptions can be made to facilitate finding optimal or near-optimal spoke-to-hub assignment solutions that will satisfy a minimal or near-minimal cost represented by the cost model of equation 1. For instance, a minimal or near-minimal cost can be found using practical approximation algorithms with approximation factors (C). Such approximation algorithms may not necessarily result in finding the most optimal spoke-to-hub assignment solution, but the solutions that can be found using these algorithms have costs within a factor (C) of the optimum solution. Also, while solving for the minimal or near-minimal cost, certain assumptions regarding characteristics of PE routers can be made. For instance, because memory is the property of PE routers most affected when implementing routing tables, the cost model of equation 1 above can be evaluated assuming that PE utilization (ui) represents memory utilization, which is proportional to the number of routes installed in a hub router. In addition, the cost model can be evaluated assuming that the transport cost is linear and proportional to the distance traversed (lijk) and that the cost of traffic traversing the network for the weighted average distance of today's traffic is, for example, $4 per Mbps. In the illustrated examples described herein, the cost of PE routers will vary linearly with the memory utilization and be amortized over 36 months. More specifically, the up-front or initial cost for a PE router is assumed to be, for example, $200K when empty and reach a total cost of, for example, $400K if the memory is fully utilized.
In some example implementations, the cost model of equation 1 can be simplified to implement hub selection and spoke-to-hub assignment processes. In particular, an example cost model of equation 2 below shows a simplified form of the cost model shown in equation 1 above.
A model instance (I1) based on the cost model of equation 1 can be transformed into a model instance (I2) that is based on the cost model of equation 2. As shown, the cost model of equation 2 does not have the third term (i.e., the memory utilization term (Σiui)) of the cost model of equation 1, nor does it have the coefficient multipliers (α), (β), and (γ). Instead, the cost model of equation 2 accounts for the initialization costs of selected hubs (H) and for the costs associated with traffic volume per time (νijk) (i.e., bandwidth usage) and latency (lijk) to effectively select optimal or near-optimal hub selections and spoke-to-hub assignment solutions as could otherwise be selected using the cost model of equation 1. The simplification of equation 2 enables performing analyses similar to those that could be performed using equation 1, while using a relatively less complex process implementation to perform such analyses.
The similarity of the cost models of equation 1 and equation 2 can be shown as follows. For any constant bandwidth cost error factor (ε′>0), an instance (I1) satisfying the cost model in the form of equation 1 can be transformed in polynomial time to an instance (I2) satisfying the simplified cost model of the form of equation 2 with a blowup factor of at most (1+ε′). If a particular one of the PE routers PE1-PE5, referred to as router (Ri), is made a hub and gets a memory utilization (ui), then it pays a price of (α+γuij) in the cost model of equation 1. In the second instance (I2), multiple ones of the PE routers PE1-PE5 are used to model the router (Ri) of the first instance (I1). Each of the PE routers of the second instance (I2) has a particular memory limit (spanning the range from zero to (Ui), which is the total memory of the modeled router (Ri)). The cost of such a set of PE routers of the new instance (I2) is (α+γUij), where (Uij) is the new memory limit of the set of PE routers. Thus, analyzing a relatively small number of sets of PE routers suffices to satisfy the blowup factor bound of at most (1+ε′).
A hub selection process using the cost model of equation 2 above can be implemented by greedily selecting an (approximately) most efficient hub to which a set of VPE routers is assigned and subsequently selecting other hubs until all VPEs have been assigned to those hubs. At each step of the process, there is a set of yet unassigned VPEs and the target is to find the most efficient hub (i.e., a hub whose selection cost divided by the number of VPEs it satisfies—is assigned to—is a minimum) for some of them. To assign a candidate hub to one or more VPEs (j), a quantity (n) of VPEs (items) are partitioned into (k) VPNs (groups). Each VPN has a memory footprint si (number of routes needed for this VPN in each hub), and each VPE (j) has a bandwidth cost cj (summation of traffic volume multiplied by the latency increase for the traffic volume per time (i.e., bandwidth usage) generated at that particular VPE). In addition, a global one-time-pay cost F (i.e., F=fh if the hub is fixed to be h). Denoting the group of VPNs using (V), and denoting the set of VPEs corresponding to VPN (i) using (Pi), the target is to: (1) pick some VPNs V′⊂V having a total memory footprint that does not exceed a memory capacity (M) of the current hub (i.e., ΣiεV′si≦M); (2) deploy their memory footprint in the memory of the current hub; and (3) use the current hub to serve some VPEs from each of the selected VPNs (i.e., serve a subset of UiεV′Pi) to minimize the ratio of cost-to-quantity of VPEs served by the hub. In addition, the sum of the bandwidth usages (νj) for each VPE should not exceed a bandwidth limit (B) for the hub.
In some example implementations, VPEs can be assigned to hub routers while ignoring bandwidth limits (B) for those hubs and reliability requirements. For instance, assuming costs are small integers, the cost model of equation 2 can be solved using a dynamic programming (DP) technique. For example, c(i,l) can denote the sum of the bandwidth costs (νj) of the (l) cheapest VPEs in VPN (i). The DP can be implemented by selecting a quantity (l) of VPEs from the VPN (i) and using the selected VPEs to populate a data structure D(i,m,c), which defines the minimum memory usage required to serve (m) VPEs (items) from VPNs 1 through (i) at a bandwidth usage cost (c). Finally, the best spoke-to-hub assignment solution whose memory utilization does not exceed the memory capacity (M) can be selected by analyzing the table for (i=k) (k is the number of VPNs). In this manner, in example implementations in which costs are assumed to be small integers, VPEs can be assigned to hub routers while ignoring bandwidth limits (B) and reliability requirements.
In other example implementations, in which costs are not assumed to be small integers, VPEs can be assigned to hub routers using techniques similar to a Polynomial-Time Approximation Scheme (PTAS) (i.e., a (1+ε)-approximation algorithm for any ε>0) for a standard knapsack problem. In a standard knapsack problem there is a knapsack capacity (e.g., the bandwidth usage cost (c)) and a number of items, each having an associated value and an associated weight, and the goal is to find a maximum-value subset of items whose total weight is less than or equal to the knapsack capacity (c). Using this approach, the instance (I2) (for a particular hub router) is modified to be an instance (I3), in which all bandwidth usage costs (c) and bandwidth usage values (ν) have been discretized to be integer multiples of some granularity (a bandwidth cost granularity (τ) and a bandwidth usage granularity (κ)). The instance (I3) can be used because it is approximately equivalent to the instance (I2). That is, the value of the optimum solution for the instance (I2) is no better than that of the instance (I3). In addition, any solution of the instance (I3) having a bandwidth cost (C) is a solution for the instance (I2) in which memory utilization is the same, bandwidth cost is at most (1+ε′)C multiplied by the bandwidth cost of the instance (I3), and bandwidth usage is at most (1+δ)B.
Using techniques similar to PTAS for the knapsack problem, the goal is to find a VPE assignment for a hub router that is of minimum density (i.e., minimize the ratio of cost-to-quantity of VPEs satisfied). That is, an optimal or near-optimal spoke-to-hub assignment is one that involves assigning the highest or near-highest quantity (N) of VPEs to a single hub while maintaining a lowest or near-lowest possible cost. A cost-to-quantity ratio is used to facilitate evaluation of this measure. For instance, assuming a facility cost (F) for the hardware of a physical PE router and assuming the cheapest VPE has a (bandwidth) cost (L), the best density is at most (F+L), since one possibility is to only serve the cheapest VPE. Thus, all VPEs costlier than (F+L) can be ignored. For a given bandwidth cost error factor ε′>0, bandwidth cost granularity can be defined to be τ=ε′(F+L)/N, and costs are rounded down to the next integer multiple of the bandwidth cost granularity (τ). In addition, the bandwidth usage granularity can be defined as κ=δB|N, where (B) is the bandwidth limit of the current hub and (δ) is a bandwidth usage error factor. The number of distinct cost values is at most the number (N) of VPEs divided by the cost error factor (ε′) (i.e., N|ε′), and the number of distinct bandwidth usage values is at most the number (N) of VPEs divided by the bandwidth usage error factor (δ) (i.e., NM). Thus, these values can be divided by the bandwidth cost granularity (τ) and the bandwidth usage granularity (κ), respectively to get an instance in which bandwidth costs and bandwidth usage values are polynomially bounded integers. This facilitates using a dynamic programming process to find an instance (I3) that will satisfy assigning a number of VPEs to a particular hub router.
Turning now to
To select hub routers and assign spoke routers to those hub routers, the multi-VPN hub selector 404 is provided with a multi-VPN hub selection module 604. In the illustrated example, the hub selection module 604 can find spoke-to-hub assignments based on the cost models of equations 1 and 2 above until every VPE router is assigned to at least a minimum reliability number (ρ) of hub routers. The hub selection module 604 performs the spoke-to-hub assignment process as discussed below in connection with the pseudo-code and flow diagrams of
To store constraint information such as maximum usage values, maximum cost values, reliability constraints, etc., the multi-VPN hub selector 404 is provided with a constraints database 606. To store cost information associated with PE router procurement costs, PE router deployment costs, PE router maintenance costs, bandwidth costs (e.g., cost per Mbps), etc., the multi-VPN hub selector 404 is provided with a costs database 608. The constraints database 606 and the costs database 608 may be stored in any number and/or type(s) of memory(-ies) and/or memory device(s) using any number and/or type(s) of data structures.
The hub selection module 604 may periodically (e.g., once a week) or aperiodically (e.g., upon occurrence of an event) update or change hub selections and spoke-to-hub assignments. This allows the MPLS-based multi-VPN network 400 of
To configure and/or provision hub and spoke routers, the multi-VPN hub selector 404 includes a BGP policy manager 610. Based upon hub router selections made by the hub selection module 604, the example BGP policy manager 610 creates, generates, customizes, installs, configures and/or provisions corresponding BGP policies and/or configurations into the example PE routers PE1-PE5 of
While an example manner of implementing the multi-VPN hub selector 404 of
Turning now to
Turning now to the example flow diagrams of
Turning to
The hub selection module 604 identifies all of the unassigned PE routers (e.g., PE routers PE1-PE5 of
The hub selection module 604 selects a first candidate hub router (h) (block 1108) from the candidate hub routers identified at block 1104. The hub selection module 604 identifies which physical PE routers (of those identified at block 1102) are not assigned to the selected candidate hub router (h) (block 1110). The hub selection module 604 then determines for which of the physical PE routers not assigned as a spoke router to the selected candidate hub (h) it would be feasible to make such an assignment (block 1112). For example, the hub selection module 604 can determine that it would be feasible to assign a particular physical PE router as a spoke router (sp) to the selected candidate hub (h) if the transmission latency of relaying the traffic generated at the physical PE router through the selected candidate hub (h) is less than a maximum latency threshold (θ). In the illustrated example, such a spoke-to-hub assignment is feasible if the distance between the physical PE router (sp) and the candidate hub (h) (i.e., d(sp,h)) plus the distance between the candidate hub (h) and a destination PE router (dst) (i.e., d(h,dst)) minus the distance directly between the physical PE router (sp) and the destination PE router (dst) (i.e., d(sp,dst)) is less than or equal to the maximum latency threshold (θ) (i.e., feasibility=true if d(sp,h)+d(h,dst)−d(sp,dst)≦θ).
For the physical PE routers identified at block 1112, the hub selection module 604 identifies all VPE routers in those physical PE routers (block 1114). For each VPE router (j), the hub selection module 604 determines the amount of traffic volume per time (νj) that needs to be sent by that VPE router (j) (block 1116). For example, the hub selection module 604 can retrieve traffic volume data from the traffic data collector 602 (
As shown in
The hub selection module 604 selects a first one of the VPNs (e.g., VPN1-VPN4 of
If minimum costs have been determined for all VPNs (block 1126), control advances to block 1130 at which the hub selection module 604 determines the least memory utilization costs that can be achieved for the candidate hub (h) to serve different quantities of spoke routers without violating a maximum bandwidth capacity constraint (bi) for the candidate hub (h) (block 1130). An example process that can be used to implement the operation of block 1130 is discussed below in connection with the flow diagram of
The hub selection module 604 then retrieves an optimal or near-optimal spoke-to-hub assignment solution based on the per-hub DP tables for the candidate hub (h) (block 1132). In the illustrated example, an optimal or near-optimal spoke-to-hub assignment solution is one that has the smallest ratio of cost-to-quantity of spoke VPEs that can be assigned to the candidate hub router (h). In this manner, the selected candidate hub (h) can store a full VRF table having all of the routes corresponding to the VPN installed in the VPEs assigned to the candidate hub (h). An example process that can be used to implement the operation of block 1132 is discussed below in connection with the flow diagram of
The hub selection module 604 then determines whether there is another candidate hub router remaining to be analyzed (block 1134). If another candidate hub router remains (block 1134), the hub selection module 604 selects another candidate hub router (h) (block 1136) and control returns to block 1110 of
If unassigned PEs still remain (block 1144), control returns to block 1108 (
The hub selection module 604 sets the VPE quantity value (cnt) to one (i.e., cnt=1) (block 1206) and clears the bandwidth usage test value (bw) to zero (i.e., bw=0) (block 1208). The hub selection module 604 then selects a first VPE (i) (block 1210) from a plurality of (n) VPEs of the VPN (vpn). The hub selection module 604 then determines whether the VPE traffic volume per time (νi) (i.e., VPE bandwidth usage) of the selected PE (i) is less than or equal to the bandwidth usage test value (bw) (i.e., νi≦bw) (block 1212). In this manner, the hub selection module 604 can determine whether the currently selected VPE (i) can be a spoke router of a spoke-to-hub assignment solution without exceeding the bandwidth usage test value (bw).
If the VPE traffic volume per time (νi) of the selected VPE (i) is less than or equal to the bandwidth usage test value (bw) (block 1212), the hub selection module 604 determines whether the per-VPN memory utilization cost value (T′[vpn, cnt, bw]) for the VPN (vpn), VPE quantity value (cnt), and bandwidth usage test value (bw) is greater than the sum of a VPE bandwidth usage cost (ci) and an adjusted per-VPN memory utilization cost value (T′[vpn, cnt−1, bw−νi]) (i.e., T′[vpn, cnt, bw]>T′[vpn, cnt−1, bw−νi]+ci) (block 1214). In the illustrated example, the adjusted per-VPN memory utilization cost value (T′[vpn, cnt−1, bw−νi]) corresponds to a VPE quantity value (cnt) decremented by one PE (i.e., cnt−1) and a bandwidth usage test value (bw) from which the VPE traffic volume per time (νi) is subtracted.
If the sum of the PE bandwidth cost (ci) and the adjusted per-VPN memory utilization cost value (T′[vpn, cnt−1, bw−νi]) (i.e., T′[vpn, cnt−1, bw−νi]+ci) is greater than the per-VPN memory utilization cost value (T′[vpn, cnt, bw]) (block 1214), the hub selection module 604 stores the sum of a VPE bandwidth usage cost (ci) and the adjusted per-VPN memory utilization cost value (T′[vpn, cnt−1, bw−νi]) in the per-VPN memory utilization cost cell (T′[vpn, cnt, bw]) (i.e., T′[vpn, cnt, bw]←T′[vpn, cnt−1, bw−νi]+ci) (block 1216). The hub selection module 604 adds the selected VPE (i) to an adjusted per-VPN spoke-to-hub assignment solution (A′[vpn, cnt−1, bw−νi]) (i.e., A′[vpn, cnt−1, bw−νi]∪{i}) (block 1218) and stores the selected VPE (i) and adjusted per-VPN assignment solution (A′[vpn, cnt−1, bw−νi]) (i.e., A′[vpn, cnt−1, bw−νi]∪{i}) in the per-VPN memory utilization cost value (T′[vpn, cnt, bw]) (block 1220).
After the operation of block 1220 or if the hub selection module 604 determines at block 1214 that the sum of the VPE bandwidth usage cost (ci) and the adjusted per-VPN memory utilization cost value (T′[vpn, cnt−1, bw−νi]) (i.e., T′[vpn, cnt−1, bw−νi]+ci) is not greater than the per-VPN memory utilization cost value (T′[vpn, cnt, bw]) or if the hub selection module 604 determines at block 1212 that the VPE traffic volume per time (νi) of the selected PE (i) is not less than or equal to the bandwidth usage test value (bw), then control advances to block 1222 shown in
Turning to
If the bandwidth usage test value (bw) is equal to the bandwidth capacity constraint (bi) (block 1226), control advances to block 1230 at which the hub selection module 604 determines whether it has finished filling the per-VPN DP table. In the illustrated example, the per-VPN DP table is filled when the VPE quantity value (cnt) equals the quantity of the plurality of (n) VPEs of the VPN (vpn) (i.e., cnt=n). If the per-VPN DP table is not yet filled (block 1230), the hub selection module 604 increments the VPE quantity value (cnt) (i.e., cnt=cnt+1) (block 1232) and control returns to block 1208 of
In a per-hub spoke-to-hub assignment solution table (A), each table cell stores a particular assignment solution and is addressed by (A [cost, vpn, served, bandwidth]). Each per-hub memory utilization value (T [cost, vpn, served, bandwidth]) corresponds to a particular per-hub spoke-to-hub assignment solution (A [cost, vpn, served, bandwidth]). To initialize the per-hub DP tables at block 1304, the memory utilization cells (T[cost, vpn, served, bandwidth]) are set to a relatively high number (denoted as infinity (∞) in mathematical notation) meaning that a least memory utilization is yet to be found. In addition, the hub selection module 604 sets a first one of the memory utilization cells addressed by (T [fh, 0, 0, 0]) to zero (where (fh) is the cost of deploying and maintaining the candidate hub (h)) and sets a first one of the spoke-to-hub assignment solution cells addressed by (A[fh, 0, 0, 0]) to an empty set (Ø) meaning that a solution does not yet exist.
The hub selection module 604 selects a VPN (vpn) (block 1306) from, for example, the plurality of VPNs identified at block 1106 of
If the hub selection module 604 determines that the bandwidth usage cost (cst) is less than or equal to a bandwidth usage cost limit (ci) for the candidate hub (h) (block 1312) and that the bandwidth usage (bw) is less than or equal to the bandwidth usage limit (bi) for the candidate hub (h) (block 1314), control advances to block 1316 shown in
The hub selection module 604 then initializes and sets a served count variable (srvd cnt) equal to one (i.e., srvd cnt=1) (block 1320), a bandwidth usage test variable (bw tst) equal to one (i.e., bw tst=1) (block 1322), and a cost test variable (c) equal to a memory utilization cost value stored in a cell addressed by (T′[vpn, srvd cnt, bw tst]) (block 1324) within the memory utilization cost DP table (T′) generated in the example process of
If the tests of blocks 1326 and 1328 are both true, the hub selection module 604 stores the sum of a VPN memory utilization requirement (svpn) and an adjusted VPN memory utilization value (T[cst−c, vpn−1, srvd−srvd cnt, bw−bw tst]) (i.e., T[cst−c, vpn−1, srvd−srvd cnt, bw−bw tst]+svpn) in the memory utilization cell addressed by (T[cst, vpn, srvd, bw]) (block 1330). In addition, the hub selection module 604 adds the spoke-to-hub assignment solution addressed by (A′[vpn, srvd cnt, bw tst]) in the per-VPN spoke-to-hub assignment DP table (A′) generated using the example process of
After block 1332 or if one or both of the tests of block 1326 and 1328 are not true, control advances to block 1334 at which the hub selection module 604 determines whether the bandwidth usage test value (bw tst) is less than the bandwidth usage (bw) retrieved at block 1310 (block 1334). If the bandwidth usage test value (bw tst) is less than the bandwidth usage (bw) (block 1334), the hub selection module 604 increments the bandwidth usage value (bw) to the next incremental value based on the bandwidth usage granularity (κ) (block 1336) and control returns to block 1324. Otherwise, if the bandwidth usage test value (bw tst) is not less than the bandwidth usage (bw) (block 1334), the hub selection module 604 determines whether the served count value (srvd cnt) is less than the quantity (srvd) of VPEs served by the candidate hub (h) as selected at block 1308 (block 1338). If the served count value (srvd cnt) is less than the served quantity (srvd) of VPEs (block 1338), the hub selection module 604 increments the served count value (srvd cnt) (i.e., srvd cnt=srvd cnt+1) (block 1340) and control returns to block 1322. Otherwise, if the served count value (srvd cnt) is not less than the served quantity (srvd) of VPEs (block 1338), control returns to block 1342. In the illustrated example, control is also advanced to block 1342 if the tests o blocks 1312 and 1314 discussed above are not true.
The hub selection module 604 determines whether the maximum served quantity (srvd) of VPEs in the selected VPN (vpn) has been reached (block 1342). If the maximum served quantity (srvd) of VPEs has not been reached, control returns to block 1308 at which another served quantity (srvd) of VPEs is selected. Otherwise, the hub selection module 604 determines whether the last VPN of the VPNs identified at block 1106 of
The hub selection module 604 then initializes and sets a served quantity (served) of VPE routers to one (i.e., served=1) (block 1406), a bandwidth usage cost (cost) to zero (i.e., cost=0) (block 1408), and a bandwidth usage value (bandwidth) to zero (i.e., bandwidth=0) (block 1410). The hub selection module 604 then determines whether a memory utilization (T[cost, vpn, served, bandwidth]) is less than a memory capacity (mi) of the full VRF table size of the candidate hub (h) (block 1412). If the memory utilization (T[cost, vpn, served, bandwidth]) is less than a memory capacity (mi), the hub selection module 604 determines whether a ratio of the bandwidth usage cost (cost)-to-quantity of PE routers served (served) (i.e., cost/quantity-served) is less than a previously determined best ratio (i.e., a ratio value previously stored in the BestRatio solution variable initialized at block 1101 of
After updating the solution variables BestRatio, BestSet, and BestHub at blocks 1414, 1416, and 1418 or if the cost-to-quantity ratio (cost/quantity-served) is not less than a previously determined best ratio (BestRatio) (block 1414), the hub selection module 604 determines whether the bandwidth usage value (bandwidth) is less than the bandwidth usage capacity constraint (bi) (block 1422). If the bandwidth usage value (bandwidth) is less than the bandwidth usage capacity constraint (bi) (block 1422), the hub selection module 604 increments the bandwidth usage value (bandwidth) to the next incremental value based on the bandwidth usage granularity (κ) (block 1424) and control returns to block 1412.
If at block 1422, the bandwidth usage value (bandwidth) is less than the bandwidth usage capacity constraint (bi), the hub selection module 604 determines whether the bandwidth usage cost (cost) is less than the maximum cost value (MAX COST) (block 1426). If the bandwidth usage cost (cost) is less than the maximum cost value (MAX COST), the hub selection module 604 increments the bandwidth usage cost value (cost) to the next incremental value based on the bandwidth cost granularity (τ) (block 1428) and control returns to block 1410.
If at block 1426 the bandwidth usage cost (cost) is not less than the maximum cost value (MAX COST), the hub selection module 604 determines whether the served quantity (served) of VPE routers is less than a total quantity (N) of VPEs in the multi-VPN network 400 of
The processor 1512 of
In general, the system memory 1524 may include any desired type of volatile and/or non-volatile memory such as, for example, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, read-only memory (ROM), etc. The mass storage memory 1525 may include any desired type of mass storage device including hard disk drives, optical drives, tape storage devices, etc.
In the illustrated examples described herein, the system memory 1524 may be implemented using a relatively fast memory such as an SRAM or TCAM to store the FIB 106a (or the FIBs 106b-e) of
The I/O controller 1522 performs functions that enable the processor 1512 to communicate with peripheral input/output (I/O) devices 1526 and 1528 and a network interface 1530 via an I/O bus 1532. The I/O devices 1526 and 1528 may be any desired type of I/O device such as, for example, a keyboard, a video display or monitor, a mouse, etc. The network interface 1530 may be, for example, an Ethernet device, an asynchronous transfer mode (ATM) device, an 802.11 device, a digital subscriber line (DSL) modem, a cable modem, a cellular modem, etc. that enables the processor system 1510 to communicate with another processor system.
While the memory controller 1520 and the I/O controller 1522 are depicted in
Of course, persons of ordinary skill in the art will recognize that the order, size, and proportions of the memory illustrated in the example systems may vary. Additionally, although this patent discloses example systems including, among other components, software or firmware executed on hardware, it will be noted that such systems are merely illustrative and should not be considered as limiting. For example, it is contemplated that any or all of these hardware and software components could be embodied exclusively in hardware, exclusively in software, exclusively in firmware or in some combination of hardware, firmware and/or software. Accordingly, persons of ordinary skill in the art will readily appreciate that the above-described examples are not the only way to implement such systems.
At least some of the above described example methods and/or apparatus are implemented by one or more software and/or firmware programs running on a computer processor. However, dedicated hardware implementations including, but not limited to, an ASIC, programmable logic arrays and other hardware devices can likewise be constructed to implement some or all of the example methods and/or apparatus described herein, either in whole or in part. Furthermore, alternative software implementations including, but not limited to, distributed processing or component/object distributed processing, parallel processing, or virtual machine processing can also be constructed to implement the example methods and/or apparatus described herein.
It should also be noted that the example software and/or firmware implementations described herein are stored on a tangible medium, such as: a magnetic medium (e.g., a disk or tape); a magneto-optical or optical medium such as a disk; or a solid state medium such as a memory card or other package that houses one or more read-only (non-volatile) memories, random access memories, or other re-writeable (volatile) memories. Accordingly, the example software and/or firmware described herein can be stored on a tangible medium such as those described above or equivalents and successor media.
To the extent the above specification describes example components and functions with reference to particular devices, standards and/or protocols, it is understood that the teachings of the invention are not limited to such devices, standards and/or protocols. Such devices are periodically superseded by different, faster, and/or more efficient systems having the same general purpose. Accordingly, replacement devices, standards and/or protocols having the same general functions are equivalents which are intended to be included within the scope of the accompanying claims.
Further, although certain methods, apparatus, systems, and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. To the contrary, this patent covers all methods, apparatus, systems, and articles of manufacture fairly falling within the scope of the appended claims either literally or under the doctrine of equivalents.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6772226 | Bommareddy et al. | Aug 2004 | B1 |
| 6789121 | Lamberton et al. | Sep 2004 | B2 |
| 7200114 | Tse-Au | Apr 2007 | B1 |
| 7376084 | Raghunath et al. | May 2008 | B2 |
| 7489635 | Evans et al. | Feb 2009 | B2 |
| 7564802 | Andrapalliyal et al. | Jul 2009 | B2 |
| 7796607 | Gerber et al. | Sep 2010 | B2 |
| 20020101826 | Giacopelli et al. | Aug 2002 | A1 |
| 20030096610 | Courtney et al. | May 2003 | A1 |
| 20060239199 | Blair et al. | Oct 2006 | A1 |
| 20080298250 | Larsson | Dec 2008 | A1 |
| 20090161658 | Danner et al. | Jun 2009 | A1 |
| 20090180600 | Blackwell et al. | Jul 2009 | A1 |
| 20090296714 | Gerber et al. | Dec 2009 | A1 |
| 20100316022 | Kakadia et al. | Dec 2010 | A1 |
| 20110188409 | Chen et al. | Aug 2011 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20110069634 A1 | Mar 2011 | US |
