1. Field of the Invention
The invention relates generally to routing protocols used in computer networks and, more particularly, to a technique that speeds up graceful restart of a routing protocol executing on an intermediate node in a computer network.
2. Background Information
A computer network is a geographically distributed collection of interconnected communication links used to transport data between nodes, such as computers. Many types of computer networks are available, with the types ranging from local area networks to wide area networks. The nodes typically communicate by exchanging discrete packets or messages of data according to pre-defined protocols. In this context, a protocol consists of a set of rules defining how the nodes interact with each other.
Computer networks may be further interconnected by an intermediate node, such as a switch or router, to extend the effective “size” of each network. Since management of a large system of interconnected computer networks can prove burdensome, smaller groups of computer networks may be maintained as routing domains or autonomous systems. The networks within an autonomous system (AS) are typically coupled together by conventional “intradomain” routers. Yet it still may be desirable to increase the number of nodes capable of exchanging data; in this case, intradomain routers executing interdomain routing protocols are used to interconnect nodes of the various autonomous systems (ASs).
An example of an interdomain routing protocol is the Border Gateway Protocol version 4 (BGP), which performs routing between autonomous systems by exchanging routing (reachability) information among neighboring interdomain routers of the systems. An adjacency is a relationship formed between selected neighboring (peer) routers for the purpose of exchanging routing information messages and abstracting the network topology. Before transmitting such messages, however, the peers cooperate to establish a logical “peer” connection (session) between the routers. BGP establishes reliable connections/sessions using a reliable/sequenced transport protocol, such as the Transmission Control Protocol (TCP).
The reachability information exchanged by BGP peers typically includes destination address prefixes, i.e., the portions of destination addresses used by the routing protocol to render routing (“next hop”) decisions. Examples of such destination addresses include Internet Protocol (IP) version 4 (IPv4) and version 6 (IPv6) addresses. A prefix implies a combination of an IP address and a mask that cooperate to describe an area of the network that a peer can reach. Each prefix may have a number of associated paths; each path is announced to a peer router by one or more of its peers. Note that the combination of a set of path attributes and a prefix is referred to as a “route”; the terms “route” and “path” may be used interchangeably herein. The BGP routing protocol standard is well known and described in detail in Request For Comments (RFC) 1771, by Y. Rekhter and T. Li (1995), Internet Draft <draft-ietf-idr-bgp4-23.txt> titled, A Border Gateway Protocol 4 (BGP-4) by Y. Rekhter and T. Li (April 2003) and Interconnections, Bridges and Routers, by R. Perlman, published by Addison Wesley Publishing Company, at pages 323-329 (1992), all disclosures of which are hereby incorporated by reference.
The interdomain routers configured to execute an implementation of the BGP protocol, referred to herein as BGP routers, perform various routing functions, including transmitting and receiving routing messages and rendering routing decisions based on routing metrics. Each BGP router maintains a routing table that lists all feasible paths from that router to a particular network. The routing table is a database that contains routing information used to construct a forwarding table of a forwarding information base (FIB) that is used by the router when performing forwarding decisions on packets.
Periodic refreshing of the routing table is generally not performed; however, BGP peer routers residing in the as exchange routing information under certain circumstances. For example, when a BGP router initially connects to the network, the peer routers exchange the entire contents of their routing tables. Thereafter when changes occur to those contents, the routers exchange only those portions of their routing tables that change in order to update their BGP peers' tables. These update messages are thus incremental update messages sent in response to changes to the contents of the routing tables and announce only a best path to a particular network.
Broadly stated, a BGP router generates routing update messages for an adjacency, to also known as a peer router, by “walking-through” the routing table and applying appropriate routing policies. A routing policy is information that enables a BGP router to rank routes according to filtering and preference (i.e., the “best path”). Routing updates provided by the update messages allows BGP routers of the as to construct a consistent view of the network topology. The update messages are typically sent using a reliable transport, such as TCP, to ensure reliable delivery. TCP is a transport protocol implemented by a transport layer of the IP architecture; the term TCP/IP is commonly used to denote this architecture. The TCP/IP architecture is well known and described in Computer Networks, 3rd Edition, by Andrew S. Tanenbaum, published by Prentice-Hall (1996).
Often, maintenance of routers (such as BGP routers) in a network is planned, thereby leading to shutdown or reset of a BGP process executing in the router. For example, a BGP router may be shutdown and removed from service in response to, e.g., upgrading of certain hardware or rebooting of the router following a software upgrade. In addition, the router may be reset in response to changing of BGP parameters, such as when a BGP router identifier is changed. However, a planned router shutdown or reset can result in temporary outages (i.e., loss of routing information exchange) for certain routes for which the shutdown router was the best path.
In order to implement a planned shutdown or reset of BGP, the shutdown router sends a conventional BGP Notification message with error code Cease in order to close its connections with its BGP peers. Subsequently to sending the Notification message, the shutdown router closes the TCP sessions over which the connections are established. In some implementations, the Notification message may be omitted. In response to closing the connections, all original routes advertised on those connections are immediately removed (withdrawn) from service (from the FIBs) by the BGP peers. As a result, some time (i.e., a convergence time) elapses before the network re-converges. In this context, the convergence time is the time that elapses between withdrawal of a route and the time when all subsequent messages triggered by the initial route withdrawal have been exchanged. In general, this can be characterized by the time needed for a BGP router to receive and process update messages from all of its peers, select best paths for each prefix, install those best paths into the routing table and advertise the best paths back to its peers. However, in some networks, multiple such rounds of messages may be required or other factors may also play a part. This approach of simply “halting” the router and re-converging the network leads to temporary loss of routing information due to route withdrawal.
A BGP router may shutdown or become unavailable as result of a timeout or a failure condition. As such, a failed router's functions may be taken over by a designated failover (backup) router. In either a planned shutdown (above) or a sudden failure, eventually BGP connectivity within the failed router will be reestablished. The prior art defines a so-called “graceful restart” (GR) procedure that allows the BGP connection to be reestablished in a manner that causes the least disruption to other connections and avoids further timeouts. A more-detailed discussion of this procedure is provided in Internet Draft <draft-ietf-idr-restart-10.txt> titled, Graceful Restart Mechanism for BGP by S. Sangli, Y. Rekhter, et. al. (December 2004), the teachings of which are expressly incorporated herein by reference.
A graceful restart assumes that the restarting router's peers have first detected that the router's connection has shut down. Subsequently the peers detect that the shutdown router is coming back up and a graceful restart procedure is implemented in an attempt to limit the negative effects on routing caused by the restart of BGP. These negative effects result in part from the need to recompute BGP routes/paths. These processes consume significant system resources.
The conventional graceful restart approach outlined in the above-incorporated Graceful Restart Mechanism for BGP involves entry by the BGP peers into a read-only mode in which they send and receive updates of routes. The peers retrieve their local routing information and generate updates for the restarting router. The restarting router then receives route updates, and thereby updates its local FIB. Once updates are complete, any “stale” paths that are no longer employed by BGP are deleted and read only mode is exited. At this time, all other activity is completed and the “best path” procedure is run on the restarting router. Best path updates are then transmitted out to peers based upon the best path computation. Once updates are complete an end-of-RIB (routing information base) marker is sent out by peers to indicate that their updates are now complete. In this approach, the end-of-RIB marker is specified by an update message with no reachable network layer reachability information (NLRI) and empty withdrawn NLRI.
This procedure for graceful restart can be relatively slow to complete. In particular, it takes time to receive all BGP peers' routes and then to send out best paths to peers. A technique that reduces this latency is highly desirable.
An approach to preserving a BGP connection, with less latency used in so-called “high-availability” implementations of BGP architecture, is to store all state information related to the connection in a standby BGP process also termed a “stateful switchover” (SSO) that mirrors the primary BGP process. While this approach allows immediate reestablishment of the connection, and preserves the entire session, including TCP information, it is expensive both in terms of hardware and processing overhead.
This invention overcomes the disadvantages of the prior art by providing a system and method for speed up of graceful restart of a BGP router in a computer network. According to an aspect of the invention, a local store is employed on the restarting router that stores a redundant copy of all the prefixes that were received by all remote peers with notation of selection preference i.e. what was selected as “best-path.” This local store is used to run best path computations on the restarting router, rather than first awaiting receipt of route information from peers to speed the restart process. The end-of-RIB message can then be immediately sent to all remote peers. Thereafter the restarting router processes incoming updates from peers as received (or these peers routes are timed-out), remaining stale paths are deleted and any changed best paths, based upon the newly received updates, are then transmitted to peers.
In one embodiment, the local store can be provided as a non-volatile memory within the primary board of the restarting router and/or on an interconnected standby board. By preserving the prefixes in a store that survives a restart, the end-of-RIB marker can be transmitted immediately by the restarting router, thereby preventing its apparent timeout by failing to respond in a timely manner.
The above and further advantages of the invention may be better understood by referring to the following description in conjunction with the accompanying drawings in which like reference numbers indicate identical or functionally similar elements:
A key function of the interdomain router 300 is determining the next router to which a packet is sent; in order to accomplish such “routing” the interdomain routers cooperate to determine best paths through the computer network 200. The routing function is preferably performed by an internetwork layer of a conventional protocol stack within each router.
The lower network interface layer 408 is generally standardized and implemented in hardware and firmware, whereas the higher layers may be implemented in the form of software. The primary internetwork layer protocol of the Internet architecture is the IP protocol. IP is primarily a connectionless protocol that provides for internetwork routing, fragmentation and reassembly of exchanged packets—generally referred to as “data-grams” in an Internet environment—and which relies on transport protocols for end-to-end reliability and other service characteristics. An example of such a transport protocol is the TCP protocol, which is implemented by the transport layer 404 and provides connection-oriented, end-to-end reliability services to the upper layer protocols of the Internet architecture. The term TCP/IP is commonly used to denote the Internet architecture.
In particular, the internetwork layer 406 is concerned with how packets are forwarded through the network, although it is not directly concerned with how the FIBs are constructed. An interdomain routing protocol, such as BGP, is used to perform interdomain routing (for the internetwork layer) through the computer network. The interdomain routers 300 (hereinafter “peers”) exchange routing and reachability information among the autonomous systems over a reliable transport layer connection, such as TCP. An adjacency is a relationship formed between selected peer routers for the purpose of exchanging routing messages and abstracting the network topology. The BGP protocol “runs” on top of the TCP transport layer 404 to ensure reliable communication of routing messages among the peer routers.
In order to perform routing operations in accordance with the BGP protocol, each interdomain router 300 maintains a routing table that lists all feasible paths from that router to each network. The routing table is a database that contains routing information used to construct a forwarding table of the FIB 330 by the route processor 302 when perm forming forwarding decisions on packets. The routers exchange the routing information using routing update messages when their routing tables change. The routing update messages are generated by an updating router to advertise best paths to each of its neighboring peer routers (peers) throughout the computer network. These routing updates allow the BGP routers of the autonomous systems to construct a consistent and up-to-date view of the network topology.
The loc-RIB 520 stores routes that are similar to the routes in the forwarding table; to that end, the loc-RIB 520 is generally responsible for selecting the best route per prefix from the union of all policy-modified Adj-RIB-In routes, resulting in routes referred to as “best paths”. The set of best paths is then installed in the global RIB 530, where those paths may contend with best paths from other protocols to become the “optimal” path ultimately selected for forwarding. A BGP Routing Information Base (bRIB) process maintains the loc-RIB 520, including processing and downloading the best path to the global RIB 530. The global RIB 530 illustratively maintains the routing table for the router. Note that the routing table typically denotes a database containing all available routes, including ones that have been selected for forwarding (optimal paths) as well as, in some cases, backup routes that are not currently selected for forwarding. In contrast, the forwarding table denotes those optimal best paths that have actually been selected for forwarding. Thereafter, the set of best paths has outbound policy 560 run on it, the result of which is placed in appropriate Adj-RIB-Outs 540 and announced to the respective peers via the same TCP connections 502 from which routing update messages were learned.
Referring further to
In this further embodiment, the bus 305, or another connectivity within the router 300, interconnects to the redundant standby board 350 that is provided with its own power source or a non-volatile memory 360 that includes a copy of a bRIB store 362. Notably, the bRIB store 362 stores the group best path of all BGP speaker processes. Conversely, the store 362 can act as a “hot standby” for the router 300, which stores all the best paths normally in the router memory 304. In any case, the store 362 allows BGP routing information that existed prior to a failure to be retrieved immediately upon restart without awaiting receipt of updates from peers. Note that the store 362 contains current routing information because it is updated by the processor 302 as its own internal best path information is updated using a mirroring approach (or the like).
The standby board 350 may include its own standby processor 364 for controlling memory and communicating with the router 300. It may also execute a failover process 366 that loads the store 362 into router memory 304 when needed at restart. In general, the standby processor 364 functions to provide the same BGP route processing functions as the primary board's route processor 302 in the event of the failure of the primary board 303. It communicates directly with appropriate network interfaces to allow continued operation of BGP. As such, the bRIB on the standby board 350 is maintained and acts as the local store 340 in the event of failure. In this example, where the primary board 303 has not failed (for example, where the BGP process is merely restarting), the bRIB continues to be maintained in a non-volatile memory (represented by dashed line 370) on the primary board 303 that is associated with the local store 340. In the case of a non-failure, the primary board's route processor 302 maintains control over BGP using the non-volatile local store 340.
Note, it is expressly contemplated that any of the following options can be employed in accordance with this invention: (a) a non-volatile area may be provided to the primary board with no standby board so that the bRIB can, at a minimum survive board restart; (b) a standby board with appropriate mechanisms for maintaining the standby bRIB (a mirror of the current local bRIB) can be provided an used in all cases to maintain the bRIB in the event of restart, failure, etc.; or (c) a standby board and a non-volatile local store can be provided and used in appropriate circumstances (standby board for primary failure and local non-volatile store for restart).
To speed up the restart process, and thus avoid the potential for a timeout condition while the restarting router awaits receipt of all updates to thereby compute best paths, the procedure 600 of
First, restart BGP occurs in 602 with the procedure in read only (RO) mode. The restart may be the result of a power-up, reset or other event that temporarily broke connection with BGP peers. In one example, the restart can involve the running of a “hot standby” process and the triggering of a “go active” signal that connects a dependent process. Other peers will discover that the router is restarting through conventional approaches. In step 604, peer discovery occurs and a peering session is opened and negotiated between the restarting peer and the discovered peers. Thereafter, the restarting peer populates its Adj-RIB-Out database with data from the old local RIB in step 606. This is possible because the old bRIB has been maintained, either on the primary board or on the standby board. By repopulating with the old RIB data, the peer avoids completing the long, involved update process before sending its end-of-RIB marker.
In step 608, the procedure then directs that the contents of the Adj-RIB-Out database be sent to peers in the form of a BGP update message. The restarting peer now sends an End-of-RIB marker to peers in step 610. The procedure then exits read only (RO) mode in step 612.
Note that the end-of-RIB marker is sent very early in the restart process, speeding restart and avoiding the possibility of timeout in reestablishment of the connection. Since the downtime before restart is likely to be very small, it is unlikely that the updates will contain much new data, thus it is generally safe to send the end-of-RIB in advance of update completion.
According to the procedure 600, the restarting peer now receives incoming updates from peers in step 614. Based upon these updates, the restarting peer runs the best path algorithm in step 616. Thereafter, appropriate adjustments to best path are made. The procedure reviews the updates from each peer and determines whether any of its own best paths information has changed. If, so then updates are sent to the peers including the changes (step 618). Thereafter, the restarting peer awaits receipt of an end-of-RIB message from each peer, indicating that the peer has sent all information, and deletes any stale best path information in its memory that is no longer applicable, as well as any timed-out peers. If the deletion of that peer results in best path changes, then the procedure sends updates to peers with such changed paths (step 620). At this point BGP convergence has occurred and the procedure 600 is complete (step 622).
The above-described approach effectively halves convergence time over prior approaches. It thus allows the system to address changes in routing topology more quickly and thereby better avoids “black holes.”
The foregoing description has been directed to specific embodiments of this invention. It will be apparent; however, that other variations and modifications may be to made to the described embodiments, with the attainment of some or all of their advantages. For instance, it is expressly contemplated that the teachings of this invention, including the various processes and modules described herein, can be implemented as software, including a computer-readable medium having program instructions executing on a computer, hardware, firmware, or a combination thereof. In addition, it is understood that the data structures described herein can include additional information while remaining within the scope of the present invention. Furthermore, the inventive technique may apply generally to other routing protocols such as, e.g., IGRP, EIGRP, RIP, or other unicast, multicast routing or signaling protocols. Accordingly this description is to be taken only by way of example and not to otherwise limit the scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.
Number | Name | Date | Kind |
---|---|---|---|
5088032 | Bosack | Feb 1992 | A |
5519704 | Farinacci et al. | May 1996 | A |
5617421 | Chin et al. | Apr 1997 | A |
5917820 | Rekhter | Jun 1999 | A |
6269099 | Borella et al. | Jul 2001 | B1 |
6339595 | Rekhter et al. | Jan 2002 | B1 |
6370119 | Basso et al. | Apr 2002 | B1 |
6400681 | Bertin et al. | Jun 2002 | B1 |
6463061 | Rekhter et al. | Oct 2002 | B1 |
6526056 | Rekhter et al. | Feb 2003 | B1 |
6539427 | Natarajan et al. | Mar 2003 | B1 |
6553423 | Chen | Apr 2003 | B1 |
6577597 | Natarajan et al. | Jun 2003 | B1 |
6584093 | Salama et al. | Jun 2003 | B1 |
6584502 | Natarajan et al. | Jun 2003 | B1 |
6658595 | Thamattoor | Dec 2003 | B1 |
6718376 | Chu et al. | Apr 2004 | B1 |
6751188 | Medved et al. | Jun 2004 | B1 |
7023808 | Ball et al. | Apr 2006 | B2 |
7286468 | Scudder et al. | Oct 2007 | B2 |
7318108 | Sreekantiah et al. | Jan 2008 | B2 |
7359377 | Kompella et al. | Apr 2008 | B1 |
20020021675 | Feldmann | Feb 2002 | A1 |
20030174653 | Basu et al. | Sep 2003 | A1 |
20040090913 | Scudder et al. | May 2004 | A1 |
20050074001 | Mattes et al. | Apr 2005 | A1 |
20050074003 | Ball et al. | Apr 2005 | A1 |
20050135256 | Ball et al. | Jun 2005 | A1 |
20060171404 | Nalawade et al. | Aug 2006 | A1 |
20060291446 | Caldwell et al. | Dec 2006 | A1 |