Generally described, computing devices utilize a communication network, or a series of communication networks, to exchange data. In a common embodiment, data to be exchanged is divided into a series of packets that can be transmitted between a sending computing device and a recipient computing device. In general, each packet can be considered to include two primary components, namely, control information and payload data. The control information corresponds to information utilized by one or more communication networks to deliver the payload data. For example, control information can include source and destination network addresses, error detection codes, and packet sequencing identification, and the like. Typically, control information is found in packet headers and trailers included within the packet and adjacent to the payload data.
In practice, in a packet-switched communication network, packets are transmitted between multiple physical networks, or sub-networks. Generally, the physical networks include a number of hardware devices that receive packets from a source network component and forward the packet to a recipient network component. The packet routing hardware devices are typically referred to as routers. Generally described, routers can operate with two primary functions or planes. The first function corresponds to a control plane, in which the router learns the set of outgoing interfaces that are most appropriate for forwarding received packets to specific destinations. The second function is a forwarding plane, in which the router sends the received packet to an outbound interface.
To execute the control plane functionality, routers can maintain a forwarding information base (“FIB”) that identifies, among other packet attribute information, destination information for at least a subset of possible network addresses, such as Internet Protocol (“IP”) addresses. In a typical embodiment, the FIB corresponds to a table of values specifying network forwarding information for the router. In one aspect, commercial level routing hardware components can include customized chipsets, memory components, and software that allows a single router to support millions of entries in the FIB. However, such commercial level routing hardware components are typically very expensive and often require extensive customization. In another aspect, commodity-based routing hardware components are made of more generic components and can be less expensive than commercial level routing hardware components by a significant order of magnitude. However, such commodity-based routing hardware components typically only support FIBs on the order of thousands of entries.
The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:
Generally described, the present disclosure corresponds to a distributed routing architecture. Specifically, the present disclosure corresponds to a hierarchical distributed routing architecture including at least three logical levels, or layers, for receiving, processing and forwarding data packets between network components. In one embodiment, the three logical levels can corresponds to a core level, a distribution level and a transit level. Illustratively, the core level corresponds to one or more router components that receive an incoming packet from a network component and processes the destination address information associated with the received packet. The core level router component then identifies a distribution level router component based on a subset of the destination address associated with the received packet. The distribution level corresponds to one or more router components that receive a forwarded packet from a core level router component and further processes the destination address information associated with the received packet. The distribution level router component identifies a transit level router component based on at least a subset of the destination address associated with the received packet. Each distribution level router component is associated with, or otherwise corresponds to, a subset of the FIB associated with the distributed routing architecture. Finally, the transit level router components correspond to one or more router components that receive the forwarded packet from a distribution level router component and forward the packet “upstream” to a respective network, or network node. The mapping, or other assignment, of portions of the FIB associated with the distributed routing environment is managed by a router management component.
In one embodiment, each of the router components associated with the core level, distribution level and transit level can correspond more closely to commodity based router components/hardware. In another embodiment, the core level, distribution level and transit level router components correspond to logical router components that do not necessarily have a corresponding hardware router component. For example, one or more logical router components within each level may be implemented in the same hardware router component. Likewise, the logical router components associated with different levels of the distributed routing architecture may be implemented in the same hardware router component. In both embodiments, however, because responsibility for maintaining the FIB associated with the distributed routing environment is divided among several router components, the processing and memory restraints associated with commodity based router components/hardware can be mitigated. Various implementations, combination, and applications for dividing the FIB associated with the distributed routing environment will be described in accordance with the distributed routing environment. However, one skilled in the relevant art will appreciate that such embodiment and examples are illustrative in nature and should not be construed as limiting.
Turning now to
With continued reference to
In communication with the first communication network 104 is a first level of the distributed routing environment 100, generally referred to as the core layer or core level. In one embodiment, the core level corresponds to one or more logical router components, generally referred to as core level routers 106A, 106B, and 106C. As previously described, within the distributed routing environment 100, the core level routers 106A, 106B, 106C receive an incoming packet from a component from the network 104 and process the destination address by identifying a distribution level router component based on a subset of the destination address associated with the received packet. Illustratively, the subset of the destination address can correspond to less than the entire destination IP address, such as the highest most values of the IP address. As previously described, the core level routers 106A, 106B, 106C can correspond to logical router components implemented on one or more hardware components. In one embodiment, each logical router component can correspond with a dedicated physical router component. In another embodiment, each logical router component can correspond to a physical router component shared by at least one other logical router component in the distributed router environment 100. In an alternative embodiment, at least some portion of the core layer may be implemented by components outside the distributed routing environment 100. In such an embodiment, such external components would directly address a distribution level router component (described below) of the distributed routing environment 100.
The distributed routing environment 100 can further include a second level of logical router components, generally referred to as the distribution layer or distribution level. In one embodiment, the distribution level corresponds to one or more router components, generally referred to as distribution level routers 108A, 108B, and 108C. As previously described, within the distributed routing environment 100 the distribution level routers 108A, 108B and 108C receiving an incoming packet from a core routing component 102 and process the destination address by identifying a transit level router component based on at least a subset of the destination address associated with the received packet. Illustratively, the subset of the destination address can correspond to a larger subset of the destination IP address used by the core level routers 106A, 106B, 106C. In this embodiment, the routing performed by the distribution level can correspond to a more refined routing of the received packet relative to the core level routing. As described above with the core level routers 106A, 106B, 106C, the distribution level routers 108A, 108B, and 108C can correspond to logical router components implemented on one or more hardware components. In one embodiment, each logical router component can correspond with a dedicated physical router component. In another embodiment, each logical router component can correspond to a physical router component shared by at least one other logical router component in the distributed router environment 100.
In communication with the distribution level router components is a third level of router components, generally referred to as the transmit layer or transit level. In one embodiment, the transit level corresponds to one or more router components, generally referred to as transit level routers 110A, 110B, and 110C. As previously described, the transit level routers 110A, 110B, 110C receive the forwarded packet from a distribution level router component 108A, 108B, 108C and forward the packet “upstream” to another communication network 112 node. Illustratively, each transit level router 110A, 110B, 110C can be configured to communicate with one or more upstream peers such that all packets destined for an associated peer network component will be transmitted through the assigned transit level router 110A, 110B, 110C (or a redundant router). As described above with the core level routers 106A, 106B, 106C and the distribution level routers 108A, 108B and 108C, the transit level routers 110A, 110B, and 110C can correspond to logical router components implemented on one or more hardware components. In one embodiment, each logical router component can correspond with a dedicated physical router component. In another embodiment, each logical router component can correspond to a physical router component shared by at least one other logical router component in the distributed router environment 100
Similar to communication network 102, communication network 112 may encompass any suitable combination of networking hardware and protocols necessary to establish packet-based communications to the distributed routing environment 100. For example, the communication network 112 may include private networks such as local area networks (LANs) or wide area networks (WANs) as well as public or private wireless networks. In such an embodiment, the communication network 112 may include the hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall/security software, etc.) necessary to establish a networking link with the distributed routing environment 100. As described above with regard to the communication network 104, the communication network 112 may implement one of various communication protocols for transmitting data between computing devices. One skilled in the relevant art will appreciate, however, that present disclosure may be applicable with additional or alternative protocols and that the illustrated examples should not be construed as limiting.
In an illustrative embodiment, the logical router components (106, 108, 110) in
As illustrated in
The memory 160 contains computer program instructions that the processing unit 152 executes in order to operate the dynamic classifier. The memory 160 generally includes RAM, ROM and/or other persistent memory. The memory 160 may store an operating system 162 that provides computer program instructions for use by the processing unit 152 in the general administration and operation of the router component 150. The memory 160 may further include computer program instructions and other information for implementing one or more of the logical router components in the distributed routing environment 100. For example, in one embodiment, the memory 160 includes a router module 164 that implements the functionality associated with any of the routers 106, 108, 110. In the event that multiple logical routers are implemented by the same router component 150, memory 160 may have each instance of a router module 164.
In an illustrative embodiment, each router component 150 may be embodied as an individual hardware component for implementing one or more logical routers 106, 108, 110. Alternatively, multiple router components 150 may be grouped and implemented together. For example, each router component 150 may correspond to an application-specific integrated circuit (ASIC) having a processing unit 152, memory 154 and memory 160 (or other components with similar functionality). The router components 150 may share one or more components, such as the network interface 156 and computer readable medium 158, via a common communication bus.
With reference now to
Based on the processing of the first subset of the destination address, the core level router 106 forwards the packet to a distribution level router, in this case illustratively 108A. As previously described, the receiving distribution level router 108A processes the destination address of the received packet and also utilizes a subset of the destination IP address to identify a third level router component that will forward the packet to a next network destination (outside of the distributed routing environment 100). Similar to the core level router 106, the receiving distribution level router can be configured to utilize a selection of the most significant bits of the IP address (e.g., the prefix) to route the packet. In an illustrative embodiment, the prefix used by the distribution level router 108A is greater than the prefix used by the core level router 106. Based on the processing by the distribution level router 106A, the transit level router 110B receives the forwarded packet and forwards the packet to a designated designation associated with the communication network 112.
Turning now to
In an illustrative embodiment, the router management component 102 (
In still a further embodiment, multiple distribution level routers 108 may be selected for a subset of IP addresses. In this embodiment, each core level router 106 can select from multiple distribution level routers 108 based on an equal cost multi-path routing (ECMP) technique in which a specific distribution level router 108 is selected based on a standard load sharing technique. Other factors that can be utilized to select from multiple assigned distribution level router 108 include carrier preference, Internet weather, resource utilization/health reports, an allocated or determine routing cost, service level agreements (SLAs), or other criteria.
In one embodiment, each distribution router 108 can maintain the portion of the FIB that is associated with the subset of IP addresses assigned the respective distribution level router 108. In another embodiment, each distribution level router 108 can maintain the entire FIB associated with the distributed routing environment 100 in a memory component, such as computer readable medium 158 (
In still a further embodiment, as a variation to the above embodiment, each distribution level router can be allocated a larger portion of the FIB associated with the distributed routing environment 100 than is capable of being maintained in a first memory component of the router, such as memory component 154 (e.g., a processor level cache memory). If a core level router 106 routes to a distribution level router 108 and the corresponding prefixes of the destination IP address do not correspond to the FIB maintained in the first memory component of the distribution level router, the distribution level router can recall the necessary information from the larger subset of the FIB maintained in a different memory component (e.g., computer readable medium 158 (
In yet another embodiment, lower traffic prefixes may be assigned to multiple distribution level routers 108. In one example, each assigned distribution level router 108 does not maintain the lower traffic routing portion of the assigned FIB in the primary memory component. Rather, routing requests for the lower traffic prefixes can be directed to a specific distribution level router based on selection techniques, such as ECMP, and can be processed by a selected distribution level router 108 based on the larger FIB maintained in a different memory component within the selected distribution level router.
With reference now to
At block 306, the selected core level router 106 identifies a distribution level router 108 corresponding to a second level of the distributed routing environment 100. The core level router 108 selects the distribution level router 108 based on processing the destination IP address and utilizing a subset of the destination IP addresses (e.g., the prefix) to determine the appropriate distribution level router 108. Illustratively, in accordance with an embodiment corresponding to the IPv4 communication protocol, the core level router 106 processing can be based on consideration of a prefix of the eight most significant bits. At block 308, the selected distribution level router 108 identifies a transit level router 110 based on processing the destination IP address and utilizing a subset of the destination IP address to determine the appropriate transit level router 110. Illustratively, in accordance with an embodiment corresponding to the IPv4 communication protocol, the distribution level router 108 processing can be based on a larger subset of IP address (e.g., a longer prefix such as 16 or 24 bits, as needed to select the appropriate transit level router 110). One skilled in the relevant art will appreciate, however, the blocks 306 and 308 may be implemented in a manner such the core level router 106 and distribution level router 108 may utilize additional or alternative attributes (including different portions of a destination IP address) of received packets in identifying the next router component to forward the received packet.
At block 310, the selected transit level router 110 transmits the receive packet to the destination recipient associated, or otherwise configured, with the transit level router 110. At block 312, the routine 300 terminates.
With reference now to
Alternatively, if at decision block 404 the subset of the destination IP address associated with the routing request is not maintained in the portion of the FIB table maintained in the primary memory of the selected distribution level router 108, at block 410, distribution level router 108 attempts to obtain additional transit routing information from a separate memory component associated with the distribution level router. At block 410, the distribution level router 108 can update the forwarding table information maintained in the primary memory component with the information obtained from the other memory component. Alternatively, block 410 can be omitted or is otherwise optional. At block 412, the routine terminates.
While illustrative embodiments have been disclosed and discussed, one skilled in the relevant art will appreciate that additional or alternative embodiments may be implemented within the spirit and scope of the present disclosure. Additionally, although many embodiments have been indicated as illustrative, one skilled in the relevant art will appreciate that the illustrative embodiments do not need to be combined or implemented together. As such, some illustrative embodiments do not need to be utilized or implemented in accordance with the scope of variations to the present disclosure.
Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements, or steps. Thus, such conditional language is not generally intended to imply that features, elements or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements or steps are included or are to be performed in any particular embodiment. Moreover, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey utilization of the conjunction “or” in enumerating a list of elements does not limit the selection of only a single element and can include the combination of two or more elements.
Any process descriptions, elements, or blocks in the flow diagrams described herein and/or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted, executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those skilled in the art. It will further be appreciated that the data and/or components described above may be stored on a computer-readable medium and loaded into memory of the computing device using a drive mechanism associated with a computer-readable medium storing the computer executable components, such as a CD-ROM, DVD-ROM, or network interface. Further, the component and/or data can be included in a single device or distributed in any manner. Accordingly, general purpose computing devices may be configured to implement the processes, algorithms, and methodology of the present disclosure with the processing and/or execution of the various data and/or components described above. Alternatively, some or all of the methods described herein may alternatively be embodied in specialized computer hardware. In addition, the components referred to herein may be implemented in hardware, software, firmware or a combination thereof.
It should be emphasized that many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.
This application is a continuation of U.S. patent application Ser. No. 13/620,363, entitled DISTRIBUTED ROUTING ARCHITECTURE, and filed Sep. 14, 2012, which is a continuation of U.S. patent application Ser. No. 12/641,260, entitled DISTRIBUTED ROUTING ARCHITECTURE, and filed Dec. 17, 2009, now U.S. Pat. No. 8,331,371, the entireties of which are incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5649185 | Antognini et al. | Jul 1997 | A |
6182125 | Borella et al. | Jan 2001 | B1 |
6192051 | Lipman et al. | Feb 2001 | B1 |
6243761 | Mogul et al. | Jun 2001 | B1 |
6377257 | Borrel et al. | Apr 2002 | B1 |
6529910 | Fleskes | Mar 2003 | B1 |
6560610 | Eatherton et al. | May 2003 | B1 |
6633324 | Stephens, Jr. | Oct 2003 | B2 |
6697805 | Choquier et al. | Feb 2004 | B1 |
7009943 | O'Neil | Mar 2006 | B2 |
7023465 | Stephens, Jr. | Apr 2006 | B2 |
7065496 | Subbloie et al. | Jun 2006 | B2 |
7096193 | Beaudoin et al. | Aug 2006 | B1 |
7114160 | Suryanarayana et al. | Sep 2006 | B2 |
7120874 | Shah et al. | Oct 2006 | B2 |
7146560 | Dang et al. | Dec 2006 | B2 |
7185084 | Sirivara et al. | Feb 2007 | B2 |
7320131 | O'Toole, Jr. | Jan 2008 | B1 |
7502836 | Menditto et al. | Mar 2009 | B1 |
7523181 | Swildens et al. | Apr 2009 | B2 |
7555542 | Ayers et al. | Jun 2009 | B1 |
7623460 | Miyazaki | Nov 2009 | B2 |
7624169 | Lisiecki et al. | Nov 2009 | B2 |
7650376 | Blumenau | Jan 2010 | B1 |
7653725 | Yahiro et al. | Jan 2010 | B2 |
7676570 | Levy et al. | Mar 2010 | B2 |
7707071 | Rigole | Apr 2010 | B2 |
7707173 | Nanavati et al. | Apr 2010 | B2 |
7739400 | Lindbo et al. | Jun 2010 | B2 |
7773596 | Marques | Aug 2010 | B1 |
7787380 | Aggarwal et al. | Aug 2010 | B1 |
7865594 | Baumback et al. | Jan 2011 | B1 |
7873065 | Mukerji et al. | Jan 2011 | B1 |
7904875 | Hegyi | Mar 2011 | B2 |
7930393 | Baumback et al. | Apr 2011 | B1 |
7933988 | Nasuto et al. | Apr 2011 | B2 |
7937456 | McGrath | May 2011 | B2 |
7961736 | Ayyagari | Jun 2011 | B2 |
8051166 | Baumback et al. | Nov 2011 | B1 |
8069231 | Schran et al. | Nov 2011 | B2 |
8117306 | Baumback et al. | Feb 2012 | B1 |
8286176 | Baumback et al. | Oct 2012 | B1 |
8296429 | Baumback et al. | Oct 2012 | B2 |
8331370 | Hamilton et al. | Dec 2012 | B2 |
8902897 | Hamilton et al. | Dec 2014 | B2 |
8971328 | Judge et al. | Mar 2015 | B2 |
20020135611 | Deosaran et al. | Sep 2002 | A1 |
20020138443 | Schran et al. | Sep 2002 | A1 |
20020150094 | Cheng et al. | Oct 2002 | A1 |
20020194382 | Kausik et al. | Dec 2002 | A1 |
20030149581 | Chaudhri et al. | Aug 2003 | A1 |
20030182413 | Allen et al. | Sep 2003 | A1 |
20040049579 | Ims et al. | Mar 2004 | A1 |
20040083307 | Uysal | Apr 2004 | A1 |
20040194085 | Beaubien et al. | Sep 2004 | A1 |
20050038967 | Umbehocker et al. | Feb 2005 | A1 |
20050086645 | Diao et al. | Apr 2005 | A1 |
20050102683 | Branson et al. | May 2005 | A1 |
20050157712 | Rangarajan et al. | Jul 2005 | A1 |
20050171959 | Deforche et al. | Aug 2005 | A1 |
20050259672 | Eduri | Nov 2005 | A1 |
20060209701 | Zhang et al. | Sep 2006 | A1 |
20060218304 | Mukherjee et al. | Sep 2006 | A1 |
20070021998 | Laithwaite et al. | Jan 2007 | A1 |
20070086429 | Lawrence et al. | Apr 2007 | A1 |
20070198982 | Bolan et al. | Aug 2007 | A1 |
20070219795 | Park et al. | Sep 2007 | A1 |
20070245010 | Arn et al. | Oct 2007 | A1 |
20080065724 | Seed et al. | Mar 2008 | A1 |
20080086559 | Davis et al. | Apr 2008 | A1 |
20080172488 | Jawahar et al. | Jul 2008 | A1 |
20080183721 | Bhogal et al. | Jul 2008 | A1 |
20080215755 | Farber et al. | Sep 2008 | A1 |
20090187575 | DaCosta | Jul 2009 | A1 |
20090248893 | Richardson et al. | Oct 2009 | A1 |
20090327517 | Sivasubramanian et al. | Dec 2009 | A1 |
20100128638 | Navas et al. | May 2010 | A1 |
20100150155 | Napierala | Jun 2010 | A1 |
20140129707 | Baumback et al. | May 2014 | A1 |
20140143320 | Sivasubramanian et al. | May 2014 | A1 |
20140257891 | Richardson et al. | Sep 2014 | A1 |
Number | Date | Country |
---|---|---|
1511399 | Jul 2004 | CN |
101473598 | Jul 2009 | CN |
3748216 | Feb 2006 | JP |
Entry |
---|
Al-Fares, M. et al., A Scalable, Commodity Data Center Network Architecture, SIGCOMM '08 Proceedings, Aug. 17, 2008, pp. 63-74, 66-68, 70-71, Seattle, WA. |
Bennami, M., et al., Resource Allocation for Autonomic Data Centers Using Analytic Performance Models, 2005, IEEE, 12 pages. |
Chang, F., et al., Automatic Configuration and Run-time Adaptation of Distributed Applications, 2000, IEEE, 10 pages. |
First Office Action in Chinese Application No. 201080057229.9 dated May 14, 2014. |
Greenberg, A. et al., Networking the Cloud, 29th IEEE International Conference on Distributed Computing Systems (ICDCS 2009), Jun. 22, 2009-Jun. 26, 2009 [online] retrieved from the Internet on Mar. 10, 2011: http://www.cse.ohio-state.edu/icdcs2009/Keynote—files/greenberg-keynote.pdf, pp. 1-45. |
Greenberg, A. et al., Towards a Next Generation Data Center Architecture: Scalability and Commoditization, SIGCOMM '08: Proceedings of the 2008 SIGCOMM Conference and Co-Located Workshops NSDR '08, WOSN '08, MOBIARCH '08, NETECON '08, & Presto '08, Seattle, WA, Aug. 17-28, 2008, ACM, Aug. 17, 2008, pp. 57-62, New York, NY. |
Greenberg, A. et al., VL2: A scalable and flexible data center network, SIGCOMM '09, Proceedings of the ACM SIGCOMM 2009 Conference on Data Communication, Aug. 17, 2009, vol. 39, Issue 4, pp. 51-62. |
International Preliminary Report on Patentability and Written Opinion in PCT/US2010/060567 mailed on Jun. 19, 2012. |
International Preliminary Report on Patentability and Written Opinion in PCT/US2010/060569 mailed Jun. 19, 2012. |
International Preliminary Report on Patentability and Written Opinion in PCT/US2010/060573 mailed Jun. 19, 2012. |
International Search Report and Written Opinion in PCT/US2010/060567 mailed on Mar. 28, 2012. |
Kounev, S., et al., Autonomic QoS-Aware Resource Management in Grid Computing Using Online Performance Models, 2007, ICST, Valuetools, 2007, 10 pages. |
Mysore, R.N. et al., Portland: a scalable fault-tolerant layer 2 data center network fabric, SIGCOMM '09, Proceedings of the ACM SIGCOMM 2009 Conference on Data Communication, Aug. 17, 2009, pp. 39-50. |
Nilsson et al., IP-Address Lookup Using LC-Tries, IEEE Journal on Selected Areas of Communication, Jun. 1999, vol. 17, Issue 6, pp. 1083-1092. |
Office Action in Canadian Application No. 2784699 dated Apr. 28, 2014. |
Office Action in Canadian Application No. 2784706 dated May 22, 2014. |
Office Action in Chinese Application No. 201080057155.9 dated Jul. 24, 2014. |
Office Action in Chinese Application No. 201080057225.0 dated Jul. 2, 2014. |
Second Office Action in Chinese Application No. 201080057229.9 dated Mar. 18, 2015. |
Number | Date | Country | |
---|---|---|---|
20150172178 A1 | Jun 2015 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 13620363 | Sep 2012 | US |
Child | 14632962 | US | |
Parent | 12641260 | Dec 2009 | US |
Child | 13620363 | US |