1. Field of the Invention
The invention relates to the field of networks. More specifically, the invention relates to network elements.
2. Background of the Invention
A network element hosts multiple processes to maintain data for network communication. These processes relay information to each other with inter-process communication (IPC). The middleware of the network element will maintain process identification numbers for the processes running on the network element. One process will communicate directly with another process using these process identification numbers. Often within a network element, multiple processors run different operating systems.
If a process wants to communicate with a process that is dead, the process continues passing requests to the dead process. The requesting process detects the failure of the dead process through a response to the request or via timeouts. Although the operating system can detect when a process dies, it does not immediately communicate state of the process to other processes.
One method of IPC utilizes heartbeat messaging between processes. Once communication is established between two processes on a network element, the two processes periodically transmit heartbeat messages or signals indicating that they are alive and running. Death of one of the processes is detected by the other process when a heartbeat message has not been received within a given time period. Once a process is dead, however, the living process is ignorant of the dead process restarting. In addition, if both communicating processes die, when they restart different scenarios can occur. If both processes restart within the same time period, then they will both send requests. If one process restarts while the other remains dead, then the requesting process will repeatedly transmit requests to the dead process until it restarts.
Processes communicate with each other to disseminate information. One process on a network element may gather information about the interfaces of the network element while another process gathers routing information. This information is exchanged and/or passed on to other processes to facilitate processing and transmission of network traffic.
When a process requires information from another process, the process will send an IPC message to the other process requesting information or data. The other process will then pass a response back to the requesting process with the requested data.
If a requesting process does not receive a response within a certain time period, then the requesting process will mark the data from the timed out process as stale. Since the requesting process is unaware of the state of the timed out process, it sets a long timer on the stale data. When the timer expires, the stale data is removed.
Unfortunately, without information about the state of the timed out process, the requesting process cannot function intelligently. The stale data may be used beyond its life. Traffic processed with the stale data may be dropped or delayed. The length of time the data should be considered stale begins at some point before the timeout until the time expires. The amount of traffic impacted increases in proportion to this length of time.
The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention. In the drawings:
In the following description, numerous specific details are set forth to provide a thorough understanding of the invention. However, it is understood that the invention may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the invention.
The Process State Manager
Assuming the process 103 is interested in the process 105, the process 103 registers interest in the process 105 with the PSM 101 by sending a lookup request to the PSM 101. The lookup request can identify the process 105 by a symbolic name, an identifier provided by the process 105. After the process 105 has registered with the PSM 101 and the process 103 registers interest of process 105 with the PSM 101, the PSM 101 passes process 105's communication key to process 103. If the process 105 has not registered with the PSM 101, then the process 103 waits until the process 105 registers with the PSM 101 or polls the PSM 101 for the process 105's communication key. The process 103 uses the communication key for inter-process communication (IPC) with the process 105. The process 105 will compare the communication key transmitted by process 103 with its communication key. If the keys do not match, then process 105 rejects messages from process 103.
The communication key includes a process identifier and an incarnation identifier. The process identifier is unique for each process registered with the PSM 101. The unique process identifier identifies a process. The incarnation identifier indicates an incarnation or version of the process. When a process first starts, its incarnation identifier is an initial value. Each time the process restarts, its incarnation identifier is updated to reflect the new version or incarnation.
For example, if the process 105 dies, then it will no longer transmit heartbeat messages to the PSM 101. The PSM marks the process 105 as dead when the defined time period for receiving a heartbeat message from the process 105 expires. Since the processes 103 is interested in the process 105, the PSM 101 will transmit a death notification to the interested process 103. With this information, the process 103 can function intelligently and perform other tasks without expending time attempting communication with the dead process 105. When the process 105 restarts, it will request a communication key from the PSM 101. The PSM 101 will find the already created process identifier for process 105 and update the incarnation identifier for the process 105 to indicate the new incarnation. After updating the incarnation identifier, the PSM 101 transmits the new communication key to the process 105. The PSM 101 then transmits the new communication key for process 105 to the interested process 103. Process 103 receives the new communication key for process 105 asynchronously. Once received, the process 103 can begin communication with process 105.
In one embodiment of the invention, the process 103 also transmits heartbeat messages to the PSM 101. In another embodiment of the invention, an operating system running on the processor 201 determines when the process 103 dies. If the process 103 dies, then the operating system will notify the PSM 101 of process 103 dying.
The described embodiments of the invention provide intelligence to processes. Processes can efficiently and intelligently perform tasks with knowledge of which processes are available for communication. A process does not expend time attempting to establish communications with a dead process. Instead, the process can complete other tasks until the dead process restarts.
If the PSM determines the received request to be a register request at block 503, then at block 511 the PSM determines if the requesting process was dead. If the requesting process was not dead (i.e., the requesting process has previously registered, then the PSM creates a unique process identifier and a new incarnation identifier for a communication key at block 515. Control flows from block 515 to block 517. If the requesting process was dead, then at block 513 the PSM uses the process identifier for the requesting process and updates the requesting process' incarnation identifier to create a new communication key. At block 517, the PSM transmits the communication key to the requesting process. At block 519, the PSM transmits a birth notification indicating the new communication key to processes interested in the requesting process.
As stated above, the described embodiments of the invention provide intelligence to processes. With this intelligence, processes can performs tasks efficiently. In addition, the incarnation identifier of the communication key provides intelligence of a process restarting. A process may have a different task set upon determining another communication has restarted. For example, an interested process may request a refresh of data from the restarted process.
Process Sync Restart
As an illustration, assume the network process 901 discovers 3 interfaces on its host network element: interface 1, interface 2, and interface 3. The network process 901 determines that all 3 interfaces are up and have cables connected. The network process 901 communicates this information to the network process 903. The network processes 903 stores this information from the network process 901 and uses it to determine routing information. The network process 903 determines the routing information as indicated in Table 1.
The network process 901 dies and restarts. The network process 901 discovers that the interface 2 is down, but discovers an interface 4 is up and has a cable connected. The network process 901 communicates this new information to the network process 903. The network process 903 synchronizes this new information with the stored information. The network process 903 then modifies its routing information as indicated in Table 2.
The change in information ripples through the communicating network processes. The network process 907 previously stored the information shown in Table 1. The network process 907 will receive the information shown in Table 2 from the network process 903. When the network process 907 synchronizes the two sets of data, the absence of information for 2.2.2.2 implies that it should be removed. The network process 909 transmits data using the information from the network process 907. Although some traffic transmitted to 2.2.2.2 may be lost because the interface 2 goes down, the network process 909 can still transmit traffic to 1.1.1.1 and 3.3.3.3 despite the death of the network process 901. In addition, if the network process 903 dies, the network process can still transmit traffic without interruption. The described mechanism for seamlessly synchronizing data from restarted processes avoids service delay and service interruption typically caused by internal errors. Hence, the described invention increases robustness and reliability of a network element.
Referring to the example described above, the information gathered by the network process 901 is stored in its memory area 1001. The information gathered by the network process 903 is stored in the area of memory 1003. Since the network process 903 has requested information from the network process 901, information gathered by the network process 901 is also stored in the area of memory 1001. The network process 907 stores information from the network process 903 in the area of memory 1007. Therefore, referring the to above described example, interface information is stored in the areas of memory 1001, 1003 and 1007. If the network process 909 requests interface information, then interface information will also be stored in the area of memory 1009. Routing information collected by the network process 903 will be stored in the area of memory 1001, 1003, and possibly 1009 if the network process 909 requests such information from the network process 903 directly or via the network process 907. In another embodiment of the invention, the memory 1002 is multiple memories.
The described embodiments of the invention improve reliability of a network element. Providing intelligence to the processes of a network element enables processes to function efficiently as previously stated. In addition, intelligence about other processes enables processes of a network element to function independently despite failures without interrupting service. Each process can use stored data from other processes to facilitate processing and/or transmission of traffic even though other processes are dead. Knowledge of other process' states also enable processes to determine how long data can be used and if the data can be refreshed.
The described network elements include line cards and control cards executing the described processes. The line cards and control cards of the network elements include memories, processors, and/or Application Specific Integrated Circuits (“ASICs”). Such memory includes a machine-readable medium on which is stored a set of instructions (i.e., software) embodying anyone, or all, of the methodologies described herein. Software can reside, completely or at least partially, within this memory and/or within the processor and/or ASICs. For the purpose of this specification, the term “machine-readable medium” shall be taken to include any mechanism that provides (i.e., stores and/or transmits) information in a form readable by a machine (e.g., a computer). For example, a machine-readable medium includes read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, and flash memory devices.
While the invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the embodiments described. The method and apparatus of the invention can be practiced with modification and alteration within the spirit and scope of the appended claims. The description is thus to be regarded as illustrative instead of limiting on the invention.
Number | Name | Date | Kind |
---|---|---|---|
4325120 | Colley et al. | Apr 1982 | A |
4847749 | Collins et al. | Jul 1989 | A |
4864557 | Morton et al. | Sep 1989 | A |
5220668 | Bullis | Jun 1993 | A |
5230051 | Quan | Jul 1993 | A |
5301309 | Sugano | Apr 1994 | A |
5440726 | Fuchs et al. | Aug 1995 | A |
5748882 | Huang | May 1998 | A |
5774717 | Porcaro | Jun 1998 | A |
5838659 | Kainulainen | Nov 1998 | A |
5907701 | Hanson | May 1999 | A |
5923832 | Shirakihara et al. | Jul 1999 | A |
5938775 | Damani et al. | Aug 1999 | A |
5956719 | Kudo et al. | Sep 1999 | A |
5978933 | Wyld et al. | Nov 1999 | A |
5991794 | Hodges et al. | Nov 1999 | A |
6012150 | Bartfai et al. | Jan 2000 | A |
6044475 | Chung et al. | Mar 2000 | A |
6049838 | Miller et al. | Apr 2000 | A |
6148415 | Kobayashi et al. | Nov 2000 | A |
6178522 | Zhou et al. | Jan 2001 | B1 |
6233623 | Jeffords et al. | May 2001 | B1 |
6292905 | Wallach et al. | Sep 2001 | B1 |
6370656 | Olarig et al. | Apr 2002 | B1 |
6381653 | Feray et al. | Apr 2002 | B1 |
6393582 | Klecka et al. | May 2002 | B1 |
6401216 | Meth et al. | Jun 2002 | B1 |
6405262 | Vogel et al. | Jun 2002 | B1 |
6446134 | Nakamura | Sep 2002 | B1 |
6463446 | Wollrath et al. | Oct 2002 | B1 |
6477663 | Laranjeira et al. | Nov 2002 | B1 |
6564341 | Sundaram et al. | May 2003 | B1 |
6571270 | Lai et al. | May 2003 | B1 |
6594779 | Chandra et al. | Jul 2003 | B1 |
6598094 | Wollrath et al. | Jul 2003 | B1 |
6647432 | Ahmed et al. | Nov 2003 | B1 |
6651185 | Sauvage | Nov 2003 | B1 |
6654948 | Konuru et al. | Nov 2003 | B1 |
6681389 | Engel et al. | Jan 2004 | B1 |
6684396 | Brittain et al. | Jan 2004 | B1 |
6694450 | Kidder et al. | Feb 2004 | B1 |
6718376 | Chu et al. | Apr 2004 | B1 |
6721898 | Hasha | Apr 2004 | B1 |
6721907 | Earl | Apr 2004 | B2 |
6763384 | Gupta et al. | Jul 2004 | B1 |
6792604 | Hickson et al. | Sep 2004 | B1 |
6823474 | Kampe et al. | Nov 2004 | B2 |
6834326 | Wang et al. | Dec 2004 | B1 |
6876625 | McAllister et al. | Apr 2005 | B1 |
6907605 | Ault et al. | Jun 2005 | B1 |
6973023 | Saleh et al. | Dec 2005 | B1 |
6983317 | Bishop et al. | Jan 2006 | B1 |
20020049776 | Aronoff et al. | Apr 2002 | A1 |
Number | Date | Country |
---|---|---|
08204823 | Aug 1996 | JP |
PCTUS0215322 | Mar 2003 | WO |
Entry |
---|
Heimbigner, Dennis. “The Process Wall: A Process State Server Approach to Process Programming.” ACM. 1992. |
PCT/US02/15322 Search Report, mailed Jul. 22, 2002. |
Larry Wall et al., “Programming Perl”, Sep. 1996, pp. 337-355, O'Reilly and Associates, Sebastopol, California. |
Aeleen Frisch, Essential System Administration, Sep. 1995, pp. 41-50, O'Reilly and Associates, Sebastopol, California. |
Heimbigner, Dennis et al., “Software Environments in Support of Wide-Area Development Final Report.”, Dec. 26, 2000, pp. 85 total. |
Number | Date | Country | |
---|---|---|---|
20020199023 A1 | Dec 2002 | US |