System and method for fault notification and correlation

Abstract

In a communications network, a system and method for fault notification and correlation. The system includes a notification collector that serves as an intermediary for processing fault notifications. If the fault notification volume exceeds a defined threshold, such that in the prior art could involve dropped notifications, the notification collector processes the critical fault notifications. The notification collector may also assign a higher priority to processing the fault notifications than the faulty components do for generating the fault notifications. In this manner, the need for a fault correlation subsystem is reduced.

Description

CROSS-REFERENCES TO RELATED APPLICATIONS

[0001] NOT APPLICABLE

STATEMENT AS TO RIGHTS TO INVENTIONS MADE UNDER FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] NOT APPLICABLE

REFERENCE TO A “SEQUENCE LISTING,” A TABLE, OR A COMPUTER PROGRAM LISTING APPENDIX SUBMITTED ON A COMPACT DISK

[0003] NOT APPLICABLE

BACKGROUND OF THE INVENTION

[0004] The present invention relates generally to networking systems and specifically to a system and method for reporting and correlating network faults for network fault management.

[0005] Telecommunications networks often consist of a large number of distinct components and sub-systems, referred to as managed entities, that require configuration and monitoring. The state of a particular managed entity often depends on, and is partially driven by, the states of other managed entities.

[0006] When each of the managed entities change state, they often have to notify a management system of the state change. The state change is communicated to the management system by a Simple Network Management Protocol (SNMP) notification, for example. Since managed entities often depend on the state of other managed entities to determine their current state, a state change in one managed entity could cause state changes in tens of other managed entities, and could therefore trigger a cascade of tens or hundreds of notifications to the management system.

[0007] As a result, such a large number of notifications can potentially create a broadcast storm and flood the management system. In the case of an external management system, the communication channel between a node in the telecommunication network and management system can become severely congested due to the high number of notifications, thus reducing the overall throughput of the telecommunication network. Furthermore, a flood of notifications may cause one or more of the notifications to be dropped. In such a case, the management system will be unaware of the problems at the nodes corresponding to the dropped notifications.

[0008] Therefore, it is an object of the present invention to obviate or mitigate the above-mentioned disadvantages.

BRIEF SUMMARY OF THE INVENTION

[0009] In accordance with an aspect of the present invention, there is provided a network fault management system for managing a plurality of managed entities. Ones of the managed entities have a predefined dependency relationship with others of the managed entities. The network fault management system includes the following elements. A management system manages the plurality of managed entities. A local notification collector receives fault notifications generated by corresponding ones of the managed entities in response to a fault, and reports the fault notifications to the management system, thereby reducing the likelihood of dropping fault notifications.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] An embodiment of the present invention will now be described by way of example only with reference to the following drawings in which:

[0011] FIG. 1 is a block diagram of a network of managed entities;

[0012] FIG. 2 is a block diagram of a notification collector according to an embodiment of the present invention; and

[0013] FIG. 3 is a flowchart of a method according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0014] For convenience, like numerals in the description refer to like structures in the drawings. As previously described, many telecommunications networks include distinct components and sub-systems that require management by a management system. These component and sub-systems are referred to as managed entities. The management system is a remote entity that observes and configures one or more telecommunication networks. Managed entities include hardware components, physical interfaces, logical interfaces and services. Managed entities typically have interfaces to include or exclude them from the telecommunication network, and to display state information, statistics, and configuration parameters. Managed entities have a cohesive data set that describes and configures them, and may have state behavior that is influenced by operating stimuli.

[0015] A managed entity is often related to other managed entities in a dependency relationship. The relationship includes referential integrity between a dependent entity and an entity upon which it depends, referred to as the “dependee”. The referential integrity is generally in the form of a foreign key, which is an attribute (or group of attributes) that is used to refer to a primary key of a dependee. The primary key is an attribute (or group of attributes) that uniquely identifies the dependee. Further, there is often a state observation relationship as described by the “observer” pattern detailed in the 1995 book “Design Patterns” by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides.

[0016] An example of the relationship between dependent and dependee managed entities is illustrated as follows. Referring to FIG. 1, a portion of a telecommunication network is illustrated generally by numeral 100. The illustrated portion 100 includes a management system 108, a pair of asymmetric digital subscriber loop (ADSL) channels 106, an ADSL physical line 104, and a line card 102. The two ADSL channels 106 are multiplexed over the ADSL physical line 104, which is coupled to the line card 102. Thus, the ADSL channels 106 are dependent on the ADSL physical line 104, which is dependent on the line card 102.

[0017] The primary key for the line card 102 is <shelf, slot> and the primary key for the ADSL physical line 104 is <shelf, slot, port>. It can be seen that there is referential integrity between the <shelf, slot> portion of the primary key of the ADSL physical line 104 and the <shelf, slot> of the primary key of the line card 102. Thus, the <shelf, slot> portion of the ADSL physical line primary key is referred to as the foreign key. If the state of the line card 102 is “inactive”, the state of the ADSL physical line 104 is also “inactive” because of their dependency.

[0018] Similarly, the primary key for the ADSL channel 106 is <shelf, slot, port, type>. The <shelf, slot> portion of the ADSL channel primary key is the same as the <shelf, slot> primary key of the line card 102 to which the ADSL channel 106 is coupled. The <shelf, slot, port> portion of the ADSL channel primary key is the same as the <shelf, slot, port> primary key of the ADSL physical line 104 on which the ADSL channel 106 is multiplexed. Thus, if the state of either the line card 102 or the ADSL physical line 104 is “inactive”, the state of the ADSL channel 106 is also “inactive” because of their dependency.

[0019] Thus it can be seen that if the line card 102 deactivates, the associated ADSL physical line 104 deactivates, as do the ADSL channels 106, which can result in hundreds of fault notifications. For example, although only one line card 102 and two channels 106 are labeled in FIG. 1, it is not uncommon for a line card 102 to be coupled to 24 ADSL physical lines 104 and for each ADSL physical line 104 to have one or two ADSL channels 106. Further, each ADSL channel 106 may be coupled to an Asynchronous Transfer Mode (ATM) interface (not shown). Thus, if a line card 102 goes down, it changes state. Each of the 24 ADSL physical lines 104 also has to change state. Each of the ADSL channels 106 also has to change state. Each ATM interface (not shown) also has to change state. As a result, between 73 and 121 notifications may be generated for one deactivated line card 102, depending on the number of ADSL channels 106 per line. However, when this many notifications are sent, the management system can lose messages, thus giving it an incorrect picture of the system.

[0020] The number of notifications sent when a managed entity fails may be reduced as follows, with reference to FIG. 2, which is a block diagram showing a notification collector according to an embodiment of the present invention. Managed entities that depend on other managed entities register at initialization or provisioning time with the dependee entity for state change notifications. Typically, the management system 108 configures the managed entities and, thus, is aware of the system configuration. Managed entities that are a root cause of a state change send a state change notification 202 to a local notification collector 204 on the system. Since the notification collector 204 is a local network component, it can avoid the message flooding problem of the prior art by appropriately programming its task priority setting. That is, the notification collector 204 is assigned a higher task priority than all other tasks that generate notifications. Since the notification collector 204 is assigned a higher task priority than the managed entities, the managed entities will not be able to generate a notification while notification collector 204 is processing a notification. Therefore, the notification collector 204 will be able to process the notifications faster than they can be generated. The identity transmitted by the managed entity to the notification collector 204 includes its own identity as well as the identity of the managed entity that is the root cause of the state change. In this case, both identities are the same.

[0021] The managed entity also sends an internal state change message to all managed entities that depend on it, with its own identity as the root cause. Each dependent managed entity so notified sends a state change notification 202 to the notification collector 204. The notification 202 includes the identity of the managed entity that is the root cause of the state change and the identity of the managed entity sending the notification 202. In this case, the identities are different.

[0022] The dependent managed entities also send and internal state change notifications to their own dependent managed entities, along with the identity of the entity that was the root cause of the fault. Each dependent managed entity so notified sends a state change notification 202 to the notification collector 204. The notification 202 includes the identity of the managed entity that is the root cause of the state change and the identity of the managed entity sending the notification 202. In this case, the identities are different. This process continues until all dependent, managed entities have been notified and have notified the notification collector.

[0023] The notification collector 204 sends the received notifications 202 to the management system 108. If there are too many notifications to send, the notification collector 204 discards any notifications where the root cause identifier is not the same as the entity that sent the notification. A threshold is defined for determining what constitutes too many notifications. The threshold is programmable and is determined by a system operator as will be appreciated by a person skilled in the art. Generally, the threshold depends on the bandwidth of the communication link, the capacity of the management system 108, and the amount of notification buffers on the system.

[0024] The discarding of notifications is an effective notification filtering mechanism because, as previously described, the management system 108 is aware of the system configuration and dependencies between managed entities. Thus, the management system 108 is programmed to deduce the state changes of the dependent managed entities from the system configuration. In the present embodiment, the management system 108 includes codified fault correlation rules encapsulating associated managed entities in a directed acyclic state dependency graph, as illustrated in FIG. 1. In the diagram, managed entities are represented as nodes and dependency relationships as unidirectional edges. As previously described, dependents (represented by a tail of the graph edge) register with dependees (represented by an arrowhead of the graph edge) for state change notifications.

[0025] Continuing the example previously described, if the line card 102 is faulty, a state change notification 202 is sent to the notification collector 204 with the identity of the line card 102 as the root cause. The line card 102 also notifies each of the ADSL physical lines 104, and each of them deactivates. Each of the ADSL physical lines 104 sends a notification 202 to the notification collector 204 with the identity of the line card 102 as the root cause. Each ADSL physical line 104 notifies its dependent ADSL channels 106. Each of the ADSL channels 106 sends a notification 202 to the notification collector 204 with the identity of the line card 102 as the root cause. Each ADSL channel 106 notifies its dependent ATM interface (not shown), and each ATM interface (not shown) sends a state-change notification 202 to the notification collector 204 with the identity of the line card 102 as the root cause.

[0026] Therefore, the notification collector 204 has between 73 and 121 notifications to send to the management system 108. If that number is above the threshold that defines “too many”, only the line card state change notification is sent to the management system 108 because that notification is the case where the root cause identity is the same as the notification sender identity. The management system 108 can deduce the corresponding state change in the ADSL physical lines, the ADSL channels, and the ATM interfaces accordingly.

[0027] FIG. 3 is a flowchart summarizing the steps performed by a system that includes a notification collector according to an embodiment of the present invention. (More detail on these steps is provided above with reference to FIG. 2.) When the network is configured, some of the managed entities have a predefined dependency relationship with others of the managed entities. In step 302, each of the managed entities is registered with a dependee. In step 304, when a fault is detected, a notification is transmitted from a faulty entity to receiving entities. The receiving entities are all those entities dependant on the faulty entity. In step 306, upon receipt of the notification, a further notification is transmitted from each of the receiving entities to all managed entities dependant on those receiving entities. In step 308, step 306 is repeated until all dependant entities have been notified, thereby removing the need for a fault correlation subsystem.

[0028] The system described above inhibits a single fault from causing a storm of fault notifications. This is achieved by tagging notification events with the identity of the root cause of the fault. As a result, congestion of the communication channel to the management system 108 is reduced, as the notification collector 204 decides whether or not to filter out notifications that are not sent by the managed entity that is the root cause of the problem. The management system 108 deduces the rest of the state changes in the system using its knowledge of the system configuration and the relationships between managed entities. Thus, the correct states of all managed entities managed by the management system 108 can be reflected with a minimum number of notifications.

[0029] A further advantage of the system described above is that it eliminates the need for a fault correlation subsystem. Typically, fault correlation is performed by a fault correlation subsystem, which operates as follows. The managed entity fault is sent to the fault correlation system. The fault correlation system runs rules based on a system configuration, and updates the visible state of the managed entities.

[0030] However, this approach is error-prone because the state correlation logic has to be coded twice, once in the managed entity control software and once in the fault-correlation subsystem. Having two components of software calculating the same state information is risky because it may not produce the same results. Thus, having each managed entity report to the dependee that it received status notifications, reduces the need for a fault correlation subsystem.

[0031] Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto.

Claims

1. A network fault management system for managing a plurality of managed entities, wherein ones of said plurality of managed entities have a predefined dependency relationship with others of said plurality of managed entities, said network fault management system comprising: a management system that manages said plurality of managed entities; and a local notification collector that receives a plurality of fault notifications generated by corresponding ones of said plurality of managed entities in response to a fault, and that reports said plurality of fault notifications to said management system, thereby reducing a likelihood of dropping ones of said plurality of fault notifications.

2. A network fault management system as defined in claim 1, wherein each of said plurality of fault notifications includes an identity of a corresponding managed entity and a root cause managed entity that are associated with each of said plurality of fault notifications.

3. A network fault management system as defined in claim 2, wherein said notification collector determines whether a number of fault notifications exceeds a predefined threshold.

4. A network fault management system as defined in claim 3, wherein if said number of fault notifications exceeds said predefined threshold, said notification collector identifies said root cause managed entity to said management system.

5. A network fault management system as defined in claim 4, wherein said notification collector identifies said root cause managed entity to said management system by transmitting an associated one of said plurality of fault notifications.

6. A network fault management system as defined in claim 3, wherein if said number of fault notifications does not exceed said predefined threshold, said notification collector identifies all faulty ones of said plurality of managed entities to said management system.

7. A network fault management system as defined in claim 6, wherein said notification collector identifies all faulty ones of said plurality of managed entities to said management system by transmitting associated ones of said plurality of fault notifications.

8. A network fault management system as defined in claim 1, wherein said notification collector has a higher task priority for receiving said plurality of fault notifications than said plurality of managed entities do for generating said plurality of fault notifications.

9. A network fault management system as defined in claim 1, wherein said management system includes codified fault correlation rules encapsulating associated ones of said plurality of managed entities in a directed acyclic state dependency graph.

10. A method of fault correlation in a network comprising a plurality of managed entities, wherein ones of said plurality of managed entities have a predefined dependency relationship with others of said plurality of managed entities, comprising the steps of: (a) registering each of said plurality of managed entity identities with a dependee; (b) transmitting a notification, upon detection of a fault, from a faulty entity to a plurality of receiving entities being all managed entities dependant on said faulty entity; (c) transmitting a further notification, upon receipt of said notification, from said plurality of receiving entities to all managed entities dependant on said plurality of receiving entities; and (d) repeating step (c) until all dependant entities have been notified, thereby removing the need for a fault correlation subsystem.

11. A method as defined in claim 10, wherein said notification includes an identity of a managed entity transmitting said notification and an identity of a root cause managed entity.