Patent Grant
10797962
The present invention relates generally to improved network management. More specifically, the present invention relates to improved methods for configuring, monitoring, and analyzing an adaptive private network.
Wide area network (WAN) standards include, for example, digital subscriber line (DSL), asymmetric digital subscriber line (ADSL), and multiprotocol label switching (MPLS), to mention a few. WANs are used to connect local area networks (LAN's) allowing devices in one location to communicate with devices and their users in other locations. In a WAN having a large number of remote sites, direct connections between the sites are many times statically configured. The dynamics of the network system may also change over time making repeated static configurations of the network inefficient and costly to implement. Further, static connections involve reservations of network resources. As data flow patterns change in the network, the reserved resources create non-optimal static connections which cause the network to reserve bandwidth that could be better used elsewhere in the network.
A model-view-controller (MVC) design pattern was articulated in the 1988 paper “A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System” by Krasner and Pope. This MVC design pattern divides an application into the three components: a model, a view, and a controller. The model component maintains state of underlying data and applies operations on that data. The view component is responsible for rendering of the data controlled by the model to a user. The view component is notified about changes to the data by the model and updates the rendered view of the data. The controller component is responsible for taking input from the user and sending appropriate messages to the model. A view component and controller component are typically paired and communicate with the same model component. A model component may interact with multiple sets of views and controllers. It is generally assumed in a framework of the MVC design pattern that the model has the ability to broadcast changes in the model to the views and controllers that it is connected with. However, this assumption does not hold in web applications. In web applications, a web browser updates its view only based on a request to the server that holds the model. As a result, changes to the model cannot be automatically pushed to the user interface. The MVC design pattern is a theoretical framework which provides a guide for system developers and is not related to any specific system. Further, the MVC design pattern framework by itself does not provide improved performance, reliability, and predictability of a network.
As networks become larger and more complex, administrative techniques for managing the network are increasingly more complex and costly. Prior techniques to separately configure, monitor, and analyze each node of a large network may not provide accurate information and are prone to errors.
Among its several aspects, the present invention recognizes what is needed is a management technique that provides more accurate and lower cost techniques to configure, monitor, analyze a network, and to present related data in a user friendly manner. Among its several aspects, the present invention addresses systems and techniques which improve discovery, database management, graph tree reporting, replay, and time correlation. To such ends, an embodiment of the invention applies a method for display management in an adaptive private network (APN). A configuration database and a statistics database are installed according to rules that define tables of data in a first node of a centralized management system including a network control node (NCN) coupled through an adaptive private network (APN) to a plurality of client nodes, wherein the NCN is separate from each client node including the first node and the NCN administers and controls client nodes within the APN. Data is queried in the configuration database and in the statistics database in the first node, wherein the configuration data is associated with a user specified time period of operation of the APN and includes statistics regarding operation of conduits in the APN. The queried data is retrieved by the centralized management system. The retrieved data is formatted for display according to a network configuration, a user selected display type, and attributes selected by the user for display, wherein the configuration data is used to display status of conduit operations for each conduit displayed in a map of the NCN and selected client nodes of the APN.
Another embodiment of the invention addresses a method for display management of an adaptive private network (APN). A database is installed, according to rules that define tables of data, in a first node of a centralized management system including a network control node (NCN) coupled through the APN to a plurality of client nodes, wherein the NCN is separate from each client node including the first node and the NCN administers and controls client nodes within the APN. A plurality of timing messages are sent from the NCN to a client node, wherein each timing message is time stamped with a send time according to a first clock in the NCN. A network time is calibrated in the client node based on an evaluation of an average of send times, an average of arrival times for the plurality of timing messages received at the client node, current time in the client node, and a round trip time between the NCN and the client node, wherein timestamps of data gathered from the NCN and each client node of the plurality of client nodes are correlated to a master time specified by the first node. The NCN and client nodes of the APN are polled for user specified data to be stored in the database in the first node, wherein the data is associated with a user specified period of time of operation of the APN as correlated to the master time and includes statistics regarding operation of the NCN, client nodes, and paths in the APN. The user specified data is accessed from the database in the first node for display according to a network configuration, a user selected display type, and attributes selected by the user for display of a network map including operating statistics for each path between wide area network (WAN) links.
A further embodiment of the invention addresses a computer readable non-transitory medium storing a computer program which causes a computer system to perform a method for display management of an adaptive private network (APN). A database is installed, according to rules that define tables of data, in a first node of a centralized management system including a network control node (NCN) coupled through the APN to a plurality of client nodes, wherein the NCN is separate from each client node including the first node and the NCN administers and controls client nodes within the APN. A plurality of timing messages are sent from the NCN to a client node, wherein each timing message is time stamped with a send time according to a first clock in the NCN. A network time is calibrated in the client node based on an evaluation of an average of send times, an average of arrival times for the plurality of timing messages received at the client node, current time in the client node, and a round trip time between the NCN and the client node, wherein timestamps of data gathered from the NCN and each client node of the plurality of client nodes are correlated to a master time specified by the first node. The NCN and client nodes of the APN are polled for user specified data to be stored in the database in the first node, wherein the data is associated with a user specified period of time of operation of the APN as correlated to the master time and includes statistics regarding operation of the NCN, client nodes, and paths in the APN. The user specified data is accessed from the database in the first node for display according to a network configuration, a user selected display type, and attributes selected by the user for display of a network map including operating statistics for each path between wide area network (WAN) links.
A more complete understanding of the present invention, as well as other features and advantages of the invention, will be apparent from the following detailed description, the accompanying drawings, and the claims.
Exemplary embodiments of the invention will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only exemplary embodiments and are, therefore, not to be considered limiting of the invention's scope, the exemplary embodiments of the invention will be described with additional specificity and detail through use of the accompanying drawings in which:
An APN path is a logical connection established between two WAN links located at different geographic sites across a WAN.
An APN conduit is a virtual connection between two APN nodes, also referred to as client sites, and formed by aggregating one or more APN paths and their allocated WAN link resources. The conduits overlay a virtual network on top of the underlying network.
A conduit maximum transmission unit (MTU) is a minimum link MTU of the one or more APN paths between a source site and a destination site.
An APN appliance (APNA) is a device that contains APN client site functionality including all software modules within. A high availability site contains two APNAs, one that is active and one that is in a standby mode of operation and available to become active in place of the other APNA if required.
A WAN link represents a physical access point to the wide area network (WAN), such as a digital subscriber line (DSL) connection or a cable modem. The distinctive characteristic of a WAN link is the bandwidth, or in other words, the amount of data capacity available for transmission and reception. WAN links can be shared among APN conduits, and intranet and Internet network services. In the present embodiments, the APN appliances do not directly attach to WAN links. APN appliances communicate with WAN links through logical connections, such as the WAN routers 1101-1103 of
A private WAN link provides a physical access point to non-public WAN destinations. Examples of such private WAN links include an asynchronous transfer mode (ATM) link with an ATM virtual circuit, a frame relay link with a frame relay circuit, a multiprotocol label switching (MPLS) tunnel, a virtual private network (VPN) tunnel, or a leased point-to-point line. Connectivity on a network having a private WAN link is made to a private list of destinations on the other end of the network. A public WAN link represents a physical access point to the Internet. It can be assumed that any public WAN link can establish a connection to any other public WAN link.
A local WAN link (LWL) is an APN client site's access point to a WAN. A site A's LWL is coupled to a corresponding remote WAN link for a site B. For a conduit between a site A and a site B, site A's local WAN links are site B's remote WAN links.
A routing domain represents a group of sites that can reach each other via an intermediate site that has WAN-to-WAN forwarding enabled. All local routes of each site in the routing domain are added to all other sites in the routing domain.
A static conduit is a conduit configured in a configuration file and created at startup time of an APNA. A static conduit is not removed without changing the configuration file.
A dynamic conduit is a conduit created between APN clients when needed and which can be removed when no longer needed.
An APN service is a set of processing steps performed on packets that are transmitted through the APN. As illustrated in
An APN conduit service associated with path 112 manages network traffic packets that are transmitted through the APN 100 from the APN appliance 105 through router 1101, through the WAN 102, through another router 1103 to APN appliance 104. The APN conduit service for path 112 operates on both APN appliances 104 and 105. The APN conduit service sends and receives data between a first geographic location that has an APN appliance 105 and a different geographic location that has an APN appliance 104 utilizing the full benefits provided by the APN conduit service for WAN resource allocation and network adaptation. An APN intranet service associated with path 114 is used to manage the sending and receiving of data between a first geographic location that has the APN appliance 105 and a different geographic location within an enterprise non-APN site 120 that does not have an APN appliance by way of a WAN link that is also utilized by other APN services.
In another embodiment, an APN intranet service, such as the one associated with path 112, may be used to send and receive data to and from a different geographic location that has an APN appliance, but an administrator selectively configures the APN not to use the APN conduit service 112 for a particular type or class of traffic. An APN Internet service associated with path 116 is used to send and receive data between a first geographic location that has the APN appliance 105 and a different geographic location that is external to an enterprise network by way of a WAN link that is also utilized by other APN services. For example, traffic using the APN Internet service may be associated with a network user accessing a public Internet web server 122. An APN pass through service 118 is used to send and receive data between a first geographic location that has an APN appliance 105 and a local site 124 within the same first geographic location. In another embodiment, an APN pass through service may be used to send and receive data between a first geographic location that has the APN appliance (APNA) 105 and a different geographic location within an enterprise network that does not have an APN appliance and does not traverse the WAN using any WAN links associated with any other APN services.
Dynamic conduits address changes in statically configured networks that are not just slow, gradual changes in network usage, but are happening in real time throughout a day across a global network. In real time, dynamic conduits dynamically optimize network performance adapting to changing communication patterns between nodes in the network. Dynamic conduits can also be used to offload traffic from intermediate nodes that may be experiencing congestion.
An adaptive private network (APN) software product according to the present invention runs as a centralized management system within a virtual machine to create APN configurations and to monitor system resources, analyze system resources, and manage a configured APN in operation as addressed further herein. Aware is a product name for a presently preferred embodiment of the centralized management system that includes capabilities that monitor, analyze, and provide discovery, timestamp correlation, and database schema migration processes of the present invention. The APN software of the invention, also referred to as APN virtual machine (VM) software, provides analysis and monitoring capabilities that are timely with respect to events to be tracked and monitored while the APN is in operation and provides storage for historical data as taught further herein. The APN system, also referred to as an APN VM system, reduces the time to configure APN appliances and the number of errors that can occur in configuring a system, as well as, to provide detailed performance data correlated across the WAN. The APN VM system further allows a centralized virtual single point of control by a network control node (NCN) for a physical network in which the NCN provides system wide timing synchronization. The centralized single point of control, also referred to as a centralized control point, is not limited to a central location within a network of nodes, may be at any point within the network, and may be coupled at a point that would be considered outside the boundary of a network. Centralized indicates the single point of control aspects of the APN as described further herein.
An onboard configuration facility is a software component designed to plugin to the APN VM system of the invention and provide an APN configuration compiler, APN configuration editing capabilities, and to provide an ability to create and edit network maps showing nodes of the APN and conduits between the nodes. Each version of the APNA software produces a version of the onboard configuration facility that understands an object model and configuration options for that version of APNA software. The APN VM system supports installation of multiple concurrent onboard configuration facility plugins so that a single APN software version can manage a variety of APNA software configuration versions. Each version of APNA software, the appliance code, is provided with a default version of the configuration facility, which is also referred to as a configuration plugin. Multiple configuration plugins may be installed. So, the term “onboard” is in reference to the configuration facility or “plugin” when it is running on the APN VM system or on an NCN.
An onboard configuration editor is a component of the onboard configuration facility that represents an APN configuration as a hypertext markup language (HTML) tree and accepts changes to the APN configuration from a user. The onboard configuration editor is closely coupled with a configuration compiler to make changes to the configuration HTML tree. The onboard configuration editor also integrates with a network map facility to display site nodes in a visual map representation of the APN.
An APN configuration file is a text file which describes a configuration of the APN. This configuration file serves as an input to the configuration compiler which generates registries for each APNA in the network.
The configuration compiler is a software program, such as a Java™ program, that can run on an APN VM system and converts an APN configuration file into either registries for use by APNAs or into an extensible markup language (XML) representation of the object model for use by the onboard configuration facility.
A configuration package is a software data file which contains the APN configuration file along with metadata. Such metadata includes the network maps that are derived from a specific APN configuration.
An onboard configuration facility package consists of the onboard configuration facility in a format which can be installed onto the APN VM system.
Adaptive private network appliance (APNA) settings are management settings that can be set directly on an APNA. These APNA settings include time parameters, such as for a time zone or time zones and for network time protocol (NTP) including an NTP server address, settings for a Netflow server, user authentication, simple network management protocol (SNMP), event handling, and periodic status reports. These APNA settings are generally not configurable through the APN configuration file. Rather, the APNA Settings are managed on a network-wide basis through the APN controls and software of the invention.
A dashboard, in the context of the APN VM system, is a user configurable display screen which may be customized to display a subset of items from the rest of the APN VM system. Multiple dashboards may be created with one being chosen as a default home screen for a particular user.
Workspaces are a construct which allow a user to organize a set of objects, allowing the user to save and recreate a state of a management session. Workspaces are used in a similar manner to use of a ‘project’ in a software integrated development environment (IDE) which collects a set of source code files and associated build scripts and resources such as help text and images to create a complex graphical application.
As illustrated in
The APN is capable of using disparate asymmetric WAN links which frequently vary in behavior of bandwidth, latency, jitter, packet loss and congestion over time. For example, the APN can use an asymmetric DSL WAN link that transmits data at 512 kbps upstream to the WAN and 6 Mbps from the WAN through the public network combined with a private symmetric leased circuit T1 WAN link that transmits data at 1544 kbps upstream and downstream and a cable broadband connection that transmits data at 312 kbps upstream to the WAN and 3 Mbps from the WAN to a peer having adequate aggregation bandwidth of these rates for a single transmission control protocol (TCP) file transfer session at a theoretical transmit rate of 2368 kbps and receive at 10544 kbps or 10.544 Mbps. Practically, under good network behavior, the actual rate would approach 90% of these rates. If the behavior of the connection was to change, for example the paths to the DSL link were to have dramatic levels of loss, the APN would, using its high frequency performance feedback mechanism, adapt the network to avoid or mitigate the issues by using alternative resources or attempting to recover from the loss.
In all path selections, conduit paths are evaluated and the best available path is selected. Any paths currently in a path quality good state are eligible to be chosen first. If multiple paths are in a path quality good state, then an estimated end to end time is evaluated and compared for each path, and the path with the lowest end to end time is chosen. If no path is in path quality good state, then a path with the highest bandwidth path quality bad state is chosen. A “one way time” (OWT) refers to the amount of time it takes for a packet to traverse a network from source to receiver. In the context of this invention, the one way time is measured by subtracting a receive time stamp from a WAN Egress Module 166 from the send time stamp from a WAN Ingress Module 160,
APN path processing services are responsible for providing a means of communicating user data and control information from one APN node to another APN node across the network. In particular, user data and control information may be transmitted from the WAN ingress processor module 160 of one APN node across the WAN and received at the WAN egress processor module 166, as shown for example in
1. Universal path tagging of all conduit traffic sent across the WAN with high resolution and highly synchronized APN time stamps to enable the highly predictive estimation of transmission latency and statistical variation of latency, subsequently in tandem a control plane modules' path state monitoring service is used to detect optimal paths for traffic to use across the APN.
2. Use of the above optimal path identification to provide, in tandem with a WAN link accounting module, WAN bandwidth reallocation from low performing paths to higher performing paths.
3. Universal path tagging, of all conduit traffic sent across the WAN APN path with path sequence numbers, enables sub second detection of packet loss enabling fast retransmission of user packets with little to no negative effect to the end users.
4. Continual monitoring of and characterization of network behavior at times of lower utilization using heartbeats for fast reaction when network demand does arrive, such as provided by a heartbeat generator.
5. The ability to identify and proactively solicit retransmission when network traffic has been extraordinarily delayed or if the network has ceased to function using a Nag method, as provided by a Nag process, operating on the path state monitoring module.
6. Universal path tagging of all conduit traffic with network utilization and non-utilization of WAN link resources enabling early detection and avoidance of network congestion prior to the packet loss that is typical of normal TCP like congestion methods.
7. The ability to transmit time sensitive control messages without typical internal scheduling delays for software process staging to rate schedulers, while still maintaining proper long utilizations to the APN network to do retransmission of lost packets without the highly predictive estimation of transmission latency and statistical variation of latency.
The APN client node uses timing data to adjust or calibrate a network time by using a linear algebraic calculation based on the slope-intercept form. In a current implementation, y is the time at an APN control node, also referred to as a network control node (NCN), and x is the client node local time, b is the base offset between the two, and m is the rate of change of y versus x which is the slope. Using these definitions, an equation in slope-intercept form y=mx+b is expressed as network time=slope*client local time+base.
The slope is calculated by taking two samples over a pre-specified period and averaging the samples together. The base offset is calculated by taking the difference of the value between the network control point time and the client time, adjusted for one half round trip time (RTT).
Using queuing theory, Poisson distribution assumptions, and a highly accurate APN wide APN clock sync that allows for accurate one way time measurement, a method is provided that is typically capable of estimating path latency and statistical jitter with an accuracy approaching ˜99%. An equation which may be suitably used is best one way time (BOWT)+(Mean WAN Jitter)+3*(√(mean WAN jitter)). This equation provides a very accurate inference with just a few samples of traffic over a short period.
A path state represents the most current condition of the network path as determined by feedback received by the WAN egress APN node's path state monitoring process. As packets are received, the sequence numbers of the packets are tracked to see if any packets were lost in transit between the WAN ingress APN node and the WAN egress APN node. A method is used to trigger path state transitions that are biased toward more tolerance for loss in the short periods of packets received with substantially less tolerance of loss over longer periods. A unique aspect of this approach is the ability to track the path's packet loss thresholds over numerous durations nearly simultaneously and continually while still maintaining low processor overhead. This aspect is obtained through the universal path tagging of conduit traffic sent across the WAN with high resolution and highly synchronized APN time stamps to enable the highly predictive estimation of transmission latency and statistical variation of latency. In tandem, a control plane modules' path state monitoring service is used to detect packet loss and optimal paths for traffic to use across the APN. The result is an ability to detect a difference between occasional incidental short term network loss and long term persistent problems.
In a presently preferred embodiment, the APN node's software modules at a client site are stored and operate in the same physical APN appliance; however, the modules may also exist in separate physical APN appliances in alternative embodiments. The methods described in connection with the embodiments disclosed herein may be embodied directly in one or more software modules executed by a processor and memory complex such as a rack mounted processing device, a personal computer, a server, or the like having one or more central processing unit devices. The processor and memory complex, for example, may be configured to execute instructions that access data and operate on data under control of a software module program stored on a computer readable non-transitory storage medium either directly associated locally with the processor and memory complex, such as may be available through an instruction cache, or accessible through an I/O device. A software module may reside in a computer readable non-transitory storage medium which may include random access memory (RAM), flash memory, dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), hard disk, a removable disk, a CD-ROM, digital video disk (DVD), other types of removable disks, or any other suitable non-transitory storage medium. A non-transitory storage medium may also be coupled to the processor and memory complex such that the hardware processor can read information from, and write information to, the storage medium over an intranet or the Internet.
An adaptive private network node (APN client site) contains software modules required to participate in an adaptive private network. An APN node may exist in one or more APN appliances at a location. An APN node contains a collection of software modules which govern its participation within an APN such as control plane modules 156 and 158, WAN ingress processor modules 160 and 162, and WAN egress processor modules 164 and 166 in
The WAN ingress processor module 160 may suitably be embodied as software and hardware components responsible for processing network traffic for transmission from a local area network (LAN) to a WAN. The WAN egress processor module 164 may suitably be embodied as software operating on hardware components, such as a processor and memory complex that is responsible for processing network traffic for transmission from a WAN to a LAN. WAN ingress and WAN egress processor modules are discussed in further detail below. The APN client site's control plane module 156 may suitably be embodied as software operating on hardware components, such as a processor and memory complex that utilizes the APN client site's WAN ingress processor module 160 and WAN egress processor module 164 as the means for transmitting and receiving APN node to APN node control data across the WAN.
Software packages for an APN are distributed through the WAN using control packets, termed Tapplication protocol (TAP), that is part of change management software or through administrative interfaces, such as downloading software using interfaces 168 and 170 to the APN client sites. The TAP is a protocol that is run on the WAN to allow processes outside of t2_app on different appliances to communicate with each other. The t2_app is a program that is running on each APNA communicating with other APNAs in the APN while forwarding user data. After a software update, the APN services on the APN client sites 152 and 154 are then restarted thus bringing the APN software node configuration into synchronization.
Each of the sites 204-219 and primary NCN site 202 contains an APN appliance to provide APN functionality. The configuration of the APN 200, generally provides for connectivity between a site A, such as site 205, and for a site B, such as site 208, where the connectivity from the site A's perspective is site A→LWL→“WAN”→RWL→site B. The connectivity from the site B's perspective is site B→LWL→“WAN”→RWL→site A. The WAN 201 represents allocated WAN link resources and APN selected paths. In
In one presently preferred embodiment, APN conduits exist between the NCN and, for example, sixteen APN client sites as shown in
For a definition of APN path states, a description of path processing services is provided below. Any paths currently in a path quality good state are eligible to be chosen first. If multiple paths are in a path quality good state, then an estimated end to end time is evaluated and compared for each path, and the path with the lowest end to end time is chosen. If no path is in a path quality good state, then a path in a path quality bad state with the highest bandwidth is chosen.
As used herein, an APN appliance is a device that contains APN node functionality according to software modules, such as the control plane module 156 and 158, the WAN ingress processor module 160 and 162, and the WAN egress processor module 164 and 166, as described in more detail above with reference to
A dynamic conduit is a conduit created between APN clients when needed and can be removed when no longer needed, based on a configured first threshold and a configured second threshold. For example, client site 205 can be configured with two local WAN links, one from a first network provider and one from a second network provider. Multiple conduits may be connected to site 205 which may be configured to use one or both of the local WAN links. In an exemplary scenario where all of the conduits that are connected to site 205 use both local WAN links, then when usage for either local WAN link passes a configured second threshold, creation of a dynamic conduit can be triggered as described in further detail below.
The centralized monitor, analysis and management software in accordance with the present invention installed on a server associated with the APN provides several functions which are useful in managing an APN. For example, a monitoring service monitors events data and provides results in a tabular format. A number of additional services are briefly described in this section.
In the APN, a network control node (NCN) is connected to the client nodes of the network, such as shown in
The APN VM 302 includes a plurality of components including a database server 306, a local storage 308, an APN VM user interface (UI) 310, a web server 312, an authentication function 314, a monitor and manager 318, a virtual file system (VFS) 319, a logger 320, a rate limiting function 322, and a port driver 324. The port driver 324 connects to a hardware interface such as an Ethernet interface to a network 326, such as a management network. For security reasons, the same interface used to communicate with the appliances is also the interface used to access the APN software from the processing device 329 to prevent a backdoor into the network. A management network is a separate logical or physical network that separates user data plane application traffic from management plane traffic. The management network as used herein could be thought of as a private management network.
The APNA 330 comprises a plurality of components including an APNA UI 332, an APNA database server 334, an APNA local storage unit 336, a local server identified as an appliance agent (AAgent) 338, a port A driver 340 which is the management network interface port on the appliance, and a data plane application (app) 342, a port B driver 344, and a port C driver 346. The data plane app 342 is an executable program which performs all communication processing of latency, loss, and jitter calculations on packets received and transmitted on the APNA, such as NCN clock synchronization packets and data communication packets. On a periodic basis, in response to a poll from the APN VM 302, such as every minute, the data plane app 342 updates the APNA database server 334 with statistics about the traffic processed over that minute for storage in the APNA local storage 336. Upon request from the monitor and manager 318 on the APN VM 302, the AAgent 338 gathers statistics from the APNA database server 334 or from the APNA local storage 336. The AAgent 338 packages up the gathered statistics into a report package file and sends the report, using the port A driver 340, through the management network 326, to the monitor and manager 318. The monitor and manager 318 unpacks the report package and sends the unpacked data to the database server 306 for storage in the datastore 304.
On the APNA that operates as a network control node (NCN), such as APNA 330, a discovery service is provided by the AAgent 338 for use by APN VM 302. A user may use the discovery service to configure the APN VM 302 by use of the management Internet protocol (IP) address of the NCN. The APN VM 302 uses the NCN management IP address to access the discovery service and obtain the complete list of APNAs in the network.
The APN VM 302 may suitably operate as a virtual machine on a hypervisor, such as VMware ESXi. The APN VM 302 stores and accesses statistics, information associated with network maps, and configuration data associated with the APNA 330 and APNA client sites 348 in the datastore 304. While the datastore 304 is shown outside of the APN VM 302, the APN centralized monitor, analysis and management system is not so limited. The location of the datastore 304 is configurable by a user. A base operating system, application software, and operation logs are stored in the local storage 308. A logger 320 records logs of operating events and takes care of rotating and pruning log files. A port driver 324, provides a communication interface such as an Ethernet interface, coupled between the APN VM 302 and the management network 326. A user may use a web browser 328 connected to the management network 326 to access the APN VM UI 310. The management network 326 is also connected to APNA client sites 348 and APNA 330 by means of management ports, such as port A driver 340. The network monitor portion of the monitor and manager 318 communicates with the AAgent 338 component of the APNA 330 and other agent components of the APNA client sites 348 to gather data and perform operations on the plurality of APNAs. The monitor and manager 318 uses a virtual file system (VFS) 319 to accept requests and communicate status to the rest of the user interface.
As an example, in a particular customer installation, an administrator installs the APN VM 302 on a processing node 303 running a hypervisor, such as VMWare ESXi 5.1, that, for example was already available on the customer's system. The administrator of the network uses the VM ware login facilities of the processing node 303 to determine the IP address of the APN VM instance, allocated by use of a dynamic host configuration protocol (DHCP) or uses a command line tool to set the IP address of the APN VM 302. The administrator then logins to the APN VM UI 310 with a web browser 328 and adds himself and other administrators as users of the APN VM 302. The administrator configures the domain name system (DNS), network time protocol (NTP), and time zone settings. The administrator instructs the APN VM 302 about the APN by configuring the IP address of the NCN, downloads security credentials certificates from the APN VM 302 and installs them on the NCN. The NCN automatically pushes those security credentials to all appliances in the APN. The APN VM 302 connects to the NCN, discovers the management IP addresses of all of the clients in the APN, connects to each of those appliances, and then displays to the administrator information about each appliance in the network, the APNA 330 and the APNA client sites 348. This information may suitably include the name of the appliance, management IP address, model number, serial number, software revision, registry timestamp, connectivity information, and polling status. A user enables polling of the network on the APN VM UI 310 and the APN VM 302 starts gathering statistics, starting with the earliest statistics available on the APNAs. The user does not need to set the polling period as a suitable default value, such as five minutes, is automatically used. The APN VM 302 displays graphs quickly from when the APN sites were initially installed and within two hours, for example, statistics from all of the appliances for an entire two week period are available in APN VM 302. The APN VM 302 systematically collects statistics starting with the oldest data and working towards the newest. The data is gathered at a specified pace so as not to cause performance problems on the management network or the APNAs using rate limiting module 322.
When an additional site is added to the APN using the onboard configuration component of the APN VM UI 310 on the APN VM 302, the new site is learned dynamically once the configuration is exported to the currently running NCN 330. The new site will be learned dynamically by the monitoring process if the site is added by the onboard configuration component of the NCN 330. The onboard configuration facility and the discovery and monitoring process are separate functions. The configuration facility does not contain the IP addresses of the sites that are being monitored. A “site” is added to the configuration, but in the process to add the site, the site's management IP address is not required for the configuration file. The monitoring process learns when a new site is added or removed from the network.
The administrator is able to use the APN VM 302 to create a new configuration with the new site and audit this new configuration for errors prior to exporting the newly created configuration with the new site to the NCN 330 for installation on the NCN. The administrator is also able to use the APN VM 302 to specify a link which opens a change management screen on the NCN to allow the administrator to activate the new configuration. Upon applying the new configuration to the APN, the APN VM 302 automatically detects, learns, that a new appliance has appeared on the APN through its monitoring process. The discovery process of
The APN VM 302 then discovers the management IP address of the new site and adds the new site to the list of appliances in the APN as monitored, analyzed, and managed by the APN VM 302. After the APN VM 302 exports the new configuration with the added site, the NCN automatically pushes the security credentials to the new site. The APN VM 302 checks periodically in the background for network additions or removals and polls for statistics from the appliances. The discovery process detects appliances added or removed.
The APN VM UI 310 stores user session information in local storage 308 and stores configuration files and network maps in the datastore 304. The database server 306 comprises a management plane (MP) configuration (MP_Config) database file server utility 352 and a network management system (NMS) statistics (stats) file server utility 354 associated with corresponding databases in the datastore 304. The MP_Config database file server utility 352 accesses a MP_Config database in the datastore 304 that contains data related to the APN VM UI 310 and the monitoring and management of APNAs as provided by the monitor and manager 318. The monitor and manager 318 comprises an adaptive private network appliance (APNA) monitor 356 and an APN manager 358. The APNA monitor 356 periodically polls information from the APNA, NCN, or Client 330. The APNA monitor 356 includes capability to generate a statistics (stats) request and unpack a response to the stats request. The APNA monitor 356 also includes capability to export appliance settings to the NCN and one or more client appliances. Export in this context means transmitting appliance specific settings to individual appliances and applying those settings on the appliance.
The APN manager 358 includes support for managing an APN discovery service which interacts with agents on an active NCN, such as AAgent 338 of
The NMS statistics file server utility 354 accesses a NMS_Stats database in the datastore 304 that contains the statistical data gathered from the APNAs. The NMS_Stats database has tables for each type of object, such as specified below. By a specified convention, for each <object type>, a set of tables is specified, including: <object type>s, <object type>_minutes, and <object_type>_hours. The <object type>s table contains the names and id numbers for the objects. The <object type>_minutes tables contain the minute samples for the objects. Each row in an <object type>_minute table contains an object_id which corresponds to an id in the <object type>s table. Each row in an <object type>_minute table also contains an update_epoch_time_s column which reflects the network management system (NMS) time at the time the minute sample was taken on the appliance and generally not when statistics are gathered. The rest of the columns in the row reflect the counts of the data for the event that is being tracked for that minute. The update_epoch_time_s is the epoch at which a sample was gathered on an APNA relative to the system clock on the APN VM 302. An update_epoch_time_s entry reflects the network management system (NMS) time at the time the minute sample was taken. So, the update_epoch_time_s is a converted time from when the minute sample was taken on an appliance, since the time on the appliance is most likely not the same as the time on APN VM 302 which is the NMS time. The rest of the columns in the row also reflect the counts of various metrics important to the object for that minute. The various metrics include packet counters, byte counters, latency measurements, and the like. The count values are values of data gathered during that minute and not a cumulative total since the start of the tracking. The <object type>_hours table is similar to <object type>_minutes except that it aggregates an hour's worth of data from the <object type>_minutes table into a single row. This approach improves the APN VM UI 310 performance in cases where data across a large span of time needs to be aggregated. For example, performance of data transfers can be captured by tracking packets in each direction as they traverse every path of the WAN without using probes or injecting test data. An hour row in the <object type>_hours table represents data from the top of the hour to the end of that hour, for example, from the first <object type>_minutes row whose update_epoch_time_s is after 1:00 to the latest <object type>_minutes row whose update_epoch_time_s is before 2:00. A column in the <object type>_hours row shows how many minute samples are represented in that row. It is legal to have <object type>_hours rows with less than 60 minute samples since for the given hour the system may not have been collecting data for the entire hour, such as an hour interrupted by the data plane app 342 being disabled. For example, if a conduit or path is disabled, or if an entire site is offline, there will be minute intervals that contain no data. Thus, in the hours tables, this lack of data is represented by a count of minutes <60. It is also legal to have <object type>_hours rows with more than 60 minute samples, such as may occur in response to a high availability (HA) switchover to a backup processor device and have an overlap of minutes from both processor devices or as a result of a time change.
The limit_bandwidth_usage file limits how much bandwidth the APN VM 302 will use when communicating with the APNAs. Note that this controls the aggregate bandwidth and is not a per connection setting. The unit used by this file is kbps. The default is 1000 (1 Mbps).
The poll_all_apnas file controls whether all appliances in the APN should be polled for stats.
The stat_poll_interval_min file controls how often the APN VM 302 polls APNAs for stats. The default is five minutes. The unit used by this file is minutes.
The APN VM 302 is configured to regularly request the latest network statistics from each appliance, such as statistics since a previous request was made. Also, if the APN VM 302 is ever down or loses communication with a specific APN appliance (APNA), the APN VM 302 provides a “catching up” function that polls for stored statistics up to and including current operations on the specific APNA. The APN VM 302 maintains a list of APNAs that it is polling for statistics in a managed_appliances table of the MP_Config database file server utility 352 and associated MP_Config database in datastore 304.
Table 1 shows the expected size of the NMS_Stats database for networks of various sizes. A product definition specifies that 1 year of data in 2 TB of storage is requested to be supported. For customer networks in the field, these numbers show that the APN VM system, such as the APN centralized management system 300, can easily support that. For the largest networks, the data may exceed a 1 TB limit.
When the web browser 402 tries to access the uniform resource locator (URL), utility code 404 parses the URL and determines which web page controller 406 should receive the request and what the action should be. A web page controller is considered one of the process application statistic thread instances 406. For example, an http GET request on dashboard/view/1 results in the view action being called on dashboardController with the parameter value of 1. When the selected web page controller receives the request, it responds to the request and calls an appropriate view 4091, 4092, . . . 409M to render a response. Dashboard is a “type”, page one is an “instance”, and it is possible to have multiple dashboards. Typically, handling the request involves dealing with one or more models 4081, 4082, . . . , 408N. These models control access to how data is stored in a database in accordance with the present invention and manipulated. For example, the dashboard/view/1 request would use a dashboard model, which provides access to a data server database which holds the dashboard settings. The dashboard model provides a way to lookup the dashboard information based on a dashboard ID which is 1 in this example. After processing a lookup and setting variables as required to store the results of the lookup, the “view” is rendered by calling a dashboard/view file. In this example, the utility code 404 implicitly renders the page, but, depending on implementation, the utility code 404 could be explicitly called by the page controller 406 to render a view. Such configurable dashboards allows a user to customize their displays, creating views that fit how they perceive their network, and store multiple dashboards in a single database for easy retrieval. For example, geographic oriented dashboards display the performance of individual regions, quality of service dashboards display selected attributes, such as performance, of identified mission critical applications, and carrier specific dashboards may be used to highlight WAN link metrics for easy monitoring of various service providers. Additional capability is provided by tool tips and drill downs that provide quick access to more detailed information.
A web page controller 406 often interacts with several models to deal with the data related to the request and typically selects one view to use and render the requested data. In different APN VM systems, the number of models each web page controller uses and the number of views that a selected web page controller might use can vary between the different APN VM systems. The data flow in the APN VM UI 310 includes significant scripting within the browser. This scripting is used for many reasons including polling of data to provide live updates, improve interactivity, and validate the input. When a view renders a page, it provides a template to the browser JavaScript™ that is necessary for the proper operation of the page.
The adaptive private network (APN) manager 358 of
When statistics polling is enabled for an APNA, the APN manager 358 requests stats from all of the appliances of the APN at the requested poll interval. A stats package is downloaded from each appliance in the network in response to the request to gather statistics and have the stats placed into the NMS_Stats database in datastore 304 under control of the NMS stats utility 354. The stats package and how it is processed is described in further detail below.
The APN manager 358 periodically polls the appliances in the network for information and updates the MP_Config database with the information that is gathered. This provides the APN VM UI 310 with updated information about which software versions are running on the APN, which appliances in a high availability (HA) configuration are active, and which appliance is serving as the active NCN for geo-diverse NCN configurations, for example.
The APN manager 358 uses a virtual file system (VFS) 319 as shown in
To communicate with appliances, the APN manager 358 utilizes a middleware layer of software that provides a framework to build applications using remote procedure calls (RPCs). Such a middleware layer of software may be a software package such as Apache Thrift™, also referred to more simply as Thrift, which is an open-source middleware software hosted by developer Apache Software Foundation. Thrift, or an alternate middleware layer of software of similar capabilities, provides an interface that allows the APN manager 358 to send messages to and get responses from APN appliances (APNAs) with simple function calls. Such a Thrift interface or a Thrift-like interface is specified to support use of a secure sockets layer (SSL) cryptographic protocol as a transport mechanism so that all traffic over the management network 326 is encrypted.
The Thrift interface or Thrift-like interface is defined to support compatibility between the Thrift client and the Thrift server when the interface definition changes. For example, new members of structures and new parameters for methods can be defined without requiring both the client and server to be upgraded nearly simultaneously. This allows the APN software releases to be kept separate from APNA releases.
The APNAs each implement a Thrift server function locally and the appliance agent (AAgent) 338 uses the Thrift interface or Thrift-like interface. Each APNA software release has a file which defines the interface to APNAs running that release. The APNA software releases update the file in such a way that an APN VM release that can talk with an older APNA software revision is also able to talk with the new APNA software. The APN software release, in general, always includes the latest Thrift file or Thrift-like file from the latest APNA software release.
The APN manager 358 creates a virtual file system using FUSE or a VFS software module and specifies a path address to a server where the virtual file system is located. When the APN VM UI 310 needs to request an action from the APN manager 358, the APN VM UI 310 writes a value into the appropriate VFS file. For example, informing the APN manager 358 of the management IP address of the NCN is done by writing the IP address to the VFS file. This approach provides a very simple interface that anything from shell scripts to compiled code can use to interface to the APN manager 358.
To discover the APN, APN VM 302 uses Thrift to connect to the IP address that the user configured as the NCN address and issues the get_network_mgt_ip_addresses function call, which is one of the Thrift or Thrift-like function calls. This function is defined as:
This function provides a way of identifying what an error is if there is no valid login. The function is an example of a call and the data returned in the following structure. This returns a list of ApplianceMgtIPInfo structures. Each structure for an APNA is defined as:
This structure contains the name of the APNA site, the appliance ID, which specifies whether it is a primary NCN or a secondary NCN in a high availability (HA) pair, and the IP address of the APNA site. A secondary NCN is a geographically different client that can become the NCN if the primary dies, for example. The APN can be configured with an HA pair at a primary NCN site and a HA pair at a secondary NCN site. In this case, the secondary NCN site becomes the NCN when the primary NCN site, including both the active and standby HA pair, went down. At the secondary Site, there's also an active and standby appliance. So there are four appliances that may become the active NCN. The NCN receives this information from each APNA site in the APN using an APN Mem shared memory mechanism as described in U.S. Pat. Nos. 8,775,547 and 8,452,846 entitled “Adaptive Private Network Asynchronous Distributed Shared Memory Services”. Specifically, regarding U.S. Pat. No. 8,452,846, see FIGS. 2A, 2B, 3, 4A-4D, 5A, and 5B, and associated description at col. 9, line 60 to col. 21, line 28 and at col. 21, line 57 to col. 24, line 8. Also, regarding high availability networks, see FIG. 6, and associated description at col. 21, lines 29-56. When a discovery 30 minute timer expires, a query is issued to the NCN using this api:
Once the list of appliances is obtained, then stats are retrieved from the list of appliances, since the NCN has the list of management IP which each appliance sends to the NCN in a clients_to_NCN_shared_MEM_table_entry. Users are also allowed to discover manually by activating a button on the APN discovery page if they don't want to wait for the timer.
Each APNA client pushes to a clients_to_NCN_shared_MEM_table_entry structure to the NCN over a connecting conduit. This process may be configured to run at specific intervals, such as once a minute, in support of system operations. This table structure is defined as:
The standby HA appliance also pushes this structure to the active HA appliance. This gives the NCN the management IP addresses of every APNA in the network and the NCN uses this information to fulfill the get_network_mgt_ip_addresses request as used in the discovery process.
After APN VM 302 receives the list of ApplianceMgtIPInfo structures, the APN VM 302 adds this information into a managed_appliances table of the MP_Config database. It then cycles through all of the APNAs and tries to connect to them to get more information through the get_appliance_info Thrift or Thrift-like function call. This function is specified as:
Selected parameters are used to populate columns in the managed_appliances table of the MP_Config database.
APN VM 302 makes use of a relational database management system, using a structured query language, such as MySql, to store configuration and statistics data. Structured query language databases require a strict schema to be defined before storing data. A database schema defines an organization of data that is stored in a particular database, such as storing data in tables governed by a predefined set of rules or constraints. As an example, a relational database schema defines tables, indexes, fields, types, relationships, synonyms, queues, views, packages, functions, procedures, triggers, sequences, materialized views, database links, directories, extensible markup language (XML) schemas, and may also include other items. In APN VM 302, the database specification defines how data is organized for statistics and events, as well as defining APN configurations.
A database schema is created when the APN VM software is initially installed. It is also likely that new installed versions of the APN VM software will require a different database schema to work properly. To prevent customer's from losing data, it is necessary to migrate the existing database using an existing schema into a new schema when the software is upgraded.
The relational database management system allows developers to define a schema that is desired for a given release. Based on the schema definition, code is auto generated in the APN VM system to correctly configure the database on a particular APN VM 302 configuration instance. The auto generated code is designed to transform any database schema into the correct schema without losing data when upgrading the APN VM software. The code is auto-generated at compile time based on the schema, filling in the specified databases, tables and columns with migrated and updated information. Hand written code is not required to migrate the data into the new schema. A database migration does not depend on hard coded version numbers to determine how to migrate a database to the new schema.
One option for updating a configuration and a statistics database begins with archiving the databases when a software install is done. This archival process basically moves those databases to different names. The software install process then creates those databases from scratch. This process technically does not lose data as the data from before the software install is still on disk under the old names. Unfortunately, there is no guarantee that the new software will be able to properly read the archived database because the new software generally does not have a backward compatibility mode to support old schemas. This lack of backward compatibility means that some if not most software upgrades could cause the old data to become unreadable which is really no different than losing data.
A second option takes a different approach by making changes very carefully. There is no autogenerated code to setup these databases. If the databases do not exist when the system is started after new software is installed, then some specially created structured query language (SQL) files are run to create the databases. Generally, hand coded script is created and runs to check for any columns that might be missing and if found adds them. This hand coded script is updated anytime a developer makes a change to the database schema.
Neither of these two options are desirable for the APN VM software system. The first option would cause data loss. The second option requires too much care on the part of developers and is prone to errors due to the hand coded script. It is expected that the APN VM software system and APN VM databases will go through many schema changes, so a different approach is advantageously employed as discussed further below.
Each of the APN VM database schemas is defined in an extensible markup language (XML) file. The root tag is specified as a <db> tag which specifies the database this file defines. This tag has a name attribute which is set to the name of the database. There is also a type attribute that is set to “DB”. Within the <db> tag, there is one or more <class> tags. A <class> tag has a name attribute which is the name of a table to create within this database. Also, within the <class> tag, there is one or more >var> tags. Since a <var> tag defines a column within the table, it supports many attributes. Table 1 lists a number of <var> attributes and Table 2 lists <var> types.
Within a <class> tag, there can be <array> tags that contain <var>, <index>, and <unique key> tags within them. The <array> tags are treated just as if their parent was the <class> tag.
The <class> and <array> tags can have an <on_delete> as a child. This <on_delete> tag has two attributes, name and function. The purpose of on delete is to allow some custom logic related to migrating data values to be inserted into the generated script. This migration of data values is useful in the case where a column is being removed, but some other column in the database needs to be updated based on the old column. The name value defines the column that is being deleted so this is a column that may exist in some older version of the database that does not exist in the current schema. A stylesheet db_gen_pl.xslt includes rules governing how different tags, attributes, listed in Tables 2 and 3, are to be handled. This file when applied on a corresponding XML, file generates a PERL file to implement the process shown in
Custom logic may be added in many other places such as if there is a need to rename a class or do a transformation of the data in an existing column. This type of custom logic extension, for example, is the only time a developer would need to hand-write migration code.
An <index> tag can exist within a <class> or <array> and is used to define an index in the database across multiple columns. The tag must have a name attribute to name the index. Within the <index>, must be multiple <item> tags that have a name attribute which references column names of the table. The database then uses those columns in the order they appear within the <index> as indexes to the table.
The <unique key> tag serves a similar function to the index function and has the same format as >index> with a name attribute and <item> children. The <unique key> tag and the index function are similar in the way they are defined. This causes the database to form a unique key with the specified columns.
Other tags which can appear within the <db> tag are <user> and <insert>, for example. The <user> tag accepts a name, password, and grant attribute. These tags are used to add a user to the database and grant the user access to this database. If the user already exists in the database, then the user is granted permissions to access the database. In this case, the password will not be changed. The password attribute is only used when adding a new user to the database.
The <insert> tag is used to specify that certain rows should be inserted into the database. This insertion is generally done in cases such as using a table as an enum mapping that maps an integer, typically representing some state, to a string that is a human readable version of the state, but the integer can support other use cases. SQL insert commands that correspond to the <insert> are generated after all tables in the database have been fully created.
The <insert> tag requires the table attribute to be set to the name of the table to receive this data. The children of this tag are one or more <field> tags. The <field> must have a name attribute which specifies the name of a column and a value attribute which specifies the value of the column. The <insert> can also have a no_overwrite or noreplace attribute set, in which case neither attribute takes a value. With neither of these attributes set, the row is inserted with a REPLACE INTO SQL command. If no_overwrite is set, then the INSERT INTO SQL command is used. If noreplace is set, the INSERT IGNORE INTO command is used. It is not valid to have both no_overwrite and noreplace set at the same time.
The APN databases are used to store configuration information, status information, and historical statistics and events that have been collected from all the appliances that APN VM 302 manages. These APN databases store data in tables in data partitions of datastore 304 as described in more detail below.
The statistics database and events database are populated by the polling process in the APN VM 302. This polling process communicates with each of the managed appliances in the network requesting blocks of multiple minutes of data which are synchronized and correlated with the APN VM 302 time and stored in the APN VM databases. In addition to APN VM 302 time synchronization, a local APN VM identification (ID) is given to each of the objects, such as sites, conduits, paths, WAN links, and other network elements and the polling process manages the mappings of APN VM IDs to objects and saves this in the APN VM 302 statistics and events database.
A statistics database, supporting the display APN application, stores and maintains statistics data from all managed appliances (APNAs) in the APN. For example, statistics data supporting APN VM displays for reports, graphs, and network maps are accessed from the statistics database.
One year of history for up to N sites, such as up to 256 sites, allows a user to view trends, zoom in and compare events from any time-period during the previous year of data, and provide a histogram of events for a specified time period.
The statistics database is unified with the naming convention identifying configuration information. By including this configuration information into the database, the database supports APN configuration updates without having to archive and restart the statistics database.
The APN VM 302 has a separate configuration and statistics database that links data according to at the object IDs and names. As a result, an APN configuration update can change object names and IDs, resulting in the updates to the statistics database and configuration database. For APN VM 302, the databases adapt across configuration updates and APN VM processes continue with the new data allowing for data visualization across all APN VM time values stored in the databases.
As a result of configuration updates, new sites may be added and old ones removed, and associated conduits, paths, WAN links, and the like are also adjusted according to the change. The databases are designed to support these types of configuration updates by storing the naming and id information persistently for objects in a history storage of the databases, but only storing minute/hour data when the objects actually exist in the network. This approach allows for viewing of objects that existed in the past, but do not exist in the current database, such as a removed or added site for example.
By pre-calculating and storing aggregated hour data based on the minute data that is collected from the appliances allows improved performance of APN VM graphing, network map preparation and display, and updating of reports display screens. For example, when large time ranges are selected for display, having the aggregated hour data already available significantly reduces the time to retrieve and analyze information from the APN VM databases. Also, the events database is an aggregate of all events on every managed appliance in the APNA that APN VM 302 is managing. Various APN VM configurations, user/workspace information, and status information are stored in the APN VM databases.
When a new software version is installed, there may be new tables of statistics or new columns added in existing tables. APN VM 302 retains older data and display reports and graphs on objects and properties that are still valid in a current software release. APN VM 302's retention of the older data prevents loss of data that would normally be thrown away during new software installations.
Both the APN VM statistics and events databases are a combination of all the statistics and events from each of the appliances managed by the APN VM software. Blocks of data may be requested by APN VM 302 from each of the appliances based on time stamps. When these blocks are received, the timestamps are synchronized with the time on APN Vm 302 and stored in the statistics and events databases as appropriate.
Tables of counters are defined for the statistics database. For each of the listed data types, generally three database tables are created. For example, a first table contains the id->name mapping, a second table contains a listing of synchronized <type>_minutes for various objects tracked, and a third table contains a listing of <type>_hours containing the aggregated and synchronized hour data from the managed appliances. In each of the “minutes” and “hours” tables, there are two UINT32 values which are an object_id and an update_epoch_time_s which together form a primary key for these tables. Having a short key provides for fast query performance. Specifics of the data fields include:
WAN Link or conduit usage on ingress or egress. This is dynamic and different from the fair rate seen in the configuration editor. This rate is the actual rate allocated to WAN link or conduit usage. This rate can change based on how much traffic is actually go through the WAN link for each conduit using that WAN link. This is different from the fair rate shown in configuration editor which is a fixed value
An extensible markup language (XML) file is used to define the tables for each type of object and columns (properties of the objects). For example, the minute and hour tables are defined as well as named→ID mapping tables which allow for storing a simple ID in the stats table and use the name for display purposes. Properties are associated with objects and attributes are associated with tags, for example.
The XML format is defined with the following tags:
Additionally, there are optional tags which allow a user to insert data into the database at creation or update time via the .xml file. These optional tags include:
There are two main tables containing event information in the APN VM NMS_Events database. An events table contains all the raw events collected by APN VM 302 from all the managed appliances in the network. A timer queue table contains events that have occurred but haven't been in a state long enough to trigger a time setting. The timer queue table is used to filter out or mark, without removal from the table, spurious events that toggle between states in a time period shorter than a specified time period. This is configurable for different object types including service, conduits, paths, WAN links, dynamic conduits. Events histogram data is also pre-calculated and stored in the NMS_Events database.
An extensible stylesheet language transformations (XSLT) template file is used along with the XML files defined in the previous discussion to generate PERL scripts which are then run during installation and update operations to create or modify the database schemas. The scripts that are generated based on these XSLT templates fulfill an APN VM 302 requirement to support automatic migration and schema updates on software updates.
The APN VM 302 provides a web-based APN VM UI 310 that allows APN configuration packages to be created and modified. This UI provides the ability to create and edit the APN configuration CFG file 474 as well as any network maps stored in the map file 470 that correspond to the stored CFG files. The onboard configuration facility gets packaged so that it can run on an APNA or the APN VM 302.
The compiler 434 takes an APN configuration file and converts it into a representation of the object model that the onboard configuration file editor 448 can work with. The network map editor 458 integrates with the onboard configuration file editor 448 to allow site nodes from the editor to be displayed and positioned on the map.
The onboard configuration file editor 448 leverages the Java™ compiler software to validate a configuration and create output files, including listing file output 440, registry output 442, and XML output 444 that provides representations of the object model.
There are two major components of the UI, the code, such as JavaScript™ code, for example, that runs in the browser, such as browser 328, and the Web backend code 446 that runs on the server, such as the processing node 303. The JavaScript™ code is responsible for handling graph tree navigation and generating requests to the server running the APN VM 302 when a user wants to view or modify items in the configuration. The Web backend code 446 takes requests from the browser, makes necessary updates to the configuration file, and interfaces to the compiler to get the updated object model.
A graphical user interface (GUI) of the onboard configuration file editor 448 displays the network configuration using three tree structures. The trees are labeled sites, connections, provisioning. The sites tree describes basic parameters of a site, such as the Ethernet interfaces, high availability (HA), site name and model, available WAN links, and IP addressing. The connections tree describes how sites are connected to each other with conduits, the rules and classes for the conduits, and whether the sites connect to the Internet and internal intranets. The provisioning tree describes how the services defined in the connections tree are to utilize the bandwidth of the available WAN links. It is in the provisioning tree that users are able to specify which WAN links are assigned to which service and how much of the WAN links the services are entitled to utilize.
The tree structures used by the GUI allows a user to open arbitrary parts of the tree nearly simultaneously. Viewing, adding, deleting, and editing objects is done within the tree itself. This approach allows users to, for example, edit WAN links on one site while viewing the configuration of another site. Branches in the tree structures can contain either additional branches or leaves in the tree. The leaves in the tree each contain an editable section. Editable sections contain parameters that can be edited. For simple items, such as type and model of a site, the editable section is a simple form that contains the parameters that can be edited. For more complex sections, such as rules, the editable section contains more complex widgets like tables, which allow rows to be added, deleted, and edited. When a user opens an editable section of a tree, a JavaScript™ frontend requests, from the UI backend, hypertext markup language (HTML) necessary to render that section.
The object model of the APN configuration has a lot of interrelations between objects. For example, paths depend on the WAN links on both the local and remote sites of the conduit. The relationships mean that a change in one part of the tree may have ramifications to other parts of the tree. The number and variety of interrelation types is addressed by using an architecture that does not require duplicate logic to handle this variability. The architecture allows the user to edit a single section at a time. When the user is finished modifying a section, an apply button is “pushed” to send that change to the server. The server then processes that change by doing a full compile of the configuration that includes this edit to regenerate the object model. The update of the tree in the UI also causes the editable sections to be collapsed. This approach prevents outdated information about the object model from being displayed to the user.
In the case where network objects are renamed, those name changes may need to be propagated to other objects. For example, if a site name changes, then any remote_site_name attributes of add_conduit_service statements that reference the old site name must change. The CFGXML component 454 accepts a rename operation, configured with information concerning which renames necessitate changes in other parameters of the configuration, and regenerates the names of dependent objects.
The compiler 434 has the current network's listing of netobjects in the netobjects facility 438. The netobjects hold the actual objects in the configuration and the relations between them. It also has the per object and network wide audits. For example, if a route with service type as Internet is added, but at a local site the Internet service is not configured, then an audit error would occur. In another example, regarding enabling a dynamic conduit on a client site, but in the network, there is no site enabled as an intermediate site, then the dynamic conduit cannot be created and an audit error also would occur. The compiler 434 has the capability to generate registries from the registry output 442 and the listing file from the listing file output 440. The registry generator in the registry output 442 is used in change management on the APNA. An XML output 444 is also added to the compiler. The XML output 444 allows the configuration file to be processed by the compiler 434 and have the netobjects model output as XML to transfer to the UI as a way to view and process all of the objects that are generated in the configuration.
The onboard configuration facility provides an XML output format. The compiler 434 has supported outputting registries and listing files. This capability is implemented by having individual netobjects output in a relevant format and then walking through all the stored netobjects for the proper output. In a current implementation of the compiler 434, a limited amount of XML output is provided for certain objects so that the APN VM database understands what appliances and models are in the configuration. The APN VM architecture requires the compiler 434 to have an XML output 444 that provides a complete representation of the netobjects. The implementation of the XML output 444, takes into account that netobjects store properties that contain a hash data structure which indexes the properties by name. This allows netobjects to be handled in generic way when processing the XML output.
The management plane (MP) configuration (MP_Config) database in datastore 304 contains a mixture of configuration and run-time status information for the APN VM 302. The following objects are defined and stored in this database:
At step 656, a determination is made whether the map background is acceptable for viewing the selected sites, connections, and provisioning. If the map background is not acceptable for the view task, the process 640 proceeds to step 658. At step 658, the map background is changed. At step 660, a determination is made whether to select sites to be viewed or to auto-populate the map with sites to be viewed. If sites are to be selected for viewing, the process 640 proceeds to step 662. At step 662, sites are dragged and dropped onto the selected map background. If sites are to be auto-populated, the process 640 proceeds from step 660 to step 664. At step 664, the selected map background is auto-populated with an appropriate set of sites. At step 666, the map view with selected sites is saved.
At step 668, a monitor map is selected. At step 668, a monitor map option is selected. At step 670, in the monitor map, a map configuration is selected to view. At step 672, a specific time or a time range is selected to view data for the selected map configuration. At step 674, the current state of the APN, the state of the APN at the selected specific time, or a play option is selected. Tooltips and select operations may be selected to view additional data. At step 676, based on observations, adjustments to map view parameters may be made or displayed results analyzed, or if the map view process is determined to be completed, the process 640 returns to the calling program.
Multiple graph tree attributes are selected to allow the user to see the selected attributes in a graph over time. The graph tree can be viewed to look for trends in data or to compare multiple graph attributes at a time. For example, conduit send/receive bandwidth or loss are graph tree attributes which may be selected. The selection of these two attributes allows the user to analyze the send/receive bandwidth and see if there is a correlation between the loss attribute and the rate attributes when the rate attributes increases.
At step 758, multiple options are presented to view the selected graph tree associated with current settings, to zoom to a sub-range of the selected graph tree, to view the selected graph tree after a reset to an initial time period, to save the selected graph tree settings, or to play the selected graph tree view at multiple time points. If view the current settings is selected, the process 740 proceeds to step 760. At step 760, the current attributes selected are viewed in the graph tree and the process returns to step 758 for further option selection. If the zoom option is selected, the process 740 proceeds to step 762. At step 762, a timeline of attributes are zoomed into a sub-range period to provided a finer view of attribute changes in the selected graph tree for the zoomed in period. The process 740 then returns to step 758 for further option selection. The ability to reset the graph view allows the user to start with a graph looking at a range of times and then zoom into a sub range, and then zoom back out or reset, as described in more detail below, to the original range of time. For example, the user could select to graph a path send/receive bandwidth for all paths that are contained within a given conduit for the month of March. The timeline allows the user to select a finer range such as the Fridays of the month. The user may then want to zoom out or reset the graph view timeline back to viewing the entire month, and then zoom back into a finer range of the first Mondays in the month.
If the reset option is selected, the process 740 proceeds to step 764 and any time period that is currently being viewed is reset back to the initial time period at the beginning of the graph tree view process. The process 740 then returns to step 758 for further option selection. If the save graph tree view is selected, the process 740 proceeds to step 766. At step 766, the graph tree view as currently configured is saved for future viewing. At step 768, the graph tree view is added to a dashboard for ease of selection for future viewing. At step 770, the process 740 returns to the calling program.
The report attributes allow a user to view multiple attributes at a time for a pre-specified object, such as, a conduit or path for a specific moment in time or a range of time. Multiple reports can be viewed over a period of time to look for trends in data or to compare multiple attributes at once. Examples of report attributes include conduit send/receive bandwidth or loss. From these particular attributes, the user could analyze the send/receive bandwidth and see if there is a correlation between the loss attribute and the rate attributes when the rate attributes increase. The report views are similar to graph tree views except in the way the data is visually displayed. Pre-specified objects include sites, services, conduits, paths, WAN links, classes, applications, Ethernet interfaces, and events, including counts of event types.
At step 854, a specific time or a time range is selected to view data for the selected report and selected map configuration. At step 856, the user views the selected report for the current state of the APN, the state of the APN at the selected specific time, or by use of a play option to play the selected report view at multiple time points. Tooltips and select operations may be selected to view additional data.
At step 858, the report view as currently configured is saved for future viewing. At step 860, the report view is added to a dashboard for ease of selection for future viewing. At step 862, the process 840 returns to the calling program.
At step 952, one or more events are selected. At step 954, a specific time or a time range is selected to view event data for the selected map configuration. At step 956, the user views any selected events that have occurred and associated with the current state of the APN, the state of the APN at the selected specific time, or by use of a play option to play the selected events as they may occur at multiple time points. Tooltips and select operations may be selected to view additional data.
At step 958, the event view as currently configured is saved for future viewing. At step 960, the event view is added to a dashboard for ease of selection for future viewing. At step 962, the process 940 returns to the calling program.
While the present invention has been disclosed in the context of various aspects of presently preferred embodiments, it will be recognized that the invention may be suitably applied to other environments consistent with the claims which follow.
This application is a divisional of U.S. patent application Ser. No. 14/973,193, filed on Dec. 17, 2015, which claims the benefit of U.S. Provisional Application Nos. 62/096,049 titled “APN Aware Architecture (Part A)”; 62/096,071 titled “APN Aware Architecture (Part B)” and 62/096,086 titled “APN Aware Architecture (Part C)”, all of which were filed on Dec. 23, 2014 and from U.S. Provisional Application Ser. Nos. 62/132,625 titled “Aware: An Adaptive Private Network Centralized Management System Discovery Process”; 62/132,987 titled “Aware: An Adaptive Private Network Centralized Management System Timestamp Correlation Process”; 62/133,071 titled “Aware: Adaptive Private Network Database Schema Migration and Management Processes” and 62/133,094 titled “Aware: Adaptive Private Network Centralized Management System Data Visualization Process” all of which were filed on Mar. 13, 2015 and from U.S. Provisional Application Ser. No. 62/187,516 titled “Methods and Apparatus for Providing Adaptive Private Network Centralized Management System Time Correlated Playback of Network Traffic” which was filed on Jul. 1, 2015 and all of which are incorporated by reference herein in their entirety. Related implementations of the present inventions have been disclosed in four other copending U.S. patent applications claiming the benefit of the provisional applications cited above and that have the same assignee as the present patent application. The related copending U.S. patent applications are 1) U.S. patent application Ser. No. 14/972,270, filed Dec. 17, 2015 entitled “Methods and Apparatus for Providing Adaptive Private Network Centralized Management System Discovery Processes”; 2) U.S. patent application Ser. No. 14/972,353, filed Dec. 17, 2015 entitled “Methods and Apparatus for Providing Adaptive Private Network Centralized Management System Timestamp Correlation Processes”; 3) U.S. patent application Ser. No. 14/972,514, filed Dec. 17, 2015 entitled “Methods and Apparatus for Providing Adaptive Private Network Database Schema Migration and Management Processes” and 4) U.S. patent application Ser. No. 14/973,343, filed Dec. 17, 2015 entitled “Methods and Apparatus for Providing Adaptive Private Network Centralized Management System Time Correlated Playback of Network Traffic”. The four related copending U.S. patent applications are hereby incorporated by reference in their entirety. The present application is also related to U.S. patent application Ser. No. 14/146,786 filed on Jan. 3, 2014 which is a divisional of U.S. patent application Ser. No. 13/592,460 filed on Aug. 23, 2012 which issued as U.S. Pat. No. 8,644,164 which is a continuation of U.S. patent application Ser. No. 13/353,693 filed on Jan. 19, 2012 which issued as U.S. Pat. No. 8,274,891 which claims the benefit of and priority to U.S. patent application Ser. No. 12/482,766 filed on Jun. 11, 2009 which issued as U.S. Pat. No. 8,125,907 entitled “Flow-Based Adaptive Private Network with Multiple WAN-Paths”, all of which claim the benefit of U.S. Provisional Patent Application No. 61/060,846 entitled “Flow-based Adaptive Private Network with Multiple WAN-Paths” filed Jun. 12, 2008; U.S. patent application Ser. No. 14/291,776 filed on May 30, 2014 which is a continuation of U.S. patent application Ser. No. 13/850,411 filed on Mar. 26, 2013 which issued as U.S. Pat. No. 8,775,547 and which is a continuation of U.S. patent application Ser. No. 13/208,825 filed on Aug. 12, 2011 entitled “Adaptive Private Network Asynchronous Distributed Shared Memory Services” which issued as U.S. Pat. No. 8,452,846, all of which claim the benefit of U.S. Provisional Patent Application Ser. No. 61/372,904 entitled “Adaptive Private Network Asynchronous Distributed Shared Memory Services” filed Aug. 12, 2010; U.S. patent application Ser. No. 13/719,433 filed on Dec. 19, 2012 entitled “An Adaptive Private Network with Geographically Redundant Network Control Nodes”; U.S. patent application Ser. No. 14/019,723 filed on Sep. 6, 2013 entitled “An Adaptive Private Network with Path Maximum Transmission Unit (MTU) Discovery Process”; U.S. patent application Ser. No. 14/481,335 filed on Sep. 9, 2014 entitled “Adaptive Private Network with Dynamic Conduit Process”; all of which have the same assignee as the present application, are related applications, and are hereby incorporated by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3789653 | Brejaud | Feb 1974 | A |
| 5121383 | Golestani | Jun 1992 | A |
| 5185860 | Wu | Feb 1993 | A |
| 5661719 | Townsend et al. | Aug 1997 | A |
| 5710885 | Bondi | Jan 1998 | A |
| 6016307 | Kaplan et al. | Jan 2000 | A |
| 6262996 | Kainulainen et al. | Jul 2001 | B1 |
| 6456594 | Kaplan et al. | Sep 2002 | B1 |
| 6458594 | Baszczynski et al. | Oct 2002 | B1 |
| 6490617 | Hemphill et al. | Dec 2002 | B1 |
| 6611863 | Banginwar | Aug 2003 | B1 |
| 6662221 | Gonda et al. | Dec 2003 | B1 |
| 6665702 | Zisapel et al. | Dec 2003 | B1 |
| 6738582 | Moshe et al. | May 2004 | B1 |
| 6738900 | Hardjono et al. | May 2004 | B1 |
| 6775235 | Datta et al. | Aug 2004 | B2 |
| 6822943 | Mantin | Nov 2004 | B1 |
| 6822944 | Mantin | Nov 2004 | B1 |
| 6894972 | Phaal | May 2005 | B1 |
| 7010607 | Bunton | Mar 2006 | B1 |
| 7036049 | Ali et al. | Apr 2006 | B2 |
| 7349961 | Yamamoto | Mar 2008 | B2 |
| 7373661 | Smith et al. | May 2008 | B2 |
| 7469281 | Kaneda et al. | Dec 2008 | B2 |
| 7542485 | Bingham et al. | Jun 2009 | B2 |
| 7633870 | Elliot | Dec 2009 | B2 |
| 7782787 | Karol et al. | Aug 2010 | B2 |
| 7870246 | Davis et al. | Jan 2011 | B1 |
| 7877477 | Wookey | Jan 2011 | B2 |
| 7886031 | Taylor et al. | Feb 2011 | B1 |
| 7903585 | Feng et al. | Mar 2011 | B2 |
| 7966388 | Pugaczewski et al. | Jun 2011 | B1 |
| 8125907 | Averi et al. | Feb 2012 | B2 |
| 8274891 | Averi et al. | Sep 2012 | B2 |
| 8452846 | Fredette et al. | May 2013 | B2 |
| 8582502 | Conte | Nov 2013 | B2 |
| 8644164 | Averi et al. | Feb 2014 | B2 |
| 8775547 | Fredette et al. | Jul 2014 | B2 |
| 9407733 | Mizrahi | Aug 2016 | B1 |
| 9858060 | Barros et al. | Jan 2018 | B2 |
| 9860245 | Ronda et al. | Jan 2018 | B2 |
| 9929915 | Erickson et al. | Mar 2018 | B2 |
| 10038601 | Becker et al. | Jul 2018 | B1 |
| 10039018 | Splitz | Jul 2018 | B2 |
| 10225146 | Miller et al. | Mar 2019 | B2 |
| 10333808 | Coleman, Jr. et al. | Jun 2019 | B2 |
| 10476765 | Martin et al. | Nov 2019 | B2 |
| 20030115508 | Ali et al. | Jun 2003 | A1 |
| 20030123446 | Muirhead et al. | Jul 2003 | A1 |
| 20050086363 | Ji | Apr 2005 | A1 |
| 20050094567 | Kannan et al. | May 2005 | A1 |
| 20060095554 | Kuhles et al. | May 2006 | A1 |
| 20060271670 | Blomquist et al. | Nov 2006 | A1 |
| 20070230361 | Choudhury | Oct 2007 | A1 |
| 20080225749 | Peng et al. | Sep 2008 | A1 |
| 20090070486 | Visser | Mar 2009 | A1 |
| 20090119630 | Binder et al. | May 2009 | A1 |
| 20090147806 | Brueckheimer | Jun 2009 | A1 |
| 20090257361 | Deshpande et al. | Oct 2009 | A1 |
| 20090310485 | Averi et al. | Dec 2009 | A1 |
| 20110289134 | de Los Reyes et al. | Nov 2011 | A1 |
| 20120042032 | Fredette et al. | Feb 2012 | A1 |
| 20120117273 | Averi et al. | May 2012 | A1 |
| 20120127977 | Copeland et al. | May 2012 | A1 |
| 20120314578 | Averi et al. | Dec 2012 | A1 |
| 20130238743 | Fredette et al. | Sep 2013 | A1 |
| 20140173331 | Martin | Jun 2014 | A1 |
| 20140185445 | Averi et al. | Jul 2014 | A1 |
| 20140207971 | Lecourtier | Jul 2014 | A1 |
| 20140376379 | Fredette et al. | Dec 2014 | A1 |
| 20150071067 | Martin et al. | Mar 2015 | A1 |
| 20160182305 | Martin et al. | Jun 2016 | A1 |
| 20160182319 | Martin et al. | Jun 2016 | A1 |
| 20160182327 | Coleman, Jr. et al. | Jun 2016 | A1 |
| 20180046469 | Johansson et al. | Feb 2018 | A1 |
| 20180123892 | Mellquist | May 2018 | A1 |
| 20190356567 | Martin et al. | Nov 2019 | A1 |
| Entry |
|---|
| Commonly-assigned, co-pending U.S. Appl. No. 16/528,092 for “Methods and Apparatus for Providing Adaptive Private Network Centralized Management System Discovery Processes,” (Unpublished, filed Jul. 31, 2019). |
| Notice of Allowance and Fee(s) Due for U.S. Appl. No. 14/973,193 (dated Feb. 20, 2019). |
| Examiner's Answer for U.S. Appl. No. 14/972,270 (dated Jan. 28, 2019). |
| Notice of Allowance and Fee(s) Due for U.S. Appl. No. 14/973,193 (dated Sep. 19, 2018). |
| Final Office Action for U.S. Appl. No. 14/973,193 (dated Jun. 12, 2018). |
| Final Office Action for U.S. Appl. No. 14/972,270 (dated Mar. 15, 2018). |
| Non-Final Office Action for U.S. Appl. No. 14/972,270 (dated Nov. 27, 2017). |
| Non-Final Office Action for U.S. Appl. No. 14/973,193 (dated Oct. 20, 2017). |
| Restriction and/or Election Requirement for U.S. Appl. No. 14/973,193 (dated Jun. 28, 2017). |
| “Adaptive Private Networking Configuration Editor User's Guide APNware Release 2.5”, Talari Netwoks, pp. 1-75 (2013). |
| “HP Network Node Manager i Software”, for the Windows®, Linux, HP-UX, and Solaris operating systems, Software Version: 9.21, pp. 1-567 (Aug. 2012). |
| Notice of Allowance and Fee(s) Due for U.S. Appl. No. 13/353,693 (dated May 24, 2012). |
| Non-Final Office Action for U.S. Appl. No. 13/353,693 (dated Apr. 2, 2012). |
| Notice of Allowance and Fee(s) Due for U.S. Appl. No. 12/482,766 (dated Oct. 20, 2011). |
| Final Office Action for U.S. Appl. No. 12/482,766 (dated Sep. 28, 2011). |
| Non-Final Office Action for U.S. Appl. No. 12/482,766 (dated Apr. 6, 2011). |
| Srisuresh et al., “IP Network Address Translator (NAT) Terminology and Considerations,” RFC 2663, pp. 1-30 (Aug. 1999). |
| Krasner, Glenn E. and Pope, Stephen T., “A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System”, ParcPlace Systems, 1988. |
| Notice of Allowance and Fee(s) Due for U.S. Appl. No. 14/972,270 (dated Jun. 27, 2019). |
| Applicant-Initiated Interview Summary for U.S. Appl. No. 14/972,270 (dated Apr. 10, 2019). |
| Number | Date | Country | |
|---|---|---|---|
| 20190253325 A1 | Aug 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62096049 | Dec 2014 | US | |
| 62096071 | Dec 2014 | US | |
| 62096086 | Dec 2014 | US | |
| 62132625 | Mar 2015 | US | |
| 62132987 | Mar 2015 | US | |
| 62133071 | Mar 2015 | US | |
| 62133094 | Mar 2015 | US | |
| 62187516 | Jul 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14973193 | Dec 2015 | US |
| Child | 16362307 | US |
