Patent Application
20170374180
The invention disclosed herein generally relates to distributed and highly scaled network device (e.g., a router). In particular, it also relates to using software architecture that adheres to a micro-service architecture to build networking software. The invention assimilates key components into a platform that effectively delivers capabilities to end users. The primary use case for this invention is the control plane on a network device like a router, however the platform can be leveraged from a software perspective for applications like network telemetry, high performance networking in gaming and high frequency financial trading.
In current generation of network devices, the software architecture is a tightly coupled monolithic having the following functionalities:
The components can be arranged in a pipeline such that a module's output (e.g., A) will be an input for others (e.g., B). In other words, B will start operating/functioning only when the Module A is done. This works perfectly in a small scale environment. But, in case of scaled scenario, this kind of pipelining may become a bottleneck for the entire system. Few of the challenges of monolithic systems are:
What is needed in the art are methods and systems that would overcome the limitations of existing monolithic systems for network devices.
In one aspect, provided herein is a software architecture for enabling a distributed and high scalable network device (e.g. router). The software architecture adheres to the micro-service architecture to build networking software. The invention assimilates varying concepts from the realm of distributed computing to build on a new set of design patterns that are the basis for a new generation of routing control plane software.
Since the capabilities described in this patent are assimilated into a platform, the building blocks can be leveraged in realm of high performance networking for varying control plane applications in the networking domain like telemetry, traffic engineering or foundation for high frequency trading.
The key building block for this architecture is a Brick Daemon (BD), a generic block analogous to a stem cell in the body. A BD provides a basic run time environment block for any application in the system. A BD can be transformed into any kind of application by defining it's personality through a JSON configuration file when it is started. A BD can be transformed into a routing protocol application or some other user defined application
Some of the capabilities of such a software are based on a data store that stores state of one or more routing apparatus.
In some embodiments, the data store retains information in a binary format and the same format is used to represent the object in memory, Inter-Process Communication and a Data store. All the applications that run on this platform including the components that read state from on the wire convert information into this format.
In some embodiments, the system allows for horizontal scaling of processing via sharding based on ranges, prefixes and hash thereby allowing distribution of processing load into smaller granular units working in parallel, thereby increasing resiliency in the system.
In some embodiments, the modules in the system allow plug-n-play driven by underlying JSON configuration, this allows easy addition and removal of features in a modular fashion.
In some embodiments, the architecture also introduces a new paradigm for recovery from managed migration or re-start known as Anti-Fragile Restart (AFR). This then can become the basis to migrate a working unit from one compute processor to another allowing Live BD migration with minimum down time.
Other characteristics of this system, for example, include a recursive architecture which uses routing protocols inside the system to discover end points on other processors running the same platform. All the processing nodes assume no locality of reference thereby getting away from lock and synchronization and allowing exploitation of parallelism in processing on different CPU cores.
Scaling numbers associated with this architecture are documented and based on the initial indication, the measured performance on some parameters highlighted herein is order of magnitude better than similar known systems.
In one aspect, disclosed herein is a distributed, modular and highly available network routing apparatus that is enabled to perform IP, MPLS VPN v4 and v6 services. The network routing apparatus comprises a plurality of functional modules that is selected from the group consisting of a processor, a data storage module, an input/output module, a shared memory, and a network module. In some embodiments, the network routing apparatus is run by an operating system comprising a system space and a user space.
In some embodiments, each functional module in the plurality functions modules is driven by a micro-services based software architecture comprising one selected from the group consisting of a TCP/IP stack, a protocol serializer, a protocol de-serializer, an in-memory data store, and one or more utility applications.
In some embodiments, the software architecture is at least partially retained in the user space of the operating system of the network routing apparatus.
In some embodiments, the software architecture is completely retained in the user space of the operating system of the network routing apparatus.
In some embodiments, the in-memory data store stores one or more state of the network routing apparatus. In some embodiments, the in-memory data store stores one or more state of the software architecture and network routing apparatus.
In some embodiments, each of the one or more state comprises an in-memory representation of all data organized in semi structured data tables.
In some embodiments, the one or more utility applications generate a control plane of the network routing apparatus.
In some embodiments, information in and out of the network routing apparatus is organized into a common format accessible by each of the one or more utility applications. In some embodiments, the common format is Brick Data Store Format (BDSF).
In some embodiments, a protocol serializer serializes objects from the user space as tables that are understood by BDSF.
In some embodiments, the one or more utility applications comprise an input/output daemon (IOd), a forwarding daemon (FWDd), an application daemon (APPd), or combinations thereof. In some embodiments, the software architecture allows an Anti-Fragile Restart (AFR), which comprises an instant re-start of an APPd based on a disk snapshot of a previous incarnation of a system state thereby allowing seamless restart from the previous incarnation.
In some embodiments, sharding of the software architecture is based on i) a set of neighbors of the routing apparatus within a network to create shard per neighbor, or ii) prefixes learnt from the network. In some embodiments, the sharding is based on range, hash, or lookup based on number of processors.
In some embodiments, the software architecture comprises a JSON configuration file that allows plug-n-play of features. In some embodiments, a JSON configuration file is used to specify the function of an application in the plurality of applications.
In some embodiments, the one or more utility applications allow division and separation of information in the form of prefixes and BGP address families, thereby allowing BGP protocol in the routing control plane to scale by diving processing in smaller atomic units.
In some embodiments, a planned restart is based on a stored state of the one or more utility applications, wherein the stored state is retained by persisting all tables of each of the one or more utility applications to a non-volatile memory. In some embodiments, the stored state of the one or more utility applications can be accessed by a different processor, and wherein the different processor re-populates tables using the stored state of the one or more utility applications.
In some embodiments, an unplanned restart is based on a state of the one or more utility applications that was saved in a non-volatile memory of the network routing apparatus and sequence numbers associated with such state. In some embodiments, a state for unplanned restart is created periodically as a system backup point. In some embodiments, the system backup point is saved in a non-volatile memory of the network routing apparatus.
In some embodiments, persisted state and sequence numbers are used to perform a complete recovery. In some embodiments, the one or more utility applications allow distributed sequence block management without rolling over of sequence numbers thereby allowing restart without collision post restart.
In some embodiments, the one or more utility applications allow manipulation of object attributes without assumption on the locality of reference, thereby allowing each of one or more utility applications to create tables, add objects and subscribe for remote updates without concern for locality of the tables.
In some embodiments, the one or more utility applications allow a recursive architecture where an instance of routing protocol (e.g., IS-IS) used for discovering nodes in a network is also used for discovery of service endpoints and state synchronization.
In some embodiments, the one or more utility applications allow an Anti-Fragile Restart (AFR) along with localization assumption, thereby allowing state associated with the one or more utility applications to migrate from a first node to a second node (e.g., VMs or chassis within a router or between routers).
In another aspect, provided herein is a distributed network routing system comprising a network including a plurality of network routing apparatus distributed across the network at network access points. In some embodiments, each network routing apparatus with the system comprises a plurality of functional modules selected from the group consisting of a processor, a data storage module, an input/output module, a shared memory, and a network module.
In some embodiments, each network routing apparatus is run by an operating system comprising a system space and a user space.
In some embodiments, each functional module in the plurality functions modules is driven by a micro-services based software architecture comprising one selected from the group consisting of a TCP/IP stack, a protocol serializer, a protocol de-serializer, an in-memory data store, and one or more utility applications.
In some embodiments, the software architecture is at least partially retained in the user space of the operating system of the network routing apparatus.
In one aspect, disclosed herein is a method of providing microservices-based network services in a network comprising a plurality of interconnected network routing apparatus. Here, each network routing apparatus comprises a plurality of functional modules selected from the group consisting of a processor, a data storage module, an input/output module, a shared memory, and a network module. In some embodiments, each network routing apparatus is run by an operating system comprising a system space and a user space. In some embodiments, the network services are driven by a software architecture comprising a TCP/IP stack, a publish-subscribe bus, a protocol serializer, a protocol de-serializer, an in-memory data store and one or more utility applications. In particular, the method of providing microservices-based network services comprises the steps of: initializing each network routing apparatus, creating, via one or more forwarding utility applications in the one or more utility applications, one or more tables for forwarding information from an incoming control packet to one or more input/output utility applications in the one or more utility applications, thereby sharding the information from the incoming control packet based on one or more neighboring network routing apparatus; and creating, via each of the one or more input/output utility applications, one or more new tables for forwarding information from the incoming control packet to one or more route computation utility applications in the one or more utility applications, thereby further sharding the information from the incoming control packet.
In some embodiments, initializing each network routing apparatus comprises: reading configuration stored in JSON format from the data storage module and instantiates the one or more utility applications to re-creating run time state from tables persisted in the data storage module.
In some embodiments, the sharding and further sharding is based on a mechanism selected from the group consisting of range, prefixes learned from the network, a hash function, and lookup based on number of processors. In some embodiments, the hash function is mapped to internet protocols corresponding to one or more neighboring network routing apparatus. In some embodiments, the prefixes learned from the network are spread across different route computation utility applications via prefix sharding.
In some embodiments, the method further comprises: computing, via one or more route computation utility applications, the best route for downloading a specific prefix, thereby generating a publish-subscribe update that is processed by one or more the forwarding utility applications.
In some embodiments, the one or more forwarding utility applications install the best route and starts relaying data traffic as a router.
In some embodiments, sharding is bi directional. For example, sharding can also be applied from the input/output utility applications to download a fraction of routes to various forwarding utility applications.
In one aspect, provided herein is a computer program product comprising a computer-readable medium having computer program logic recorded thereon arranged to put into effect the method of an embodiment disclosed herein.
It would be understood by one of skill in the art that any embodiments disclosed herein can be used, without any limitation, in connection with any aspect of the invention when applicable.
Those of skill in the art will understand that the drawings, described below, are for illustrative purposes only. The drawings are not intended to limit the scope of the present teachings in any way.
Unless otherwise noted, terms are to be understood according to conventional usage by those of ordinary skill in the relevant art.
The major components of the system include: Forwarding Daemon: FWDd that enables data forwarding in the platform, APPd: Any application that runs on the platform, ETCd: An application on the platform that is a registry for all the tables across APPds and IOd: Input Output daemon that performs pre-processing and session management for control plane protocols that run on the platform.
A few features have been implemented to build this platform, including, for example, identifying design patterns that enable building an abstraction called brick stem cell, which is a key building blocks for all IOds, APPds and FWDds on the platform. The building block provides an on board and off board Inter Process Communication bus and a memory format to relay messages across the system. Underlying data model on JSON, e.g., semi structured data organized in tables, is used. This is applying concepts of relational algebra/database to protocol developments by storing protocol information in tables and objects. The stem cell abstraction ensures that protocol code foot print is a fraction of the code to deliver similar capabilities in other routing architectures. Lesser code for same functionality implies lesser bugs and fewer test cases and yet broad coverage. Publish subscribe bus for inter-process communication to carry object information across the system and assume no locality of reference in information processing. This allows state generated to be used in computation without locks or synchronization across processing units. Decision to allow memory usage to expand since objects without locality of reference imply that tables are replicated multiple times. Given the falling price per unit of memory we decided to increase memory usage using it as a currency to give higher return in processing.
As disclosed herein, an architecture is applied for building a distributed network device including but not limited to an Internet scale router.
The proposed software architecture adheres to the micro-service architectural style where each module is designed and developed as a single application block which runs on its own and interacts with other application blocks through well-defined interfaces and a publish-subscribe (Pub-Sub) receiver driven distribution channel. Following are some of the attributes of the resulting system.
The back end of the system is a distributed data store, this is semi-structured data model created in memory from JSON representation. Similar format examples include the structured SQL, unstructured NoSQL and XML. Software functionality is built around with components that encapsulate the logic store for the control plane running on the platform. Each of these components implement a set of narrowly, related functions.
The state of the system is represented by tables and objects. Maintenance of these tables and objects is taken care by the underlying software infrastructure. This enables application developer to concentrate on the application logic and corresponding algorithms. This in turn reduces the programming complexity, time to completion and number of lines of code.
Each component is developed and deployed independently of one another. This makes new features to be developed, tested and deployed faster. If a problem does occur, it can be isolated quickly to an individual component, making fast rollback and hot fixes easy to achieve. This enables plug and play environment.
Scaling is achieved through scale-out concept, where components are scaled horizontally. An important part of this solution is sharding that allows processing by dividing incoming prefixes/protocol peers across multiple instances there by enabling horizontal scaling.
The following table summarizes various exemplary architectural blocks that can be used to build a micro-services architecture, their unique characteristics and attributes.
The system architecture is based on distributed system of small software components called Bricks. Each individual component is designed to play a small role in an overall big picture. Each of these bricks or components are implemented as a standard UNIX daemon on a contemporary Linux operating systems called Brick Daemon (BD). It will be understood that the system architecture can be implemented in other operating systems, including but not limited to, for example, Unix OS, Mac OS and etc. Unlike traditional networking devices, the information (state) is stored as objects. These objects are stored in the back stores called Brick Data Store (BDS) in Brick Data Store Format (BDSF).
Each of the daemon/components own a set of tables. They advertise this information to other components in the eco system via a Pub-Sub. The components express the interest in specific tables owned by other components, which are either onboard or off-board. Applications develop functionality implemented around the data while Pub-Sub plays the role of distributing the states (Objects) across the system via IPC's or typical UDP Packet hiding the transport from the application.
Advantageously, distribution of processing enables high survivability in face of failure. Subsequently, fast restart allows quick re-creation of state using snap shot of objects. This substitutes fancy concepts like ISSU/NSR with a simple and deterministic solution.
Some of the components that enable the above architecture are the following:
Brick Daemon
A Brick Daemon (BD) as disclosed herein is a generic building block in the system analogous to a stem cell in the body. BD provides basic runtime environment block for any application in the system. BD can be transformed into any kind of application by defining it's personality through a JSON configuration file when it is started. A BD can be transformed into a routing protocol application or some other user defined application The JSON configuration file can be changed post startup and this will entail a daemon restart to get the new personality.
Most networking applications are monoliths composed of I/O component, housekeeping functionality for liveness enablement, protocol state machinery and processing algorithms. These capabilities are mapped to, for example, a component in an RtBrick system known as IOd or an input-output daemon.
The application logic that computes best routes, and best paths through a network or any other user defined telemetry data is known as APPd or an application daemon. In a networking system, an application that enables traffic forwarding between separate instances and different systems is known as the forwarding daemon and this is built as FWDd in RtBrick system.
All three components (IOd, APPd, FWDd) need to discover each other using a service discovery daemon called ETCd. ETCd is an internal daemon that is a registry for publishers and subscribers of tables and object in the system. To some extent, an ETCd functions as an application layer registry. ETCd provides transport-IP address information and port-numbers, such that off-board components can exchange BDS state. ETCd is a center piece for bootstrapping the entire system.
Thus, a BD can manifest itself into broadly one of the three components under the following categories:
Initialization Sequence
ETCd helps bootstrap components on a local system start-up. Boot strapping involves table advertisements, sequence number block management and systematic handshaking between the various components to have a structured bootup. ETCd also plays a role in Anti-Fragile Restart (AFR) explained in later section.
Bootstrapping of a BD
Each BD spawned advertises the tables it owns to ETCd. This enables ETCd to have a system table view importing information from various BDs. This information is advertised back to all other BD's. Each ETCd communicates with other node's ETCd and exchange its table information. This enables each ETCd to complete view of the system.
Sequence Number Management
Sequence number is a monotonically increasing number for a given table. This ensures synchronization of tables in the system. ETCd helps recover sequence number block for a table in a BD post restart.
Input Output Daemon (IOd)
This component is mainly designed to do input/output processing of the packets. A system might have more than a single IOd for scaling purpose. Each of these IOd subscribes to protocol specific packet tables with FWDD (which is designed for sending/Receiving of RAW packet on the wire). Essentially, IOd is a runtime programmable message parser, supporting several input and output protocol formats along with an input and output protocol caching machinery. It is responsible for:
Forwarding Daemon (FWDd)
The forwarding daemon us a user-space, software based forwarder. It receives its inbound packets directly from the NIC or Chipset using direct DMA technologies like e.g. DPDK/netmap. The lookup pipeline is fully programmable, such that rapid prototypes for new data-plane functionality can be done easily. FWDd does not program its full forwarding table down to the hardware chipset. Rather it is the bottom of a forwarding cache hierarchy, which downloads only “HOT” forwarding entries.
APPds upon their route calculations download their best routes to FWDd. Each of these routes in turn are the rib-entry objects. FWDd, translates these rib-entry objects to local OS representations to tables, objects and next hop. This is the end point of objects where they are converted and sent to driver for programming. FWDd also serves as a next hop resolver for Border Gateway Protocol (BGP) APPd.
Application Daemon (APPd)
APPd is the home for protocols application logic. For example, BGP's Filtering, BGP's Best route selection, ISIS's SPF, route selection, and etc. happen in APPd. In general, APPd operate table-centric.
RtBrick In Action
In this section, we explain the end-to-end flow of data within the system in the current architecture using an example of BGP application. Assumption: BGP is enabled only for default Virtual Routing and Forwarding (VRF) instance with IPv4 Unicast address family. The design (e.g., in
We can clearly distinguish entire functionality of BGPd into three major portions:
Applying the micro-service design as shown in
It should be noted that there is no rule on how to divide an application in this micro service architecture. All these can be combined in one single process or multiple process.
Bootup Sequence
The boot-up sequence is divided in to 2 phases. In the first phase as shown in the sequence diagram in
Application Modularity and Scale Out
In a large monolithic architecture (a whole) all components have to be uniformly scaled up. A small part of overall system is constrained in performance, but if that behavior is locked up in a giant monolithic application, all elements of scaling everything. This implies we have to provide more computing resources to scale up.
In micro service architecture, application's module is divided into granular components (a.k.a. BD). BD's acting as sum of parts interact to provide overall functionality. These discrete elements can be independently scaled thereby allowing BDs to execute on different hardware. More instances of the BD can be spawned so that work load can be evenly distributed across these BD's, in effect a scale out.
The above concept is explained taking BGP as an example. A single BGP BD can be run to achieve the functionality (Which is same as monolithic). But for better scaling and performance, we can:
It should be noted that there is no restriction on how BGP is modularized. It can be based on the running environment.
Sharding
Scale out does not stop just at BGP address family level. It can be at a granularity of a prefix level also. In the sense, a large number of BD's can be spawned just to service range of object's key space.
Each of these BD's are logically separated based on the object's key space which is prefix. For e.g., if there are 4 different BD's spawned to service the IPv4/Unicast address family, then each of the BD's will be serving for prefixes in the range of [1.0.0.0-55.0.0.0], [56.0.0.0-110.0.0.0]-[111.0.0.0-165.0.0.0] & [166.0.0.0-220.0.0.0]. The range defined can be derived from user defined has function.
Sharding is not restricted just to prefix but rather to any object attribute.
This structure enables the following:
Coupling both of them there is capability to perform both sharing for load sharing, horizontal scaling and redundancy by building small replaceable units of APPds.
There are several sharding modes available, including, for example, i) Split, and ii) Split and replicate.
Split
This mode is enables when sharding is done based on a hash key based on the number of processing units defined by the user.
Split and Replicate Mode
This mode is enabled where sharding done over set of local BD's and a copy is replicated on the standby nodes. Enabling this mode makes the application to support NSR.
Localization Assumption
Each BD depending on their role, create their own tables and add data to those tables as objects. They subscribe to specific set of tables to get their functionality. These set of tables will be owned by other BD's in the system. These BD's might be located in same node or altogether in different node. This information is totally abstracted from BD and it is handled by underlay infra (Pub-Sub). BD's will act on a table content; it does not make any assumption on how it gets it. Underlay infra will be responsible to get the data required for a given BD.
A BD might reboot can occur owing to multiple causes ranging from a software bug or any kind of failure. Resilience is an attribute of a system that enables it to deal with the failure in a way that doesn't cause the entire system to fail. A complex system like the proposed architecture can be constantly undergoing varying degrees of failure. Resilience is how it can recover or be insulated from failure. Restart can be either planned or unplanned restart and the ability to instantly recover from partial failures is called AFR.
Table 3 below shows a key building block for AFR. The uniform memory format a.k.a. BDSF allows information interchange across file system, IPC and In Memory Database.
Commonality of BDSF allows periodic snapshot of information prior to In-Service-Software Upgrade (ISSU) from In-Memory Database to file system. This permits constant back up of state information from the database to file.
Interactions between IPC and In-Memory data base leading to information interchange are benefited to a large extent by not requiring the expensive serialization and de-serialization of information rather using binary format and embedded information to access various attributes of data/information in the object.
A key capability that can be availed on account of this is the Anti-Fragile re-start which allows in-memory data base to be instantly re-created from the file system and state replayed to re-start the application within matter of milliseconds.
Decentralizing responsibility for data across micro services requires strong information sharing and no assumption on locality of reference while having loosely coupled interactions. Another key aspect of the de-coupling is that the software parts can perform as individual redundant units that can be re-started instantly by replaying the state in binary format in which it is retained.
Unplanned Restart
BD crashes and reboots. Table 4 below depicts an exemplary sequence of unexpected BD reboot steps.
Planned Restart
In case of a planned restart (e.g., it can be for ISSU upgrade, maintenance, etc.), we can take a complete view of the BD by dumping all its tables to a non-volatile memory. This is called taking a snapshot. We can again spawn the new BD reading back the dumped file, which will resume where it was at the time of the dumping.
This feature is not enabled by default. If enabled, snapshot of the system is taken frequently. Snapshot of the system can be taken by issuing the “snapshot” command also.
Every state is represented as an object. TCP/IP stack is also part of the infra. Each and every TCP/UDP block are viewed as objects. Snapshotting includes even dumping these TCP/UDP block into volatile memory and reincarnating those on new BD. This is also one of the way of achieving NSR. A sequence involving snapshotting is shown in
BD Motion
One of the immediate advantage of snapshot feature is a Live BD migration. Live migration of a BD from one node to another is enabled in the three steps:
Fault Domains
If a BD reboots, it will be impacting only a small portion in the overall functionality. In this architecture set of BD's serving each customer can be segregated. This helps to restrict propagation of false routing. This concept as shown in
Vector Clock and Archival Server
To ensure temporal consistency in the system the micro-services based router can use the well-known concept of vector clock for interaction between components that derive their clock from different clock sources especially for off-board interactions. The vector clock schema allows for synchronization and causality of events in object table access and consistent processing allowing various different publishers of state publishers updating a table to get into synch during the course of operation.
In addition to the above the router can send object state generated by the routers with time stamps to an off-line server thereby allowing state to be persisted via frequent updates. This information allows for easy rewind and replay of events on the system at a given time shot to allow for triaging and debugging issues besides plotting information for both operational and strategic analysis.
Characteristics of System
System and method disclosed herein offer high availability. Any system can and will fail at some point. High availability can achieved by letting the system fail but fix it efficiently. The following table 4 summarizes features of a typical system and of the RtBrick system.
The system and method disclosed herein offer numerous advantages. For example, the user level TCP/IP stack allows all complete state machine and data to be persisted. In some embodiments, the current system allows TCP/IP state to be persisted and spliced on processor cores that are located on servers. Applications are in user space completely all the way to Layer 3 (e.g., the network layer) without retaining state in the kernel.
In some embodiments, objects are serialized from user space as objects and tables that can be understood by BDSF format. Objects can also be de-serialized on restoration from persistent store and restored to BDSF format/utilities.
In some embodiments, the current method and system allow easy migration of application objects is easier since the format does not assume any locality of memory which allows seamless migration to processor cores that are located anywhere.
In some embodiments, in the current system, an active/active state does not require periodic sharing of information between applications on 2 cores rather the information snapshots can be taken periodically via configuration or a job and used to re-create state.
In some embodiments, information is organized in BDSF and applications use this as a building block. State re-creation latency is relatively low and allows close to seamless migration.
The method disclosed herein can be implemented as a computer system and/or a computer program product that comprises a computer program mechanism embedded in a computer readable storage medium. Further, any of the methods of the present invention can be implemented in one or more computers or computer systems. Further still, any of the methods of the present invention can be implemented in one or more computer program products. Some embodiments of the present invention provide a computer system or a computer program product that encodes or has instructions for performing any or all of the methods disclosed herein. Such methods/instructions can be stored on a CD-ROM, DVD, magnetic disk storage product, or any other computer readable data or program storage product. Such methods can also be embedded in permanent storage, such as ROM, one or more programmable chips, or one or more application specific integrated circuits (ASICs). Such permanent storage can be localized in a server, 802.11 access point, 802.11 wireless bridge/station, repeater, router, mobile phone, or other electronic devices. Such methods encoded in the computer program product can also be distributed electronically, via the Internet or otherwise, by transmission of a computer data signal (in which the software modules are embedded) either digitally or on a carrier wave.
Some embodiments of the present invention provide a computer system or a computer program product that contains any or all of the program modules as disclosed herein. These program modules can be stored on a CD-ROM, DVD, magnetic disk storage product, or any other computer readable data or program storage product. The program modules can also be embedded in permanent storage, such as ROM, one or more programmable chips, or one or more application specific integrated circuits (ASICs). Such permanent storage can be localized in a server, 802.11 access point, 802.11 wireless bridge/station, repeater, router, mobile phone, or other electronic devices. The software modules in the computer program product can also be distributed electronically, via the Internet or otherwise, by transmission of a computer data signal (in which the software modules are embedded) either digitally or on a carrier wave.
Having described the invention in detail, it will be apparent that modifications, variations, and equivalent embodiments are possible without departing the scope of the invention defined in the appended claims. Furthermore, it should be appreciated that all examples in the present disclosure are provided as non-limiting examples.
The following non-limiting examples are provided to further illustrate embodiments of the invention disclosed herein. It should be appreciated by those of skill in the art that the techniques disclosed in the examples that follow represent approaches that have been found to function well in the practice of the invention, and thus can be considered to constitute examples of modes for its practice. However, those of skill in the art should, in light of the present disclosure, appreciate that many changes can be made in the specific embodiments that are disclosed and still obtain a like or similar result without departing from the spirit and scope of the invention.
The scaling number mentioned below are the expected scale numbers of the system based on this architecture.
The following table illustrates the recovery efficiency under the system as disclosed herein.
The various methods and techniques described above provide a number of ways to carry out the invention. Of course, it is to be understood that not necessarily all objectives or advantages described may be achieved in accordance with any particular embodiment described herein. Thus, for example, those skilled in the art will recognize that the methods can be performed in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other objectives or advantages as may be taught or suggested herein. A variety of advantageous and disadvantageous alternatives are mentioned herein. It is to be understood that some preferred embodiments specifically include one, another, or several advantageous features, while others specifically exclude one, another, or several disadvantageous features, while still others specifically mitigate a present disadvantageous feature by inclusion of one, another, or several advantageous features.
Furthermore, the skilled artisan will recognize the applicability of various features from different embodiments. Similarly, the various elements, features and steps discussed above, as well as other known equivalents for each such element, feature or step, can be mixed and matched by one of ordinary skill in this art to perform methods in accordance with principles described herein. Among the various elements, features, and steps some will be specifically included and others specifically excluded in diverse embodiments.
Although the invention has been disclosed in the context of certain embodiments and examples, it will be understood by those skilled in the art that the embodiments of the invention extend beyond the specifically disclosed embodiments to other alternative embodiments and/or uses and modifications and equivalents thereof.
Many variations and alternative elements have been disclosed in embodiments of the present invention. Still further variations and alternate elements will be apparent to one of skill in the art.
In some embodiments, the numbers expressing quantities of ingredients, properties such as molecular weight, reaction conditions, and so forth, used to describe and claim certain embodiments of the invention are to be understood as being modified in some instances by the term “about.” Accordingly, in some embodiments, the numerical parameters set forth in the written description and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by a particular embodiment. In some embodiments, the numerical parameters should be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of some embodiments of the invention are approximations, the numerical values set forth in the specific examples are reported as precisely as practicable. The numerical values presented in some embodiments of the invention may contain certain errors necessarily resulting from the standard deviation found in their respective testing measurements.
The application claims priority to U.S. Provisional Application No. 62/353,486, filed on Jun. 22, 2016 and entitled “METHOD TO BUILD A MODULAR DISTRIBUTED NETWORK DEVICE USING MICROSERVICES ARCHITECTURE,” which is hereby incorporated by reference herein in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62353486 | Jun 2016 | US |
