Patent Application
20170093616
The present disclosure generally relates to the field of firmware upgrading. More particularly, and not by way of any limitation, the present disclosure is directed to a method and apparatus for providing in-service firmware upgradability in a piece of equipment, e.g., a network element.
Use of programmable devices in various applications, including network router applications, has been steadily increasing due to a number of benefits such as dedicated performance, quick time-to-market and prototyping, reprogrammability, low NRE (nonrecurring engineering) cost, etc. For example, Field-Programmable Gate Arrays (FPGAs) have become particularly ubiquitous in implementations where they can be useful for off-loading processor-intensive applications that a CPU host may not be optimized in its design to perform.
One desirable feature of a reprogrammable device is that its firmware may be re-downloaded and upgraded as needed. However, in a typical upgrade scenario, the device is powered down or taken off-line, which can result in unacceptable levels of downtime and concomitant disruption of service.
The present patent disclosure is broadly directed to a system, apparatus and method for providing in-service firmware upgradability in a network element having a programmable device configured to support a plurality of application service engines or instances. A static core infrastructure portion of the programmable device is architected in a multi-layered functionality for effectuating an internal packet redirection scheme for packets intended for service processing by a particular application service engine that is being upgraded, whereby the remaining application service engines continue to provide service functionality without interruption.
In one aspect, an embodiment of a programmable device adapted to perform an application service is disclosed. The claimed embodiment comprises, inter alia, an aggregation layer component configured to distribute ingress packets received from a host device to a plurality of crossbar distributors forming a crossbar layer component of the programmable device. An admission layer component is operably coupled between a plurality of application service engines and the crossbar layer component for facilitating transfer of ingress packets and processed egress packets, wherein each crossbar distributor may be configured by the host device in either a default mode or a redirect mode of operation. When configured to operate in default mode, a crossbar distributor forwards or bridges the ingress packets to a specific corresponding application service engine for processing. On the other hand, if a particular crossbar distributor is configured to operate in a redirect mode, it is adapted to distribute received ingress packets to a subset of the plurality of the application service engines excluding the specific application service engine corresponding to the particular crossbar distributor, which specific application service engine may be undergoing a reconfiguration or upgrading process.
In another aspect, an embodiment of a method operating at a network element configured to support in-service application upgradability is disclosed. The claimed method comprises, inter alia, receiving, at a first-level ingress distributor of a programmable device of the network element, ingress packets from a host component coupled to the programmable device, each ingress packet having a first-level distribution tag, a second-level distribution tag and a host identifier configured by the host component, wherein the programmable device comprises a dynamic component including a plurality of application service engines, each configured to execute an instance of an application service with respect to the ingress packets. Responsive to the first-level distribution tag, an ingress packet may be forwarded by the first-level ingress distributor to a specific one of a plurality of second-level ingress distributors, each corresponding to a particular application service engine of the plurality of application service engines. A determination may be made if a particular second-level ingress distributor is in a default mode or in a redirect mode, wherein the redirect mode corresponds to a condition in which an application service engine associated with the particular second-level ingress distributor is in a state of unavailability and the default mode corresponds to a condition in which the application service engine corresponding to the particular second-level ingress distributor is in an active state. If the particular second-level ingress distributor is in default mode, the ingress packets are forwarded to the particular application service engine associated with or corresponding to the particular second-level ingress distributor for processing. Otherwise, if the particular second-level ingress distributor is in redirect mode, the ingress packets are distributed to remaining active application service engines for processing, responsive to the second-level distribution tags of the ingress packets. In one example implementation, the first-level distribution and the second-level distribution tags each comprise N-bit random numbers provided by the host component, which tags may be used for indexing into respective Look-Up Tables (LUTs) for determining where the ingress packets should be forwarded or redirected.
In another aspect, an embodiment of a network element is disclosed which comprises, inter alia, one or more processors and a programmable device supporting a plurality of application service engines configured to execute an application service, wherein the programmable device comprises a layered packet distribution mechanism that includes an aggregation layer component for distributing ingress packets to a crossbar layer component configured to selectively bypass a particular application service engine and redirect the ingress packets to remaining application service engines. A persistent memory module coupled to the one or more processors and having program instructions may be included for configuring the aggregation layer and crossbar layer components under suitable host control in order to effectuate in-service firmware upgradability of the programmable device.
In a still further aspect, an embodiment of a non-transitory, tangible computer-readable medium containing instructions stored thereon is disclosed for performing one or more embodiments of the methods set forth herein. In one variation, an embodiment of a network element having in-service firmware upgrade capability may be operative in a service network that is architected as a Software Defined Network (SDN). In another variation, the service network may embody non-SDN architectures. In still further variations, the service network may comprise a network having service functions or nodes that may be at least partially virtualized.
Benefits of the present invention include, but not limited to, providing non-stop application service functionality in a network element even during an upgrade of service firmware embodied in one or more programmable devices of the network element. The multi-layered core infrastructure of a programmable device according to an embodiment herein advantageously leverages recent advances in partial reconfiguration of such devices whereby equipment-level requirements such as high availability, etc. may be realized. Further features of the various embodiments are as claimed in the dependent claims. Additional benefits and advantages of the embodiments will be apparent in view of the following description and accompanying Figures.
Embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the Figures of the accompanying drawings in which like references indicate similar elements. It should be noted that different references to “an” or “one” embodiment in this disclosure are not necessarily to the same embodiment, and such references may mean at least one. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
The accompanying drawings are incorporated into and form a part of the specification to illustrate one or more exemplary embodiments of the present disclosure. Various advantages and features of the disclosure will be understood from the following Detailed Description taken in connection with the appended claims and with reference to the attached drawing Figures in which:
In the following description, numerous specific details are set forth with respect to one or more embodiments of the present patent disclosure. However, it should be understood that one or more embodiments may be practiced without such specific details. In other instances, well-known circuits, subsystems, components, structures and techniques have not been shown in detail in order not to obscure the understanding of the example embodiments. Accordingly, it will be appreciated by one skilled in the art that one or more embodiments of the present disclosure may be practiced without such specific components-based details. It should be further recognized that those of ordinary skill in the art, with the aid of the Detailed Description set forth herein and taking reference to the accompanying drawings, will be able to make and use one or more embodiments without undue experimentation.
Additionally, terms such as “coupled” and “connected,” along with their derivatives, may be used in the following description, claims, or both. It should be understood that these terms are not necessarily intended as synonyms for each other. “Coupled” may be used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” may be used to indicate the establishment of communication, i.e., a communicative relationship, between two or more elements that are coupled with each other. Further, in one or more example embodiments set forth herein, generally speaking, an element, component or module may be configured to perform a function if the element is capable of performing or otherwise structurally arranged to perform that function.
As used herein, a network element or node (e.g., a router, switch, bridge, etc.) may comprise a piece of networking equipment, including hardware and software that communicatively interconnects other equipment on a network (e.g., other network elements, end stations, etc.). Some network elements may comprise “multiple services network elements” that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer-2 aggregation, session border control, Quality of Service, and/or subscriber management, and the like), and/or provide support for multiple application services (e.g., data, voice, and video). In some implementations, a network element may also include a network management element and/or vice versa. End stations (e.g., servers, workstations, laptops, notebooks, palm tops, mobile phones, smartphones, multimedia phones, Voice Over Internet Protocol (VOIP) phones, user equipment, terminals, portable media players, GPS units, gaming systems, set-top boxes, etc.) me be operative to communicate via any number of network elements or service elements in order to access or consume content/services provided over a packet-switched wide area public network such as the Internet through suitable service provider access networks. Some end stations (e.g., subscriber end stations) may also access or consume content/services provided on virtual private networks (VPNs) overlaid on (e.g., tunneled through) the Internet. Whereas some network nodes or elements may be disposed in wired communication networks, others may be disposed in wireless infrastructures. Further, it should be appreciated that example network nodes may be deployed at various hierarchical levels of an end-to-end network architecture. Regardless of the specific implementation, one skilled in the art will recognize that an embodiment of the present patent disclosure may involve a network element (e.g., a router) wherein one or more services or service functions having multiple instances (i.e., “service function replicas”) that may be placed or instantiated with respect to one or more packet flows (e.g., bearer traffic data flows, control data flows, etc.) traversing through the network element according to known or otherwise preconfigured service requirements and/or dynamically (re)configurable service rules and policies. Additionally and/or alternatively, one or more embodiments of the present disclosure may be practiced in the context of network elements disposed in a service network that may be implemented in an SDN-based architecture, which may further involve varying levels of virtualization, e.g., virtual appliances for supporting virtualized service functions or instances in a suitable network function virtualization (NFV) infrastructure. In a still broader aspect, an embodiment of the present patent disclosure may involve a generalized packet processing node or equipment wherein one or more packet processing functionalities, e.g., services, applications, or application services, with respect to a packet flow may be off-loaded to a reconfigurable device that may require in-service upgradability.
One or more embodiments of the present patent disclosure may be implemented using different combinations of software, firmware, and/or hardware. Thus, one or more of the techniques shown in the Figures (e.g., flowcharts) may be implemented using code and data stored and executed on one or more electronic devices (e.g., an end station, a network element, etc.). Such electronic devices may store and communicate (internally and/or with other electronic devices over a network) code and data using computer-readable media, such as non-transitory computer-readable storage media (e.g., magnetic disks, optical disks, random access memory, read-only memory, flash memory devices, phase-change memory, etc.), transitory computer-readable transmission media (e.g., electrical, optical, acoustical or other form of propagated signals—such as carrier waves, infrared signals, digital signals), etc. In addition, such electronic devices may typically include a set of one or more processors coupled to one or more other components, such as one or more storage devices (non-transitory machine-readable storage media), user input/output devices (e.g., a keyboard, a touch screen, a pointing device, and/or a display), and network connections. The coupling of the set of processors and other components may be typically through one or more buses and bridges (also termed as bus controllers), arranged in any known (e.g., symmetric/shared multiprocessing) or heretofore unknown architectures. Thus, the storage device or component of a given electronic device may be configured to store code and/or data for execution on one or more processors of that electronic device for purposes of implementing one or more techniques of the present disclosure.
Turning now to
In the context of the present patent application, a programmable device for effectuating application services on behalf of a host component may comprise a variety of (re)configurable logic devices including, but not limited to, Field-Programmable Gate Array (FPGA) devices, Programmable Logic Devices (PLDs), Programmable Array Logic (PAL) devices, Field Programmable Logic Array (FPLA) devices, and Generic Array Logic (GAL) devices, etc. At least portions of such devices may be responsible for executing application service functionalities and may be configured to be upgradable either in field, in lab, and/or remotely. By way of illustration, one or more embodiments will be described in detail hereinbelow by taking occasional reference to FPGA implementations, although one skilled in the art will recognize that the teachings herein may be applied in the context of other types of programmable devices as well, mutatis mutandis.
It should be appreciated that FPGAs may be implemented as critical components in virtually every high-speed digital design, including the design of router applications such as Non-Stop Routing (NSR), In-Service Software/Firmware Upgradability (ISSU/ISFU), etc. Unlike Application-Specific Integrated Circuits (ASICs), an FPGA-based application service implementation may be configured to ensure maximum availability with minimal downtime resulting from device maintenance and/or upgrade processes. By way of illustration, an FPGA implementation may be used in the context of router applications for providing the necessary processing with respect to services such as, inter alia, IPSec encapsulation where the CPU/NPU off-loads applicable packet encryption processes, which typically use CPU-intensive techniques.
Since the FPGA firmware is downloadable, it advantageously provides an upgrade path from software release to software release during the course of its deployment. For example, the complete FPGA binary file may be (re-)downloaded using in-system programming where the FPGA chip goes through a chip-level reset. During the FPGA upgrade process, therefore, services/applications provided by the FPGA will become unavailable for a period of time, which only increases with the ever-increasing FPGA logic gate capacity. Because newer FPGA devices supporting complex service/application functionalities may comprise tens of millions of Logic Cells (with the resultant FPGA Configuration Bitstream lengths being as large as 400 Mbits or more), ensuing disruption of services in the event of an upgrade or replacement significantly impairs the performance of the network equipment, especially when the FPGA functionality is deployed in datapath processing (e.g., on a line card or service card in NSR-capable equipment).
One skilled in the art will recognize upon reference hereto that network element 200 is illustrative of a more particularized arrangement of the node 104 disposed in communications network 102 shown in
As an example router implementation, network element 200 may include one or more routing modules 208 for effectuating packet routing according to known protocols operating at one or more OSI layers of network communications. Additionally, suitable input/output modules 206 may be provided for interfacing with a communications network, which may comprise any combination or subcombination of one or more extranets, intranets, the Internet, ISP/ASP networks, service provider networks, datacenter networks, call center networks, and the like, as described hereinabove. By way of illustration, application service cards 210-1 to 210-N as well as the remaining portions of the network element 200 may be interfaced using suitable buses, interconnects, high-speed packet interfaces, etc., collectively shown as transmission infrastructure 232 in
Taking reference to both
Returning to
Continuing to refer to
The application admission layer component 220 of the static core infrastructure of the programmable device 230 may be configured to include the engine-specific second-level FIFO pool, wherein each second-level ingress FIFO is equipped with a scheduler that services requests from the FIFO-crossbar distributor layer component 218. In one example implementation, scheduling may be performed by a Round Robin (RR) scheduler configured to serve the requests received from one or more crossbar distributors. Based on the dual-mode operation of the crossbar distributors, it should be appreciated that an ith scheduler of the application admission layer component 220 may receive requests in a normal/default operation (e.g., non-upgrade scenario) only from the corresponding ith second-level ingress distributor of the crossbar layer component 218. On the other hand, however, during upgrade of a jth service/application engine, the ith scheduler may receive requests from both ith and jth distributors due to the second-level LUT entries based on the Second-level_RN indexing. In other words, the requests that would have gone to the jth scheduler (for servicing by the associated jth application service engine) are now redistributed or redirected to the remaining active application service engines (via their corresponding schedulers). In one example embodiment, only one application/service engine may be configured to be upgraded at any single time such that an application admission scheduler may receive requests only from its corresponding second-level ingress distributor (in default mode) and requests from the second-level ingress distributor (in redirect mode) corresponding to the particular application/service engine being upgraded. It should be appreciated, however, that multiple engines may also be upgraded but such an arrangement may result in unacceptable performance degradation (since the remaining active engines/schedulers will be burdened with additional extra loads).
Turning to
Upon completion of application service processing, processed egress packets 400B may be returned to the host component via a default return path that may be effectuated in a number of ways wherein the prepended host identifier tag 404 may be used for properly directing the egress packets all the way to the correct host component and/or for tracking purposes. Accordingly, in one arrangement, egress packets may simply be bridged from a pool of second-level egress FIFOs of the application admission layer 220 (that receive the processed packets from corresponding application service engines) to the corresponding pool of first-level egress FIFOs (due to the 1-to-1 correspondence relationship in the FIFO crossbar layer 218 in normal mode similar to the ingress FIFO relationship). Thereafter, the aggregation layer 216 may utilize suitable scheduling techniques (e.g., RR scheduling) to retrieve the packets from the first-level egress FIFOs and forward them to the host component via applicable high-speed packet interfacing.
An example programmable device using a 4-bit based packet distribution scheme for supporting ISFU capability is provided below by way of illustration.
Aggregation layer 504 may be configured to include a first-level ingress distributor 518 that is interfaced with a host 502, wherein an ingress packet 520 is provided with a 4-bit first-level distribution tag and a 4-bit second-level distribution tag as described previously. A first-level LUT 522 is associated with the first-level ingress distributor 518 for determining a specific first-level ingress FIFO (and corresponding second-level ingress distributor or crossbar distributor).
In normal mode of operation, all four crossbar distributors 530A-530D are operative to forward the ingress packets to the respective particular application service engines for processing, wherein the crossbar distributors 530A-530D receive the ingress packets as distributed by the first-level ingress distributor 518. In an illustrative ISFU scenario, assuming that application service engine 550A is being upgraded, the crossbar distributor 530A corresponding to that engine is configured or reconfigured to operate in redirect mode whereby the ingress packets received from the first-level distributor 518 may be redirected or redistributed based on a second-level LUT that may be initialized by the host 502 at an appropriate time, preferably prior to initiating the IFSU procedure.
As noted hereinabove, egress packet flow remains unaffected insofar as the active application service engines emit the processed packets that are normally bridged from the corresponding second-level egress FIFOs 543B-543D to the corresponding first-level egress FIFOs 527B-527D. Thereafter, a scheduler 560 operating as part of the aggregation layer 504 is operative to transmit the processed packets to the intended host device 502, as illustrated by a dotted line communication path 561.
Turning to
Reference numeral 600B in
In the above-description of various embodiments of the present disclosure, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and may not be interpreted in an idealized or overly formal sense expressly so defined herein.
It should be appreciated that although service engine replacement has been described herein, packet redistribution in the context of incremental patches, upgrades, etc. pertaining to the firmware within an engine may also be practiced in accordance with the teachings herein. Additionally, packet redistribution in a scenario where multiple service engines, potentially performing different applications on a programmable device, are being replaced are upgraded is also deemed to be within the ambit of the present disclosure.
At least some example embodiments are described herein with reference to block diagrams and/or flowchart illustrations of computer-implemented methods, apparatus (systems and/or devices) and/or computer program products. It is understood that a block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits, logic gate arrangements, etc. For example, such computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and/or other programmable data processing circuit to produce a machine, so that the instructions, which execute via the processor of the computer and/or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions/acts specified in the block diagrams and/or flowchart block or blocks, and thereby create means (functionality) and/or structure for implementing the functions/acts specified in the block diagrams and/or flowchart block(s). Additionally, the computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions/acts specified in the block diagrams and/or flowchart block or blocks.
As alluded to previously, tangible, non-transitory computer-readable medium may include an electronic, magnetic, optical, electromagnetic, or semiconductor data storage system, apparatus, or device. More specific examples of the computer-readable medium containing program instructions and/or application service engines for replacement would include the following: a portable computer diskette, a random access memory (RAM) circuit, a read-only memory (ROM) circuit, an erasable programmable read-only memory (EPROM or Flash memory) circuit, a portable compact disc read-only memory (CD-ROM), and a portable digital video disc read-only memory (DVD/Blu-ray). The computer program instructions may also be loaded onto or otherwise downloaded to a computer and/or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer and/or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks. Accordingly, embodiments of the present invention may be embodied in hardware and/or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as “circuitry,” “a module” or variants thereof.
Further, in at least some additional or alternative implementations, the functions/acts described in the blocks may occur out of the order shown in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Moreover, the functionality of a given block of the flowcharts and/or block diagrams may be separated into multiple blocks and/or the functionality of two or more blocks of the flowcharts and/or block diagrams may be at least partially integrated. Finally, other blocks may be added/inserted between the blocks that are illustrated and blocks from different flowcharts may be combined, rearranged, and/or reconfigured into additional flowcharts in any combination or subcombination. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction relative to the depicted arrows.
Although various embodiments have been shown and described in detail, the claims are not limited to any particular embodiment or example. None of the above Detailed Description should be read as implying that any particular component, module, element, step, act, or function is essential such that it must be included in the scope of the claims. Reference to an element in the singular is not intended to mean “one and only one” unless explicitly so stated, but rather “one or more” or “at least one”. All structural and functional equivalents to the elements of the above-described embodiments that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Accordingly, those skilled in the art will recognize that the exemplary embodiments described herein can be practiced with various modifications and alterations within the spirit and scope of the claims appended below.
