The present application claims priority from Japanese patent application JP 2011-199737 filed on Sep. 13, 2011, the content of which is hereby incorporated by reference into this application.
The present invention relates to a node apparatus, a system, and a packet processing method.
One problem about a packet stream processing and a data stream processing in a network is to provide multiple advanced services to multiple users, and further to cope with a variation of a load during an operation of the network.
Japanese Unexamined Patent Application Publication No. 2010-193366 discloses a method for providing multiple advanced services to multiple users without causing mutual interference by virtualizing a network node and making it programmable. However, in Japanese Unexamined Patent Application Publication No. 2010-193366, a processor is assigned fixedly for each service, and load distribution in which a load of one service is distributed to multiple processors is not described.
Japanese Unexamined Patent Application Publication No. 2004-135106 discloses the load distribution performed when the load of the network node increases. Here, when the load increases, a packet is processed in a processor unit in low load.
In the case where the processor is assigned to each service as in a conventional technology of Japanese Unexamined Patent Application Publication No. 2010-193366, depending on a time, a deviation in load of each processor becomes large and it becomes impossible to use the processor efficiently. Moreover, when load forecast for every service is difficult, it becomes difficult to apply a technology of Japanese Unexamined Patent Application Publication No. 2004-135106. Furthermore, the technology of Japanese Unexamined Patent Application Publication No. 2004-135106 has a possibility that a high speed processing cannot be performed while performing load distribution.
An object of the present invention is to provide a node apparatus capable of distributing a load of a packet processing flexibly even in a situation where the load of the packet processing varies dynamically and is difficult to predict, and capable of high-speed processing of the packet.
A typical aspect of the present invention is as follows. That is, a node apparatus connecting to the network includes: multiple packet processor units; an interface unit that connects to the multiple packet processor units via a switch, stores association information for associating each packet with information of the packet processor unit capable of processing this, and forwards a packet that is associated with information of a first packet processor unit in the association information to the first packet processor unit; and a control unit of, when the first packet processor unit is determined to be in high load, updating the association information so that the packet being processable by the first packet processor unit may also be associated with information of a second packet processor unit in the association information; in which the interface unit forwards a packet that is associated with the information of the second packet processor unit in the updated association information to the second packet processor unit.
According to the aspect of the present invention, the node apparatus is enabled to perform flexible load distribution by dynamically increasing/decreasing the number of the packet processor units each for processing a packet. Moreover, since the interface unit has the association information for associating the packet with the information of the packet processor unit capable of processing this, packets can be distributed to the packet processor units at high speed, which makes possible a high speed processing of the packets.
The packet stream processor 101 has N packet processing boards 111 and M interfaces 151. However,
The control board 141 is a computer, which has a CPU (central processing unit) 142 and memory 143. The memory 143 includes a program loader 144 that is a program executed by the CPU 142, a load table 145, a program table 146, and a program storing unit (program store) 147. The program loader 144 loads programs P0, P1, . . . , Pm stored in the program storing unit 147 into memory (below-mentioned DRAM 113) of the packet processing board 111. The program loader 144 may load the programs P0, P1, . . . , Pm existing in URL (Uniform Resource Locator) stored in the program storing unit 147 into the memory (the DRAM 113) of the packet processing board 111. The processing that the program loader 144 performs using the CPU 142 will be explained in detail with reference to
In
In
Contents of the load table 145 are updated by a scheduled or unscheduled notification from each packet processing board 111. Alternatively, the control board 141 may inquire the load periodically to each packet processing board 111 and may update the contents of the load table 145. That is, each packet processing board 111 autonomously, namely when an internal timer has operated for a fixed time or when a specific processing in the each packet processing board 111 starts or ends, observes the index value of the load of the each packet processing board 111 and transmits a packet containing the value to the control board 141. Alternatively, the control board 141 transmits a packet containing inquiry information to each packet processing board 111, and answering to it, the each packet processing board 111 observes the index value of the load and transmits a packet containing the value to the control board 141.
The program management table 146 stores a combination of an identifier SID that is a value contained in a specific field of the header of the packet, an identifier PID of the program for processing a packet having the SID, and information (queue list QAVAIL) of the PPB queue of the packet processing board 111 that is associated with the program. The identifier SID is an identifier showing the kind of packet, and is a service identifier that specifies a service that the packet should receive in this embodiment. In the program management table 146 of
In this embodiment, although the SID is fixed in size and has a value, such as 617, 800, and 715, the SID may be variable in size. Moreover, in this embodiment, although the program management table 146 stores a value that should fully coincide with a value (identifier) contained in a field in the packet as the SID, a value that a part of this value is extracted and converted may be stored instead. When the extracted value is stored in the program management table 146, a value contained in the field in the packet is extracted and converted and then is compared with the identifier SID contained in the program management table 146. A relationship between the SID and the identifier PID of the program indicates which program should processes the packet containing an identifier corresponding to the SID: for example, the program is specified by a program load request from the network management device 12 in initialization of the packet stream processor 101 (the node apparatus).
In the program management table 146, information of the PPB queue of a certain one of the programs P0, P1, . . . , Pm (the queue list QAVAIL) in the program management table 146 is a bit vector representing the PPB queue that leads to execution of the program, and represents the PPB queue that holds a packet that the program processes. Here, elements (bits) of the bit vector are associated with the identifiers QID (e.g., Q00, Q01, . . . , Qi0, Qi1, Q(N−1)0, Q (N−1)1) of the PPB queue in a predetermined order from the head. When a packet sent to the PPB queue with a certain identifier is processable by a program with an identifier PID, an element (a bit) corresponding to that certain identifier QID becomes unity in the bit vector (QAVAIL) to the identifier, and when it is not so, this element (the bit) becomes zero. For example, in the case where the program that has only the identifiers Q00 and the PPB queue of Qi1 hold the packet that a certain program should processes, bits of a bit vector QAVAIL to this program are associated with the PPB queue from the head (left) as shown in
When loading the programs P0, P1, . . . , Pm into memory (the DRAM 113) of the packet processing board 111, the program loader 144 registers information (QAVAIL) of the PPB queue of the packet processing board 111 into the program management table 146.
The program storing unit 147 of the memory 143 stores the executable programs P0, P1, . . . , Pm. In this embodiment, although the programs P0, P1, . . . , Pm are stored in the memory 143, they may be stored in a mass storage, such as a semiconductor disk and a hard disk.
Referring to
The packets of the PPB queues 114a, 114b (in detail, the header and the descriptor of the packet) are forwarded to either of the NPUs 112a, 112b being in a state that can perform the packet processing (for example, an idle state) and are processed. The NPUs 112a, 112b execute a corresponding program stored in the DRAM 113 in order to process the packets of the PPB queues 114a, 114b, respectively. The packets of the PPB queues 114a, 114b may be sent to the NPUs 112a, 112b that are associated with the PPB queues in advance.
The packets after the processing (in detail, the header and the descriptor of the packet) are stored in the output queues 115a, 115b corresponding to the NPUs 112a, 112b, and are forwarded to the interface 151A, 151B, or 151C via the switch fabric 181.
The interface (NIF) 151 has an arithmetic processing circuit 156 for controlling the operation, such as ASIC, and one or more pieces of memory. The memory of the interface 151 stores a distribution table 152, input queues 154a, 154b, queue lists 155a, 155b, and a load table 153. Incidentally, hereafter, the input queues 154a, 154b of the interface 151 may be called an NIF queue. The queue lists 155a, 155b show information of the available packet processing boards 111.
The interface 151 searches the NIF queue corresponding to the identifier SID (namely, a classification of the packet) by referring to the distribution table 152, and distributes the packets to the NIF queues 154a and 154b for every identifier SID. Since the queue list makes the packets and the information of available packet processing board correspond to the packets for every pair of NIF queues 154a and 154b, the information of the available packet processing board 111 will be associated for every identifier SID or every distributed packet. Although the number K of NIF queues is two in this embodiment, the number K of NIF queues may be three or more. Moreover, it is also possible to eliminate the queue distribution table 152 by setting the number K of NIF queues to unity.
Furthermore, the interface 151 outputs the packet to resources of the packet processing board 111 based on the information of the packet processing board 111 that is associated for each distributed packet. As the resources of the packet processing board 111, there are the packet processing board 111 itself, and the PPB queue and the network processing unit NPU in the packet processing board 111. For example, the information of the packet processing board 111 includes the identifier (ID) and address of the PPB queue, the identifier (ID), the address, and a port number of the packet processing board itself, and a number of the network processing unit NPU.
The distribution table 152 shows a relationship between the identifier SID contained in the header of the inputted packet and the NIF queues 154a, 154b. The NIF queues 154a, 154b are held until the packet inputted into the interface 151 is forwarded to the packet processing board 111. The queue lists (or queue tables) 155a, 155b show queues that the packet can use. The queue lists 155a, 155b are association information each of which associates the packet with the information of the packet processing board 111 to which this packet is forwarded and where it can be processed by the program. Here, the information of the packet processing board 111 is information showing the PPB queue (input queue) that when the packet is forwarded to there, can process it with a corresponding program.
In this embodiment, the load table 153 stores the number of packets contained in each PPB queue etc. as the load of the packet processing board 111 after associating it with each PPB queue. The contents of the load table 153 are equal to the contents of the load table 145 of the control board 141. Moreover, the load table 153 is updated by a scheduled or unscheduled notification from the packet processing board 111 like the load table 145. The load may be notified to each interface 151 from each packet processing board 111 at fixed intervals, and the load shown in the load table 153 may be updated based on the notified load.
The PPB queue such that a load shown in the load table is low and corresponding bits of the available queue lists (the bit vector QAVAIL) 155a, 155b are set to unity (namely, available) is selected, and packets held in the NIF queues 154a, 154b are forwarded to the selected PPB queue. For example, if the bit vector of the available queue list corresponding to the NIF queue is (01 . . . 0), the packet of the NIF queue will be forwarded to the PPB queue of the identifier Q01.
The general purpose processing core 311 mainly controls units in the network processing unit 112. The multiple packet processing cores 321 mainly execute data processings in parallel. The I/O controller 302 is connected to a switch 361 existing outside the network processing unit 112, i.e., on the packet processing board 111.
Although the SRAM 331 has a small capacity, it is a main storage with a small delay. Moreover, the memory controller 341 is connected to the DRAM 113 existing outside the network processing unit 112, i.e., on the packet processing board 111. Although the DRAM 113 has a large capacity, it is a main storage with a somewhat large delay.
The switch 361 forwards packet data that arrives at the packet processing board 111 via the switch fabric 181 to the I/O controller 302. The I/O controller 302 forwards the packet data to the SRAM 331 through the bus 351 and forwards it to the DRAM 113 via the memory controller 341. The packet data stored in the SRAM 331 or the DRAM 113 is processed in the packet processing core 321 to be stored again in the SRAM 331 or the DRAM 113, or is outputted to the outside of the network processing unit 112 through the I/O controller 302.
However, when the program loader 144 receives the program load request, the program has not been stored in the program storing unit 147 yet. For this reason, in this case, the CPU 142 stores a program specified by the program load request or a program contained in the program load request in the program storing unit 147, and subsequently executes a procedure of FIG. 6. Incidentally, in the program load request, the program may be specified by a URL where it exists.
In the program loading processing, the program is loaded into the packet processing board 111. Furthermore, the association information (the queue list QAVAIL) that associates each packet with information of the packet processor unit capable of processing this is updated in each interface 151 so that the packet that the loaded program should process may be sent to the packet processing board 111. Thereby, the packet having arrived at the interface is forwarded to one of the packet handlers into each of which the program capable of processing it is loaded according to this association information at high speed, and is processed by the loaded program at high speed. Each packet handler 111 notifies the load to the program loader 144, and the program loader 144 having received the notification can load a program whose load is large into other packet handlers 111, before the load of the specific packet handler 111 becomes excessive.
Furthermore, when a program with a large processing load exists but an unused packet handler 111 does not exist, it is possible for the program loader 144 to select a program that has been distributed to multiple packet handlers 111 but whose processing load is low. The program loader 144 can delete the contents of the distribution table 152 related to a service (the service identifier SID) of a program whose processing load is low, can unload the program from one of the multiple packet handlers 111, and can load a program whose processing load is high instead.
When the program loading processing is started, first, at Step S411, the program loader 144 selects the packet processing board 111 not used or in low load that can perform the packet processing and either of the PPB queues 114a, 114b in this board by referring to the load table 145. Incidentally, when the processing of Step S611 of a below-mentioned load monitoring processing has been executed, the PPB queues 114a, 114b in low load selected at Step S611 and the packet processing board 111 containing it may be selected in Step S411. For example, the packet processing board 111 having a minimum load of the load table 145 may be determined to be in low load.
The program loader 144 registers the selected PPB queue in the program management table 146. The selected PPB queue is shown in the list of the PPB queues (i.e., the bit vector QAVAIL), that is, a bit corresponding to the selected PPB queue is set to unity. Incidentally, when an item about the identifier SID (the service identifier etc.) of the packet that the program to be loaded should process does not exist in the program management table 146, the item is generated, and the identifier PID (a pointer or a URL to the program storing unit 147) of the program and the selected PPB queue are registered. Moreover, when an item about the identifier SID of the packet that the program to be loaded should process exists in the program management table 146, the selected PPB queue is added to the list of the PPB queues of the packet processing board 111, that is, a bit corresponding to the selected PPB queue is set to unity.
Next, at Step S412, the program loader 144 loads the program into the memory (the DRAM 113) of the selected packet processing board 111 and activates it.
Furthermore, at Step S413, the program loader 144 associates the identifier SID contained in the program load request with the NIF queue in each interface 151. For this reason, the program loader 144 issues an instruction to each interface 151, and registers a pair of the identifier SID and the NIF queue number assigned to this in the distribution table (qDT) 152. For example, in
Finally, at Step S414, in response to the instruction of the program loader 144, each interface 151 registers all the PPB queues selected at Step S411 in the list QAVAIL of the PPB queues by setting bits corresponding to them to unities.
The contents of the list QAVAIL of the PPB queues memorized in the memory of the interface 151 become equal to the list QAVAIL of the PPB queues memorized in the memory of the control board 141. Thereby, when there does not exist the PPB queue registered in the list of the PPB queues before execution of the program loading processing (when all the bits of the bit vector QAVAIL are zeros), the packet having arrived at the interface 151 can be newly processed by the program on the packet processing board 111. Moreover, when an already registered PPB queue exists in the list QAVAIL of the PPB queues (when one of bits of the bit vector QAVAIL is unity), the packet having arrived at the interface 151 is forwarded not only to the already registered PPB queue but also to a newly registered PPB queue, and the load of the packet processing board 111 can be distributed.
The list (QAVAIL) of the PPB queues will also exist in each interface 151 in addition to the control board 141, and thereby the packet will be able to be processed in each interface 151 at high speed. In the interface 151, the list (QAVAIL) of the PPB queues is stored in memory capable of being accessed at high speed, such as CAM (content addressable memory), so that the packet can be processed at wire rate.
Next, the program load request for activating the program loading processing in the packet stream processor 101 will be explained. The network management device 12 sends the program load request to the control board 141 of the packet stream processor 101 as per instruction of the administrator or the management program for the initialization etc. before the packet processing start of the packet stream processor 101. The program load instruction includes the following items (i) to (iii): (i) URL of a program or the program itself, (ii) service identifier (SID), and (iii) load forecast information (resource information).
Regarding the item (i), when the program load request includes the program itself, the control board 141 stores the program, as it is, in the program storing unit 147. When a URL of the program is contained in the program load request, the control board 141 accesses a specified URL by HTTP (HyperText Transfer Protocol), receives the program, and stores it in the program storing unit 147, or stores the URL itself in the program storing unit 147. The program is an object program or a source program. In the case of the source program, the control board 141 includes a compiler for converting the source program into an object program, and stores the object program obtained by the compilation in the program storing unit 147. Instead, the control board 141 may compile the source program as a part of the program loading processing. In doing compilation, the compiler of the language is used according to a language with which the source program was described. For example, if a fact that the source program is for a specific architecture (for example, a CPU of a specific network processor) is described in the source program, the compiler for the network processor will be used.
Incidentally, the program loader 144 having received the program load request containing neither the program nor its URL can assume that when an element containing the service identifier specified in the program load request exists in the program management table 146, the program containing the element is specified in the program load request.
Regarding the item (ii), the service identifier (SID) is a value that the packet contains in order to specify a program that processes itself. The service identifier corresponds to a program, in a one-to-one manner, or in a many-to-one manner.
Regarding the item (iii), the load forecast information is information to forecast a load that the program gives the packet stream processor 101 (the node apparatus). The load forecast information includes a resource amount required to execute the program one time or information for estimating it as first information, and the number of times of execution of the program or information for estimating it as second information. The resource amount expressed with a processing time, a memory quantity, etc. as the first information may be given by a value or a function according to an attribute of the packet that should be received. Incidentally, the resource amount required to execute the program one time can also be estimated by a static analysis or dynamic analysis of the program by executing the program with standard input data given. In that case, the first information is unnecessary. The number of times of execution of the program as the second information is given in a form of the number of packets, but may be given in a form of a bandwidth (a bit rate). If the bandwidth is specified, the number of packets can be estimated using an estimate of average packet length.
When the program loader 144 in the control board 141 receives the program load request, a specified program to the packet having the specified service identifier is loaded into the selected packet processing board 111 and is executed (S412). The packet generated or forwarded by the specified program can initialize the packet stream processor 101 so that its destination (for example, one of adjacent node apparatuses) may be decided by the program. However, the packet stream processor 101 can be configured to function as a normal Ethernet (a registered trademark, the same shall apply hereafter) or an IP router and to be initialized so that the packet may be forwarded automatically. Moreover, the packet stream processor 101 can also be initialized so as to discard all packets that have arrived at it as its initial state.
Moreover, when the service identifier of the packet having arrived does not exist in the queue distribution table 152, the NIF 151 can also transmit the program load request to the program loader 144. In this case, it is also possible to transmit the program load request containing neither a program nor its URL. Such a program load request is equivalent to a request of inquiring a server of a switch operation when a new flow arrives at an OpenFlow switch. (Here, OpenFlow is a network control technology that the OpenFlow Switching Consortium is advocating.) That is, in the OpenFlow switch, although its operation is selected from a range decided in advance, an operation of the packet stream processor 101 on a packet having the service identifier can be given by an arbitrary program in this embodiment. In this embodiment, although one field in the packet called the service identifier is used for selection of the program, values of the multiple fields in the packet can also be used as in the OpenFlow switch.
Moreover, when the control board 141 receives the program unloading request sent from the network management device as per instruction of the administrator or the management program, the program loader 144 executes the program unloading processing for all the packet processing boards 111 into each of which the program of an object of the unloading processing was loaded. After that, the program loader 144 deletes this program or a corresponding URL from the program storing unit 147. The program unloading request contains the service identifier. Since the service identifier corresponds to a program in a one-to-one manner or in a many-to-one manner, the program to be unloaded can be specified with it. After the program unloading processing is performed, the same processing as that before the program corresponding to this service was loaded is performed on the packet having the service identifier contained in the program unloading request.
Referring to
Next, at Step S512, when the PPB queue that the program can use dose not exist in the program management table 146, the program loader 144 sends an instruction to every interface 151 to erase the NIF queue assigned to a service corresponding to this program (namely, the identifier SID of this service) in the distribution table qDT of each interface 151. When the list of the PPB queues that the program management table 146 contains becomes empty (that is, all the bits of QAVAIL are zeros), it is determined that the PPB queues used by the service does not exist.
Furthermore, at Step S513, all the PPB queues obtained at S511 are erased from the list (QAVAIL) in each interface 151 that the service uses. Thereby, it does not happen that when the PPB queue registered in the list (QAVAIL) no longer exists, a packet having arrived at the interface 151 is processed by a program on the packet processing board 111. That is, the packet having arrived at the interface 151 is discarded by the initialization, or is processed as a usual Ethernet packet or an IP packet. Moreover, when the PPB queue registered in the QAVAIL remains, an effect that the load is concentrated on the remaining PPB queue and consequently on the packet processing board 111 containing this PPB queue arises.
Finally, at Step S514, the program loader 144 stops and unloads the program on the packet processing board 111 obtained at S511. Moreover, if the program is unloaded from all the packet processing boards 111, this program is deleted also from the program table 146. The program loader 144 deletes the program from the memory (the DRAM 113) of this packet processing board 111. In the packet processing board 111 from which the program is unloaded, although it becomes impossible to execute the program, the memory that the program has occupied is released, an other program is loaded into this memory and uses it.
At Step S610, it is determined whether the PPB queue qh corresponding to the selected element QLEN of the load table 153 is in high load. If the PPB queue qh is in high load, processings of Steps S611 to S613 will be executed. In this embodiment, the element QLEN indicating a load in the load table 153 is the number of packets accumulated in the corresponding PPB queue qh. When the load (here, the number of packets) exceeds a fixed threshold, it is determined that the PPB queue qh is in high load. For example, the threshold is a value of three as the number of packets. However, the threshold does not need to be a fixed value, and the threshold may be set larger as a value of an other element QLEN of the load table becomes higher.
At Step S611, the load table 153 is referred to, and the PPB queue ql un-assigned (unused) or in low load and the packet processing board PPBpl containing it are selected. The PPB queue ql un-assigned or in low load can be found by scanning the load table 153. However, it is also possible to obtain the PPB queue ql in the last scanning and to save this. Here, the PPB queue that has a minimum load in the load table 153 can be selected as the PPB queue ql in low load.
At Step S612, when the selected PPB queue ql has already been assigned (namely, being in low load), according to the unloading processing of
At Step S613, the program that was processing the packet of the PPB queue qh in high load is loaded also into the memory (the DRAM 113) of the packet processing board PPBpl containing the PPB queue ql un-assigned or in low load according to the program loading processing of
At Step S614, it is determined whether the processing is completed for all the elements that show loads of the load table 153. When the processing is not completed for all the elements, the next element is selected at Step S615 and the routine returns to Step S610. When the processing is completed for all the elements, the routine is ended.
In the load monitoring processing of
At Step S801, first, a number i of the packet processing board (PPB) 111 is set to zero (i=0), and a number x of the input queue (PPB queue) in the packet processing board 111 is set to zero (x=0).
At Step S802, as the determination processing of forwardability, it is determined whether QAVAILk [(i·L)+x] is equal to unity and QLEN[(i·L)+x] is smaller than a specified value (e.g., 2). If QAVAILk[(i·L)+x]=1 and QLEN[(i·L)+x] is smaller than the specified value, the routine will proceed to Step S803; if it is not so, the routine will proceed to Step S804. Here, QAVAILk [ ] represents a [ ]-th bit from the highest bit in the bit vector QAVAILk that represents the PPB queue capable of processing a packet of a k-th NIF queue qjk. When the packet of the k-th NIF queue qjk is forwarded to the x-th PPB queue of the i-th packet processing board 111, if it is processable by the program, QAVAILk[(i·L)+x] will become unity, and if it is un-processable, QAVAILk[(i·L)+x] will become zero. Moreover, QLEN[ ] represents the [ ]-th element (load) from the highest element in the load table 153. QLEN[ ] is a value representing the load, and is the number of packets contained in the PPB queue here.
At Step S803, as the forwarding processing, a top packet is taken out from the NIF queue qjk, and is forwarded to the x-th PPB queue Qix in the i-th packet processing board 111.
At Step S804, it is determined whether the determination processing of forwardability at 5802 is completed for all the PPB queues in the i-th packet processing board 111. That is, denoting the number of the PPB queues in the packet processing board 111 by L, it is determined whether x=L−1 holds. When x does not satisfy x=L−1, the number x increments by one at S805, and a processing of S802 is performed on the next PPB queue. Thereby, determination processing of forwardability of S802 is performed on all the PPB queues (x=0, 1, . . . , L−1) in the i-th packet processing board 111. On the other hand, when x satisfies x=L−1, the routine proceeds to Step S806.
At Step S806, it is determined whether the determination processing of forwardability of S802 is completed for all the packet processing boards 111. That is, denoting the number of the packet processing boards 111 in the packet stream processor 101 by N, it is determined whether the number i is i=N−1. When does not satisfy i=N−1, the number i increments by one at S807, and the processing of S802 is performed about the next packet processing board. Thereby, the processing of S802 is performed about all the packet processing boards 111 (i=0, 1, . . . , N−1) in the packet stream processor 101. The routine is ended when i satisfies i=N−1.
Explaining a case where k=1, L=2, N=2, and QAVAIL1=(1001) as a simple example, since QAVAIL1[0]=1 holds, if QLEN[0]<specified value (e.g., 2) holds, a head packet of the NIF queue 154b (qj1) will be forwarded to the PPB queue Q00, and will be processed by a program (already loaded into PPB#0) corresponding to the identifier SID of the packet. Since QAVAIL1 [1]=0 and QAVAIL1 [2]=0, regardless of QLEN[1] and QLEN [2], the packet of the NIF queue 154b (qj1) is not forwarded to the PPB queues Q01 and Q10. Since QAVAIL1 [3]=1, if QLEN [3]<specified value holds, a next head packet of the NIF queue 154b (qj1) will be forwarded to a PPB queue Q11, where it will be processed by the program (having loaded into PPB#1) corresponding to the identifier SID of the packet.
In the packet forwarding processing, when the list QAVAIL of available PPB queues includes representations (i.e., unity) of the multiple PPB queues, the packets related to one service identifier SID need to be distributed to the PPB queues. By a round robin scheduling as described above, as a simplest method, the packet can be forwarded by using the PPB queue whose load is lower than a specified value sequentially. On the other hand, more packets can also be forwarded to the PPB queue (or packet processing board) in low load of the load table 153. For example, PPB queues can be weighted by using a function that decreases monotonously depending on a value of the load, and packets can also be forwarded to the PPB queues according to the weights.
With reference to
According to this embodiment, the interface (NIF) 151 connects to the multiple packet processor units 111 through the switch 181, stores the association information (e.g., the queue list QAVAIL) for associating each packet with information of the packet processor unit 111 capable of processing this, and forwards a packet that is associated with the information of the first packet processor unit in the association information to the first packet processor unit (e.g., PPB#0). When the first packet processor unit is determined to be in high load more than or equal to the threshold (S610), the control board 141 (control unit) updates the association information so that a packet being processable by the first packet processor unit is also associated with information of a second packet processor unit (e.g., PPB#1) in the association information (S414, S613). Then, the interface 151 forwards a packet that is associated with information of the second packet processor unit (PPB#1) in the updated association information to the second packet processor unit.
Thereby, it becomes possible even for the second packet processor unit to process a packet being processable by the first packet processor unit in response to increasing load of the first packet processor unit. Therefore, it is possible to dynamically increase/decrease the number of the packet processor units each for processing a packet about a certain service according to the load, which makes flexible load distribution possible. Moreover, since the interface 151 has the association information, packets are distributed to the packet processor units at high speed, which makes possible the high speed processing of the packets in the node apparatus.
The interface 151 has the association information (the queue list QAVAIL) for each identifier (SID) that should be contained in each packet. The interface 151 forwards a packet containing the first identifier to the first packet processor unit that is associated in the association information (e.g., QAVAIL0) about the first identifier. The control board 141 updates the association information (e.g., QAVAIL0) about the first identifier so that the packet containing the first identifier may also be associated with the information of the second packet processor unit. The interface 151 forwards the packet containing the first identifier to the second packet processor unit.
When the service is specified by an identifier, the identifier is contained in the packet and the packet having the identifier and the information of the packet processor unit are associated with each other in the association information for each identifier of the interface 151. Thereby, the load of the processing can be distributed to multiple packet processor units for each packet that should receive a specific service.
The first packet processor unit has a first program for processing the packet containing the first identifier. When the first packet processor unit is in high load, the control board 141 loads the first program into the second packet processor unit (S412, S613). The second packet processor unit processes the packet containing the first identifier with the first program. Thereby, when a certain packet processor unit is in high load, a program that is executed by the packet processor unit will come to be executed by an other packet processor unit, which will distribute the load given by execution of the program.
The control board 141 sets the second packet processor unit as the packet processor unit in low load (S411, S611). The control board 141 deletes a second program from the second packet processor unit when loading the first program into the second packet processor unit (S612). Thereby, the memory that the second program occupies is released in the second packet processor unit, and the first program can be loaded into the memory of the second packet processor unit. Incidentally, since the second packet processor unit is in low load, even if it becomes impossible for the second packet processor unit to process the packet, there will arise little bad influences. Furthermore, the control board 141 deletes the information of the second packet processor unit in the association information about the second identifier contained in the packet that the second program processes (S513). Thereby, the packet that the deleted second program should process will not be forwarded to the second packet processor unit.
The control board 141 stores the same association information as the association information that the interface 151 has. For this reason, the control board 141 can load a program required in order to process the packet to be sent to the packet processor unit into the packet processor unit based on the association information.
Each packet processor unit sends the information of the load to the interface 151, and the interface 151 has the load table 153 for storing the information of the load. Thereby, the interface 151 can find out the packet processor unit in low load at high speed from among the packet processor units each capable of the packet processing based on the association information (S802), and can forward the packet to this packet processor unit in low load. The information of the load may be the number of the packets accumulated in each queue of each packet processor unit. Thereby, the load can be grasped easily.
In the above-mentioned embodiment, allocation of the resources in the packet processing board 111 is decided in the program loader 144 and the interface 151. However, in the program loader 144 and the interface 151, the resources may be managed and controlled using the packet processing board 111 as a unit, and the resource allocation in the packet processing board 111 may be decided in the packet processing board 111. That is, the packet inputted into the packet processing board 111 is allocated to the PPB queue in the packet processing board 111 according to a rule decided in the packet processing board 111. Then, when the NPU becomes idol, the PPB queue is selected, the packet is taken out, and the program specified by the identifier of the packet is executed in the NPU according to a decided scheduling algorithm. In this case, the bit vector QAVAIL may be not the list of the PPB queue but the list of packet processing boards each capable of processing the packet. In this case, the bits of the bit vector QAVAIL may be associated with the numbers (the port numbers may be sufficient) and the identifier of the packet processing board 111 from the top (left), as shown in
In the above-mentioned embodiment, the control board 141 performed the load monitoring in a centralized manner. However, it is also possible to perform the load monitoring in a distributed manner. That is, each packet processing board 111 performs the load monitoring, and when the load is high, the packet processing board 111 issues an instruction to all the interfaces 151 each transmitting a packet to the packet processing board 111 and stops transmissions of the respective packets to the packet processing board 111. For this reason, each packet processing board 111 holds the program management table 146 that only the control board had in the above-mentioned embodiment, and manages it in a distributed manner. Therefore, each packet processing board 111 manages only the program that it uses with the program control table.
Incidentally, in the embodiment described above, the destination of the packet from the interface 151 was determined based on the load of the PPB queue as the load of the packet processing board 111. However, it is also possible to decide the destination considering also the load of the interface 151 that is the destination of the program, i.e., a state of congestion. That is, each interface 151 grasps the interface 151 of the destination for each program on the packet processing board 111 first at the time of the program load request. Therefore, the program load request may include the information of the interface 151 of the destination, and it may be found from the contents of the program. Each interface 151 keeps the information of the interface 151 of the destination thus obtained and the congestion information periodically reported from the interface 151 of the destination, and when the interface 151 of the destination is in congestion, it delays a packet being processed by the program from being transmitted to the PPB queue. By this, the discard of the output packet by congestion of the destination interface 151 can be decreased.
It is clear that the present invention is not limited to the above-mentioned embodiments and their modification, and various alterations can be made within a range of its technological idea.
Number | Date | Country | Kind |
---|---|---|---|
2011-199737 | Sep 2011 | JP | national |