Patent Application
20170063636
This application claims priority to French Patent Application No. 1558131; filed Sep. 2, 2015, the entire content of which is incorporated herein by reference in its entirety.
The invention relates to systems including service nodes which are interconnected via links and, more specifically, the determination of interconnection topologies of such systems.
The term “service node” in this context is understood to mean an electronic device including resources which can be of any type if they provide at least one configurable service useful to the system to which they belong, or to an application operating in this system. Consequently, this may relate, for example, to a computation device including at least one processor, or an input/output device, or a router (or switch), or alternatively a data storage system.
Furthermore, the term “interconnection topology” is understood to mean the architecture (or the interconnection network) defining all the links (or interconnections) between all the service nodes of a system. This relates to a non-directional graph in which the peaks represent, for example, routers and the arcs represent the links (or wires). It will be observed that several topology structures exist, and in particular those known as “hierarchical structures”, ring structures (with direct topology), and networked structures. Of the hierarchical structures one may, for example, cite in particular those called Fat tree, Generalised Fat Tree (or GFT), Extended Generalised Fat Tree (or XGFT), and Parallel Generalised Fat Tree (or PGFT). Among the networked structures one may in particular cite the one called “HyperX”.
Bearing in mind the above two definitions, a system may, in particular, be a computer of the high-performance type (or supercomputer), or a data centre.
The determination (or dimensioning) of the interconnection topology of a system such as, for example, a supercomputer, is a complex task due to the large number of constraints which must be taken into account, and in particular performance level, cost, technical feasibility, logistical constraints (and in particular encumbrance) and thermal constraints.
Due to this complexity, determination (or dimensioning) of the interconnection topology of a system is frequently implemented manually after the structure of this interconnection topology has been chosen, leading to several disadvantages. Indeed, the determined interconnection topology is rarely optimal from all standpoints, or technically infeasible, and therefore many modifications must be made, frequently leading to substantial additional costs and/or possible delays of provision to the client and/or a possible reduction of the initially desired level of performance.
An information technology tool exists, of the “web” type, developed by the company Mellanox, intended to determine (or dimension) interconnection topologies. This tool is suitable for structures of the Fat Tree type, and enables the number of compute nodes to be chosen, together with the type of router to use on the first two levels of interconnection topology, and the blocking factor. The main disadvantage of this tool is that it only provides the number of routers and links required to establish the system, and therefore gives no information on the topological configuration (such as a formal description, for example), which is however essential for an assessment of the quality of the topology in terms of routing and quality/price ratio. Moreover, traditional or generally purpose computers or computer systems are not well suited for implementing or helping implement determination (or dimensioning) of the interconnection topology of a system.
An aspect of the invention is therefore, in particular, to improve the situation.
In particular, an embodiment proposes to this end a method intended to allow determination of an optimal interconnection topology of a system including service nodes which are interconnected using links, and some of which are routers with ports, and belonging to at least one level.
This method includes:
a first step in which a structure for an optimal interconnection topology is chosen, to be determined from among a set of structures each one of which is defined by at least one structural constraint, and at least one topology parameter is then chosen from among a maximum number of router ports, a number of levels of routers, a number of service nodes connected to a router belonging to the first level, a number of service nodes of a level i−1 connected to the same router of a level i, a number of routers of a level i connected to the same service node of a level i−1, a number of links (or connections) between a router of a level i and a service node of a level i−1, and
a second step in which a solver determines the optimal interconnection topology defined by at least one corresponding structural constraint, and each chosen topology parameter, and at least one chosen optimisation criterion.
By this means it is possible to determine the optimal interconnection topologies of different structures which are technically feasible, i.e. which match clients' requirements without requiring many modifications when they are installed.
The method according to one or more embodiments the invention can include other characteristics which can be considered separately or in combination and, in particular:
each optimisation criterion can be chosen from among (at least) the cost of the system, a level of performance, the extensibility of the system, the system's energy consumption, thermal constraints, and the system's encumbrance;
each structural constraint can be chosen from (at least) the following inequalities: pi*mi+pi+1*wi+1≦radixMax, ph*mh≦radixMax, and w1*p1+noeudsParFeuille≦radixMax, where mi is the number of service nodes of a level i−1 connected to given router of a level i, wi is the number of routers of a level i connected to given service node of a level i−1, pi is the number of links (or connections) between a router of a level i and a service node of a level i−1, radixMax is the maximum number of ports of each router, h is the number of levels of routers, and noeudsParFeuille is the number of service nodes connected to a given router belonging to the first level;
in the second step the optimal interconnection topology can also be determined in terms of at least one chosen usage constraint;
the system can be chosen from among (at least) one supercomputer and one data centre.
An aspect of the invention also proposes a computer program product including a set of instructions which, when it is executed by processing means, is able to implement a method of the type presented above to determine an optimal interconnection topology of a system including service nodes which are interconnected using links, and some of which are routers with ports, and belonging to at least one level. The computer program product may be implemented in a non-transitory machine readable medium. The computer program product includes a set of instructions for implementing each step of the method.
Another aspect of The invention also proposes a processing device, intended to determine an optimal interconnection topology of a system including service nodes interconnected using links, some of which are routers with ports, and belonging to at least one level, and able to be fitted to an electronic device including computational resources and a man-machine interface.
This processing device includes:
a programming interface designed to enable the user to choose, via the man-machine interface, a structure for an optimal interconnection topology to be determined from among a set of structures each one of which is defined by a least one structural constraint, at least one topology parameter chosen from among a maximum number of router ports, a number of levels of routers, a number of service nodes connected to a router belonging to the first level, a number of service nodes of a level i−1 connected to a the same router of a level i, a number of routers of a level i connected to the same service node of a level i−1, a number of links (or connections) between a router of a level i and service node of a level i−1, and
resolution system configured to determine the optimal interconnection topology defined by at least each corresponding structural constraint, and each chosen topology parameter, and by at least one chosen optimisation criterion.
An aspect of the invention also proposes an electronic device including computational resources, a man-machine interface, and a processing device of the type presented above.
Other characteristics and benefits of the invention will be seen clearly on an examination of the detailed description below, and the appended illustrations, in which:
An aspect of the invention is to propose a method of determination, and an associated processing device DT, intended to allow the determination of an optimal interconnection technology of a system S comprising service nodes Nij.
In what follows it is considered, as a non-restrictive example, that system S is a high-performance computer (or supercomputer). But the invention is not restricted to this type of system. Indeed, it relates to all systems including service nodes which are interconnected using links, some of which are routers with ports, and belonging to at least one level, and in particular data centres.
As stated above, an aspect of the invention proposes a method intended to allow determination of an optimal interconnection technology of a system S (in this case a supercomputer) including service nodes Nij which are interconnected using links, and some of which are routers with ports, and belonging to at least one level.
This method includes first and second steps, referenced respectively 10 and 20 in the example algorithm of
As illustrated in
For example, the programming interface INP can be of the API type (“Application Programming Interface”). As a non-restrictive example, it can be created from a Python library, such as the library called Pyomo. It should be noted that Pyomo is a Python library which allows the generation, in a particular format (.nl) supported by many equation resolution programs, of systems of equations which have been declared and instantiated beforehand by means of an API.
In the non-restrictive example illustrated in
It will be appreciated that the different steps of the method work in concert to determine the optimal interconnection topology of the system including service nodes. In particular, by virtue of performing the specific steps of the method described herein, it will be appreciated that the described method provides an interconnection topology of the system that is different from that that would be obtained using a conventional approach or a conventional approach that would have been implemented on a general purpose computer. It will also be appreciated that the processing device described herein is a special purpose computer as it is specifically constructed to perform the method described herein.
During first step 10 of the method according to an embodiment the invention, the human agent (a user of electronic device EE) starts by choosing, from among a set of structures each of which is defined by at least one structural constraint, and by the man-machine interface IH, a structure for an optimal interconnection topology which is determined for a system S. The set of structures is provided by programming interface INP of processing device DT. For example, this set may include at least one structure of the hierarchical type (such as, for example, Fat tree or Generalised Fat Tree (or GFT) or Extended Generalised Fat Tree (or XGFT) or alternatively Parallel Generalised Fat Tree (or PGFT)) and/or at least one ring structure and/or one networked structure (such as for example HyperX).
For example, each of the hierarchical structures may be defined by at least one structural constraint defined by at least two variables mi and wi, and sometimes also by a third variable pi, in which index i indexes the level to which a router within the structure belongs. mi is the number of service nodes Ni−1j of a level i−1 connected to a given router Nij of a level i. wi is the number of routers Nij of a level i connected to a given service node Ni−1j of a level i−1. pi is the number of links (or connections) between a router Nij of a level i and a service node Ni−1j of a level i−1. These variables m, w and p are vectors of dimension h. With the choice made above for indexing, the value i=0 is attributed to the level containing service nodes N0j which are connected to routers N1j (called “leaves”) of the first level (i=1).
Thus a GFT (Generalised Fat Tree) can be defined by at least. one structural constraint defined by two variables m and w (vectors of dimension equal to one), and also by h, where ∀i∈[[1,h]], each router Nij of level i is connected to m service nodes Ni−1j of lower level i−1, and each router Ni−1j of level i−1 is connected to w routers Nij of upper level i. It can therefore be noted (h,m,w).
An XGFT (Extended Generalised Fat Tree) can be defined by at least one structural constraint defined by the two variables m and w (vectors of dimension equal to h), and by h, where ∀i∈[[1,h]], mi>0 wi>0. It can therefore be noted (h,[m1, m2, . . . , mh], [w1, w2, . . . ,wh]). In addition, it enables different links to be obtained (number of ascending/descending links) for the different levels i of the tree structure.
A PGFT (Parallel Generalised Fat Tree) is an XGFT to which third vector p has been added. It can therefore be noted (h,[m1, m2, . . . , mh], [w1, w2, . . . ,wh],[p1, p2, . . . ,ph]). It enables the notion to be introduced of redundancy of connections between service nodes of two adjacent levels.
First step 10 continues by choosing at least one topology parameter from among:
a maximum number of ports of routers Nij (with i≧1),
a number h of levels of routers Nij (with i≧1),
a number of service nodes N0j connected to a router N1j belonging to the first level (i=1),
a number of routers Ni−1j of a level i−1 connected to given router Nij of a level i,
a number of routers Nijof a level i connected to a given service node Ni−1j of a level i−1, and
a number of links (connections) between a router Nij of a level i and a service node Ni−1j of a level i−1.
It will be noted that the user of electronic device EE can choose, using the man-machine interface IH, each topology parameter from a list of topology parameters supplied by programming interface INP.
This list can, for example, be stored in a storage system MS of processing device DT. This storage system MS can, for example, take the form of a memory, possibly of the software type.
An aspect of the invention therefore proposes to define in an unambiguous manner the structure of each interconnection topology on the basis of one or more structural constraints (each defined by at least one variable), due to the regularity of the theoretical interconnection topologies. It will be noted that these structural constraints, and therefore the interconnection topology structures, can even be expressed in the form of mathematical equations, as indicated below.
As an example for PGFT structures, each structural constraint can be chosen from the following inequalities ∀i∈[1,h−1], where h is the number of levels of routers of the interconnection topology:
pi*mi+pi+1*wi+1≦radixMax (this structural constraint enables it to be specified that the number of links connected to a router does not exceed the number of ports available on this router),
ph*mh≦radixMax, and
w1*p1+noeudsParFeuille≦radixMax,
where radixMax is the maximum number of ports of each router Nij, and noeudsParFeuille is the number of service nodes N0j connected to a given router N1j, called a “leaf” since it belongs to the first level.
Also as an example for XGFT structures, each structural constraint can be chosen from the following inequalities ∀i∈[1,h−1], where h is the number of levels of routers of the interconnection topology:
mi+wi+1≦radixMax,
mh≦radixMax, and
w1+noeudsParFeuille≦radixMax.
Also as an example for GFT structures, each structural constraint can be chosen from the following inequalities ∀i∈[1,h−1], where h is the number of levels of routers of the interconnection topology:
m+w≦radixMax,
m≦radixMax, and
w+noeudsParFeuille≦radixMax.
During the second step (20) of the method according to an embodiment of the invention, one determines with a solver (defined by resolution system MR) the optimal interconnection topology of system S, defined by at least one corresponding structural constraint and each topology parameter chosen in the first step (10) and at least one chosen optimisation criterion.
The choice of each optimisation criterion can, for example, be made by the user in first step 10. As a variant, it can be made in second step 20. This choice can be made using the man-machine interface IH, from a list of optimisation criteria supplied by programming interface INP. This list can, for example, be stored in storage system MS of processing device DT.
Due to the definition of the structure of each interconnection topology on the basis of at least one structural constraint defined by at least one variable, it is now possible to determine the optimal interconnection topology of a system S by resolution of an optimisation problem using a solver (or resolution means MR). In other words, the separation between the modelling of the structure and the resolution method enables the search for optimal solutions to be delegated to a reliable and robust mathematical tool (or solver).
As a non-restrictive example, the solver called “Bonmin” may be used, which is capable of resolving systems of non-linear equations with integer variables. It enables accurate solutions to be found in short periods (typically several seconds in the case of topologies of 64,000 service nodes). As a variant, the solver Ipopt may for example be used.
For example, each optimisation criterion can be chosen from among at least the cost of system S, the desired performance level of system S (for example its computational power or its computation speed), the extensibility of system S, the desired energy consumption of system S, thermal constraints, in particular relating to the hosting environment of system S, and the encumbrance of system S (logistical criterion). In general, an optimisation criterion can be any property of an interconnection topology.
It will be noted that in certain situations use solely of structural constraints may lead to a correct interconnection topology, but one which is not usable in practice. Consequently, in second step 20, the optimal interconnection topology can also be determined in terms of at least one chosen usage constraint. For example, each usage constraint may be chosen from among, at least:
a prohibition against connecting to a router Nij (with i>1) belonging to a given level i more links (connections) to the level above this given level than links (or connections) to the level below this given level (indeed, the lower links will not be able to generate sufficient traffic for all the upper links to be used). With the example definitions of variables m, w, p and h given above, this usage constraint may be expressed in a mathematical form using the inequality wi+1≦mi,
a conservation of an input and output bandwidth of each router Nij (with i>1). With the example definitions of variables m, w, p and h given above, this usage constraint may be expressed in a mathematical form using the equation facteursMini=facteursMaxi=1, where facteursMini and facteursMaxi are minimal and maximal blocking factors for level i in question (where a blocking factor is a ratio of numbers of service nodes between a level above a given level i and a level below this given level i). For example, it is possible to use the inequalities facteursMini≦pi+1*wi+1/pi*mi≦facteursMaxi and facteursMin1≦p1*w1/noeudsParFeuille≦facteursMax1. In this case, if there are only components equal to 1 for facteursMini and facteursMaxi, then 1≦p1*w1/noeudsParFeuille≦1 (for level 1, and the same for the other levels), and therefore the bandwidth is conserved. Conversely, if there are, for example, for level 1 facteursMin1=0.5 and facteursMax1=1, up to one ascending link for two descending links is permitted, and therefore there is no conservation of bandwidth. It will be noted that it would be possible to use other formulae in this case, such as, for example, facteursMini=1 and facteursMaxi=1 for all i or pi+1* wi+1=pi*mi,
a minimum number of service nodes Nij. With the example definitions of variables m, h and noeudsParFeuille given above, this usage constraint may be expressed in a mathematical form using the inequality noeudsParFeuillemi (of i=1 to h)≧minNoeuds, where minNoeuds is the number of service nodes N0j connected to leaf routers Nij of the first level (i=1), and
a maximum number of links (connections) between two routers Nij. This usage constraint enables reduced tolerance to malfunctions to be prevented if a router is defective. With the example definition of variable p given above, this usage constraint may be expressed in a mathematical form by means of the inequality 1≦pi≦maxRépétition, where maxRépétition is the maximum number of connections between two routers.
It will be noted that it is also possible to use specific usage constraints to eliminate certain special cases. This is the case, in particular, when one of the components of vector m and/or one of the components of vector w is equal to 1. Indeed, in this case it is possible to construct a similar PGFT structure by deleting one level (for a lower price, and equally resistant to malfunctions). In this case it is possible to use a usage constraint which is expressed in a mathematical form using the inequalities wi+1>1 and mi>1.
It will also be noted that the user of electronic device EE can choose, using the man-machine interface IH, each usage constraint from a list of usage constraints supplied by programming interface INP. This list can, for example, be stored in storage system MS of processing device DT.
A non-restrictive example of system S with a PGFT structure defined by (h,m,w,p)=(2, [4,2], [2,2], [1,2]) has been illustrated diagrammatically in
For the construction of the first level of routers N1j, the value of m1 is used. In this example, m1 is equal to 4 and therefore routers N2j of the second level will be connected to four different leaf routers N1j of the first level. After this the value of w1 is used. In this example, w1 is equal to 2 and therefore each leaf router N1j of the first level is connected to two routers N2j of the second level. After this the value of p1 is used. In this example, p1 is equal to 1 and therefore a single (non-redundant) link is used between the first and second levels. The term “level 1 island” is given hereinafter to a group containing four routers N1j (j=1 to 4) of the first level and two routers N2j (j=1 to 2) of the second level. In this example the system contains two level 1 islands.
After this the value of m2 is used. In this example, m2 is equal to 2 and therefore routers N3j of the third level will be connected to two routers N2j of two different level 2 islands. After this the value of w2 is used. In this example, w2 is equal to 2 and therefore each router N2j of the second level will be connected to two different routers N3j belonging to the third level (and more specifically to different level 2 islands). Since each level 1 island has two routers N2j of the second level, there are therefore two routers N3j in each level 2 island.
After this, the missing connections are completed between the second and third levels, and then, since the value of p2 is equal to 2, each connection between routers of the second and third levels is duplicated. Finally, in the case of a supercomputer S, compute nodes N0j are added, connecting them to routers N1j of the first level. In this case, the connection between each compute node N0j and corresponding router N1j is single, in order that the number of available ports on this router N1j is not exceeded.
This construction method may be implemented by the solver (or resolution system MR) for system S of
The invention is not restricted to the embodiments of determination method, processing device, and electronic device described above only as examples, but encompasses all variants which those skilled in the art might envisage solely in the context of the claims hereinafter.
It will be appreciated by one skilled in the art that the disclosed method and system described herein represent a solution to the technological problem currently faced by designers for determining (or dimensioning) the interconnection topology of a system including service nodes such as, for example, a supercomputer.
In the above description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the specification. It will be apparent, however, to one skilled in the art that the disclosure can be practiced without these specific details. In other implementations, structures and devices are shown in block diagram form in order to avoid, obscuring the description. For example, the present implementation is described in one implementation primarily with reference to user interfaces and particular hardware. However, the present implementation applies to any type of computing device that can receive data and commands, and any peripheral devices providing services.
Reference in the specification to “one implementation” or “an implementation” or “an embodiment” or “an aspect” means that a particular feature, structure, or characteristic described in connection with the implementation, embodiment or aspect is included in at least one implementation of the description. The appearances of the phrase “in an embodiment” in various places in the specification are not necessarily all referring to the same implementation.
Some portions of the detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms including “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
The present implementation of the specification also relates to an apparatus for performing the operations herein. This apparatus is specially constructed for the required purposes, or it comprises a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, including, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, flash memories including USB keys with non-volatile memory or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.
The specification can take the form of an entirely hardware implementation, an entirely software implementation or an implementation containing both hardware and software elements. In an implementation, the specification is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.
Furthermore, the description can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.
Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers.
Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
Finally, the specification is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the specification as described herein.
The foregoing description of the implementations of the specification has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the specification to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the disclosure be limited not by this detailed description, but rather by the claims of this application. As will be understood by those familiar with the art, the specification may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Likewise, the mechanisms that implement the specification or its features may have different names, divisions and/or formats. Furthermore, as will be apparent to one of ordinary skill in the relevant art, the modules, routines, features, attributes, methodologies and other aspects of the disclosure can be implemented as software, hardware, firmware or any combination of the three. Also, wherever a component, an example of which is a module, of the specification is implemented as software, the component can be implemented as a standalone program, as part of a larger program, as a plurality of separate programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future to those of ordinary skill in the art of computer programming. Additionally, the disclosure is in no way limited to implementation in any specific programming language, or for any specific operating system or environment. Accordingly, the disclosure is intended to be illustrative, but not limiting, of the scope of the specification, which is set forth in the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 1558131 | Sep 2015 | FR | national |
