1. Technical Field
The present invention relates in general to data processing systems and, in particular, to improvements in multiprocessor data processing systems.
2. Description of the Related Art
In parallel computing environments, the Reduce operation is frequently used to combine multiple input vectors of equal length utilizing a desired mathematical or logical operation to obtain a result vector of the same length. As an example, the reduction operation can be a vector summation or a logical operation, such as a maximum. A long term profiling study has indicated that a conventional application having parallel processing capabilities may spend 40% or more of the cycles used for parallel processing to perform Reduce operations.
The Message Passing Interface (MPI) is a language-independent communications protocol commonly used to program parallel computers. MPI defines several collective communication interfaces for Reduce operations, such as MPI_REDUCE and MPI_ALLREDUCE. In MPI, the MPI_REDUCE operation is a global operation across all members of a process group defined by the MPI communicator. The MPI Reduce operation returns the result vector at one process specified by the user, known as the root. In the following discussion, it will be assumed that a commutative Reduce operation is employed and the N participating processes are ranked from 0 to N−1, with the root having a rank of 0.
To reduce vectors having short lengths, the best known MPI_REDUCE algorithm is the Minimum Spanning Tree (MST) algorithm, in which the root process is the root of a MST consisting of all participating processes. A process in the tree first receives vectors from all of its children, combines received vectors and its own input vector, and then sends the result vector to its parent. This bottom-up approach continues until the root produces the result vector by combining its input vector and all input vectors of its children.
The MST algorithm does not work as well, however, if the vectors are long. Efficient long vector algorithms for MPI_REDUCE focus on better exploitation of available network bandwidth and processor computing power, instead of minimized message start-up and latency overhead. One of the algorithms widely adopted for reduction of long vectors is the Recursive Halving Recursive Doubling (RHRD) algorithm, in which each of the N participating processes first takes log(N) steps of computation and communication.
During each step k, process i and process j exchange half of their intermediate reduction results of step (k−1), where j=i^ A mask, a carat (^) denotes a bitwise exclusive-OR operation, and mask is the 1's binary representation left shifting (k−1) bits. If i<j, process i sends the second half of the intermediate result, receives the first half of the intermediate result, and combines the received half with the half that was not sent. This procedure continues recursively, halving the size of exchanged/combined data at each step, for a total of log(N) steps. At the end of the steps, each process owns 1/N of the resulting vector: process 0 owns the first 1/N, process 1 owns the second 1/N, process i owns the (i+1)th 1/N and so on.
All processes then perform a gather operation to gather the pieces back by the root process from other processes. The gather operation also consists of log(N) steps, for a total Reduce operation length of 2*log(N) steps. During step k of the gather, process i sends the partial results it has to process j, where both i and j are less than N/2k,j<i,j=^mask and mask is the 1's binary representation left shifting (k−1) bits. These steps continue recursively, doubling the size of the data passed at each step. Finally, the root process 0 obtains the final result of the reduce operation of the entire vector.
The above description of RHRD algorithm applies to cases where N is an integer power of two. When N is not an integer power of two (as is common), the RHRD algorithm includes an extra preparation step prior to the exchange of intermediate results. In the preparation step, processes from N′ to N−1 send their input vectors to processes from 0 to r−1, where r=N−N′ and N′ is the largest integer power of two less than N. Specifically, process i sends its vector to process i−N′ if i>N′. Processes from 0 to r−1 then perform local reduce operations on the received vectors and their own input vectors and will use the results as their input vectors in the above-described algorithm. Processes from N′ to N−1 do not participate in the remainder of the MPI_REDUCE operation, and processing by the rest of the processes remains the same as in the case in which N is an integer power of two.
The conventional RHRD algorithm does not scale well. Modern parallel high performance computing applications often require MPI_REDUCE operations on tens of thousands or even hundreds of thousands of processes. Even When the vectors are lone (e.g., 1 MB), the RHRD algorithm breaks the vector into very small chunks, e.g., 64 bytes when N=16K, since each of N′ processes gets 1/N′ th of the final result vector at the end of the recursive halving and before the recursive doubling. With a large number of short messages each carrying a small chunk of the vector, message start-up overhead is high and available network link bandwidth is not fully utilized.
In one embodiment, a parallel processing data processing system builds at least one data structure indicating a communication schedule for a plurality of processes each having a respective one of a plurality of equal length vectors formed of multiple equal size chunks. The data processing system, based upon the at least one data structure, communicates chunks of the plurality of vectors among the plurality of processes and performs partial reduction operations on chunks in accordance with the communication schedule. The data processing system then stores a result vector representing reduction of the plurality of vectors.
With reference now to the figures, wherein like reference numerals refer to like and corresponding parts throughout, and in particular with reference to
As utilized herein, the term processing node (or simply node) is defined as the set of computing resources that form the domain of a coherent operating system (OS) image. For clarity, it should be understood that, depending on configuration, a single physical system may include multiple nodes. The number of processing nodes 102 deployed in a given system is implementation-dependent and can vary widely, for example, from a few nodes to many thousands of nodes.
Each processing node 102 may be implemented, for example, as a single integrated circuit chip (e.g. system-on-a-chip (SOC)), multi-chip module (MCM), or circuit board, which contains one or more processing units 104 (e.g., processing units 104a, 104b) for processing instructions and data. Each processing unit 104 may concurrently execute one or more hardware threads of execution.
As shown, each processing unit 104 is supported by cache memory 112, which contains one or more levels of in-line or lookaside cache. As is known in the art, cache memories 112 provide processing units 104 with low latency access to instructions and data received from source(s) within the same processing node 102a and/or remote processing node(s) 102b. The processing units 104 within each processing node 102 are coupled to a local interconnect 114, which may be implemented, for example, with one or more buses and/or switches. Local interconnect 114 is further coupled to HFI 120 to support data communication between processing nodes 102.
As further illustrated in
Those skilled in the art will appreciate that processing system 100 of
Referring now to
Each physical system 200 further includes an instance of a hypervisor 202 (also referred to as a Virtual Machine Monitor (VMM)), such as hypervisor 202a and hypervisor 202b. Hypervisor 202 is a program that manages the full virtualization or paravirtualization of the resources of physical system 200 and serves as an operating system supervisor. As such, hypervisor 202 governs the creation and destruction of nodes and the allocation of the resources of the physical system 200 between nodes.
At least some (and in the illustrated embodiment, all) of operating systems 204 include and/or support a message passing interface (MPI) 206, which may each comply, for example, with the MPI 1.2 (commonly referred to as MPI-1) or MPI 2.1 (commonly referred to as MPI-2) standard promulgated by the MPI Forum. As will be appreciated, each MPI 206 provides a set of routines callable from applications 208 of any language (e.g., C, C++, Fortran, Java, etc.) capable of interfacing with the MPI routine libraries. The library routines of MPI 206 can include computation routines (e.g., MPI_REDUCE), message passing, data synchronization and process management routines, as well as other routines supporting parallel programming.
In accordance with a preferred embodiment, MPI 206 implements an MPI_REDUCE routine in a manner that utilizes communication links of data processing system 100 in accordance with a predetermined schedule in order to complete the communication between processes efficiently and without contention. In particular, the MPI_REDUCE process decomposes a plurality of vectors into M chunks of equal size and pipelines each chunk individually along log(N) different pipelines consisting of N processes in parallel. The MPI_REDUCE routine is efficient because communication on the pipelines is scheduled to be non-conflicting and up to all available network links are utilized. The MPI_REDUCE routine is also scalable because the number of chunks M is not dependent on the number of the processes, but can instead be selected based on trade-off between fully utilizing the bandwidth of any network link and utilizing as many network links as are available.
With reference now to
The process of
MPI 206 determines at block 310 whether integer N is an integer power of two. If so, N′, a number of “units” each containing at least one process, is equal to N, and the process shown in
In an exemplary embodiment, the data structures built at block 320 include two integer arrays each including log(N′) elements, which are referred to herein as BR (binary representation) and DIS (distance). For a process i, BR[k] stores the value of bit k in the binary representation of integer i, and DIS[k] stores the distance between bit k and the first non-zero bit to its right in the binary representation of integer i. If bit k is the rightmost bit in the binary representation of integer i, then k=0, or if bits to the right of bit k are all zeros, then define DIS[k]=k+log(N′)−l, where bit l is the leftmost nonzero bit in the binary representation of i. Furthermore, if i=0, then BR[k]=0 and DIS[k]≡log(N′) for all 0<k<log(N′). For each process belonging to a multi-process unit, the process additionally builds a third integer array SW (switch) of log(N′) elements. For unit i and 0<k<=log(N′), SW[k] records the number of nonzero bits between bit 0 and bit k in i's binary representation.
As indicated at block 322 of
Referring now to
The method depicted in
In a one-process unit, the single process is considered to be both the input process and the output process of the unit. In cases in which N is not an integer power of two, meaning that at least one unit contains multiple processes, a preferred embodiment groups processes such that units I through r each have two processes and other units each have one process. More specifically, in an exemplary embodiment, unit i consists of process i and N′+i−1 if 0<i<=r. For simplicity, if 0<i<=r, it will be assumed that process i will be the initial output process of the unit and process N′+i−1 be the input process.
In cases in which N is not an integer power of two, the input process of each multi-process unit sends its chunk 0 to the output process of the same unit, as also shown at block 402. In response to receiving the chunk 0 of the input process, the output process performs the reduction operation on the received chunk and its own chunk 0. Of course, for units containing a single process, no communication of chunk 0 or partial reduction is performed.
At block 404 of
j=i^mask and mask is 1's binary representation left shifting k′ bits
s=k−BR[k′]*DIS[k′]
t=k−(1−BR[k′]*DIS[k′]
Block 408 illustrates that the process then takes possible action depending upon its role in the present step k. The actions taken by an input process are depicted at blocks 409-412, and the actions taken by an output process are depicted at blocks 413-416. Referring first to blocks 409-410, if the process is an input process, the process receives chunk t from the output process of unit j and performs the specified partial reduction operation on the received chunk t and its local intermediate result, unless t<0 or t>M−1 in which case the receive operation and partial reduction operation are skipped. As further indicated at blocks 411-412, the process also sends chunk k+1 to the output process of unit i, unless the input process is the root process, when k+1<M−1 (block 412).
As indicated at blocks 413-414, if the process is an output process and not the root process, the process sends chunks s to the input process of unit j during step k, unless s<0 or s>M−1 in which case the sending operation is skipped. This output process also receives chunk k+1 from the input process of unit i, when k+1<M−1 (block 416).
Following any of blocks 411, 412 or block 416, the process determines at block 418 whether or not step index k has reached log(N′)+M−1. If so, the method depicted in
In one preferred embodiment, the two processes within a given multi-process unit recalculate roles before the scheduled communication and computation for the step k if BR[k′]=1 and otherwise keep their roles from the previous step. To recalculate roles for step k, a process calculates two values u and v by:
u=SW[ log(N′)−1]*(k/log(N′))+SW[k′]
v=u+k/log(N′)+1
With these two values, the roles of the processes in the two-process unit i and its partner unit j can then be determined by:
Returning now to
Referring now to
Table I below shows the unit organization and the scheduling arrays built at step 320 of
Table II additionally summarizes the calculated values (u, v, s, t, j) for each process during each or the steps of the exemplary reduction operation.
As has been described, in at least one embodiment, a plurality of vectors each comprising M equal size chunks is reduced through parallel pipelined processing by N processes. During each of log(N)+M steps of processing (for N an integer power of two), a process may send one chunk of the vector (original input data or intermediate result of the reduce operation) and may receive a different chunk. The communication of the chunks follows a predetermined communication schedule such that a maximum of one chunk is sent to the same process during any given step and most processes are involved in both sending and receiving chunks.
While various embodiments have been particularly shown as described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the claims. For example, although aspects have been described with respect to a computer system executing program code that directs the functions of the present invention, it should be understood that present invention may alternatively be implemented as a program product including a computer readable storage medium (e.g., flash memory, DRAM, hard disk, optical disk) storing program code that can be processed by a data processing system. In addition, although a Reduce operation has been described herein with reference to the specific example of an MPI_REDUCE operation, it should be understood that the Reduce operation disclosed herein is equally applicable to other implementations of Reduce operations employing parallel processing. Further, although the disclosed Reduce operation builds two or three integer arrays governing the inter-process communication schedule, it should be understood that one or more other types of data structures can alternatively be employed and that the inter-process communication schedule can be governed by at least one data structure.
This invention was made with United States Government support under Agreement No. HR0011-07-9-0002 awarded by DARPA. THE GOVERNMENT HAS CERTAIN RIGHTS IN THE INVENTION.
Number | Name | Date | Kind |
---|---|---|---|
5533169 | Burnod et al. | Jul 1996 | A |
6023751 | Schlansker et al. | Feb 2000 | A |
6104374 | Sullivan et al. | Aug 2000 | A |
6212617 | Hardwick | Apr 2001 | B1 |
6292822 | Hardwick | Sep 2001 | B1 |
6788302 | Ditlow et al. | Sep 2004 | B1 |
6895115 | Tilton | May 2005 | B2 |
7313582 | Bhanot et al. | Dec 2007 | B2 |
7376812 | Sanghavi et al. | May 2008 | B1 |
7451278 | Feind et al. | Nov 2008 | B2 |
7937559 | Parameswar et al. | May 2011 | B1 |
8041551 | Pakyari et al. | Oct 2011 | B1 |
20020181390 | Mody et al. | Dec 2002 | A1 |
20020198709 | Takagi | Dec 2002 | A1 |
20060041610 | Hokenek et al. | Feb 2006 | A1 |
20080092124 | Archambault et al. | Apr 2008 | A1 |
20080127203 | Huang | May 2008 | A1 |
20080148013 | Jia | Jun 2008 | A1 |
20080183779 | Jia | Jul 2008 | A1 |
20080301683 | Archer | Dec 2008 | A1 |
20090007140 | Jia | Jan 2009 | A1 |
20090064176 | Ohly et al. | Mar 2009 | A1 |
20090089544 | Liu | Apr 2009 | A1 |
Number | Date | Country | |
---|---|---|---|
20110060891 A1 | Mar 2011 | US |