1. Field of the Invention
The present invention relates to message broadcasting in large computing system and in particular to an algorithm that achieves optimal results in message broadcasting in a large computing system.
2. Description of Background
Large computing environments often include a network of computers that are in processing communication with one another. The networked computers can be loosely coupled forming a computer cluster. In such arrangements the loosely coupled computers work together closely so that in many respects they can be viewed as one computer. In addition, many such large environments provide parallel processing capabilities. The term parallel processor is sometimes used for a computer with more than one processor, available for parallel processing. Systems with thousands of such processors are known as massively parallel.
There are many different kinds of parallel computers or parallel processors. They are distinguished by the kind of interconnection between processors, known as processing elements (PEs) and between processors and memories. Parallel processor machines are also divided into symmetric and asymmetric multiprocessors, depending on whether all the processors are capable of running all the operating system code and, say, accessing I/O devices or if some processors are more or less privileged.
Computer clusters and parallel processing is used both to achieve greater speed better performance. However, the speed and performance of the system greatly depends on how data is processed and transmitted during computing operations. In this regard, message broadcasting has become of utmost importance in the design of large computing systems. Broadcasting in a computer network refers to transmiting a packet that will be received (conceptionally) by every device on the network. The packets are usually small and of fixed size. At parallel application level, broadcasting refers to transmiting a message from one process to other processes running on the parallel computers. The messages can be large and consist of multiple packets.
A particular challenge in the design of large computing environments, and particulalry in parallel computing, is to use algorithms that are often essentially sequential in nature in a cooperative fashion to achieve the desired parallel result. Most algorithms must be completely redesigned to be effective in parallel environments. This is particularly true in circumstances where multiple copies of the same program may interfere with one another.
As mentioned earlier, broadcasting is widely used by parallel applications in such environments and therefore the prior art has struggled with providing algorithms that can efficiently boradcast a large message among the group of processes. Unfortunately, the prior art algorithms currently being used either do not provide an optimal communication schedule or the scheduling is not practical for implementation of large message broadcasts on different kind of processes such as power of two and non power of two processes. For example the implementation of Scatter-Allgather algorithm is straightforward, but it does not fully and optimally uses the available bandwidth. By contrast, Edge-disjoint Spanning Binnomial Tree algorithm performs better but has a very complex structure and only works on power of two processes. Another algorithm that works both owth power of two and non power of two processes is Partition_exchange algorithm but while the scheduling is simple for the communication content part, the communication partner determination is highly complicated. Consequently, a novel algorithm for broadcasting large messages is needed that can overcome the above mentioned shortcomings of the prior art.
The shortcomings of the prior art are overcome and additional advantages are provided through the method and associated program storage device readable by a machine embodying the method for message broadcasting in a parallel computing environment is enclosed. The method comprises the steps of first performing a set up process phase by first determining how the message is sliced to be broadcasted and then establishing parent-child relationship among processes such that parent will be responsible to pass any message received to said child. Thereafter, ensuring that all non-root processes get one slice of the message and pass it along to their designated children and performing a pipelining process phase consisting of multiple sub-steps during which broadcasted message slices are further pipelined by establishing partner relationship among processes and having each process exchange a slice of message with its partner based on preselected criteria.
Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. For a better understanding of the invention with advantages and features, refer to the description and to the drawings
The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
The present invention provides for a novel algorithm for broadcasting large messages. The algorithm provides an optimal communication schedule and the scheduling is practical for implementing large message broadcast in communication libraries. Specially, the algorithm performs optimal broadcasting of large messages on both power of two and non power of two processes.
In MPI_BCAST, the process that has the message initially is called the root. The root sends the message to a group of processes. For the convenience of discussion, we assume process 0 is the root. At the end of MPI_BCAST, every process has a copy of the message. We discuss single large message broadcast algorithms only in the present instance to keep the understanding of the discussion relatively simple with the understanding that the same idea can be applied to broadcasting a series of messages from the same source.
To efficiently broadcast a large message among the group of processes and nodes 110 (of
The differences among those algorithms are their pipeline schedules and scheduling methods, i.e. how schedules are constructed. According to the pipeline schedule, a process can determine at each communication step: 1) its communication partners, i.e. the source process of the incoming slice and the target process of the outgoing slice, and 2) the communication content, i.e. which slice comes in and which slice goes out. An optimal schedule is the key to performance while low scheduling complexity and overhead make an algorithm practical to be implemented and incorporated in communication libraries such as MPI.
It may be useful to carefully examine the workings of Scatter-Allgather, Edge-disjoint Spanning Binomial Tree and Partition-Exchange algorithms previously discussed briefly to achieve a better understanding of the workings of the present invention.
In a Scatter-Allgather algorithm often used for large message MPI_BCAST, the root split the message up into P slices, where P is the number of process in the group, including the root. It then scatters the entire message out to all P participating processes by every process calling MPI_SCATTER. As a result of the scatter, every process gets at least one slice of the message. Then the scattered slices are collected back at each process by calling MPI_ALLGATHER. In the MPI_ALLGATHER, each of the P processes acts as if it has only one distinctive slice after the scatter and it contributes this distinctive slice to the group.
This algorithm is easy to implement. The schedulings in both the scatter phase and the allgather phase are simple and straightforward. However, the performance of this algorithm is not optimal because available bandwidth is not fully utilized in the scatter phase and there are redundant data transfers in the allgather phase. The communication partner relationship in each step is shown in the following table for an example of broadcasting on 4 processes using this algorithm. The communication content can be seen in the following table which shows the slices available at each process. Process 0 is the root and there are 4 processes in the group. Table 1 and 2 below, show an example of this:
The n Edge-disjoint Spanning Binomial Trees (nESBT) algorithm was developed for hypercubes systems. Its idea is to construct nESBT graph by merging n spanning binomial trees (SBT). The SBTs are extracted from the n-cube and then rotated certain number of times. The root sends slices to the SBTs in a round robin manner and each slice is broadcasted along one SBT. This algorithm performs better than the Scatter-Allgather algorithm but the implementation could be highly complicated. First, the nESBT graph construction is very complex. Secondly, the communication partner part of scheduling is simple but to determine communication content during each step, nESBT graph traverse is required, which results in high complexity and overhead. This algorithm can be ported to other platforms but only works on power of two processes.
Another algorithm deals both power of two and non power of two processes. It's called Partition-Exchange algorithm in this application. The idea is to partition the P processes in to several subsets during each communication step. There are logP subsets for power of two processes and └logP┘+3 subsets for even number non power of two processes. For odd number non power of two processes, a dummy process is added into the group.
The dummy process does not participate in message passing and the algorithm acts as if there are P+1 processes. In each step, a process is paired up with another process from a different subset and slices are exchanged between the two. What slices a process has received previously determines which subset it belongs to and in turn decides what slice it should send and receive during the current step.
For power of two processes, the communication schedule is essentially the same as the nESBT algorithm but constructed differently. Unlike the nESBT algorithm, the communication content part of the scheduling is simple but the communication partner determination is highly complicated, especially for non power of two processes. The nESBT and the Patition-Exchange algorithms perform better theoretically than the Scatter-Allgather algorithm. However their schedulings are impractical for MPI implementations.
Referring back to
Both the communication partner and the communication content are easily determined with the new scheduling by simply checking the binary representation of the process ID. For power of two processes, the communication schedule generated by this algorithm is relatively similar to that as discussed in conjunction with the nESBT algorithm and the Partition-Exchange algorithm, but with novel, low complexity and low overhead scheduling. For non power of two processes, both the communication schedule and the scheduling are very different from prior art solution.
For broadcasting among P (P is power of two) processes, as illustrated in the flowchart illustration of
In the setup phase 200, the first n=logP slices of the message are passed along a binomial tree 220 whose parent-children relationship is defined as follows:
Definition 1:
for i=(in−1 in−2 . . . ir . . . i0) and r satisfies:
ir=1 and in−1=in−2= . . . =ir+1=0 if i ≠0; or r=−1 if i=0,
the parent and children of process i are:
Child (i, s)={(in−1 . . .
Parent (i)=(in−1 in−2 . . .
According to the definition, process 0 has n children. As per workings of the invention, it sends out the first n slices, one to each of its children as shown at 225. Each process, except for process 0, gets one slices from its parent in the setup phase. Once received a slice, a process sends it to each of its children, one by one, as shown at 226.
Consequently for a process i, process i=(in−1 in−2 . . . ir . . . i0) expects from its parent slice t where t satisfies: it=1 and it−1=it−2= . . . =i0=0. This setup takes n steps and at the end, every non-root process gets one slice of the message (227).
The pipeline phase 300 also consists of several steps. Similar to the setup phase. During this phase the parent-child relationship is replaced by a partner-shipping of pairs as depicted by process step 320. This partnership can be achieved in a number of ways as will be discussed in detail below. Note that the communication partner and the communication content can be easily determined by checking the binary representation of the process ID as shown at 310.
Fore each process, once a partner is determined as referenced at 320, then at least one slice of the message is then exchanged with the partner as shown at 325 until the process is completed as depicted by 327.
That is to say that for example, in step k, process i exchanges one slice with a partner process. The partner is determined by flipping bit k % n of i:
Definition 2:
for i=(in−1 in−2 . . . ir . . . i0), Partner (i, k)=(in−1 in−2 . . .
For example, when process 0 is the partner, the process receives slice k+n from process 0 if k<q−n, or slice q−1 otherwise.
Process 0, on the other hand, sends to its partner slice k+n or slice q−1 if k>=q−n.
When neither i nor Partner(i, k) is process 0, i sends slice k+s(i, k) to Partner(i, k) and receives slice k+t(i, k) from Partner(i, k). If k+s(i, k) or k+t(i, k)>=q, then slice q−1 is sent or received instead. s(i, k) and t(i, k) are given by:
Definition 3:
for i=(in−1 in−2 . . . ir . . . i0), s(i, k) satisfies
An example of the algorithm is shown in Tables 3 and 4 below:
Communication Content: slices received after each steps on power of two processes, P = q = 8
A simple approach is used to extend the algorithm to non power of two processes. The idea is to pair up the processes into power of two pairs, self-pair allowed, and then follow the algorithm as if each pair is one process. When broadcasting on P′ processes, where P <P′<2*P, we first define a simple scheme that pairs up processes i≧P with process 1 to P′−P:
Definition 4: for any process i,
Process i participates in the setup phase if and only if i=Rep(i). The setup phase is exactly the same as in the power of two processes case. In the pipeline phase, Rep(i) is used instead of i in partner calculation. During each step of the pipeline, process i and Pair(i) cooperate to accomplish slice exchanging with their partner or partners. One of the pair sends the outgoing slice to the partner and is called the output of the pair. The other, labeled input of the pair, receives the incoming slice from the partner. Note the sources of the incoming slice and the target of the outgoing slice are different if the Partner(i, k) ≠ Pair(Partner(i, k)). The input also passes a slice it received during previous steps to the output.
More specifically, at the beginning of the pipeline, when i ≠Pair(i), process i is labeled as the output of the pair if i=Rep(i), and Pair(i) is the input. Otherwise it is the other way around. If i=Pair(i), i is both the output and the input. During step k of the pipeline, the output sends slice k+s(Rep(i), k) to the input of the partner pair {Partner(Rep(i), k), Pair(Partner(Rep(i), k))} if k<q−s(Rep(i), k). If k≧q−s(Rep(i), k), it sends slice q−1 instead. The input of the pair receives slice k+t(Rep(i), k) from the output of the partner pair if k<q−t(rep(i), k). It receives slice q−1 if otherwise. When k >0, the input also sends slice k−1 to the output. A process's role in the pair may change. If Rep(i)k=1, the input and output of {i, Pair(i)} switch roles after step k. To determine which of {i, Pair(i)} is the output, we define u(i, k) and v(i, k):
Definition 5: u(i, k) is the number of 1s in binary representation of i from bit 0 to bit k. v(i, k) is the number of role switches before step k (k>0), and v(i, k) is given by:
v(i, k)=└k/n┘*u(i, n−1)+u(i, k % n).
According initial role assignment, if v(i, k) is a odd number, then process i is the input of the pair if i=Rep(i) and Pair(i) is the output. Otherwise it is the other way around. Finally, after q−1 steps of the pipeline, the input of the pair sends slice q−2 to the output and the output send slice q−1 to the input, if i≠ Pair(i). Tables 5 and 6 depict an example of the algorithm.
Communication Content: slices receive at each step with the scheduling on non power of two processes, P = q = 6, 1 is paired up with 4, 2 is paired up with 5
While the preferred embodiment to the invention has been described, it will be understood that those skilled in the art, both now and in the future, may make various improvements and enhancements which fall within the scope of the claims which follow. These claims should be construed to maintain the proper protection for the invention first described.