Patent Application
20220050707
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Further, it should not be assumed that any of the approaches described in this section are well-understood, routine, or conventional merely by virtue of their inclusion in this section.
Graphics Processing Unit (GPU) hardware in Single Instruction Multiple Thread (SIMT) architectures often includes the capability to coalesce multiple small requests from different threads into a single larger request to improve efficiency and avoid overwhelming the memory subsystem. For example, GPU hardware may coalesce separate requests from different threads for the same operation and to contiguous memory addresses into a single larger request that is more efficiency processed.
In emerging GPU communication models that implement a SIMT architecture, threads issue network commands to access memory locations across a network in a similar manner that the threads access local memory locations. For example, threads issue remote store and retrieve commands to store data to and retrieve data from memory locations on other nodes.
GPU hardware typically includes a coalescing mechanism that combines memory accesses from different threads to reduce overhead and improve efficiency. While GPU hardware memory can be very effective for coalescing accesses to local memory locations, it is only partially effective for coalescing accesses to remote memory locations. Accesses made by threads to and from local memory to prepare command packets, e.g., to stage data in local buffers, etc., are coalesced efficiently using existing GPU hardware in the same manner that other accesses to local memory are coalesced. Once the command packets are prepared however, the GPU hardware is unable to coalesce separate network command packets into a single network command packet, even for the same operation to contiguous memory addresses on the same processing element. This results in each thread issuing separate network command packets, even for very small amounts of data, which is inefficient and can degrade performance, making it difficult to satisfy network performance requirements.
Embodiments are depicted by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements.
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that the embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the embodiments.
I. Overview
II. Sharing Next Network Operation Information
III. Determine Coalescability with Other Threads
IV. Sharing Coalescability Results with Other Threads
V. Generating Coalesced Network Command Packets
An approach is provided for coalescing network commands in a GPU that implements a SIMT architecture. Compatible next network operations from different threads are coalesced into a single network command packet. This reduces the number of network command packets generated and issued by threads, thereby increasing efficiency, and improving throughput. The approach is applicable to any number of threads and any thread organization methodology, such as wavefronts, warps, etc.
In step 102, threads share information about their next network operation with other threads. The information describes attributes of the next network operation.
In step 104, the threads determine whether their own next network operation can be combined with the next network operation of another thread. According to an embodiment, this is determined by a thread comparing the attributes of its own next network operation to the attributes of the next network operation of another thread.
In step 106, threads share with other threads the results of the determination of step 104, i.e., whether their own next network operation can be combined with the next network operation of another thread.
In step 108, one or more network command packets are issued, for example to a Network Interface Card (NIC), where each network command packet implements the next network operations of two or more threads. As used herein, the term “implements” means that when the network packet is processed by a processing element, the next network operations of the two or more threads are performed. Threads whose next network operation is implemented by a network command packet do not generate and issue a network command packet.
As previously described above, the first step in the process is for threads to share information about their next network operation with other threads. The sharing of information may be performed using different methods that may vary depending upon a particular implementation. According to an embodiment, threads share information about their next network operation with their directly adjacent upwards neighbor thread by invoking a primitive to “shuffle” data to the adjacent upwards neighbor thread.
For purposes of explanation, the term “shuffle” refers to a thread sharing data with another thread, and the term “directly adjacent upwards neighbor thread” is thread that is next highest in an ordering or sequence of threads. Threads may be ordered numerically, or alphanumerically, and have corresponding identification data that uniquely identifies the thread. For example, four threads may have corresponding thread IDs of “0,” “1,” “2,” and “3.” In this example, thread “1” is the directly adjacent upwards neighbor thread of thread “0.”
The information shared between threads specifies attributes of the thread's next network operation. According to an embodiment, the attributes include a local address on the node of the thread where data will be retrieved from or stored to, depending upon the type of operation, a remote address on another node where data will be retrieved from or stored to, depending upon the type of operation, a processing element on the other node that will perform the operation, an operation, and a size of the data. Other attributes may be included in the information, depending upon a particular implementation, and embodiments are not limited to this example. Table 1 below includes example pseudo code for shuffling next operation information to another thread.
Where “shfl_up” corresponds to the shuffle primitive, the first parameter value is the local address (“lcl”), the second parameter is the remote address (“rmt”), the third parameter is the processing element (“pe”) that will perform the operation, e.g., a particular node and/or process, the fourth parameter is the operation (“op”), and the fifth parameter is the size of the data (“size”). The “1” parameter specifies that the information is to be provided to the directly adjacent upwards neighbor thread, i.e., the next higher thread in the sequence or order.
After threads have shared the attributes of their next network operation with another thread, each thread determines whether its next network operation can be combined with the next network operation of another thread, i.e., determines its coalescability with another thread. To determine its coalescability with another thread, a thread compares the attributes of its next network operation with the attributes of the next network operation of another thread to determine whether the two operations can be combined, i.e., implemented by a single network operation. For example, the next network operations for two threads that are both remote store operations to store that same amount of data to contiguous memory addresses on the same processing element may be combined, i.e., implemented by a single network operation, by changing the value of the size parameter in the network command packet. Thus, the network operations for the two threads can be implemented by a single network command packet issued by one of the threads instead of each thread issuing its own network command packet. On the other hand, next network operations that are for different operations, operations to different processing elements, or for different size data or non-contiguous memory addresses cannot be implemented by a single network operation, without providing additional information, and require separate network command packets be generated and issued, one for each thread.
According to an embodiment, each thread determines its coalescability with its adjacent lower neighbor thread, i.e., whether its next network operation can be combined with the next network operation of its adjacent lower neighbor thread, which is the thread having the next lower thread ID in a thread sequence. A thread makes this determination by comparing the attributes of its own next network operation with the attributes of the next network operation of its adjacent lower neighbor thread, using the data shuffled up from the adjacent lower neighbor thread.
Table 2 below includes example Boolean logic implemented by a thread for determining whether the next network operation of the thread can be combined with the next network operation of its adjacent lower neighbor thread into a single network operation. Boolean logic may be efficiently processed by threads and the result stored in memory, such as a register.
Two next network operations are coalescable when all of the conditions of the above Boolean logic are true. Threads with an inactive or absent adjacent lower neighbor thread are not coalescable with their adjacent lower neighbor thread. Threads may be deactivated, for example, to process conditional branches. The size parameter ensures that the size of data to be processed by both threads is the same. The source (src) and destination (dst) address parameters ensure that the addresses are contiguous. The operation (op) and processing element (pe) parameters ensure that the next network operation of both threads will process the same operation on the same processing element. The last variable tests whether the current thread is the lowest thread in the sequence, i.e., thread 0 in
In the examples described herein, thread 0 is designated as not coalescable to ensure that at least one thread will generate and issue a network command packet. Otherwise, if all threads were determined to be coalescable, no network commands would be generated and issued. Embodiments are not limited to the example of thread 0 being designated as not coalescable and any thread may be designated as not coalescable. Also, the highest thread in the sequence, e.g., thread 3, may be configured to shuffle the information that specifies attributes of its next network operation information to the lowest thread in the sequence, e.g., thread 0. For example, a group of threads may include 20 threads, designated as thread 0 through thread 19, with thread 10 designated as not coalescable and thread 19 configured to share its next network operation information with thread 0.
In the example of
After threads have determined the coalescability with their adjacent lower neighbor thread, the results are shared with other threads. This may be accomplished, for example, by each thread sharing its Boolean value as a predicate to a ballot voting function such as lowerNeighborCoal=_ballot(coalescable). The results may be represented by a plurality of bits that is stored locally, for example in a register, where each bit in the plurality of bits corresponds to a particular thread and the bit value specifies whether the corresponding thread is coalescable with its adjacent lower neighbor thread. Embodiments are described hereinafter in the context of the plurality of bits in the form of a bit vector for purposes of explanation, but embodiments are not limited to a bit vector per se, and other bit organization methodologies may be used.
In the example of
After all the threads have populated the bit vectors to specify whether they are coalescable, all of the threads have a consistent view of which threads are coalescable with their adjacent lower neighbor thread. Each thread examines their respective bit vector to determine whether they need to generate and issue a network command packet and if so, which network operations need to be implemented by the network command packet. Threads that are not coalescable generate and issue a network command packet that implements its own next network operation and the next network operations of all the coalescable threads in the bit vector between itself and the next thread that is not coalescable. Threads that are coalescable do not generate and issue a network command packet since their next network operation will be implemented by a network command packet generated and issued by another thread and more specifically, by the next lower thread in the sequence that is not coalescable. According to an embodiment, all threads perform this process in parallel since they are part of the same wavefront.
In the example depicted in
All the threads between thread 1 and thread six, i.e., threads 2-5, have a bit value of “1” indicating that they are coalescable, so the network command packet generated and issued by thread 1 implements the next network operation of thread 1 and the next network operations of threads 2-5. Thread 7 is coalescable with thread 6 but thread 8 is not coalescable with thread 7, so thread 6 generates and issues a network command packet that implements its own next network operation and the next network operation of thread 7. Thread 8 is not coalescable and threads 9-15 are coalescable with thread 8, so thread 8 generates and issues a network command packet that implements its own next network operation and the next network operation of threads 9-15. So, using the thread coalescing approach described herein, only four network command packets are generated and issued, by threads 0, 1, 6 and 8, instead of 16 separate network command packets using conventional approaches.
Network command packets that implement the next network operation of multiple threads include parameter values that “cover” all of the next network operations.
Table 3 below includes example pseudo code for generating and issuing a network command packet for a thread that is not coalescable.
The approach described herein for coalescing next network operations from different threads into a single network command packet can provide significant performance improvements by reducing the number of network command packets generated and issued by threads. Although embodiments are described herein in the context of the network operation of each thread operating on the same size data, this is a common scenario in GPU-SIMT architectures. The approach may be extended to threads operating on different size data by including a mechanism for coalesced threads to share their respective data size with the thread to which they are being coalesced. In the prior example, threads 1 and 2 would share their respective data size with thread 0 to enable thread 0 to include correct memory address values in the network command packet. The approach described herein for coalescing network command packets may be selectively instantiated, for example, via a configuration parameter or control value. The approach may also be instantiated in response to runtime information that indicates that it is desirable to use the approach.
