Patent Application
20090320027
Scheduling multithreaded computations on multiprocessor machines has been the source of much research. To execute multithreaded computations, the operating system runs a collection of kernel-level processes, one per processor, and each of these processes controls the execution of multiple computational threads created dynamically by the executed program. The scheduling problem is that of dynamically deciding which thread is to be run by which process at a given time, so as to maximize the utilization of the available computational resources (processors). Thread is short for thread of execution. A thread is a sequence of instructions which may execute in parallel with other sequences of instructions.
Most of today's multi-processor machines run programs in a multi-programmed mode, where the number of processors used by a computation grows and shrinks over time. In such a mode, each program has its own set of processes, and the operating system chooses in each step which subset of these processes to run, according to such factors as the nature, size, and number of processors available for that program at the time, the amount of work load waiting to be performed, the nature, size, and number of memory modules, and so on. Therefore the scheduling algorithm is dynamic (as opposed to static): at each step it schedules threads onto processes, without knowing which of the processes or work tasks are going to be run.
When a program is executed on a multiprocessor machine, the threads of computation are dynamically generated by the different processes, implying that the scheduling algorithm generated in a scheduler has processes load balance the computational work in a distributed fashion. The challenge in designing such distributed work scheduling algorithms is that performing a re-balancing, even between a pair of processes, requires the use of costly synchronization operations.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
As provided herein, systems and methods for scheduling parallel processing or concurrent work loads across processors are illustrated. The objective is to distribute what could possibly be made parallel and distribute the work (e.g., processor executions) among processor cores available in the system thereby improving processor performance.
Methods and systems are disclosed which synchronize concurrent processing for claiming operations, such as push, pop and/or steal operations, between locally owning processors of a data structure and foreign processors that are among the multiple processors on a multi-processor machine. The data structure is a lock free, wait free, structure that elides/omits the majority of fences in a work stealing pop algorithm (e.g., a foreign processor (e.g., a processor not owning the data structure) stealing work from other processors by removing work items (steal operation) and/or work descriptors off a local processor). Work stealing is a many-core parallelism technique where processors without tasks assigned to be processed will dynamically search for tasks to steal and execute for work scheduled on other processors with the result of efficient and effective dynamic load-balancing across all processing resources. Popping/pop operation refers to taking or grabbing work off of a data structure by a local processor. Stealing/steal operation is what foreign processors do (e.g., the act of trying to grab the least recently queued item). For example, stealing refers to a foreign processor grabbing work off of a local processor's data structure.
To the accomplishment of the foregoing and related ends, the following description and annexed drawings set forth certain illustrative aspects and implementations. These are indicative of but a few of the various ways in which one or more aspects may be employed. Other aspects, advantages, and novel features of the disclosure will become apparent from the following detailed description when considered in conjunction with the annexed drawings.
The claimed subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the claimed subject matter. It may be evident, however, that the claimed subject matter may be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order to facilitate describing the claimed subject matter.
Parallel processing comprises the use of more than one CPU to execute a program. Ideally, parallel processing makes a program run faster because there are more engines (CPUs) running it. In practice, it is often difficult to divide a program in such a way that separate CPUs can execute different portions without interfering with each other. Therefore, methods of improving the algorithm for allowing processors that have free resources to take work from processors that are not free are illustrated herein.
In other embodiments, device 112 may include additional features and/or functionality. For example, device 112 may also include additional storage (e.g., removable and/or non-removable) including, but not limited to, magnetic storage, optical storage, and the like. Such additional storage is illustrated in
The term “computer readable media” as used herein includes computer storage media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions or other data. Memory 118 and storage 120 are examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by device 112. Any such computer storage media may be part of device 112.
Device 112 may also include communication connection(s) 126 that allows device 112 to communicate with other devices. Communication connection(s) 126 may include, but is not limited to, a modem, a Network Interface Card (NIC), an integrated network interface, a radio frequency transmitter/receiver, an infrared port, a USB connection, or other interfaces for connecting computing device 112 to other computing devices. Communication connection(s) 126 may include a wired connection or a wireless connection. Communication connection(s) 126 may transmit and/or receive communication media.
The term “computer readable media” may include communication media. Communication media typically embodies computer readable instructions or other data in a “modulated data signal” such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” may include a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
Device 112 may include input device(s) 124 such as keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, and/or any other input device. Output device(s) 122 such as one or more displays, speakers, printers, and/or any other output device may also be included in device 112. Input device(s) 124 and output device(s) 122 may be connected to device 112 via a wired connection, wireless connection, or any combination thereof. In one embodiment, an input device or an output device from another computing device may be used as input device(s) 124 or output device(s) 122 for computing device 112.
Components of computing device 112 may be connected by various interconnects, such as a bus. Such interconnects may include a Peripheral Component Interconnect (PCI), such as PCI Express, a Universal Serial Bus (USB), firewire (IEEE 1394), an optical bus structure, and the like. In another embodiment, components of computing device 112 may be interconnected by a network. For example, memory 118 may be comprised of multiple physical memory units located in different physical locations interconnected by a network.
Those skilled in the art will realize that storage devices utilized to store computer readable instructions may be distributed across a network. For example, a computing device 130 accessible via network 128 may store computer readable instructions to implement one or more embodiments provided herein. Computing device 112 may access computing device 130 and download a part or all of the computer readable instructions for execution. Alternatively, computing device 112 may download pieces of the computer readable instructions, as needed, or some instructions may be executed at computing device 112 and some at computing device 130.
Various operations of embodiments are provided herein. In one embodiment, one or more of the operations described may constitute computer readable instructions stored on one or more computer readable media, which if executed by a computing device, will cause the computing device to perform the operations described. The order in which some or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering will be appreciated by one skilled in the art having the benefit of this description. Further, it will be understood that not all operations are necessarily present in each embodiment provided herein.
Referring now to
In other realizations, the data structure 206 may be encoded in the shared memory portion 208 and local memory section 218 and 220 may reside in or on the same physical structures. Similarly, shared memory portion 208 does not need to correspond to a single physical structure. Shared memory portion 208 may correspond to a collection of sub-portions that can be associated with a processor, respectively. The multiprocessor configuration can provide communication mechanism, such as message passing links, bus protocols, etc., to present the collection of sub-portions as shared storage. In addition, local memory section 218, and 220 can correspond to one or more underlying physical structures including addressable memory, register, stack, queue or other storage, such as the data structure 206, as local to corresponding processors. Furthermore, the shared memory portion 208 can comprise a scheduler 226 comprising software portions that allocate sections of the data structure 206 for various operations to be performed on it.
A queue is another data structure where data is stored and retrieved from it in a first-in-first out (FIFO) basis. Thus, if the queue grows in the direction of increasing addresses in the memory, new data are pushed at the back (high-address end) and popped from the front (low-address end) of the queue. The data structure 206 can have work pushed on and then have work popped off for execution by a push operation or a pop operation respectively.
Last-in-first-out (LIFO) is one way of describing an exemplary data structure, such as a stack where the last work item or data item placed on the stack is the first one removed when a retrieval process begins. A push operation and a pop operation describe placing a new item on the stack (for example, the work item 204) and removing the top item from the stack, respectively.
Multi-threaded programming has often used locks to synchronize access to shared resources. Synchronization primitives such as mutexes, semaphores, and critical sections are all mechanisms by which a programmer can ensure that certain sections of code do not execute concurrently if doing so would corrupt shared memory structures. If one thread attempts to acquire a lock that is already held by another thread, the thread will block until the lock is free.
The data structure 206 can be utilized for work stealing operations taking place in concurrent or parallel processing. The data structure 206 in one embodiment can comprise a top and a bottom in which foreign processors can steal work items from it by using a steal operation, and local owning processors that own the data structure can push and pop work items on and off respectively by a push operation and a pop operation. The data structure 206 is a lock free, wait free structure that eliminates the majority of fences in the pop/push operations performed by the owning processor.
A stealing operation can be performed in a first-in-first out (FIFO) basis by the foreign processor 310. This may be done by a local/owning processor 302 first receiving work in its data structure 306, such as a work stealing queue, for example. Work items 304 may be removed from the front (low-address end) of the queue by the foreign processor as a steal operation. The data structure 306 can have work pushed on by the local processor 302 and also have removed by a foreign processor 310 for execution by a push operation or a steal operation respectively.
In one embodiment, the foreign processor 310, which is a processor other than the local owning processor 302, can examine other data structures such as the data structure 306 or work stealing queue after processing the chore descriptors local to it. In other words, if the foreign processor can not find anything in its own queue, it can attempt to steal from other queues the amount of work that it could possibly execute. This is the functioning essence of work stealing operations.
The data structure 306 behaves in a Last-In-First-Out (LIFO) basis from the perspective of the owning processor 302 at a tail end (e.g., end with a tail pointer) and in a First-In-First-Out (FIFO) basis according to the foreign processor 310 at a head end (e.g., end with a head pointer). The data structure 306 is referred to herein as a work stealing queue which is not limited to any particular type of data structure.
Furthermore, work items such as the work items 304 of
The data structure 306 of
If the distance between work items 404 close to a tail pointer 422 and those closer to a head pointer 424 is far apart, synchronization is not as much of a concern, but as the work items 404 popping off from both ends of the data structure 406 get closer together there is a possibility that the owning processor 402 (popping off work items 404 at the tail pointer 422 end), and the foreign processor 410, (stealing at the head pointer 424 end), could try to grab the same work item. The data structure 406 of
In one embodiment, the data structure 406 is a lock free algorithm that uses the processor intrinsic to try to synchronize it. There is no mutual exclusion like a traditional mutex or assembler as with lock algorithms. In order to be able to detect the case where a pop operation from the owning processor 402 and a steal operation from the foreign processor 410 have interleave (e.g., they try to grab the same element), the changes to the head pointer 424 and 422 are made visible to foreign processors, (e.g., processor other than the owning processor 402).
When memory operations, such as a write operation, occur from memory they may go into cache, or a processor store buffer, and a foreign processor (e.g., a processor other than that performing the operation) does not necessarily see that write operation. For example, in traditional algorithms, when a owning processor pushes onto the work stealing queue it may not update or it doesn't expense that operation, a foreign processor might not yet see that push in order to steal. In that particular case the foreign processor may not steal something or it may not see that work is ready to do. Therefore, every time a pop operation is executed there is a memory barrier so it does an interlocked decrement on the pointer or an equivalent on whatever processor architecture being used.
In one embodiment, the pointers of the work stealing queue 406, the head pointer 424 and the tail pointer 424, are made visible to the other processors. Instead of putting a memory barrier on the data structure like doing an operation similar to an interlock exchange or interlock increment/decrement on the processor for foreign processors to see it, overhead is saved by using a scheduler to allocate different regions of work stealing queue 406.
Because the owning processor 402 is controlling what happens in the private region 454, no memory barriers are implemented on the work stealing queue 406. Anything that happens inside the public region still has to be sent, therefore, the head pointer utilizes a memory barrier for it to be changed. If the foreign processor 410 tries to execute steal operations on the work stealing queue 406, and it sees there is work, but it is not allowed to steal it, an advertisement flag 416 is set visibly (e.g., the flag is fenced) to other processors (e.g., the foreign processor can set the flag 416). The advertisement flag 416 indicates to the owning processor 402 that the foreign processor 410 has tried to steal from the work stealing queue 406 and was not allowed. For example, this may occur when the entire work stealing queue is privatized and/or little or no public region is allocated by the scheduler.
When the owning processor 402 does an operation, such as a claiming operation like a pop operation in the private region, it checks the flag 416. The flag can also be checked when a push operation is performed as well. The owning processor 402 checks the advertisement flag 416 and publishes if it is set. Then the owning processor 402 clears the advertisement flag 416 without fence(s) (e.g., no memory testing is done, it just sets the flag). If the flag happens to be set already then it will make more of the work sharing queue 406 public through the scheduler 426. Owning processors 402 are able to do this operation called publishing, foreign processors 410 are not.
The local owning processor 402 updates the tail pointer 422 as push and pop operations are executed, and the foreign processor 410 updates the head pointer 424 as steal operations occur. In one embodiment, illustrated by
In one embodiment, the barrier pointer 434 gets moved when the owning processor 402 publishes work items 404. The barrier pointer 434 can also be moved when the owning processor 402 privatizes the work stealing queue 406 or work items 404. The owning processor 402 is configured to privatize the data structure by moving the barrier pointer 434 towards the head pointer 424. In this regard, the entire data structure may be privatized or only a portion of it.
In one embodiment, the owning processor 402 may be within a computer system in a parallel configuration with a foreign processor 410 and is configured to privatize the work stealing queue 406 by resetting the barrier pointer 434 substantially equal to the head pointer 424. Upon privatizing the work stealing queue 406 entirely no foreign processor 410 has access to the work items 404 on the queue. In addition, the private region 454 may be increased in size by moving the barrier pointer 434 towards the head pointer 424 without necessarily resetting it to be substantially equal. In this case the private region 454 and public region 456 exists when foreign processors may steal work items from the public region.
In one embodiment, privatizing the work stealing queue 406 by setting the barrier pointer 434 equal to the head pointer 424 or moving the barrier pointer 434 towards the head pointer 424 occurs when a claiming operation, such as a pop operation by the owning processor, is executed and/or when a delta number that is a predetermined threshold has been met. This predetermined threshold is a heuristically determined number chosen by the scheduler 426 that indicates the distance between the tail pointer 422 and head pointer 424. In one example, once the predetermined threshold has been met the work stealing queue 46 can be privatized by the owning processor 402.
In one embodiment, the work stealing queue 406 operates as an array for work items 404 that are made available to the foreign processor 410 to steal when the owning processor 402 publishes the work items 404. The owning processor 402 publishes work items 404 on the work stealing queue 406 by moving the barrier pointer 434 towards the tail pointer 422 on the data structure. The owning processor will then clear the advertisement flag 416. The advertisement flag 416 is set by the foreign processor when at least one work item is on the work stealing queue 406 and the foreign processor is unable to perform a steal operation. The foreign processor 410 can execute a steal operation once its queue is clear and it is ready to execute additional work items.
The tail pointer 422 is updated without any memory fences in one example. The head pointer 424 can be updated by a single foreign processor and can be guarded with a fence in one example. In one example, the barrier pointer 434 is updated (e.g., moved) with a fence, except when pushing of new work items when the advertisement flag is clear (e.g., initially when the barrier pointer and tail pointer are kept in parity, as discussed infra).
Referring now to
The method 500 of
A scheduler allocates a public region and a private region on the data structure in order to make the work stealing data structure operable to allow for push and pop operations by an owning processor and steal operations thereon by a foreign processor. At 504 barrier pointers(s) are set to partition work stealing data structure(s) into public region(s) and private region(s). The data structure is lock free, wait free and free of most memory fences for push and pop operations. Only the owning processor can touch the private region, and as such, operations which modify the data structure in the private region need not be fenced. The public region can be accessed by foreign processors doing a steal operation much as a steal in common work stealing algorithms, however, foreign processors cannot steal into the private region. In addition, the owning processor cannot pop into the public region without moving the barrier pointer.
Initially the barrier pointer, head pointer and tail pointer are identical. When a push occurs, the barrier pointer and tail pointer are updated without a memory barrier. Initially the barrier pointer tracks the tail pointer allowing for the work stealing queue to be public and thus provide fully for steal operations. This does not need to be the case, but is one embodiment of the disclosure. The barrier pointer may likewise track the head pointer to provide more for pop operations. Both operations are claiming operations for claiming a work item off of the data structure.
If a steal operation fails when there are work elements in the data structure (tail is greater than head) and the data structure is privatized, the foreign processor advertises its intent to steal through an advertising flag at 506. Whenever a push or pop operation occurs by the owning processor, the owning processor checks the advertising flag. If the advertising flag is set the owning processor moves the barrier forward towards the tail and clears the advertising flag, therein publishing work items or chore descriptors in the data structure for stealing at 508. At 512 the method ends. A claiming operation may be a push/pop operations and/or a steal operation.
In one embodiment, a delta is heuristically chosen by the scheduler. When a pop occurs and a distance between the tail pointer and the head pointer is greater than the delta, which a predetermined threshold value heuristically chosen, then the barrier pointer is reset to be identical to the head pointer, therein privatizing the data structure at 510. The delta is dependent upon what the system looks like, for example, the number of processors, or what the work load looks like. In some cases, the delta is zero; meaning stealing is through an advertising pass (e.g., when the advertising flag is set). This can be the case when the pattern or shape of incoming work cannot be readily modeled.
When the region between the barrier pointer and the head pointer gets above a delta and pop operations are occurring from the local processor, the owning processor at that point has begun executing work. The owning processor can change and privatize the entire queue until a foreign processor advertises that it wants to steal. The owning processor may then adjust the barrier so that there is again a public region. The public and private regions are quite dynamic therefore and constantly changing.
The method of
Still another embodiment involves a computer-readable medium comprising processor-executable instructions configured to implement one or more of the techniques presented herein. An exemplary computer-readable medium that may be devised in these ways is illustrated in
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
As used in this application, the terms “component,” “module,” “system”, “interface”, and the like are generally intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term “article of manufacture” as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. Of course, those skilled in the art will recognize many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.
Although not required, embodiments are described in the general context of “computer readable instructions” being executed by one or more computing devices. Computer readable instructions may be distributed via computer readable media (discussed below). Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types. Typically, the functionality of the computer readable instructions may be combined or distributed as desired in various environments.
Moreover, the word “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous over other aspects or designs. Rather, use of the word exemplary is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims may generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.
Also, although the disclosure has been shown and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based upon a reading and understanding of this specification and the annexed drawings. The disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, resources, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the disclosure. In addition, while a particular feature of the disclosure may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms “includes”, “having”, “has”, “with”, or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising.”
