Patent Application
20110067011
The present invention relates to computing system software. More particularly, this invention relates to thread management.
Efficient operation of modern computing systems generally requires support of multiple instruction “threads”, with each thread being an instruction stream that provides a distinct flow of control within a program. To improve overall system speed and responsiveness, multiple threads can be simultaneously acted upon by computing systems having multiple processors, each processor supporting a single thread. In more advanced computing systems, multiple threads can be supported by use of processors having a multithreaded processor architecture that are capable of acting on multiple threads simultaneously. Alternatively, a single processor can be multiplexed between threads after a fixed period of time in a technique commonly referred to as time-slice multi-threading. In still another approach known as switch-on-event multithreading, a single processor switches between threads upon occurrence of a trigger event, such as a long latency cache miss.
The concept of multi-threading has been enhanced in a technique called simultaneous multi-threading (“SMT”). Simultaneous multithreading is a processor design that combines hardware multithreading with superscalar processor technology to allow multiple threads to issue instructions each cycle. SMT typically permits all thread contexts to simultaneously compete for and share processor resources. In some implementations, a single physical processor can be made to appear as multiple logical processors to operating systems and user programs, with each logical processor maintaining a complete set of the architecture state, but nearly all other resources of the physical processor, such as caches, execution units, branch predictors, control logic and buses being shared. The threads execute simultaneously and make better use of shared resources than time-slice multithreading or switch-on-event multithreading. Effective utilization of such multithread supporting processors can require procedures for automatically optimizing program behavior and identifying portions of code that are the best candidates for optimization. Optimizing regions of code identified through a set of threading mechanism increases program performance by transforming an original single-threaded application into a de facto multithreaded code. In one known technique a “speculative precomputation” (SP) thread is created to run in parallel with the original code as a main thread. The SP thread will run ahead of the main thread and encounter future cache misses, thus performing effective prefetches for the main thread. This technique is not always reliable however, due to thread synchronization issues.
The inventions will be understood more fully from the detailed description given below and from the accompanying drawings of embodiments of the inventions which, however, should not be taken to limit the inventions to the specific embodiments described, but are for explanation and understanding only.
The computer system 10 of the present invention may include one or more I/O (input/output) devices 15, including a display device such as a monitor. The I/O devices may also include an input device such as a keyboard, and a cursor control such as a mouse, trackball, or trackpad. In addition, the I/O devices may also include a network connector such that computer system 10 is part of a local area network (LAN) or a wide area network (WAN).
Examples of a system 10 include, but are not limited or restricted to a computer (e.g., desktop, a laptop, a server, blade server, a workstation, a personal digital assistant, etc.) or any peripherals associated therewith; communication equipment (e.g., telephone handset, pager, etc.); a television set-top box and the like. A “connection” or “link” is broadly defined as a logical or physical communication path such as, for instance, electrical wire, optical fiber, cable, bus trace, or even a wireless channel using infrared, radio frequency (RF), or any other wireless signaling mechanism. In addition, the term “information” is defined as one or more bits of data, address, and/or control. “Code” includes software or firm-ware that, when executed, performs certain functions. Examples of code include an application, operating system, an applet, boot code, or any other series of instructions, or microcode (i.e. code operating at privilege level and below OS).
Alternatively, the logic to perform the methods and systems as discussed above, could be implemented in additional computer and/or machine readable media, such as discrete hardware components as large-scale integrated circuits (LSI's), application-specific integrated circuits (ASIC's), microcode, or firmware such as electrically erasable programmable read-only memory (EEPROM's); or spatially distant computers relaying information through electrical, optical, acoustical and other forms of propagated signals (e.g., radio waves or infrared optical signals).
In one embodiment, a computer program product readable by the data storage unit 18 may include a machine or computer-readable medium having stored thereon instructions which may be used to program (i.e. define operation of) a computer (or other electronic devices) to perform a process according to the present invention. The computer-readable medium of data storage unit 18 may include, but is not limited to, floppy diskettes, optical disks, Compact Disc, Read-Only Memory (CD-ROMs), and magneto-optical disks, Read-Only Memory (ROMs), Random Access Memory (RAMs), Erasable Programmable Read-Only Memory (EPROMs), Electrically Erasable Programmable Read-Only Memory (EEPROMs), magnetic or optical cards, flash memory, or the like, including any methods to upgrade or reprogram or generate or activate or reserve activation of microcode enhancement)
Accordingly, the computer-readable medium includes any type of media/machine-readable medium suitable for storing electronic instructions. Moreover, the present invention may also be downloaded as a computer program product. As such, the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client). The transfer of the program may be by way of data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., a modem, network connection or the like).
In one embodiment, the methods of the present invention are embodied in machine-executable instructions directed to control operation of the computing system 10, and more specifically, operation of the processor, register, cache memory, and general memory. The instructions can be used to cause a general-purpose or special-purpose processor that is programmed with the instructions to perform the steps of the present invention. Alternatively, the steps of the present invention might be performed by specific hardware components (including microcode) that contain hardwired logic for performing the steps, or by any combination of programmed computer components and custom hardware components.
It is to be understood that various terms and techniques are used by those knowledgeable in the art to describe communications, protocols, applications, implementations, mechanisms, etc. One such technique is the description of an implementation of a technique in terms of an algorithm or mathematical expression. That is, while the technique may be, for example, implemented as executing code on a computer, the expression of that technique may be more aptly and succinctly conveyed and communicated as pseudocode that generically defines program flow logic, by formula, algorithm, or mathematical expression.
Thus, one skilled in the art would recognize a block denoting A+B=C as an additive function whose implementation in hardware and/or software would take two inputs (A and B) and produce a summation output (C). Thus, the use of formula, algorithm, or mathematical expression as descriptions is to be understood as having a physical embodiment in at least hardware and/or software (such as a computer system in which the techniques of the present invention may be practiced as well as implemented as an embodiment).
As seen in
A delay software module 34 is used to perform a wait/sleep operation on speculative precomputation threads between thread creation and activation. SP threads run only as often as their corresponding sections in their respective non-speculative threads. In most applications there is some discrete time between SP thread creation and SP thread activation, as well as time between successive SP thread activations. During these times, the SP thread perform a wait/sleep operation to allows the SP thread to yield to other processes that the system may wish to run on that logical processor.
A synchronization module 36 (which includes memory access functionality to store global variables) tracks progress of non-speculative threads through a set of global variables, allowing the speculative precomputation (SP) threads to gauge relative progress with respect to non-speculative threads. Given that both SP and non-SP threads may be reading and writing to a set of shared variables, it has been shown to be helpful to bound all accesses to this set of global variables with a fast, synchronization object. The synchronization object can be directly from OS thread API such as the event object manipulated by setEvent( ) and waitForSingleObject( ) in Win32 thread API or equivalent API in pthread. Alternatively, such synchronization object can be implemented via suitable hardware thread wait monitor that allows a thread to define a cache line aligned memory address as monitor, and a load access to this monitor object can suspend the execution of a said thread—making it semantically equivalent to waitForSingleObject( ) and a store access to the monitor can wake up the suspended thread—thus equivalent to setEvent( ). It can be noted however, that while monitor write and mwait are much more efficient than an OS level thread API, implementation of the described embodiment is applicable to any hardware, software, or mixed hardware and software mechanism that supports wait and wakeup.
In addition to use of global variables and provision of a wait state, code transformation for SP optimized operation can further include provisions to limit the frequency of communication between the SP thread and the non-speculative main thread. Defining “stride” as a variable equal to the number of iterations of loops that a SP thread is set to run ahead relative to a non-speculative main thread, the threads can be set to only access the set of shared global variables after stride operations. This minimizes communication, with thread run-ahead and fall-behind also being limited to units of size stride. In certain embodiments where the SP thread consistently runs just ahead of the non-speculative thread, and any synchronizing communication is unnecessary overhead, stride dependent communication limitations are not used. As will be appreciated, stride choice often impacts performance of the application. If the stride is set too low (with run-ahead distance is being too short, more frequent inter-thread communication needed, and frequent non-timely memory access by the SP thread), communication overhead begins to negate the benefit of the SP thread. On the other hand, if it is set too high the SP thread may run too far ahead and some previously prefetched data can be overridden before use by the main thread, there may be insufficient thread communication, and erroneous or unnecessary (i.e. untimely) prefetches may result.
In the majority of applications, the SP thread had been observed to fall behind and/or run significantly ahead of the non-speculative thread. Fall behind and/or run ahead frequency can be minimized via good communication between threads by dynamically increasing or decreasing execution of the speculative thread. If the SP thread finds it is behind the non-speculative thread, it should effectively increase its execution by attempt to jump ahead to the last communicated location. On the other hand, if the SP thread finds it has run ahead of the non-speculative thread, it can employ one of two techniques to decrease execution: wait and jump-back. With the wait technique, the SP threads simply yield and wait to be signaled by the non-speculative thread. Alternatively, a jump-back technique can be used in which SP thread execution requires jump back to the last known location of the non-speculative thread, and begin prefetching again.
A SP thread may also fall behind its non-speculative thread. If this occurs, and the non-speculative thread has completed the section of code the SP is prefetching for, the application may incur additional, unnecessary cache misses while the SP thread continues to run. In one embodiment, the SP thread includes a throttling mechanism at the end of each stride of run-ahead operation to check the relative progress of the main thread (via global variable for trip count) and then determine whether it is running too far ahead or running behind the main thread. The run-ahead strategy can be accordingly adjusted to either continue to do another round of prefetch (if not running too far ahead), or put itself to sleep and wait for the next wakeup from the main thread (if running too far ahead or behind), or sync up with the main thread's progress (by syncing prefetch's starting pointer via the global variable) and continue to run the prefetch.
To enhance efficiency, the SP thread should, at its core, contain only those instructions necessary for determining the desired long latency operation (e.g. a memory load) sequence required by the non-speculative main thread. Thus it is desirable to minimize the number of function calls from the SP thread via function inlining. Inlining is useful, for example, in applications such as a minimal spanning tree (MST) that repeatedly loops over a list of hash tables and performs a lookup on each of those tables (which requires traversing another list).
Recursive functions can also be the source of delinquent loads that would be minimized by addition of SP thread functionality. Since recursive functions can be difficult to directly transform into SP threads for 2 reasons: the stack overhead of the recursive call can be prohibitively expensive, and jump-ahead code if difficult (if not impossible) to implement, it is sometimes useful to transform the recursive function into a loop-based function for the SP thread.
To better illustrate one embodiment of a method and system for conversion of single threaded code into optimized code having speculative precomputation, consider the following single threaded pseudocode:
In one embodiment, when executed, line 4 requires 49.47% of total execution time, while line 5: requires about 49.46% of total execution time. Line 5 also has 99.95% of total L2 misses, making it an ideal candidate for optimization using speculative precomputation threads.
The following illustrates an example of pseudocode suitable for running the foregoing pseudocode with increased efficiency. A “Main” thread is generated such that:
Line 7 corresponds to Line 5 of the single threaded code, and Line 13 SetEvent is a synchronous trigger (where an API call is statically placed at specific location in the code, as contrasted to asynchronous trigger where the code location when triggered is not initially known) to launch the following speculative precomputation (SP) thread (hereafter alternatively known as a “scout”, “worker” or “helper” thread):
Line 9 is responsible for most effective prefetch due to run-ahead, while line 15 detects run-behind and adjusts by jumping ahead.
Overall, execution time of Line 7 in the main thread (corresponding to line 5 in the single threaded case) is 19% vs 49.46% in single-thread code. The L2 cache miss is a negligible 0.61% vs 99.95% in single-thread code. Line 9 of speculative precomputation thread (corresponding to Line 7 of the main thread) has an execution time of 26.21% and an L2 miss of 97.61%, indicating that it is successful in shouldering most L2 cache misses.
To achieve such performance results, the speculative precomputation (SP) worker thread T( ) essentially performs the task of pointer-chasing in the main loop, and it does not perform the work( )operations. In essence, the worker probes or scouts the load sequence to be used by the main loop and effectively prefetches the required data.
There is only one worker thread created at the beginning of the program, and it lives until there are no longer any loop iterations to be performed. In certain embodiments, processor architectures that supports two or more physical hardware thread contexts and have a relative heavy cost of creating a new thread can map the worker thread to a second hardware thread. In effect, there is no additional thread spawning, and the cost of thread spawning is amortized across the program to become virtually unnoticeable.
Once the SP thread is created, the main thread waits for the SP thread to indicate that it has completed its pre-loop work. A more elaborately tuned SP thread can probe more than one iterations of the initial pointer chasing for this pre-loop work.
Essentially, the SP worker thread performs all of its precomputation in units of size stride as previously defined. This both minimizes communication and limits thread run-ahead, while effectively setting limits to how many iterations a precomputation thread can run ahead of the main thread. If run-ahead is too far, the precomputation induced prefetches could not only displace temporally important data to be used by the main thread but also potentially displace earlier prefetched data that have not been used by the main thread. On the other hand, if the run-ahead distance is too short, then the prefetch could be too late to be useful.
In the foregoing pseudocode example of a speculative precomputation worker thread, the worker thread's pre-loop work consists of performing stride loops, i.e. prefetches, shown between Line 2-5. Every stride loops in the main thread, a global copy of the current pointer and the number of loops remaining are updated, shown between Line 10-12. Additionally, the main thread signals the worker thread that it may continue prefetching, shown at Line 13, if the worker is stalled because it ran too far ahead. After prefetching in chunks of stride length, shown at Line 8-11, the worker thread waits for a signal from the main thread to continue. Again, this prevents the worker from running too far ahead of the main thread. More importantly, before looping over another stride iterations, the worker thread examines whether its remaining iterations are greater than the global version. If so, the worker thread has fallen behind, and must “jump ahead” by updating its state variables to those stored in the global variables (Lines 13-15).
The following respective “Single Threaded Code” and modified “Speculative Computation Multithreaded Version” illustrates conversion of single threaded code using algorithms corresponding to the foregoing pseudocode:
——asm { pause };
——asm { pause };
In another specific embodiment intended to illustrate conversion of a code snippet into a form suitable for efficiently operating with speculative precomputation, the structure of the speculative precomputation thread is as follows:
The code segment to be altered to support threads of the foregoing structure is known as the MCF program:
The SP thread is setup so that:
Loop control is duplicated as follows:
While synchronization issues are handled by adjusting for fall-behind or run-ahead thread through insertion of an internal loop counter and stride counter:
Sychronization with the main thread is as follows:
Atomic update of MCF code with an internal counter is:
Other MCF code enhancements include other enhancements SP thread termination by run-ahead main thread and intermittent prefetches of delinquent loads in loop body:
Reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “other embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments, of the invention. The various appearances “an embodiment,” “one embodiment,” or “some embodiments” are not necessarily all referring to the same embodiments.
If the specification states a component, feature, structure, or characteristic “may”, “might”, or “could” be included, that particular component, feature, structure, or characteristic is not required to be included. If the specification or claim refers to “a” or “an” element, that does not mean there is only one of the element. If the specification or claims refer to “an additional” element, that does not preclude there being more than one of the additional element.
Those skilled in the art having the benefit of this disclosure will appreciate that many other variations from the foregoing description and drawings may be made within the scope of the present invention. Accordingly, it is the following claims including any amendments thereto that define the scope of the invention.
The present application claims priority to U.S. patent application Ser. No. 10/334,868 filed Dec. 31, 2002, the entire disclosure of which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10334868 | Dec 2002 | US |
| Child | 12949630 | US |
