Patent Application
20120284720
The present invention relates to computer systems and, more particularly, to hardware assisted scheduling of software tasks in such computer systems.
Modern applications are comprised of a large set of software threads that need to be dispatched to a finite set of hardware threads. A software thread (also referred to as a software task or, simply, task) is a unit of work specified by a programmer comprising a set of instructions that are to be executed. The programmer specifies this sequence of instructions assuming that they will be executed in sequence. A hardware thread is a hardware resource available for executing this software thread in a manner that conforms to the programmer's view that the instructions in that thread are executed in sequence. At any given time, a system may have multiple software threads that need to be executed, and a set of hardware threads on which they may execute. Scheduling software threads and assigning them to hardware threads has traditionally been the job of the operating system or OS. As is well known, an operating system is a software system including programs and data, which executes (runs) on a computer system and which manages computer hardware resources and provides common services for execution of various application software programs in accordance with such computer hardware resources. The operating system maintains one or more run queues of executable tasks and time shares this set of runnable tasks over the available hardware threads. In case a task is blocked on an asynchronous event (e.g., input/output or I/O), the task is removed from the run queue and re-entered when the blocking event completes.
Computer architectures are becoming increasingly more complex and need to address problems that arise from deeper architecture pipelines and relatively long memory latencies. One technique that has been deployed is that of simultaneous multi-threading where several hardware threads share the underlying resources of a compute core (e.g., pipelines, integer units, memory caches, load store queues, etc.). The hardware itself recognizes that a thread is stalled and prohibits dispatching of that thread until the stall has been resolved, thus temporarily giving more resources to the other hardware threads.
A typical operating system treats each hardware thread as a continuously available target onto which a task can be dispatched. However, dependent on the instruction mix and the memory reference pattern of each of the tasks executing, different assignments can result in different utilization of the underlying hardware resources. For instance, keeping too many of the hardware threads active at a given time can result in thrashing of the resources (e.g., memory cache), thus slowing the overall progress of the various dispatched tasks. In contrast, if dispatched tasks experience frequent stalls (e.g., on cache misses), then the resources are under-utilized. In addition to this observation, it is well established that software in many scenarios goes through different phases that often are shorter than the scheduling intervals exposed by the operating system and each of these phases exhibits different behavior with respect to the resource requirements.
It follows that, at any given time, there is an optimal number of hardware threads that should be active and this optimal number can switch rapidly based on the behavior of the software threads (tasks) that are executing upon the hardware threads at that time. In addition, given the potentially large number of tasks that are schedulable at a given time, the overhead of cycling through this task based on OS scheduler invocation creates suboptimal utilization. It is a known property that in order to create reuse in resources (e.g., cache), it often makes sense to “batch” tasks for a period of time. At the same time the overhead associated with scheduling (e.g., interrupt, examining the run queue, etc.) can create additional pressure on the underlying hardware.
Accordingly, it is impractical for an operating system to react to the rapid changes in the optimal number of hardware threads as well as having to dispatch and manage a large number of tasks.
Principles of the present invention provide apparatus and methods for hardware assisted scheduling of software tasks in a computer system.
For example, in one aspect of the invention, a computer system comprises a first pool for maintaining a set of executable software threads, a first scheduler, a second pool for maintaining a set of active software threads, and a second scheduler. The first scheduler assigns a subset of the set of executable software threads to the set of active software threads and the second scheduler dispatches one or more threads from the set of active software threads to a set of hardware threads for execution. In one embodiment, the first scheduler is implemented as part of the operating system of the computer system, and the second scheduler is implemented in hardware.
Advantageously, illustrative techniques of the invention enable better task scheduling with less involvement of the operating system software and more autonomy of the hardware to optimize for better resource utilization.
These and other objects, features, and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.
Principles of the invention will be illustratively described herein in the context of one or more illustrative computer system architectures. However, it is to be appreciated that the principles of the invention are not limited to any particular computer system architecture and are more generally applicable to any computer system in which it would be desirable to enable better task scheduling with less involvement of the operating system software and more autonomy of the hardware to optimize for better resource utilization.
As will be described in detail herein, illustrative principles of the invention advantageously introduce an additional layer of scheduling. More particularly, illustrative principles of the invention define a software thread pool and an active thread pool, wherein the active thread pool is a subset of the software thread pool. It is the task of a software thread scheduler (e.g., OS) to assign the threads from the software thread pool onto the active thread pool. A hardware thread scheduler, which is implemented in hardware, dispatches the active threads onto the hardware threads. Illustrative principles of the invention allow for more active threads than there are hardware threads. An execution unit continuously monitors the behavior of its resources and provides information to the hardware thread scheduler which in turn uses this information to dispatch more or less active threads in order to optimize for better system performance When the execution unit experiences a long delay of one hardware thread, its associated thread is returned to the active thread pool, marked as waiting for an event and a different ready active thread can be dispatched. A thread in such a state is said to be “pending.” In place of the pending thread, the hardware thread scheduler finds another ready (“non-pending”) thread in the active thread pool, and resumes its execution on the particular hardware thread.
Further, illustrative principles of the invention associate performance requirement characteristics with a software thread, and such performance requirement characteristics are also taken into account when the number and type of active hardware threads are determined.
Still further, illustrative principles of the invention use the “pending” state in the active thread pool in support of event polling, e.g., for I/O or synchronization.
Instead, illustrative principles of the invention provide, as shown in the embodiment of
A separate hardware thread scheduler 103 selects active threads based on its specific policy and associates them with hardware threads 104. By doing so, the state of the software thread is brought closer to the execution unit in order to avoid delays. For example, it is common to associate registers with fast static random access memory (SRAM). The hardware threads execute on the one or more execution units 105. As referred to herein, “execution units” are functional parts of the central processing unit (CPU) of the computer system upon which the inventive principles are implemented, while “memory units” are functional parts of the overall memory of the subject computer system.
In contrast to existing multi-threading technology where a hardware thread simply stalls, advantageously in accordance with illustrative principles of the invention, when a unit stalls for a prolonged period of time due to one of a first type of exception 110 (e.g., the memory unit 106 recognizes or anticipates a long stall due to an event such as a cache miss, an I/O request, or a lock spin), the thread is suspended by the first thread suspender 107, marked as “waiting for stall completion” and returned to the hardware thread scheduler 103. The hardware thread scheduler 103 can then return the thread to the active thread pool 102 (it is still presumed to be running) and fetch another active thread and dispatch it on the hardware thread 104. Though not explicitly required, an illustrative embodiment of the invention presumes that the state of the active thread (e.g., their architected register set) is efficiently saved in fast memory that provides significantly faster access then to dynamic RAM (e.g., cache).
It is to be appreciated that illustrative principles of the invention contemplate multiple pending states. That is, there are a variety of reasons that would cause a thread to transition from an active state. For example, on a cache miss, a thread transitions from an active state to a “waiting for stall completion,” as mentioned above. By way of another example, on an I/O operation, a thread transitions from active to a “pending I/O” state. By way of further example, when there is a stall on synchronization, a thread transitions from an active to a “waiting on synchronization state.” Given the inventive teachings herein, one of ordinary skill in the art will realize many other possible examples of pending states and how the scheduler would react to a thread in any given pending state.
Note that when the first type of exception is encountered, the “pending” thread can then be returned to the hardware thread that it was originally assigned to (or another hardware thread) after the event which caused its return to the active thread pool is cleared.
Further, a second type of exception 111 (e.g., floating point exceptions, illegal instruction, page faults) requires intervention by the system software, and thus, threads that raise one such second type of exception 111 are suspended by the second thread suspender 109 and returned to the software thread scheduler 101 which will take an appropriate action dependent upon the type of exception. For example, on a page fault exception, the operating system will initiate an I/O operation to retrieve the referenced page, and will place the thread in a suspended state. Later, upon completion of the I/O operation, the operating system will reschedule the software thread using the software thread scheduler 101. Alternatively, if the page fault is due to an invalid storage reference, the operating system can terminate the thread.
The hardware thread scheduler 103 also dispatches the optimal, or close to optimal, number of active threads into the execution units 105. However, as described in the background section above, it is undesirable to “load” the execution units with so much work that the resources of the execution units (e.g., integer unit, floating point unit, load/store pipeline, etc.) are overcommitted and the threads start to stall based on resource thrashing that go beyond the stalls related to reach memory.
Accordingly, illustrative principles of the invention utilize a monitor 108 that continuously monitors the overall performance of the system. It is to be appreciated that the monitor 108 can be part of each execution unit 105 or it can be a separate element of the computer system (for ease of reference, it is shown in
The monitor 108 continuously provides the IPC number(s) to the hardware thread scheduler 108, which then decides whether to dispatch more or fewer active threads from the active thread pool 102. Another active thread can be dispatched as long as the number of dispatched active threads has not yet reached the number of hardware threads supported by the execution unit. The hardware thread scheduler 130 will typically attempt to schedule more threads until a degradation of performance is observed at which point the number of executing threads is reduced. Reduction is typically performed when a hardware thread is stalled, at which point it is returned to the active thread pool 102, rather than being continuously associated with the hardware thread resources and instantly redispatched when the stall completes.
Metrics other than IPC may be used in alternative embodiments of the invention (such as, for example, cache miss rates, cache sharing rates, issue slot utilization), with statistics associated with each active thread by the monitor 108, and used to better schedule execution resources by the hardware thread selector 103.
It is to be appreciated that, in one illustrative embodiment, the hardware thread scheduler 103 is comprised of a set of hardware resources such as finite state machines, combinatorial logic, latches and memories, including interfaces to the memory in which the active thread pool 102 resides, as well as interfaces to the hardware threads 104 allowing any software thread to be assigned and its execution initiated. In concert, these hardware resources monitor the set of threads in the active thread pool, the status of hardware threads 104 and execution units 105 and monitor 108. In response to changes in this state, metadata is written into the active thread pool 102, architectural state is retrieved from the hardware threads and saved in the active thread pool, and new threads are assigned on available hardware threads.
As shown in the hardware thread scheduler 103, associated with each active thread is an entry in a thread quality-of-service (QoS) table 201. Each entry in table 201 specifies performance goals of the corresponding active thread. For example, it may be desirable to specify a target IPC for a particular thread. Further, the monitor (recall from
Furthermore, illustrative principles of the invention provide efficient support of event notification. In various situations, a thread may seek to suspend itself pending the occurrence of a certain event, typically in response to a specific memory location being written (e.g., the completion of a direct memory access (DMA) write by an I/O device, the release of a lock, or the release of a barrier). Such event notification can be supported via the pending state in the active thread pool 102 by associating that specific memory address with the pending state. A thread may request that it be set to pending until the specific location is written, at which point the monitor 108 will transition the thread's state from the pending state to the active state. Various embodiments of this solution may be implemented in a straightforward manner. By way of example only, in one embodiment, a cache block is marked indicating that a pending thread is waiting on the block, which is used to notify the monitor 108 that the block was written. If a block is evicted due to capacity, the monitor 108 may be allowed to spuriously awake the pending thread (which would then test the wait condition again). By way of further example, in another embodiment, the monitor 108 includes a Bloom filter to monitor pending memory locations, shielding itself from cache capacity issues.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, apparatus, method or computer program product. Accordingly, certain aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
Referring again to
Accordingly, techniques of the invention, for example, as depicted in
One or more embodiments can make use of software running on a general purpose computer or workstation. It is to be understood that the computer architecture of
With reference to
The processor 302, memory 304, and input/output interface such as display 306 and keyboard 308 can be interconnected, for example, via bus 310 as part of a data processing unit 312. Suitable interconnections, for example, via bus 310, can also be provided to a network interface 314, such as a network card, which can be provided to interface with a computer network, and to a media interface 316, such as a diskette or CD-ROM drive, which can be provided to interface with media 318.
A data processing system suitable for storing and/or executing program code can include at least one processor 302 coupled directly or indirectly to memory elements 304 through a system bus 310. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.
Input/output or I/O devices (including but not limited to keyboard 308 for making data entries; display 306 for viewing input and output information; pointing device for selecting and entering data and user feedback; and the like) can be coupled to the system either directly (such as via bus 310) or through intervening I/O controllers (omitted for clarity).
Network adapters such as network interface 314 may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
As used herein, a “server” includes a physical data processing system (for example, system 312 as shown in
It will be appreciated and should be understood that the exemplary embodiments of the invention described above can be implemented in a number of different fashions. Given the teachings of the invention provided herein, one of ordinary skill in the related art will be able to contemplate other implementations of the invention. Indeed, although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be made by one skilled in the art without departing from the scope or spirit of the invention.
