Patent Application
20140208072
The present application relates to multi-core processing.
The Intel Xeon Phi is a recently introduced x86-based 60-core, 240-thread coprocessor that is increasingly being deployed in servers and clusters. It is easier to program than other manycore processors, and runs the Linux operating system. The operating system provides services such as virtual memory and context switching, and enables multiple processes to run concurrently and share the coprocessor. Multi-processing on the manycore is also necessary to fully utilize the hardware resources of the Xeon Phi.
It is remarkably easy to offload processing the Xeon-Phi: it supports a popular ISA (x86), a popular OS (Linux), and a popular programming model (OpenMP). Unfortunately, quick and easy portability rarely results in an implementation that executes faster on the Xeon-Phi. In order to use the Xeon Phi, a programmer employs pragmas to identify code regions to be “offloaded” to the coprocessor, for which the compiler automatically generates coprocessor instructions along with glue code to transfer data. This is referred to as the “offload programming model” where the main trunk of the code runs on the host processor while regions identified by the pragmas are offloaded to run on the Xeon Phi coprocessor. Thus, if portions of a program are already parallelized using OpenMP, porting them to the Xeon Phi is easy. Unfortunately, quick and easy portability rarely results in an implementation that executes faster on the Xeon Phi. Rather, additional programmer effort such as carefully selecting the number of threads, mapping them to cores, ensuring no thread or memory oversubscription, and load balancing the application across multiple coprocessors is necessary.
Such application tuning can work well for individual applications when they “own” the coprocessor, but not in a multi-processing environment. The Xeon Phi coprocessor runs Linux, which makes it easy for multiple processes to share the coprocessor. Such a use case of a manycore like the Xeon Phi where multiple processes compete for coprocessor resources is likely not only in cluster and cloud deployments, but also in servers since good hardware utilization is essential.
In a multi-processing environment, multiple application processes compete for coprocessor resources, and one programmer is unaware of another programmer's intentions. Thus any programmer-driven steps taken to improve performance can in fact degrade it. In such an environment, processes must adhere to the following guidelines in order to avoid performance degradation and benefit from the manycore.
Additional programmer effort such as carefully selecting the number of threads, mapping them to cores, ensuring memory is not over-subscribed, and using multiple coprocessors well are necessary to tune application performance But programmer directives alone are insufficient: multi-processing on the manycore is required to improve hardware utilization. Linux makes it easy for processes to share the Xeon Phi, but in an environment where applications compete for manycore resources, any programmer effort intended to boost individual application performance can in fact end up doing the opposite.
In a multi-processing environment, directives introduced by the programmer specifically to enhance manycore coprocessor performance can be counter-productive. For instance, programmers must select the number of threads, map them to cores, ensure memory is not over-subscribed, and manage their workload across multiple coprocessors. But these can degrade performance in a multi-processing environment when one programmer is unaware of other programmers' intentions.
Various solutions are available to help programmers take advantage of the co-processor. Programming model such as SWARM provides an API to represent workloads as tasks, compiles them to specific processors including the Xeon Phi and uses a runtime to manage the tasks on distributed heterogeneous nodes. Compilers such as CAPS can produce host and Xeon Phi code starting from OpenACC directives. Libraries with parallelized high performance numerical code for the Xeon Phi have been done. Cluster management middleware can schedule workloads on the Xeon Phi. Virtualization approaches such as ScaleMP provide a hypervisor that can “virtualize” the Xeon Phi and host into a single entity visible to programmers. An operating system runtime (MPSS) is available on top of the Xeon Phi micro kernel (OS) to perform primitive scheduling of offloads.
In one aspect, a method is disclosed to manage a multi-processor system with one or more multiple-core coprocessors by intercepting coprocessor offload infrastructure application program interface (API) calls; scheduling user processes to run on one of the coprocessors; scheduling offloads within user processes to run on one of the coprocessors; and affinitizing offloads to predetermined cores within one of the coprocessors by selecting and allocating cores to an offload, and obtaining a thread-to-core mapping from a user.
In another aspect, a server populated with one or more multiple core Xeon Phi coprocessors includes a manager to control user processes containing offload blocks by intercepting COI API calls and schedules user processes to run on one of the Xeon Phi coprocessors; schedules offloads within user processes to run on one of the Xeon Phi coprocessors; and affinitizes offloads to specific cores within one of the Xeon Phi coprocessors by selecting and allocating cores to an offload, and obtaining the thread-to-core mapping from the user.
Implementations of the above system can
Advantages of the above system may include one or more of the following. The system provides a middleware on top of the Xeon Phi micro kernel and the Intel runtime. The middleware handles multi-processing on Xeon Phi coprocessor-based servers by automatically avoiding thread and memory oversubscription and load balancing processes across the cores of the Xeon Phi and across several Xeon Phi coprocessors. The system is completely transparent to the users and requires no changes to the underlying software such as the MPSS and the Linux kernel running on the coprocessor. It uses a scheduling technique to schedule processes and Xeon Phi offload regions within processes simultaneously. It also uses algorithms to set thread affinity and load balance processes across coprocessors.
The system achieves faster operation when multiple processes share a many integrated core coprocessor system. Faster operation includes end-to-end turn-around-time per process (latency), as well as the number of processes completed per unit time (throughput).
The system protects against thread and memory over-subscription resulting in severe performance loss and crashes. Within a coprocessor, it manages cores such that offloads of different processes run on separate sets of cores, and offloads in the same process use the same cores (thus respecting data affinity). The system balances the load of multiple processes across multiple Xeon Phi coprocessors. The manager provides a transparent user-level middleware that includes a suite of run-time techniques explicitly designed to enhance performance portability in the presence of multi-processing.
To simplify memory management, one implementation requests that the programmer specify the maximum memory required on the Xeon Phi for each process. This is similar to job submission requirements in cluster schedulers. In typical cases, different offloads of the same process often share data in order to reduce data movement between the host and Xeon Phi. Thus, as long as the process exists, it will use memory on the card. However, unlike cluster schedulers, this embodiment does not require the process specify core, devices or other resources, but infers it automatically from the number of threads requested by the offload. Unlike memory that is reserved for the life of a process, threads (and cores) are given to an offload when it starts executing and released when the offload completes for use by other offloads.
Before execution, every process requests COSMIC for memory, and every offload requests COSMIC for threads. COSMIC arbitrates the requests by taking into consideration the different available coprocessors, the available cores within each device and the available memory. It then schedules and allocates resources for the offloads in such a way that thread and memory oversubscription are avoided, and the devices as well as the cores within them are load balanced.
COSMIC has several parameters that may be set by the server administrator or user that can affect its policies and behavior. An administrator can configure the following parameters of COSMIC to affect its scheduling decisions:
The COSMIC host middleware component has a global view of all processes and offloads emanating from the host, and knowledge of the states of all coprocessor devices. COSMIC is architected to be lightweight and completely transparent to users of the Xeon Phi system. As shown in
The Xeon Phi compiler converts all offload blocks that are marked by pragmas into COI calls. The user's program with offload pragmas is compiled using Intel's icc or a gcc cross-compiler for the Xeon Phi. The compiler produces a host binary, and Xeon Phi binaries for all the offload portions. The offload portions are first translated into a series of COI API calls. The figure shows the important calls for a simple example: first COIEngineGetCount and COIEngineGetHandle get a handle to the coprocessor specified in the pragma. Then COIProcessCreateFromFile creates a process from the binary corresponding to the offload portions. Each offload block is represented as a function, and COIProcessGetFunctionHandles acquires the handles to these functions. COIPipelineCreate creates a “COI pipeline” which consists of 3 stages: one to send data to the coprocessor, one to perform the computation and one to get data back from the coprocessor. Then COIBufferCreate creates buffers necessary for inputs and outputs to the offload. In this example, three COI buffers corresponding to the arrays a, b and c are created. COIBufferCopy transfers data to the coprocessor, and COIPipelineRunFunction executes the function corresponding to the offload block. Finally, another COIBufferCopy gets results (i.e., array c) back from the Xeon Phi.
Since every offload is converted into a series of COI calls (which has a standard API), COSMIC can transparently control offload scheduling and dispatch.
COSMIC is architected as two components implemented as separate processes: the COSMIC client and the COSMIC server. The COSMIC client is the front-end, while the COSMIC server is the back-end consisting of the scheduler and the monitor. The monitor comprises a host portion and a card-side portion, as depicted in
The COSMIC client is responsible for intercepting COI calls and communicating with the scheduler in the COSMIC server to request access to a coprocessor. It accomplishes this using library interposition. Every user process links with the Intel COI shared library that contains definitions for all API function modules. COSMIC intercepts and redefines every COI API function: the redefined COI functions perform COSMIC-specific tasks such as communicating with the COSMIC scheduler, and then finally calls the actual COI function. With the redefined functions, COSMIC creates its own shared library that is pre-loaded to the application (using either LD_PRELOAD or redefining LD_LIBRARY_PATH). The pre-loading ensures that COSMIC's library is first used to resolve any COI API function.
Based on the type of COI API intercepted, the client sends the following different messages to the scheduler in the COSMIC server:
NewProcess: When an offload is first encountered for a a process, the client sends a NewProcess message to the scheduler indicating that the scheduler should account for a new process in its book-keeping. Every new process is annotated with its memory requirement provided by the user.
NewOffload: For every offload, the client sends a NewOffload message to the scheduler indicating the process to which the offload belongs and the number of threads it is requesting. It also indicates the size of the buffers that need to be transferred to the coprocessor for this offload.
OffloadComplete: When an offload completes, the client sends an OffloadComplete message to the scheduler so that it can account for the newly freed resources such as coprocessor cores and threads.
ProcessComplete: When a process completes, the client sends a ProcessComplete message to the scheduler to account for the freed memory used by the process.
The COSMIC scheduler is the key actor in the COSMIC system and manages multiple user processes with offloads and several coprocessor devices by arbitrating access to coprocessor resources. It runs completely on the host and has global visibility into every coprocessor in the system. In scheduling offloads and allocating resources, it ensures no thread and memory oversubscription and load balances coprocessor cores and devices to most efficiently use them.
A key distinction between the COSMIC scheduler and traditional operating system schedulers is that COSMIC concurrently schedules processes and offloads within the processes. Each process has a memory requirement, while each offload has a thread requirement. Various coprocessors in the system may have different memory and thread availabilities.
Under these constraints, the goal of the scheduler is to schedule processes and offloads by mapping processes to Xeon Phi coprocessors and offloads to specific cores on the coprocessors. The scheduler also ensures fairness, i.e., makes sure all processes and offloads eventually get access to coprocessor resources.
The scheduler is event-based, i.e., a scheduling cycle is triggered by a new event. A new event can be the arrival of a new process, the arrival of a new offload in an existing process, the dispatching of an offload to a Xeon Phi device, the completion of an offload or the completion of a process. A queue of pending processes is maintained: each arriving new process is added to the tail of the pending process queue. A process is eventually scheduled to one Xeon Phi coprocessor. The scheduler also maintains a queue of pending offloads for each Xeon Phi coprocessor in the system. Each new offload is added to the tail of the offload queue belonging to the Xeon Phi coprocessor on which its process has been scheduled.
When a new event occurs, a pending process is selected and scheduled to a coprocessor that has enough free memory. Then offload queues corresponding to each Xeon Phi are examined, and the scheduler dispatches an offload to each coprocessor if it has enough free threads. Both processes and offloads are selected based on an aging-based first-fit heuristic.
At the start of a scheduling cycle, let P be the process at the head of the pending process queue (402). The scheduler maintains a circular list of the Xeon Phi coprocessors in the system. Let D be the next coprocessor in the list (404). The scheduler checks to see if the memory required by P fits in the available memory of D (406). If it does, P is removed from the queue and dispatched to D (408). If not, the next coprocessor in the circular list is examined (410). If P does not fit in any coprocessor, its age is incremented, and the next pending process is examined (412). When a process' age reaches a threshold, all scheduling is blocked until that process is scheduled (414). This ensures fairness since all processes will get a chance at being scheduled.
Scheduling an offload is similar to scheduling a process, with one difference. Instead of memory, an offload has a thread requirement; COSMIC checks if the threads requested by an offload are available on the coprocessor on which the offload's owner process has been scheduled. If so, the offload is dispatched. If not, it increments the offload's age, and examines the next offload in the queue.
An administrator can specify the following parameters to tailor the scheduler's behavior: (i) aging threshold, (ii) thread over-scheduling factor and (iii) memory over-scheduling factor. The latter two indicate to what extent threads and memory may be oversubscribed.
The COSMIC monitor collects data pertaining to the state of the coprocessors. It has a host-side component, and a component that is resident on each of the coprocessors. The host-side component is primarily responsible for communicating with the scheduler and all the coprocessor-side components. The coprocessor-side components monitor the load on the device, the number of threads requested by each offload and the health of each offload process. If a process dies for any reason, it catches it and reports the reason to the COSMIC scheduler.
Next, the affinity setting is discussed. COSMIC selects the cores that are used by an offload, and affinitizes threads to these cores using programmer directives. The core selection procedure for an offload is discussed next. COSMIC's core selection algorithm scans one or more lists of free physical cores to select cores until it finds enough cores for a given offload region. The number of cores assigned to an offload region is the number of threads used by the offload region divided by the thread-to-core ratio N. The order of the core lists from which COSMIC selects cores reflects the preference of COSMIC's core selection strategy. The first list of physical cores for a new offload region consists of cores that are both free and only used by the earlier offloads coming from the same offload process. If more physical cores are needed, COSMIC picks from a second list of physical cores, which are both free and not currently assigned to other offload processes. If still more cores are needed, COSMIC forms a third list of physical cores, which are the remaining free cores not yet selected.
To ensure efficient execution of offloads COSMIC adopts several policies to make sure the hardware resources of a Xeon Phi processor are well utilized but not oversubscribed. These policies and their implementations are discussed below.
COSMIC limits the total number of actively running software threads on a Xeon Phi device to ensure that the device's physical cores are not oversubscribed. When an offload region is running, all of the threads spawned by the offload process are considered active. Otherwise the threads are considered dormant. COSMIC keeps track of the number of active and inactive software threads spawned by offload processes on a Xeon Phi device. It maintains the ratio between the total number of active software threads spawned by all offload processes and the physical cores to be no more than an integer N, which is configured as 4 in our current implementation of COSMIC. Therefore COSMIC only schedules an offload region to run if the thread-to-core ratio will not exceed N after the offload region starts.
COSMIC uses several mechanisms to detect the number of software threads created by an offload process on a Xeon Phi device. First COSMIC inspects the environment variable MIC_OMP_NUM_THREADS of a submitted job. COSMIC also intercepts omp_set_num_threads_target( ) function calls on the host. Finally on a Xeon Phi device an offload process's call to omp_set_num_threads( ) is also intercepted and the number of threads is reported back the host COSMIC process.
COSMIC relies on the OMP library to get the number of physical cores on a Xeon Phi device. During COSMIC's initialization, the monitor (running on each Xeon Phi device) queries the maximum number of threads by calling omp_get_max_threads( ) and communicates the returned value to COMSIC's host process. COSMIC then divides this number by 4 to obtain the number of the processing cores on a Xeon Phi device. Notice that the number of physical cores derived in this approach is generally one less than the real number of the physical cores. We believe this is because one core is reserved for the OS, and thus we do not adjust the derived number.
To further improve the efficiency of offload executions, COSMIC creates physical-core containers and sets thread affinity automatically. The goal is to avoid threads migrating from one core to another so data in the registers and the private L1 cache can be reused, and to have multiple concurrent offload processes so the overall system utilization remains high.
COSMIC assigns different, non-overlapping sets of physical cores to execute concurrent offload regions. For an offload region the selected cores constitute a physical-core container that the offload can use exclusively. The physical-core containers minimize the interference between multiple concurrent offloads on the same Xeon Phi device due to resource contention. The physical-core container expires after the execution of its offload region completes, and the assigned cores are released for use by other offloads. Notice that a physical core on a Xeon Phi device can be assigned to multiple offload processes but only used by at most one active offload region at any given point of time. A free core is a core not currently used by any offload region (but it may be assigned to one or more offload processes.)
COSMIC's core selection algorithm scans one or more lists of free physical cores to select cores until it finds enough cores for a given offload region. The number of cores assigned to an offload region is the number of threads used by the offload region divided by the thread-to-core ratio N. The order of the core lists from which COSMIC selects cores reflects the preference of COSMIC's core selection strategy. The first list of physical cores for a new offload region consists of cores that are both free and only used by the earlier offloads coming from the same offload process. If more physical cores are needed, COSMIC picks from a second list of physical cores, which are both free and not currently assigned to other offload processes. If still more cores are needed, COSMIC forms a third list of physical cores, which are the remaining free cores not yet selected.
The core selection algorithm uses simple linear arrays to track of the status of the physical cores so that it can efficiently construct various lists of physical cores to choose from. For each Xeon Phi device the algorithm maintains two arrays, called F and G. Entry i of an array stores the status of physical core i. Array F indicates which cores are free, and array G records the number of the offload processes that a core is assigned to. Further the algorithm also maintains for each active offload process an array, called P, listing the cores assigned to the process's latest offload region (P[i] is 1 iff core i is assigned.)
From the status arrays the algorithm constructs the various lists of cores to select for an offload region. For an offload region, the first list of cores to be considered (the most preferred) consists of core i such that F[i]=1, G[i]=1, and the offload process's P[i]=1. If not enough cores can be found from the first list, the algorithm creates a second list of cores, where a core i on the list satisfies F[i]=1, G[i]=0 (and P[i]=0, which is implied by G[i]=0). Finally the third list of cores (the least preferred) consist of any core i such that F[i]=1 (and G[i]>1, that is, core i is assigned to more than one offload processes).
These status arrays are updated after offload starts and completes or when an offload process finishes. The initial values of the entries in F and G are 1 and 0, respectively. The update rules are as follows:
COSMIC uses the thread affinity API of the OpenMP library to keep the software threads of an offload region running on the selected cores. This thread-to-core binding procedure is conducted before or at the beginning of every offload region after the number of software threads is detected and the cores are selected. Since the thread binding must be conducted in the offload process at which the binding is targeted, COSMIC preloads every offload process a special function to perform the binding and calls the function when the binding is required. The binding is similar to the “compact” option provided by OpenMP: each core is assigned N=4 threads.
Oversubscribing Xeon Phi memory can lead to two undesirable results: application crashes or excessive performance degradation due to memory swapping. Therefore COSMIC ensures that the overall memory requirement of the offload processes running on a single Xeon Phi device does not exceed the amount of the device's physical memory.
To avoid memory oversubscription COSMIC keeps track of the amount of available physical memory for each Xeon Phi device. When COSMIC starts running, it queries each MIC device the amount of free physical memory using a COI function (COIEngineGetInfo). When a user submits a job to COSMIC, the user needs to inform COSMIC the total amount of memory the process needs on a Xeon Phi device through an environment variable COSMIC_MEMORY. COSMIC only launches a submitted job if there is one Xeon Phi device with enough free memory to meet the memory requirement of the job.
COSMIC can be optionally configured to terminate any running process that uses more Xeon Phi memory than the amount specified by the user. COSMIC relies on Linux's memory resource controller to set up a memory container for each offload process on a Xeon Phi device. Each container limits the real committed memory usage of the offload process to the user-specified maximum value. If a process's memory footprint goes over the limit, the memory resource controller invokes Linux's out-of-memory killer (oom-killer) to terminate the offending process.
Enforcing this maximum memory usage rule requires an extra installation procedure and incurs minor runtime performance overhead. The memory resource controller is not enabled in the default Xeon Phi OS kernel. To install a new kernel with the memory resource controller requires adding one line to the kernel configuration file, recompiling the kernel, and rebooting Xeon Phi cards with the new kernel image. The runtime performance overhead due to using the Linux memory controller ranges from negligible to about 5% in real applications.
A many integrated cores (MIC) co-processor can have the cores, PCIe Interface logic, and GDDR5 memory controllers are connected via an Interprocessor Network (IPN) ring, which can be thought of as independent bidirectional ring. The L2 caches are shown as slices per core, but can also be thought of as a fully coherent cache, with a total size equal to the sum of the slices. Information can be copied to each core that uses it to provide the fastest possible local access, or a single copy can be present for all cores to provide maximum cache capacity. In one embodiment, the co-processor is the Intel® Xeon Phi™ coprocessor that can support up to 61 cores (making a 31 MB L2) cache) and 8 memory controllers with 2 GDDR5 channels each. Communication around the ring follows a Shortest Distance Algorithm (SDA). Co-resident with each core structure is a portion of a distributed tag directory. These tags are hashed to distribute workloads across the enabled cores. Physical addresses are also hashed to distribute memory accesses across the memory controllers. Each Xeon Phi core is dual-issue in-order, and includes 16 32-bit vector lanes. The performance of each core on sequential code is considerably slower than its multi-core counterpart. However, each core supports 4 hardware threads, resulting in good aggregate performance for highly parallelized and vectorized kernels. This makes the offload model, where sequential code runs on the host processor and parallelizable kernels are offloaded to the Xeon Phi, a suitable programming model. The Xeon Phi software stack consists of a host portion and coprocessor portion. The host portion asynchronous execution and data transfer between the host and Xeon Phi. The coprocessor portion of the software stack consists of a modified Linux kernel, drivers and the standard Linux proc file system that can be used to query device state (for example, the load average). The coprocessor portion also has a SCIF driver to communicate over the PCI bus with the host and other nodes. Together the current Xeon Phi software stack is referred to as the Many Integrated Core (MIC) Platform Software Stack or MPSS for short.
The invention may be implemented in hardware, firmware or software, or a combination of the three. Preferably the invention is implemented in a computer program executed on a programmable computer having a processor, a data storage system, volatile and non-volatile memory and/or storage elements, at least one input device and at least one output device.
Each computer program is tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.
The invention has been described herein in considerable detail in order to comply with the patent Statutes and to provide those skilled in the art with the information needed to apply the novel principles and to construct and use such specialized components as are required. However, it is to be understood that the invention can be carried out by specifically different equipment and devices, and that various modifications, both as to the equipment details and operating procedures, can be accomplished without departing from the scope of the invention itself.
This application is a non-provisional of and claims priority to provisional applications with Ser. No. 61/754,371 filed on Jan. 18, 2013 and Ser. Nos. 61/761,969 and 61/761,985 both filed on Feb. 7, 2013, the contents of which are incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61754371 | Jan 2013 | US | |
| 61761985 | Feb 2013 | US | |
| 61761969 | Feb 2013 | US |
