Embodiments of the invention relate to the management of processor cores in a multicore processor system.
Most modern computing systems provide hot-plug support that allows a processor core to be powered on or off, or physically inserted or removed during operating system (OS) runtime. In a multicore processor system that supports hot-plug, the OS can unplug a processor core to remove it from the system, and can replug it back into the system on demand. A hot-pluggable system is adaptable to the changing capacity demand, as processor cores can be dynamically provisioned on demand. Moreover, for system reliability purposes, a hot-pluggable system can remove a faulty processor core during OS runtime, keeping the processor core off the system execution path.
While hot-plug saves power and enhances system reliability, the unplugging and re-plugging processes often require large amounts of time-consuming operations and memory manipulations. When a processor core is unplugged from a system, the processor core is offline and powered off. In a Linux system, the processor core is offline with respect to the OS and all of the OS subsystems, such as the scheduler, networking, memory management, power management, etc. All of its existing tasks are immediately migrated and future event-handling functions are immediately disabled. The entire, or nearly entire, data structures of its logical context are generally freed and its memory space is de-allocated. When the processor core is re-plugged back online, all of its data structures need to be re-initialized and its de-allocated memory re-claimed. Thus, the hot-plugging process is generally slow and power inefficient.
Some systems place a processor core into deep sleep when the processor core runs out of tasks and is predicted not to receive any tasks for a period of time. However, a sleeping processor core typically does not remain in the deep sleep for long as it may be frequently woken up to perform assigned tasks. Frequently transitioning a processor core between operation and deep sleep degrades system performance. Therefore, there is a need to improve the management of processor cores in a multicore processor system.
In one embodiment, a method is provided for operating a computing system that includes a plurality of processor cores. The method comprises transitioning a target processor core in the computing system from an active mode into a clearance mode according to a system policy. The system policy determines a number of processor cores to be in the active mode. The transitioning into the clearance mode further comprises: migrating work from the target processor core to one or more other processor cores in the active mode in the computing system; and configuring the computing system to prevent task assignment to the target processor core. While the target processor core is in the clearance mode, the target processor core is maintained in an online idle state in which the target processor core performs no work.
In another embodiment, a system that includes processor cores and memory. The memory contains instructions executable by the processor cores. The computing system is operative to transition a target processor core in the computing system from an active mode into a clearance mode according to a system policy. The system policy determines a number of processor cores to be in the active mode. To transition into the clearance mode, the kernel module is further configured to: migrate work from the target processor core to one or more other processor cores in the active mode in the computing system; and configure the computing system to prevent task assignment to the target processor core. While the target processor core is in the clearance mode, the target processor core is maintained in an online idle state in which the target processor core performs no work.
According to embodiments described herein, a multicore processor system provides a clearance mode for processor cores such that the system can have fast response, efficient power usage, and improved performance.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that different references to “an” or “one” embodiment in this disclosure are not necessarily to the same embodiment, and such references mean at least one. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
In the following description, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the understanding of this description. It will be appreciated, however, by one skilled in the art, that the invention may be practiced without such specific details. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.
It should be noted that the term “multicore processor system” as used herein may be arranged and managed as one or more clusters. A multicore processor system may be a multicore system or a multi-processor system, depending upon the system implementation. In other words, the proposed method may be applicable to any multicore system and multi-processor system arranged and managed as one or more clusters. For example, concerning the multicore system, all of the processor cores may be disposed in one processor. As another example, concerning the multi-processor system, each processor core may be disposed in one processor. Hence, in a multicore processor system, each cluster may be implemented as a group of one or more processors.
Embodiments of the invention provide a system and method for managing operations of processor cores in a multicore processor system. The system operates according to a system policy that determines the number of processor cores required to be in an active mode. According to the system policy, the system transitions a target processor core in the computing system from the active mode into a clearance mode. To transition the target processor core into the clearance mode, the system migrates work from the target processor core to one or more other processor cores in the active mode in the computing system, and configure the computing system to prevent task assignment to the target processor core. While the target processor core is in the clearance mode, the target processor core is maintained in an online idle state in which the target processor core performs no work.
The term “online,” as used herein, describes the perspective of the OS kernel in connection with a processor core. An “online” processor core means that the logical context of the processor core is available, and that the processor core is in an operable state from the OS kernel' view. An online processor is “alive” to the OS kernel, because it can be woken up and return to the active mode to receive task assignments. A processor core is “idle” when it performs no work.
In one embodiment, the multicore processor system 100 includes a multi-level cache structure. For example, each processor core may include or has exclusive access to an L1 cache, and processor cores of the same cluster may share the same L2 cache or additional levels of caches. In another embodiment, each processor core may include or has exclusive access to an L1 cache and an L2 cache, and processor cores of the same cluster may share the same L3 cache or additional levels of caches. It is noted that in further other embodiments, each processor core may include or has exclusive access to one or more cache, and processor cores of the same cluster may share the same one or more caches. In addition to the shared cache(s), processor cores of the same cluster may share other hardware components such as a memory interface, timing circuitry, and other shared circuitry. Processor cores in each cluster also have access to a system memory 130 via a cache coherence interconnect 110. The cache coherence interconnect 110 includes coherence circuitry (e.g., snooping circuitry) that keeps track of the status of the cache lines in the caches of each cluster to maintain cache coherence among the clusters.
In one embodiment, the multicore processor system 100 also includes an OS kernel 150 which includes or controls a number of system modules. These system modules may also be referred to as subsystems. In one embodiment, the system modules include a power management module 120, a scheduling module 140 and other system modules 160 such as networking module, memory management module, file system module, etc. The power management module 120 manages the power state of each processor core and each cluster to satisfy system design requirements such as to achieve energy efficiency. For example, the power management module 120 may transition any processor core in any cluster from a power-on state into a power-off or ultra-low power state, and any processor core in any cluster from the power-off or ultra-low power state into the power-on state. In one embodiment of the ultra-low power state, power consumed in the cluster can be as low as to retain data in the caches but not enough to afford logical calculations. The scheduling module 140 assigns and schedules tasks to the processor cores. The embodiment of
In one embodiment, any processor core and any cluster may enter the clearance mode according to a system policy that determines fewer than all of the processor cores in the system need to be active (e.g., due to reduced workload, power consumption requirement, etc.). The processor cores that enter the clearance mode include those in active operation of tasks, but do not need to be active according to the system policy. That is, a processor core enters the clearance mode not because it runs out of tasks and is predicated not to receive any task for a predetermined time period, but because the system policy dictates the clearance mode entrance and forces the existing and future tasks to be migrated or removed. A processor core in the clearance mode is in an online idle state. The processor core is “online” because it is alive to the OS kernel 150, and can be woken up and return to the active mode to receive task assignments. The processor core is “idle” because it performs no work. As the processor core performs no work, it can be powered-off or enter an ultra-low power state.
In comparison with the conventional sleep or deep sleep modes in which any processor core can be idle when there is no task in the run queue, a target processor core can be transitioned into the clearance mode according to a system-wide processor strategy to forcibly clear all workload in the target processor. Moreover, in the conventional sleep or deep sleep modes, if a processor core enters an idle state and is predicted to be woken up by a future awaking event (e.g., timer) a long time later (e.g., after a predetermined time period), the processor core can then sequentially or gradually transition to the lower power states or enter the powered off state. In contrast, if the target processor core enters an online idle state and is in the clearance mode, it can directly or immediately enter an ultra-low power state or be powered off as required. Furthermore, in the conventional sleep or deep sleep modes, the processor cores can be woken up from the sleep or deep sleep modes due to workload not being cleared from the processor cores. This can prevent the processor cores from remaining in the sleep or deep sleep modes for a longer period of time. In contrast, when the target processor core(s) is in the online idle state, the core configuration aware scheduling (CAS), system software, and applications can be aware that there are processor core(s) in the clearance mode and therefore do not assign tasks to the processor cores(s). The processor core(s) can therefore remain in the clearance mode as long as it is required by the system-wide processor strategy.
Each processor core may be associated with a logical context that contains pointers and data for its operation. For each processor core, the OS kernel 150 maintains a set of data structures 230 of the logical context associated with the processor core in one or more memory devices. In one embodiment, when a processor core enters the clearance mode, the OS kernel 150 maintains the entire data structures of the logical context associated with the processor core in one or more memory devices. In an alternative embodiment, when a processor core enters the clearance mode, the OS kernel 150 maintains at least a first portion of data structures of the logical context associated with the processor core, while freeing a second portion of the data structures of the logical context associated with the processor core. In this alternative embodiment when the processor core wakes up from (i.e., exits) the clearance mode, the OS kernel 150 only allocates and initializes the second portion of the data structures without allocating and initializing the first portion of the data structures. The first portion of the data structures may include the data structure(s) used by the scheduling module 140, and the second portion of the data structures may include the data structure(s) used by one or more other system modules 160. In an alternative embodiment, the second portion of the data structures may include the data structure(s) used by the scheduling module 140, and the first portion of the data structures may include the data structure(s) used by one or more other system modules 160. In further another embodiment, either of the first and second portions can include data structures used by either or both of the scheduling module 140 and one or more other system modules.
Compared to the conventional technologies that make a processor core go offline and remove all of the logical context associated with the offline processor core, the processor core in the clearance mode is kept online, without removing all of the associated logical context. While the disclosed clearance mode typically involve migration only, in some cases the clearance mode may involve both migration and thread parking. Thread parking refers to a facility which retains per process core thread data structure to avoid the full teardown and setup of per processor core threads. Moreover, with the conventional technologies, to restore a dead or offline processor core back to the online state, data structures associated with all modules affected by an online mask of the operating system kernel need to be initialized or re-initialized (including allocating or reclaiming memories), resulting in a heavy task and a lengthy operation time. In contrast, to restore an online processor core in the clearance mode back to the active mode is a lighter operation that can be completed in a shorter amount time.
It is noted that in the above embodiment, removing the target processor core from a scheduling configuration of the computing system is described as one method to prevent task assignment to the target processor core. However, in other embodiments, there can be different ways implemented to configure the computing system such that tasks will not be assigned to the target processor core in the clearance mode. For example, a measure of appropriateness associated with the target processor core can be changed such that in determining which one of the processor cores is more suitable or appropriate to take a task, it can be calculated from the measure that task assignment to the target processor core is less appropriate compared to task assignment to the other processor cores in the active mode. In one embodiment, a respective cost brought by task assignment to the target processor core or other processor cores may be calculated as a measure of appropriateness used for determining which processor core is more appropriate to execute the task. For example, the cost associated with task assignment to the target processor core in the clearance mode can be set to a higher value to such an extent that the target processor core in the clearance mode can avoid task assignments.
It is noted that a processor core in the clearance mode may preferably not undergo any operating parameter changes in order to maintain system stability and prevent system errors. The operating parameters may include voltage and frequency, which means that dynamic voltage frequency scaling (DVFS) is preferably not performed on the processor core in the clearance mode. To achieve this end, in one embodiment, modules in the multi-core processor system 100 can avoid adjusting operating parameters of the processor core in the clearance mode. For example, the modules may ignore any adjustment operation or requirement or request on the processor core in the clearance mode. Alternatively or additionally, a pseudo processor core, which exists logically or virtually rather than physically, can temporally stand in or serve as a substitute for the processor core in the clearance mode to undertake the adjustment of the operating parameters. The adjustment can be performed on the pseudo processor core until the processor core is changed back to the active mode. Moreover, when the processor core transitions back to the active mode, its operating perameters can be adjusted based on the final operating parameters of the pseudo processor core.
In one embodiment, the removal and migration of workload may occur at a time that is more appropriate for the processor core. For example, the target processor core may continue execution of a currently running task until the execution reaches a defined point before the currently running task is migrated for entering the clearance mode. In an alternative embodiment, the target processor core may delay migration of a currently running task from the target processor core until it is time for the target processor core to receive a next task scheduling event.
Furthermore, at step 302, the multicore processor system 100 is configured to prevent task assignment to the target processor core (e.g., by removing the target processor core from the scheduling configuration 220 of
At step 305, it is determined whether all processor cores in the cluster (in which the target processor core is located) are in the clearance mode; that is, none of the processor core in that cluster is in the active mode. If at least one processor core in the cluster is not in the clearance mode, the process 300 returns to step 301. When all processor cores of the cluster are in the clearance mode and their power states have transitioned into the power-off state or the ultra-low state, the one or more components shared by all of these processor cores may also transition into the power-off state or the ultra-low power state. These shared components may include one or more caches (such as the L2 cache), memory interfaces, etc. Before the shared components enter the power-off state, at step 306, one or more caches in the shared components may be flushed; for example, by writing their contents to the system memory, and the cache coherence between the cluster and one or more other clusters may be disabled. If the shared components are to enter the ultra-low power state, at step 307, the content of the caches in the shared components is maintained, and the cache coherence between the cluster and one or more other clusters is also maintained. Then at step 308, the shared components enter the power-off state or the ultra-low power state. At step 309, the cluster transitions into the power-off or ultra- low power state. The process 300 then returns to step 301 where the system determines whether another processor core in another cluster should enter the clearance mode according to a system policy.
When a processor core enters the clearance mode, system modules including the scheduling module 140 and one or more other system modules are prevented from waking up the processor core to handle work. The system may determine when the processor core may be woken up according to predetermined system policy. For example, when the system receives a task specifically directed to a target processor core in the clearance mode and in the power-off state or the ultra-low power state and where the task cannot be diverted or disabled, the target processor core may be powered on to handle the task. After the task is handled, the target processor core may transition back into the clearance mode and from the power-on state back to the power-off state or the ultra-low power state.
In one embodiment, a batch of processor cores may enter the clearance mode in parallel, and/or may exit the clearance mode in parallel. With the processor cores enter/exit the clearance mode in parallel, the scheduling configuration 220 may only need to be updated once to remove the processor cores from scheduling or to make the processor cores available for scheduling. In an alternative embodiment, a batch of processor cores may enter the clearance mode sequentially, and/or may exit the clearance mode sequentially.
In one embodiment, the method 500 begins when a computing system transitions a target processor core from an active mode into a clearance mode, according to a system policy that determines the number of processor cores to be in the active mode (step 501). The transition into the clearance mode further includes the steps of: migrating work from the target processor core to one or more other processor cores in the active mode in the computing system (step 502); removing the target processor core from a scheduling configuration of the computing system to prevent task assignment to the target processor core (step 503). When the target processor core is in the clearance mode, the target processor core is maintained in an online idle state in which the target processor core performs no work (step 504).
The operations of the flow diagrams of
Various functional components or blocks have been described herein. As will be appreciated by persons skilled in the art, the functional blocks will preferably be implemented through circuits (either dedicated circuits, or general purpose circuits, which operate under the control of one or more processors and coded instructions), which will typically comprise transistors that are configured in such a way as to control the operation of the circuity in accordance with the functions and operations described herein. The specific structure or interconnections of the transistors may be determined by a compiler, such as a register transfer language (RTL) compiler. RTL compilers operate upon scripts that closely resemble assembly language code, to compile the script into a form that is used for the layout or fabrication of the ultimate circuitry. RTL is well known for its role and use in the facilitation of the design process of electronic and digital systems.
While the invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the embodiments described, and can be practiced with modification and alteration within the spirit and scope of the appended claims. The description is thus to be regarded as illustrative instead of limiting.
This application claims the benefit of U.S. Provisional Application No. 62/152,224 filed on Apr. 24, 2015.
Number | Date | Country | |
---|---|---|---|
62152224 | Apr 2015 | US |