Certain computing systems provide the ability to disable or physically replace a processor without powering off or rebooting the system. For example, in certain systems a processor may be disabled or removed in response to a fault that triggers the removal of the processor. Additionally, in certain other systems a processor may be added or removed to change system capacity or to support domain partitioning.
In an operational computing system, one or more active Input/Output (I/O) devices may actively generate interrupts directed to the processors in the computing system. A computing system can malfunction if an I/O device attempts to send an interrupt to a processor that is being removed. The malfunctioning may be prevented by stopping and restarting, or suspending and resuming the operations of I/O devices when a processor is being removed. Stopping and restarting, or suspending and resuming the operations of an I/O device may decrease the performance of the computing system by consuming processing time. In certain cases, stopping and restarting, or suspending and resuming the operations of an I/O device may cause the computing system to be rebooted.
Furthermore, stopping and restarting, or suspending and resuming an I/O device may require software support from the corresponding I/O device driver stack. The software support may have to be provided by the vendor of the I/O device and may have to be replicated for each I/O device. If there are any defects or deficiencies in the I/O device driver stack, the stopping and restarting, or suspending and resuming routines may fail.
Referring now to the drawings in which like reference numbers represent corresponding parts throughout:
In the following description, reference is made to the accompanying drawings which form a part hereof and which illustrate several embodiments. It is understood that other embodiments may be utilized and structural and operational changes may be made.
The computing platform 102 may comprise a computational device, such as, a personal computer, a workstation, a server, a mainframe, a hand held computer, a palm top computer, a laptop computer, a telephony device, a network computer, a blade computer, etc. The computing platform 102 comprises a plurality of processors 106a, 106b, . . . , 106n that include a plurality of local interrupt controllers 108a, 108b, . . . 108n. For example, in certain embodiments, the processor 106a may include the local interrupt controller 108a, the processor 106b may include the local interrupt controller 108b, and the processor 106n may include the local interrupt controller 108n. In certain embodiments, the processors 106a . . . 106n may be central processing units (CPU), and the local interrupt controllers 108a . . . 108n may be Advanced Programmable Interrupt Controllers (APIC). A local interrupt controller, such as, local interrupt controllers 108a . . . 108n, may store interrupts that are to be processed by the corresponding processor that includes the local interrupt controller. For example, the local interrupt controller 108a may store interrupts that are to be processed by the processor 106a. The interrupts may be generated by the plurality of devices 104a . . . 104m, where the devices 104a . . . 104m may be I/O devices or other devices that generate interrupts.
In addition to the processors 106a . . . 106n and the local interrupt controllers 108a . . . 108n, the computing platform 102 may also comprise an operating system 110, an interrupt migrator 112, one or more device drivers 114 that include one or more interrupt service routines 116, an interrupt mapping data structure 118, and an affected interrupts data structure 120.
The operating system 110 may include system programs that allow applications, such as, the interrupt migrator 112 and the device driver 114 to execute in the computing platform 102. The interrupt migrator 112 is an application that may be implemented in hardware, software, firmware or any combination thereof. The interrupt migrator 112 allows interrupts directed towards one processor to be retargeted towards another processor. For example, in certain embodiments if processor 106a is disabled then the interrupt migrator 112 may retarget an interrupt that is supposed to be processed by processor 106a to be processed by processor 106b instead. In certain embodiments, the devices 104a . . . 104m do not have to be disabled or suspended when one of the plurality of processors 106a . . . 106n is disabled.
The device drivers 114 correspond to the devices 104a . . . 104m. For example, each device, such as, device 104a, may have a corresponding device driver that interfaces the device to the operating system 110. The interrupt service routines 116 implemented in the device drivers 114 may process interrupts received from the devices 104a . . . 104m.
The interrupt mapping data structure 118 includes mappings of interrupts to processors. For example, an entry in the interrupt mapping data structure 118 may map a first interrupt to be processed by the processor 106a, and a second interrupt to be processed by the processor 106c.
In certain embodiments, while the interrupt migrator 112 is retargeting interrupts directed towards a first processor to a second processor, additional interrupts intended for the first processor may be received at the computing platform 102. If the first processor needs to be disabled, the affected interrupts data structure 120 stores these additional interrupts, such that, these additional interrupts are not processed by the first processor that is to be disabled. Additionally, the affected interrupts data structure 120 may also include indications for interrupts that are already in the local interrupt controller of a processor that is to be disabled.
The interrupt mapping data structure 118a, that is an exemplary embodiment of the interrupt mapping data structure 118, includes entries 202, such that for each entry an interrupt source device 204 and an interrupt destination 206 is indicated. For example, in the exemplary interrupt mapping data structure 118a, entry 208 may correspond to interrupt number 0001, where the interrupt source device is I/O device #1 (I/O device 104a) and the interrupt destination is CPU #1 (processor 106a), i.e., interrupt number 0001 generated by device 104a is supposed to be directed to CPU #1, i.e., the processor 106a, for processing. Similarly, entry 210 corresponds to interrupt number 0002, where the interrupt source device is I/O device #2 (device 104b) and the interrupt destination is CPU #3, (processor 106n), and entry 212 corresponds to interrupt number 0003, where the interrupt source device is I/O device #1 (device 104a) and the interrupt destination is CPU #2, (processor 106b). In certain alternative embodiments, the interrupt mapping data structure 118a may be implemented differently from the representation shown in
The affected interrupts data structure 120 includes an indicator for interrupts pending in local interrupt controller of processor to be disabled 214 and an indicator for interrupts pending from interrupt source devices while retargeting interrupts 216. For example, in the exemplary computing platform 102, the indicator for interrupts pending in local interrupt controller of processor to be disabled 214 may indicate the pending interrupts 200a in the processor 106a, if the processor 106a is to be disabled. While the interrupt migrator 112 retargets interrupts from one processor to another, interrupts may arrive at the computing platform 102. Some of the arriving interrupts may be for the processor that is to be disabled. Such arriving interrupts may be indicated in the indicator for interrupts pending from interrupt source devices while retargeting interrupts 216.
In certain exemplary embodiments, a processor, such as, processor 106a, may need to be disabled in the computing platform 102 illustrated in
If the exemplary processor 106a is to be disabled, then in certain embodiments, the interrupt migrator 112 updates the interrupt mapping data structure 118a of
While the interrupt mapping data structure 118b is being modified, if new interrupts that target the processors to be disabled arrive from the devices 104a . . . 104m, such new interrupts are indicated in the indicator for interrupts received from interrupt source devices while retargeting interrupts 304.
In an exemplary embodiment, after modifications have been completed in the interrupt mapping data structure 118b, the pending interrupts 200a in the processor 106a that to be disabled are processed by the processor 106a that to be disabled. As each pending interrupt 200a is processed, the pending interrupt that has completed processing is removed from the indicator for interrupts pending in local interrupt controller of processor to be disabled 302.
In an exemplary embodiment, after modifications have been completed in the interrupt mapping data structure 118b, the interrupts indicated by the indicator for interrupts received for interrupt source devices while retargeting interrupts 304 are processed by a processor that is not to be disabled, such as, processor 106c. After all indicators indicated in the affected interrupts data structure 120 have been processed, the processor to be disabled is disabled. New interrupts from the devices 104a . . . 104m are processed by determining the processor corresponding to the interrupt destination 206 in the interrupt mapping data structure 118b.
Control starts at block 400, where the interrupt migrator 112 determines one of a plurality of processors 106a . . . 106n to disable, wherein the plurality of processors 106a . . . 106n are capable of processing interrupts from at least one device, such as, a device 104a included in the plurality of devices 104a . . . 104m. An indication for determining which processor to disable may be generated by the operating system 110 or by some other application, and the indication may be forwarded to the interrupt migrator 112 to determine the processor to disable. For example, in certain exemplary embodiments the interrupt migrator 112 may determine that processor 106a is to be disabled.
The interrupt migrator 112 communicates (at block 402) an interrupt directed at the determined processor 106a to at least one other processor 106b . . . 106n of the plurality of processors 106a . . . 106n while receiving the interrupts from the at least one device 104a. For example, in certain embodiments if an interrupt is directed at the processor 106a that is to be disabled, the interrupt may be redirected to one other processor, such as, processors 106b . . . 106n. In certain embodiments during the process of communicating, additional interrupts may continue to be received from the at least one device 104a. For example, in certain embodiments the devices 104a . . . 104m are neither stopped or suspended.
Subsequently, the interrupt migrator 112 disables (at block 404) the determined processor. For example, in certain embodiments, the interrupt migrator 112 disables the processor 106a and subsequent interrupts are processed by the remaining processors 106b . . . 106n.
In certain embodiments, disabling the at least one device 104a from which the interrupts are received causes an execution error in the plurality of processors 106a . . . 106b. For example, in certain embodiments if any of the devices 104a . . . 104m are stopped or suspended an execution error may occur in the computing platform 102. Additionally, if a device implements swapping or paging then the device cannot be disabled even temporarily. In certain embodiments, the devices 104a . . . 104m are not disabled while a processor is being disabled.
In certain embodiments, the plurality of processors 106a . . . 106n are CPUs, wherein the at least one device 104a is an input/output device, wherein the communicated interrupt is stored in a plurality of APICs 108a . . . 108n in the central processing units.
Control starts at block 500, where the interrupt migrator 112 disables interrupt processing in each of the plurality of processors 106a . . . 106n. Disabling the interrupt processing implies that certain interrupts from the devices 104a . . . 104m may time out and may need to be regenerated. However, the devices 104a . . . 104m are not disabled.
The interrupt migrator 112 maps (at block 502) an interrupt destination 206 corresponding to the interrupt to the at least one other processor, such as, processors 106b . . . 106n, in an interrupt mapping data structure (such as, 118, 118b), wherein the interrupts from the at least one device, such as, device 104a, are received during the mapping of the interrupt destination 206. For example, in certain embodiments the interrupt migrator 112 may substitute the entry 300 for the entry 208 in the interrupt mapping data structure 118a (causing a generation of the interrupt mapping data structure 118b) for mapping interrupts targeted at the processor 106a that to be disabled to the processor 106c that is not to be disabled.
After the interrupt mapping data structure 118, 118b has been updated to correctly map the interrupt destinations 206, the pending interrupts in the local interrupt controller of the processor to be disabled should be processed. The interrupt migrator 112 enables (at block 504) the interrupt processing in the determined processor, such as, processor 106a. The interrupt migrator 112 processes (at block 506) at least one pending interrupt in a local interrupt controller, such as, local interrupt controller 108a, of the determined processor, such as, processor 106a. At the conclusion of block 506, the pending interrupts 200a shown in
The interrupt migrator 112 processes (at block 508) the interrupts that were received from the at least one device, such as, device 104a, during the mapping of the interrupt destination 206. In certain embodiments, such interrupts received from the at least one device during the mapping of the interrupt destination 206 may have been indicated in the indicator for interrupts received from interrupt source device while retargeting interrupts 304 in the affected interrupts data structure 120.
Control starts at block 600, where the interrupt migrator 112 determines all entries in the interrupt mapping data structure 118, 118a, 118b whose possible interrupt destination 206 is the determined processor 104a.
The interrupt migrator 112 suspends (at block 602) interrupt processing corresponding to the determined entries while receiving the interrupts from the at least one device 104a. The interrupt migrator 112 changes (at block 604) interrupt destinations 206 corresponding to the determined entries to the at least one other processor 106b . . . 106n.
The interrupt migrator 112 indicates (at block 606) the determined entries in an affected interrupts data structure 120. Subsequently, the interrupt migrator 112 resumes (at block 608) the suspended interrupt processing corresponding to the determined entries.
Control starts at block 700, where the interrupt migrator 112 reads a local interrupt controller 108a of the determined processor 106a. The determined processor 106a is the processor that is to be disabled.
The interrupt migrator 112 initiates (at block 702) processing of all interrupts in the local interrupt controller 108a of the determined processor 106a. The interrupt migrator 112 calls (at block 704) interrupt service routines 116 corresponding to all the interrupts in the local interrupt controller 108a of the determined processor 106a.
Subsequently, the interrupt migrator 112 generates (at block 706) an end of interrupt command to indicate a completion of handling of all the interrupts in the local interrupt controller 108a.
Control starts at block 800, where the interrupt migrator 112 determines entries in the interrupt mapping data structure 118, 118b whose interrupt destination 206 was mapped during a time period in which a corresponding interrupt source device 204 generated an interrupt while the interrupt destination 206 was being mapped. In certain embodiments, these determined entries may be present in the indicator for interrupts received from interrupt sources devices while retargeting interrupts 216, 304 of the affected interrupts data structure 120.
The interrupt migrator 112 invokes (at block 802) a corresponding interrupt service routine 116 in a device driver 114 corresponding to the interrupt source device 204. The interrupt migrator 112 receives (at block 804) a completion indication from the interrupt service routine 116.
At the completion of the operations described in
Certain embodiments describe a set of operations that are executed in a specified order such that devices 104a . . . 104m do not have to be disabled while a processor 106a . . . 106a is being disabled and corresponding interrupts are being retargeted to a processor that is not be disabled. No modifications are needed to device driver stacks. Additionally, no interrupts are lost and after the disablement of a processor, interrupts are not targeted to the processor that is disabled.
Certain embodiments allow an operating system to dynamically load balance interrupt loads on processors in a multi-processor system. Some embodiments allow hot-plugging CPUs, adjusting number of processors based on demand, and dynamic domain partitioning.
In certain embodiments, when a processor has to be removed while a system is operating, the system ensures that no device in the system will attempt to send an interrupt to a processor that has to be removed. All devices may continue to operate without being notified that a processor was going to be disabled or physically removed.
In certain embodiments, when the single processor is disabled or physically removed, the operating system is able to reprogram the interrupt destination of interrupts asserted by the I/O devices without stopping, restarting, or suspending the operations of the I/O devices.
In certain embodiments, where an operating system may need to change the destination processor information for a storage I/O controller that hosts the paging or swap file. The embodiments do not require the operating system to stop and restart the storage I/O controller that hosts the paging or swap file, since the operating system cannot deal with a situation in which the paging or swap file is unavailable even temporarily. Certain embodiments allow an operating system to be able to reprogram interrupt destination information without stopping and restarting, or suspending and resuming I/O devices.
If an operational I/O device tries to assert an interrupt just at the time the operating system or the interrupt migrator is attempting to change the destination processor information of the I/O device a potential race condition can occur that may cause a system to enter into an unpredictable or unstable state. In certain embodiments such potential race conditions are prevented.
Certain embodiments allow the operating system to safely reprogram the interrupt destinations of I/O devices in an operational system. Certain embodiments do not require the I/O devices to be in a quiescent, i.e., stopped or suspended, state. Certain embodiments do not require I/O devices to be suspended or stopped and does not require any changes or special support in the I/O device driver software. In certain embodiments, the operating system may be able to support capacity on demand, i.e., increase or decrease the number of processors based on the processing load. Additionally, in certain embodiments processors may be dynamically inserted or removed in an operational system. In alternative embodiments, message signaled interrupts may be used instead of using the I/O APIC interrupt controller.
The described techniques may be implemented as a method, apparatus or article of manufacture involving software, firmware, micro-code, hardware and/or any combination thereof. The term “article of manufacture” as used herein refers to program instructions, code and/or logic implemented in circuitry [e.g., an integrated circuit chip, Programmable Gate Array (PGA), Application Specific Integrated Circuit (ASIC), etc.] and/or a computer readable medium (e.g., magnetic storage medium, such as hard disk drive, floppy disk, tape), optical storage (e.g., CD-ROM, DVD-ROM, optical disk, etc.), volatile and non-volatile memory device (e.g., Electrically Erasable Programmable Read Only Memory (EEPROM), Read Only Memory (ROM), Programmable Read Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, firmware, programmable logic, etc.). Code in the computer readable medium may be accessed and executed by a machine, such as, a processor. In certain embodiments, the code in which embodiments are made may further be accessible through a transmission medium or from a file server via a network. In such cases, the article of manufacture in which the code is implemented may comprise a transmission medium, such as a network transmission line, wireless transmission media, signals propagating through space, radio waves, infrared signals, etc. Of course, those skilled in the art will recognize that many modifications may be made without departing from the scope of the embodiments, and that the article of manufacture may comprise any information bearing medium known in the art. For example, the article of manufacture comprises a storage medium having stored therein instructions that when executed by a machine results in operations being performed. Furthermore, program logic that includes code may be implemented in hardware, software, firmware or many combination thereof. The described operations of
Certain embodiments illustrated in
In certain embodiments, the storage device 1014 may be absent in the system 1000. Instead of the storage device 1014, in alternative embodiments the system 1000 may include another device, such as, a video or graphics device that renders information to display on a monitor coupled to the system 1000, where the system 1000 may comprise a desktop, workstation, server, mainframe, laptop, handheld computer, etc. An operating system may be capable of execution by the system, and the video controller may render graphics output via interactions with the operating system. Alternatively, some embodiments may be also be implemented in a computer system that does not include a video or graphics controller but includes a switch, router, etc.
At least certain of the operations of
The data structures and components shown or referred to in