Patent Grant
10157155
All modern computing platforms implement a mechanism called interrupt handling. In general, a device generates a signal, called an interrupt, to the system to request an asynchronous service to be performed. In response to the interrupt, the system executes an interrupt handler. An interrupt handler is a computer program that, when executed by a processor, causes the system to perform the requested service, or other appropriate action, in response to that interrupt. The interrupt handler may in turn schedule additional operations to be performed to assist in responding to, or processing the data associated with, that interrupt.
In modern computing platforms, there typically are several devices that can generate interrupts, each with its own interrupt handler. The rate at which each device generates interrupts generally is variable. The amount of time taken to process each interrupt also can vary.
In a multiprocessor system, each individual interrupt is typically directed at a single processor; however some interrupt architectures allow an interrupt to be directed at a cluster of processors. In general, multiprocessor systems are designed such that interrupts are distributed among a subset of the system's processors while attempting to optimize one or more of overall system throughput, latency and power consumption. For example, in some systems, all interrupts are directed to a dedicated processor that executes the interrupt handlers. In some systems, each device or process that generates an interrupt, i.e., an interrupt source, is statically assigned a processor for its interrupt handler.
In some systems, a hardware-based interrupt controller can dynamically assign each interrupt to one of the processors based on information local to the interrupt controller. In some systems, the interrupt controller assigns a set of processors to handle each interrupt in a round robin fashion among the set of processors, each of which is programmed with the various interrupt handlers. In some systems, the interrupt controller broadcasts each interrupt to a set of processors, each of which selects whether to accept the interrupt. In some systems, the interrupt is directed to the processor, within a subset of processors, which is currently handling the lowest priority task.
This Summary introduces selected concepts in simplified form that are further described below in the Detailed Description. This Summary is intended neither to identify key or essential features of the claimed subject matter, nor to limit the scope of the claimed subject matter.
An operating system is provided in which an interrupt router dynamically steers each interrupt source to a processor within set of processors based on overall load information from the set of processors. An interrupt source is assigned to a processor based on the load imposed by the associated interrupts and the target overall load for the processor. For example, each processor can maintain information about each interrupt it processes over time. The operating system receives this historical load information to determine an expected load due to interrupts of a given type from a given device, an overall load on the system, and a target load for each processor. Given a set of interrupt sources, their expected loads, and target load for each processor, each interrupt source can be assigned dynamically to a processor during runtime of the system. On a regular basis, these assignments can be changed given current operating conditions of the system. The assignments also can be determined based on the current power state of each processor, so as to avoid activating an idle processor solely to process an interrupt, and to allow processors to become idle, to save power.
One challenge is measuring the overall load due to an interrupt source, because an interrupt handler can invoke additional processing on the same processor in response to handling an interrupt.
Accordingly, in one aspect, in a computer including a plurality of processors, an interrupt router receives information about interrupts. The interrupt router determines a load on the computer due to interrupt handling by the plurality of processors for the interrupts. The interrupt router assigns each interrupt source to a selected one or more of the plurality of processors, such selection being a function of the determined load so as to distribute the load among the processors. The interrupt router can periodically repeat determining a load on the computer due to interrupt handling, and assigning each interrupt source to a selected one of the plurality of processors.
In one implementation, the interrupt router identifies a number of processors, from among the plurality of processors, available for processing interrupts, and selects a number of processors, from among the identified number of processors available for processing interrupts, such that the number of selected processors matches the determined load divided by a target per-processor load. The load can be determined by, in response to each interrupt, executing an interrupt handler for the interrupt on the processor assigned to the interrupt, and storing, in memory associated with the processor assigned to the interrupt, data indicating an amount of processing time consumed due to executing the interrupt handler. The interrupt router aggregates the stored data from the plurality of processors for the interrupts.
The amount of processing time can be determined by storing a system time stamp when beginning execution of the interrupt handler, computing a difference between a system time stamp observed when ending execution of the interrupt handler, and storing the computed difference. Determining the amount of processing time can further include, for any process invoked by the interrupt handler, storing a system time stamp when beginning execution of the process, computing a difference between a system time stamp observed when ending execution of the process, and storing data indicative of the computed difference for the process and the computed difference for the interrupt handler. Determining the amount of processing time can further include, for any preemptive activity, such as an interrupt or other process preempting interrupt processing by another interrupt handler or process performing associated work, computing a difference between system time stamps observed when pausing the preempted interrupt processing and when restarting the preempted interrupt processing to have an amount of time for executing the preemptive activity, such that the computed difference for the interrupt handler excludes the amount of time for executing the preemptive activity.
These various aspects and implementations can be embodied in a computer-implemented process, computer, or an article of manufacture including computer storage media.
In the following description, reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific example implementations of this technique. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the disclosure.
The following section provides an example operating environment in which operating-system based interrupt steering can be implemented.
Referring to
With reference to
Additionally, computer 100 may also have additional features/functionality. For example, computer 100 may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in
Computer 100 may also contain communications connection(s) 112 that allow the device to communicate with other devices over a communication medium. Communication media typically carry computer program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal, thereby changing the configuration or state of the receiving device of the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Communications connections 112 are devices that interface with the communication media to transmit data over and receive data from communication media, such as a network interface.
Computer 100 may have various input device(s) 114 such as a keyboard, mouse, pen, camera, touch input device, and so on. Output device(s) 116 such as a display, speakers, a printer, and so on may also be included. All of these devices are well known in the art and need not be discussed at length here. Various input and output devices can implement a natural user interface (NUI), which is any interface technology that enables a user to interact with a device in a “natural” manner, free from artificial constraints imposed by input devices such as mice, keyboards, remote controls, and the like.
Examples of NUI methods include those relying on speech recognition, touch and stylus recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence, and may include the use of touch sensitive displays, voice and speech recognition, intention and goal understanding, motion gesture detection using depth cameras (such as stereoscopic camera systems, infrared camera systems, and other camera systems and combinations of these), motion gesture detection using accelerometers or gyroscopes, facial recognition, three dimensional displays, head, eye, and gaze tracking, immersive augmented reality and virtual reality systems, all of which provide a more natural interface, as well as technologies for sensing brain activity using electric field sensing electrodes (EEG and related methods).
Each component of this system that operates on a computer generally is implemented by software, such as one or more computer programs, which include computer-executable instructions and/or computer-interpreted instructions, such as program modules, being processed by the computer. Generally, program modules include routines, programs, objects, components, data structures, and so on, that, when processed by a processing unit, instruct the processing unit to perform particular tasks or implement particular abstract data types. This computer system may be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
Alternatively, or in addition, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
Given a computer such as described above, an operating system based interrupt router that performs dynamic interrupt steering at runtime among multiple processors assumes that the computer includes a central processing unit (e.g., 120 in
Interrupt handling generally is managed by the operating system, firmware or other low level software on the computer. In one implementation, the operating system includes an interrupt router in the kernel that is executed on one or more of the processors and that dynamically steers interrupts to a subset of processors which may include the processor on which the interrupt router is executed. Alternatively, other components of an operating system, hardware abstraction layer, firmware or other low level computer programs of a computer can implement such functionality.
Referring now to
It should be understood that the number of processors, i.e., four, in
Referring now to
Other data structures, not shown, include, for each processor, a count of the number of interrupt sources assigned to the processor as part of the interrupt data for each processor. Additionally, a current list of interrupt source-to-processor assignments also is stored by the interrupt router.
Steering of interrupts to different processors generally involves four steps. First, the interrupt router identifies which processors, of the available processors in the system, are to be used. Second, the interrupt router determines how to distribute interrupts among the identified processors. This interrupt source-to-processor assignment is set in an interrupt controller and used for a period of time. Third, the interrupt router dynamically changes the distribution of interrupts in response to system conditions during runtime. The result of this process changes the current interrupt source-to-processor assignments to new assignments. Fourth, the interrupt router tracks the impact of the interrupts from the various interrupt sources on system performance, which is used to determine future interrupt source-to-processor assignments. It should be understood that the process may assign interrupt sources to individual processor clusters as well as or instead of individual processors.
Referring now to
This process begins by identifying 400 an initial set of available processors. This initial set can be all of the available processors, a predetermined subset of the available processors, or other set of processors identified based on a specified characteristic. As an example, the initial set of processors can be all active, i.e., non-idle, processors. Next, some of these processors can be eliminated 402 based on the time for which they have been active. The amount of time can be a tunable parameter, which, when set to zero, disables this step.
The interrupt router then determines 404 the load on the system due to interrupt handling, in a manner described in more detail below. Then, a number of processors to be used for handling interrupts, given a target per-processor load and the determined load, is then computed 406.
In one example implementation, a total sum of all processor time used to handle all interrupt-related work is computed. This total amount of time is divided by the actual elapsed time for the entire system to process the interrupts, to determine a percent of time spent processing interrupts and related work. This percent of time is then divided by a target per-processor load, which can be a tunable parameter, to determine a target number of processors to handle that load. For example, if each of eight processors handled interrupts for 20 ms, the total load is 160 ms. If this processing occurred in 100 ms, then the total load is 160% (of one of the processors). If the target processing load is 40%, then the target number of processors is four.
Other algorithms can be used to select the number of processors. For example, statistics about which interrupt sources are primarily responsible for the load can be examined. If one interrupt source is responsible for a load that is significantly greater than the target per-processor load, then it is possible that this interrupt source can be assigned to one processor and fewer processors can be used for the remaining interrupts.
The computed number can be capped 408 to the number of processors in the initial set of processors identified at 400. If this number of processors is less than the size of the initial set, as determined at 410, then a subset of the initial set is selected 412.
In one example implementation, the subset can be selected by using a same algorithm used by the kernel to assign threads to processors thus maximizing a number of processors that can be idle to reduce power consumption.
In this example implementation, a set of processors is selected (see
First, interrupt sources are sorted 500 in order of the load imposed on the system by the interrupt source. Next, an interrupt source from the list is selected 502. A processor from the set of processors is selected 504 and assigned to the selected interrupt source in the sorted list. If interrupt sources remain, as determined in 506, the process continues by repeating the selection steps 502 and 504, until all interrupt sources are assigned one of the processors. After assignments are completed, the interrupt controller is set 508 to direct interrupts to their assigned processors.
There are a variety of different ways to match processors and interrupts, and the process in
Using the example of
Other implementations are possible, with the general objective of evenly dividing the load due to handling interrupts among the processors, trading off computational complexity against a more optimal distribution.
In one example, a worst-fit bin packing algorithm can be used. In this algorithm, each subsequent interrupt source is assigned to the processor that will have the least total load after the assignment.
In another example, the serpentine selection of the next processor is followed, with the condition that the current processor has a higher total load than the next processor; otherwise the current processor is used again for the next interrupt.
In another example, an entire processor is first reserved for each interrupt for which the load is greater than the target per-processor load. Then, the remaining interrupts are assigned to the remaining processors.
In another example, interrupt sources having little impact on the system can be assigned permanently to certain processors and removed from this process of assigning interrupt sources to processors.
Given an assignment of interrupt sources and processors, this assignment can be dynamically changed during operation of the computer as system conditions change. The result of this process is to allow the system to transition active processors into idle states and idle processors into active states, while not missing interrupts.
An example implementation is shown in
Thus, in
Each interrupt source-to-processor assignment is again processed 610 and the current assignment is set to the new assignment in the interrupt controller. For each interrupt source where the target processor changed, the count of interrupt sources assigned to the previous target processor for that interrupt is decremented 612. On each such increment, if the previous target processor's count of interrupt sources changes from one to zero, as determined at 614, the processor is added to a list. Instructions are then sent 616 to each processor on this list, in response to which each processor can evaluate whether it is idle and can transition into a power saving state due to no longer having any interrupts targeted at it.
The process in
An example implementation of a process for tracking the impact of interrupt handling as associated work on the system will now be described in connection with
A challenge with measuring the impact of interrupts is the fact that an interrupt handler can invoke additional processes and generate additional interrupts, or other work generated by an interrupt handler that is performed on the same processor as the interrupt handler. Such additional work, in turn, can create yet more additional work on a processor. Measurement and tracking of such work uses the data structure described above, an example of which is shown in
For each interrupt source and processor in the system, fields are allocated 700 in this data structure to accumulate the amount of time consumed by the interrupt source's associated interrupt handler and any additional work it invokes on the processor.
When an interrupt handler is executed, the amount of time it spends executing is tracked and stored 702 on its processor. For example, a time stamp is read both prior to and after such execution. The difference is calculated and stored in the data structure. If the interrupt handler schedules additional work, such as by invoking another process, the amount of time the additional work is performed also is determined. For example, when additional work is queued on the processor, a pointer to the data structure can be queued along with that work. If the interrupt handler, or its related work, is preempted by another higher priority task, then the higher priority task adjusts the time stamp data of the preempted interrupt handler or related work.
Periodically, the interrupt data from the processors is accumulated 704 by the interrupt router, and stored 706 in the allocated data structure so as to accumulate the interrupt handling statistics across the system.
It should be understood that the foregoing is only an example implementation and that other implementations of tracking also are possible. In general, each interrupt handler and its additional work track the amount of time spent processing the related interrupt. This data is collected over the multiple interrupts and multiple processors.
With such information, interrupt sources can be dynamically assigned to processors, and such assignments can be dynamically changed, during runtime of the computer. By managing the assignments efficiently, interrupts can be distributed among processors while maximizing a number of processors that can be idle to reduce power consumption.
Any or all of the aforementioned alternate embodiments described herein may be used in any combination desired to form additional hybrid embodiments. It should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific implementations described above. The terms “article of manufacture”, “process”, “machine” and “composition of matter” in the preambles of the appended claims are intended to limit the claims to subject matter deemed to fall within the scope of patentable subject matter defined by the use of these terms in 35 U.S.C. § 101. The specific implementations described above are disclosed as examples only.
This application is a continuation of U.S. patent application Ser. No. 13/917,634, filed Jun. 13, 2013 and entitled “Operating System-Managed Interrupt Steering in Multiprocessor Systems”, pending, which is hereby incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5446910 | Kennedy et al. | Aug 1995 | A |
| 5944840 | Lever | Aug 1999 | A |
| 6237058 | Nakagawa | May 2001 | B1 |
| 6604136 | Chang | Aug 2003 | B1 |
| 6813665 | Rankin et al. | Nov 2004 | B2 |
| 7328294 | Kim et al. | Feb 2008 | B2 |
| 7581052 | Solomita | Aug 2009 | B1 |
| 7694055 | Orita et al. | Apr 2010 | B2 |
| 7761720 | Kaushik et al. | Jul 2010 | B2 |
| 7805555 | Takata et al. | Sep 2010 | B2 |
| 7962771 | Song et al. | Jun 2011 | B2 |
| 8032681 | Crossland et al. | Oct 2011 | B2 |
| 8321614 | Wolfe | Nov 2012 | B2 |
| 8996773 | Ho et al. | Mar 2015 | B2 |
| 9032127 | Fischer et al. | May 2015 | B2 |
| 9424212 | Raffman | Aug 2016 | B2 |
| 20020124196 | Morrow et al. | Sep 2002 | A1 |
| 20030061423 | Rankin et al. | Mar 2003 | A1 |
| 20030105798 | Kim et al. | Jun 2003 | A1 |
| 20030120702 | Jahnke | Jun 2003 | A1 |
| 20030200250 | Kiick | Oct 2003 | A1 |
| 20050125582 | Tu et al. | Jun 2005 | A1 |
| 20060123422 | Felter et al. | Jun 2006 | A1 |
| 20070005742 | Eldar et al. | Jan 2007 | A1 |
| 20070043970 | Solomita et al. | Feb 2007 | A1 |
| 20070143514 | Kaushik et al. | Jun 2007 | A1 |
| 20090089470 | Ven | Apr 2009 | A1 |
| 20100057967 | Murakami et al. | Mar 2010 | A1 |
| 20110145461 | Zhao et al. | Jun 2011 | A1 |
| 20140359184 | Kang et al. | Dec 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 1737765 | Feb 2006 | CN |
| 101872315 | Oct 2010 | CN |
| 2013066124 | May 2013 | WO |
| Entry |
|---|
| Structured Computer Organization, Third Edition, Andrew S. Tanenbaum pp. 11-13 (Year: 1990). |
| “Chapter 4. Interrupts and Exceptions”, Retrieved from: <<http://linux.linti.unlp.edu.ar/images/0/0c/ULK3-CAPITULO4- UNNOBA.pdf>>, Retrieved on: Mar. 4, 2013, 52 Pages. |
| “Final Office Action Issued in U.S. Appl. No. 13/917,634”, dated Nov. 25, 2015, 8 Pages. |
| “Non-Final Office Action Issued in U.S. Appl. No. 13/917,634”, dated Aug. 4, 2015, 8 Pages. |
| “Notice of Allowance Issued in U.S. Appl. No. 13/917,634”, dated Apr. 27, 2016, 5 Pages. |
| McDougall, et al., “Chapter 3.11—Interrupts”, In Book—Solaris Internals: Solaris 10 and OpenSolaris Kernel Architecture, Second Edition, A Prentice Hall Publication, Jul. 10, 2006, 7 Pages. |
| Mukherjee, Satyaki, “Migrating Software to Multicore SMP”, Retrieved from: <<http://www.arm.com/files/pdf/AT-_Migrating_Software_to_Multicore_SMP_Systems.pdf>>, Retrieved on: Mar. 4, 2013, 39 Pages. |
| “International Search Report & Written Opinion Issued in PCT Application No. PCT/US2013/060243”, dated Feb. 25, 2014, 12 Pages. |
| Tumeo, et al., “An Interrupt Controller for FPGA-based Multiprocessors”, In Proceedings of International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation, Jul. 16, 2007, pp. 82-87. |
| “Summons to attend Oral Proceedings Issued in European Patent Application No. 13777361.0”, Mailed Date: Oct. 12, 2017, 8 Pages. |
| “Office Action Issued in European Patent Application No. 13777361.0”, dated Nov. 11, 2016, 6 Pages. |
| “Office Action Issued in Chinese Patent Application No. 201380077383.6”, dated Mar. 5, 2018, 9 Pages. (W/o English Translation). |
| Number | Date | Country | |
|---|---|---|---|
| 20160357689 A1 | Dec 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13917634 | Jun 2013 | US |
| Child | 15241104 | US |
