Patent Grant
10282299
This description relates to managing cache partitions based on cache usage information.
Many modern processors support hierarchical cache systems with multiple levels of cache, including one or more levels within the processor or within each core of a multi-core processor, and one or more levels external to the processor or cores, up to a last level cache (LLC) that is accessed just before main memory is accessed. At each level of the hierarchy, the cache stores copies of a subset of data to speed access to that data by the processor relative to the speed of a higher level cache (or relative to the speed of the main memory for the LLC). Lower level caches are closer to the processor (or core), whereas higher level caches are further away from the processor (or core). The LLC is typically shared by all of the cores of a multi-core processor. At each level, the cache system will load blocks of data into entries and evict blocks of data from entries in units of “cache lines” (also called “cache blocks”). Each cache line includes a number of “words” of data, each word consisting of a predetermined number of bytes.
A technique called “cache partitioning” (also called “cache allocation”) involves managing the way in which a particular cache is shared among multiple entities. For example, these entities may be different cores and the particular cache that is shared among the cores may be the LLC. In some examples, the management involves limiting a particular core to allocating new cache lines only within a particular assigned region of the LLC. By doing so, the cache will still be able to access data upon a hit in any region, but is limited to allocating (and if necessary evicting) data upon a miss only in its assigned region.
In one aspect, in general, an apparatus includes: a plurality of processor cores, with each of the plurality of processor cores including storage configured to store at least one core-specific identifier; and a shared memory system connected by interconnection circuitry to each of the plurality of processor cores. The shared memory system includes: a cache shared by more than one of the plurality of processor cores, the cache comprising multiple non-overlapping regions, a memory array configured to store partition information, where the partition information comprises entries that each include an entity identifier and cache configuration information associated with the entity identifier, and a controller configured to access the cache and the memory array. The controller is configured to manage memory requests originating from the processor cores. The managing includes: for at least a first memory request, comparing at least a portion of an address included in the first memory request with tags stored in the cache to determine whether the first memory request results in a hit or a miss, and comparing an entity identifier included in the first memory request with entity identifiers stored in the memory array to determine a matched entry, and for at least a first matched entry for the first memory request, updating the cache configuration information associated with the entity identifier in the first matched entry based at least in part on a hit or miss result for the first memory request. At least some of the entity identifiers are derived at least in part from an associated one of the core-specific identifiers. The cache configuration information associated with a particular entity identifier includes cache usage information that tracks usage of the cache by an entity associated with the particular entity identifier, and a plurality of partition descriptors that each define a different group of one or more of the regions.
In another aspect, in general, a method for managing a shared memory system accessible by a plurality of processor cores includes: storing, in storage of one or more of the plurality of processor cores, at least one core-specific identifier; and managing a shared memory system connected by interconnection circuitry to each of the plurality of processor cores. The managing system includes: operating a cache shared by more than one of the plurality of processor cores, the cache comprising multiple non-overlapping regions, storing partition information in a memory array, where the partition information comprises entries that each include an entity identifier and cache configuration information associated with the entity identifier, and operating a controller to access the cache and the memory array. The controller manages memory requests originating from the processor cores. The managing includes: for at least a first memory request, comparing at least a portion of an address included in the first memory request with tags stored in the cache to determine whether the first memory request results in a hit or a miss, and comparing an entity identifier included in the first memory request with entity identifiers stored in the memory array to determine a matched entry, and for at least a first matched entry for the first memory request, updating the cache configuration information associated with the entity identifier in the first matched entry based at least in part on a hit or miss result for the first memory request. At least some of the entity identifiers are derived at least in part from an associated one of the core-specific identifiers. The cache configuration information associated with a particular entity identifier includes cache usage information that tracks usage of the cache by an entity associated with the particular entity identifier, and a plurality of partition descriptors that each define a different group of one or more of the regions.
Aspects can include one or more of the following features.
The cache configuration information associated with a particular entity identifier includes: the cache usage information that contains a plurality of values for tracking at least two of: a total number of hits, a total number of misses, or a total number of memory requests, at least one threshold on a value that represents a ratio of hits to total memory requests or a ratio of misses to total memory requests, and the plurality of partition descriptors that each define a different group of one or more of the regions within which an entity associated with the particular entity identifier is allowed to allocate a cache line.
Each of the plurality of processor cores includes storage configured to store multiple core-specific identifiers.
The controller is configured to manage memory requests originating from the processor cores and memory requests originating from outside the processor cores.
Every entity is allowed to read a cache line in any of the one or more regions.
The controller is configured to manage memory requests originating from the processor cores and memory requests originating from outside the processor cores.
The memory requests originating from outside the processor cores are provided by an input/output bridge coupled to the interconnection circuitry.
At least some of the entity identifiers are derived at least in part from an associated one of the core-specific identifiers and at least in part from information indicating whether a memory request that includes a particular entity identifier is associated with a request for data or a request for one or more instructions.
The cache comprises a set associative cache, and the multiple non-overlapping regions comprise ways of the set associative cache.
Aspects can have one or more of the following advantages.
Previous techniques for monitoring performance of various entities with respect to cache usage are limited in their ability to apply that cache usage information effectively for cache partitioning. The techniques described herein are able to facilitate dynamic cache partitioning that is managed without requiring intervention by software. A given cache module can be configured to dynamically adjust cache allocation for a variety of entities, including entities outside of the processor cores, such as an entity accessing the cache via an input/output bridge. Assignment of entities can be based on a variety of characteristics including both hardware characteristics (e.g., different processor cores) and access characteristics (e.g., access of cached data vs. cached instructions).
Other features and advantages of the invention will become apparent from the following description, and from the claims.
The registers 106 can be arranged in any of a variety of configurations and accessible by various parts of the processor core 102. In some implementations, the registers 106 include one or more register files. The registers 106 can be configured for various purposes. For example, some of the registers 106 are general purpose registers accessible by the pipeline 104. The registers 106 can also include registers that are dedicated for storing important values such as a stack pointer, and a program counter. The registers 106 can also include various special purpose registers, including registers storing values used for cache partitioning, which are described in more detail below.
The processor memory system 108 and external memory system 112 together form a hierarchical cache system including at least a first level (L1) cache within the processor memory system 108, and any number of higher level (L2, L3, . . . ) caches within the external memory system 112. The highest level cache within the external memory system 112 (which may be the L12 cache if there are only two levels in the hierarchy) is the LLC 120, which is accessed just before main memory. Of course, this is only an example. The exact division between which level caches are within the processor memory system 108 and which are in the external memory system 112 can be different in other examples. For example, the L1 cache and the L2 cache could both be internal to the processor core 102, and the L3 (and higher) caches could be external to the processor core 102. Each processor core 102 could have its own internal L1 cache, and multiple processor cores could share an L2 cache. The external memory system 112 also includes a main memory controller 122, which is connected to any number of memory modules 124 serving as main memory (e.g., Dynamic Random Access Memory modules).
In some implementations of the computing system 100, there can be multiple sets of processor cores 102 included on separate “sockets” (not shown) that each have their own external memory system 112 connected to the set of processor cores over their own processor bus 110. Processor cores would send “local requests” to access addresses associated with memory modules 124 connected to the main memory controller 122 of their “local socket” on which they are contained. Processor cores on different sockets can also send “remote requests” to access addresses associated with different memory modules 124 connected to “remote sockets.”
The size of the main memory allows for a “physical address space” of a certain size. But, that physical memory can be virtualized using a “virtual memory” scheme, which allows executing processes to refer to virtual addresses within a “virtual address space” that is larger than the physical address space. Virtual memory involves address translation from one or more such virtual address spaces into the physical address space. The translation is performed using a “page table” that stores mappings between virtual addresses and physical addresses at a granularity of memory pages (or simply “pages”).
A variety of entities can be defined for tracking cache usage information and for applying that tracked cache usage information to cache partitioning. For example, each processor core can be defined as an entity. Additionally, other entities can be based on certain types of programs executing on the processor cores. One type of entity that is supported by some processors is called a virtual machine (VM). Multiple VMs can share the same physical system hardware, such as a processor core. Alternatively, different VMs can be assigned different processor cores, such as four different VMs each being assigned a different set of 6 processor cores in a processor with 24 total cores. In either case, different VMs could share certain physical hardware, such as a cache module within the hierarchical cache system. In such systems, a hypervisor (also called a virtual machine monitor (VMM)) runs on the physical system to manage all of the guest operating systems running within respective VMs. The hypervisor runs at higher-privileged access level than the guest operating systems. Only the hypervisor has access to the physical address space. Therefore, each guest operating system is provided its own “intermediate physical address space” into which one or more virtual address spaces are translated, and the hypervisor translates intermediate physical addresses into physical addresses.
Other examples of entities, in addition to individual processor cores, can include VMs executing on one or more processor cores, threads executing on the same processor core, or individual I/O devices. For example, some operating systems treat hardware threads executing on the same processor core as “virtual cores.”
The request sent to the CCU 302 will also include an entity identifier (EID) that corresponds to an entity making the request. The size of the EID determines the number of entities that can be assigned partitions. For example, an 8-bit ED enables 256 entities to have an entry within the partition information 304 stored in a memory array (e.g., implemented using Random Access Memory (RAM) circuitry, such as a Static RAM module or other circuitry configured to store an array). The EID is computed based on various parameters associated with the requesting entity. For example, if the EID corresponds to an entity executing on a particular processor core, a core-specific identifier stored in a special purpose register in that processor core may be used. There may be multiple different registers storing different core-specific identifiers associated with different “exception levels” (also called “privilege levels” or “access levels”) of the requesting entity. For example, there may be a lowest-privileged exception level EL0 for user programs running in a guest OS, an exception level EL1 for a guest OS, an exception level EL2 for a hypervisor, and a highest-privileged exception level EL3 for a security system. One register may store a valued used to compute an EID for requesting entities at exception levels EL0, EL1, and EL2, and another register may store a value used to compute an EID for an entity at exception level EL3.
Other parameters can also be used to compute an EID. An “I/D bit” can be used, indicating whether the request is associated with a stored instruction (using an I-value of the l/I bit) or associated with a stored data value (using a D-value of the I/D bit). This ID bit enables the regions into which the LLC 120 is partitioned to form effective instruction cache or data cache sub-regions. The same program can therefore be configured act as two different entities with respect to cache partitioning, such as in the case of self-modifying code that sends write requests using the D-value of the I/D bit and sends read requests using I-value of the I/ID bit. The ID bit can be used for computing the EID in any of a variety of ways, such as concatenating the bit with a 7-bit core-specific identifier, yielding an 8-bit EID.
If the EID corresponds to an entity outside of the processor cores, other parameters can be used to compute the EID, such as quality of service (QoS) parameter for an entity providing a request over the I/O bridge 114. In some implementations this QoS parameter or EID is retrieved by tables retrieved by the SMMU 115, and the SMMU 115 determines a unique QoS or EID value for each I/O device (e.g., devices 118A, 118B, 118C, and 118D).
The CCU 302 is coupled to the memory array storing the partition information 304. The location of that memory array closely coupled to, within circuitry of, or otherwise in proximity to the LLC 120 and/or the CCU 302 (e.g., as opposed to being located within each processor cores 102) provides a conveniently accessible and centralized location for storing partition information that may be associated with any entity, whether that entity is associated with a processor core on the same or a different socket or associated with a device outside of any processor core. The partition information 304 is, in some implementations, accessed within the memory array as a table data structure with rows that are indexed by a field (or “column”) that stores the EID) 312. Each row also includes fields storing cache configuration information that represents enough information for dynamically determining a configuration of the regions in the LLC 120 that will be part of the partition assigned to a corresponding uniquely identified entity. As described above, this assigned partition (which may overlap with partitions assigned to other entities) represents the regions within the LLC 120 in which the corresponding entity is allowed to store new cache lines, if necessary, evicting old cache lines to do so.
In this example, the configuration information includes the following additional fields within a row having a particular value of the EID 312.
A group of fields store cache usage information (CUI) 314. The CUI 314 tracks usage of the LLC 120 by the entity uniquely identified by the particular EID 312. The CCU 302 is configured to update the CUI 314 in response to each request to access the LLC 120. For example, in some implementations, the CUI 314 stores two different counts that can be used to compute an approximate hit ratio characterizing results of requests to the LLC 120 by the entity. One field would store a count of a total number of hits H resulting from requests by the entity. Another field would store a count of a total number of misses M resulting from requests by the entity. The hit ratio is then computed as H(H+M). Alternatively, instead of counting both hits and misses, only one of them is counted, and a total number of memory requests (H+M) is also counted, which also enables computation of the hit ratio (or equivalently a miss ratio M/(H+M)). In some implementations, the EID in the request is compared with EIDs in the table concurrently with comparison of the address in the request to index and tag bits in the LLC 120, as described in more detail below. If a matched entry is found, the hit count H or miss count M is incremented in the table accordingly after determining whether there is a hit or miss in the LLC 120. In alternative implementations, an approximation or equivalent to the number of hits or misses can be counted, such as counting the number of allocations into the LLC 120 of a data or instruction value retrieved from main memory that occur after a miss. Certain types of activity associated with the LLC 120 does not necessarily increment either the hit or miss counts, such as certain activity associated with “victims” and “forwards.”
Configuration fields 316 store one or more thresholds for comparing against the hit ratio (or miss ratio). If the threshold is exceeded, an update of a selection from partition descriptors 318 is performed. The CCU 302 can be configured to perform this comparison and potential update for every received request to access the LLC 120. If the request results in a miss, this comparison and potential update can be performed during the relatively long time needed for the requested data or instruction to be retrieved from main memory, without incurring additional delay. If the request results in a hit, this comparison and update can still be performed concurrently with other activity (e.g., execution of an instruction within the pipeline 104), but may result in an additional delay. But, the cost of any additional delay may be worthwhile to achieve a more efficient partitioning after updating a selection from the partition descriptors 318. In some implementations, comparison of an updated hit ratio to one or more thresholds is performed after every miss but only selectively after certain hits (e.g., after a certain number of hits, or after certain amount of time since the last hit). Alternatively, comparison of an updated hit ratio to one or more thresholds can be performed selectively regardless of whether there was a hit or a miss.
A group of fields store multiple partition descriptors 318. For a row corresponding to a particular entity, each partition descriptor defines a different group of one or more regions within the LLC 120, called a “partition,” within which the entity associated with the matched EID 312 is allowed to allocate cache lines. The manner in which the LLC 120 is divided into non-overlapping regions depends on its structure, as described in more detail below. Different partitions, each defined by a different partition descriptor, can overlap with other partitions, whether those other partitions are associated with the same EID (i.e., in the same row in the table) or different EIDs (i.e., in different rows in the table). At any given time, one of the multiple partition descriptors 318 in a row for an entity with a particular EID is selected as the “active partition descriptor” for that entity. For example, an appropriate number of bits are stored in one of the configuration fields 316 to enable indication of one of the partition descriptors 318 as active. Updating this field to indicate a that a different partition descriptor has been selected as the new active partition descriptor is performed only if the result of the threshold comparison indicates that a change is warranted.
The LLC 120 can be configured using any of a variety of cache structures, and the manner in which its non-overlapping regions are identified by the partition descriptor may depend on that cache structure. In this example, the LLC 120 includes a structure that has circuitry implementing an N-way set associative cache 320, and the non-overlapping regions are called “ways” within the cache 320. Each cache line 322 within this cache 320 includes bits for storing the data words of that particular cache line 322, bits for a tag (which contains a number of the most significant bits of an address, which are common to the words of that entry), and bits for other information (e.g., a valid bit and any flags or error correction code bits). Before comparing a tag portion of a memory address within a request to find a match to the stored tags (in the case of a hit), circuitry of the cache 320 compares an index portion of the address (a number of bits between the tag bits and the least significant bits) to determine in which of multiple “sets” the cache line containing that data may be stored (e.g., SET i in
The CCU 302 is configured to change the active partition descriptor in response to an updated hit ratio crossing (i.e., rising above, or falling below) a threshold stored in one of the configuration fields 316. For example, the CCU 302 may be configured to change the active partition descriptor of an entity from PD(j) to PD(j+1), allowing more ways for cache line allocation, after an updated hit ratio of the entity drops below a stored threshold. The low hit ratio may indicate that too many cache lines are being evicted, and more ways could reduce the likelihood of eviction. Alternatively, hit ratios of other entities can be used to change the active partition descriptor. For example, if a particular entity is found to have a much higher hit ratio than other entities, that particular entity may have its active partition descriptor changed to reduce the ways allowed for cache line allocation, and/or other entities may have their active partition descriptors changed to increase the ways allowed for cache line allocation.
There may also be other reasons why different partition descriptors 318 are stored. For example, one partition descriptor may be used if the entity associated with the matched EID 312 has certain characteristics (e.g., executing on a processor core of a local socket associated with the LLC 120) and another partition descriptor may be used if the entity associated with the matched EID) 312 has other characteristics (e.g., executing on a processor core of a remote socket). Different partition descriptors can also have different sizes for partitioning different regions of the LLC 120. If the partition descriptors 318 are way masks, for example, different way masks can have different sizes, with each being associated with a different cache structure that has a different number of ways. For example, there may be multiple caches within the hierarchy being partitioned, including one or more caches at a lower level than the LLC 120, each with a different number of ways. Alternatively, the LLC 120 may consist of different parallel cache structures for different purposes, each with a different number of ways, such as instruction LLC and a data LLC.
The partition information 304 may be initialized using certain default values for all of the fields in the table. In some implementations, the active partition descriptor all entities can initially be set to allow all ways for allocation (i.e., a way mask with all bits set to zero) by default, to start with no partitioning being used (or a single completely overlapping partition for all entities). Entities would then be dynamically partitioned, incrementally, based on changing CU 314. Alternatively, entities may start with small (possibly non-overlapping) partitions initially, which grow to use larger (possibly overlapping) partitions, again based on changing CUI 314. Also, when certain information in the table (e.g., the partition descriptors 318) is initialized or changed, the counters for computing hit ratio can be reset to zero. In some implementations, the ability to change some or all fields of the partition information 304 can be protected, for example, by limiting changes to a high exception level (e.g., EL3), with lower exception levels able to request changes be made at a higher exception level by making a request via software executing at the lower exception level to software executing at the higher exception level.
Other forms of partitioning can be combined with the cache partitioning described herein. For example, memory bandwidth can be partitioned by assigning certain entities limits on memory bandwidth that they are allowed to use, such as an assigned fraction of a total memory bandwidth. Additionally, bandwidth associated with I/O can be partitioned, or bandwidth associated with a particular I/O device such as a storage device 118A (e.g., disk drive bandwidth). Other resources can be partitioned, including partitioning core resources among multiple threads executing on a single processor core.
Other embodiments may fall within the scope of the following claims, which do not necessarily include all of the features or advantages of the embodiments described above.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5434992 | Mattson | Jul 1995 | A |
| 6587937 | Jensen et al. | Jul 2003 | B1 |
| 6591347 | Tischler et al. | Jul 2003 | B2 |
| 6662272 | Olarig et al. | Dec 2003 | B2 |
| 6871264 | Soltis, Jr. | Mar 2005 | B2 |
| 6996679 | Cargnoni et al. | Feb 2006 | B2 |
| 7318124 | Hama et al. | Jan 2008 | B2 |
| 8745618 | Lin et al. | Jun 2014 | B2 |
| 9081501 | Asaad | Jul 2015 | B2 |
| 9336147 | Kotla et al. | May 2016 | B2 |
| 9645945 | Meredith | May 2017 | B2 |
| 20090300293 | Mantor | Dec 2009 | A1 |
| 20100280806 | Bowers | Nov 2010 | A1 |
| 20140201452 | Meredith | Jul 2014 | A1 |
| 20160283246 | Fleming et al. | Sep 2016 | A1 |
| 20160342514 | Wang et al. | Nov 2016 | A1 |
| 20160357680 | Hooker | Dec 2016 | A1 |
| 20160357686 | Waldspurger | Dec 2016 | A1 |
| Entry |
|---|
| Intel, “Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit,” White Paper, Feb. 2016, Document No. 333742-001US, 25 pages. |
| Intel, “Improving Real-Time Performance by Utilizing Cache Allocation Technology, Enhancing Performance via Allocation of the Processor's Cache, White Paper,” Apr. 2015, Document No. 331843-001US, 16 pages. |
| Qureshi et al., “Utility-Based Cache Partitioning: A Low-Overhead, High-Performance, Runtime Mechanism to Partition Shared Caches,” The 39th Annual IEEE/ACM International Symposium on Microarchitecture, IEEE, 2006, 10 pages. |
| Sahu et al., “Creating Heterogeneity at Run Time by Dynamic Cache and Bandwidth Partitioning Schemes,” SAC '14, Mar. 24-28, 2014, Gyeongju, Korea, pp. 872-879. |
| Suh, et al., “Dynamic Cache Partitioning for Simultaneous Multithreading Systems,” Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, in the proceedings of the IASTED International Conference on Parallel and Distributed Computing and Systems, Aug. 2001, Computation Structures Group, Memo 446, 8 pages. |
| Wang et al., An Analytical Performance Model for Partitioning Off-Chip Memory Bandwidth, Proceeding IPDPS '13 Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing, pp. 165-176, 12 pages, May 2013. |
| Number | Date | Country | |
|---|---|---|---|
| 20180373635 A1 | Dec 2018 | US |
