Patent Application
20240202116
Many computing devices have a memory system that includes a main memory and a cache, which is typically smaller but faster than the main memory. When a memory request is made, the device attempts to satisfy the request using the cache. If the attempt to satisfy the request using the cache fails, the device satisfies the request using the main memory.
The detailed description is described with reference to the accompanying figures. Entities represented in the figures are indicative of one or more entities and thus reference is made interchangeably to single or plural forms of the entities in the discussion.
Many computer memory systems include a main memory and a cache, which is typically a smaller but faster memory than the main memory. Memory requests (e.g., load and store memory requests) that are satisfied by the cache typically have lower latency than memory requests satisfied by the main memory. However, not all requests are satisfied by a cache because the requested data or instruction is not available in the cache, which is referred to as a cache miss. Some applications, such as memory bound applications, oftentimes experience high last-level cache (LLC) miss rates. As the number of cores (e.g., in a core complex, such as a core complex die (CCD)) sharing the LLC rises, the LLC miss rate typically rises too. To sustain high throughput a large number of LLC miss status holding registers (MSHRs) is typically used to track the outstanding LLC misses. However, as the number of LLC MSHRs increases, so does the area usage and power usage by the LLC MSHRs on the CCD.
The techniques discussed herein make use of an LLC shadow tag array, which is typically used to determine whether particular cache lines are present in the previous level cache of any of multiple processor cores on the CCD. In one or more implementations, the techniques discussed herein repurpose an entry of the LLC shadow tag array to track pending LLC misses to private data in a previous level cache (e.g., an L2 cache), that also are misses to an exclusive LLC (e.g., an L3 cache) and to the LLC shadow tag array. Accordingly, LLC MSHRs need not be expended to track cache misses to private data that are already being tracked by a previous level cache MSHR.
In one or more implementations, LLC pending misses to shared data are tracked in the LLC shadow tag array or an LLC MSHR. In one or more implementations, each LLC shadow tag array entry is able to track a threshold number of pending shared LLC misses from different processor cores. If the current number of pending LLC misses to a physical address corresponding to the shadow tag array entry is equal to or less than that threshold number then the LLC shadow tag array entry tracks all of the pending LLC misses for the physical address. However, if the current number of pending LLC misses to a physical address corresponding to the shadow tag array entry exceeds that threshold number then an LLC MSHR is used to track at least one of the pending LLC misses for the physical address. Accordingly, an LLC MSHR need not be expended to track cache misses to shared data at a physical address unless the current number of pending misses for the shared data at the physical address exceeds a threshold number.
The techniques discussed herein allow cache systems to support a number of LLC misses that scales with the total number of previous level cache MSHRs in a core complex rather than with the total number of LLC MSHRs. This allows the cache system to sustain higher memory level parallelism as multicore systems integrate higher numbers of cores at the system level (e.g., a system on chip (SoC) level) as well as the core complex level.
Furthermore, the techniques discussed herein reduce area usage and power usage at least in part because the number of LLC MSHRs scales with the amount of shared data accesses that miss in the LLC or with the number of previous level cache MSHRs multiplied by the number of cores in the core complex. The former (shared data accesses pending in the LLC) is much lower than the total number of LLC misses typically observed by multithreaded applications. Accordingly, the LLC MSHR pool occupies less area and consumes less power or in some situations is not needed at all (due to additional metadata included in the LLC STA).
In some aspects, the techniques described herein relate to a method including: receiving, at a current level cache, a physical address corresponding to a cache miss at an immediately preceding level cache, and in response to a cache miss at the current level cache and a shadow tag array miss at the current level cache: recording the cache miss at the current level cache in the shadow tag array, and sending, to a memory system, a request for data stored at the physical address.
In some aspects, the techniques described herein relate to a method, the recording the cache miss at the current level cache in the shadow tag array including recording the cache miss at the current level cache in the shadow tag array without reserving a current level cache miss status holding register.
In some aspects, the techniques described herein relate to a method, wherein recording the cache miss includes setting a current level cache pending miss indicator in an entry of the shadow tag array.
In some aspects, the techniques described herein relate to a method, further including: receiving, from the memory system, the data stored at the physical address, marking, in an entry of the shadow tag array corresponding to the cache miss at the current level cache, that the cache miss at the current level cache is no longer pending, and returning the data to a first processor core requesting the data.
In some aspects, the techniques described herein relate to a method, further including: returning the data to a second processor core indicated in the entry of the shadow tag array.
In some aspects, the techniques described herein relate to a method, further including, in response to the cache miss at the current level cache and a shadow tag array hit at the current level cache: recording the cache miss at the current level cache in an existing entry in the shadow tag array without sending the request to the memory system.
In some aspects, the techniques described herein relate to a method, wherein the current level cache includes a level 3 cache and the immediately preceding level cache includes a level 2 cache.
In some aspects, the techniques described herein relate to a method including: receiving, at a current level cache, a physical address corresponding to a cache miss at an immediately preceding level cache for a first processor core, and in response to a cache miss at the current level cache and a shadow tag array hit at the current level cache: recording the cache miss at the current level cache for the first processor core in an entry of the shadow tag array, the entry of the shadow tag array already recording a cache miss at the current level cache for a second processor core.
In some aspects, the techniques described herein relate to a method, further including: recording, in response to available space being present in the entry of the shadow tag array, the cache miss at the current level cache for the first processor core in the entry of the shadow tag array, and tracking, in response to available space not being present in the entry of the shadow tag array, the cache miss at the current level cache for the first processor core in a shared data miss queue miss status holding register at the current cache level.
In some aspects, the techniques described herein relate to a method, further including recording the cache miss at the current level cache for the first processor core without sending a request for data stored at the physical address to a memory system.
In some aspects, the techniques described herein relate to a method, further including: receiving, from a memory system, data stored at the physical address, marking, in the entry of the shadow tag array corresponding to the cache miss at the current level cache, that the cache miss at the current level cache is no longer pending for the second processor core, and returning the data to the second processor core requesting the data.
In some aspects, the techniques described herein relate to a method, further including: marking, in the entry of the shadow tag array corresponding to the cache miss at the current level cache, that the cache miss at the current level cache is no longer pending for the first processor core, and returning the data to the first processor core requesting the data.
In some aspects, the techniques described herein relate to a method, wherein the current level cache includes a level 3 cache and the immediately preceding level cache includes a level 2 cache.
In some aspects, the techniques described herein relate to a device, including: a current level cache to receive a physical address corresponding to a cache miss at an immediately preceding level cache and send, to a memory system, a request for data stored at the physical address, and a shadow tag array to record, in response to a cache miss at the current level cache and a shadow tag array miss at the current level cache, the cache miss at the current level cache in an entry of the shadow tag array.
In some aspects, the techniques described herein relate to a device, wherein to record the cache miss at the current level cache in the shadow tag array includes recording the cache miss at the current level cache in the entry of the shadow tag array without reserving a current level cache miss status holding register.
In some aspects, the techniques described herein relate to a device, wherein to record the cache miss includes setting a current level cache pending miss indicator in the entry of the shadow tag array.
In some aspects, the techniques described herein relate to a device, wherein the current level cache is further to receive, from the memory system, the data stored at the physical address and return the data to a first processor core requesting the data, and the shadow tag array is to mark, in the entry of the shadow tag array, that the cache miss at the current level cache is no longer pending.
In some aspects, the techniques described herein relate to a device, wherein the current level cache is further to return the data to a second processor core indicated in the entry of the shadow tag array.
In some aspects, the techniques described herein relate to a device, wherein the shadow tag array is further to, in response to the cache miss at the current level cache and a shadow tag array hit at the current level cache, record the cache miss at the current level cache in an existing entry in the shadow tag array without sending the request to the memory system.
In some aspects, the techniques described herein relate to a device, wherein the current level cache includes a level 3 cache and the immediately preceding level cache includes a level 2 cache.
The cache system 104 includes at least one cache. Each cache is memory that is typically faster than the system memory 108. Each cache is, for example, an instruction cache, a data cache, or a combination thereof (e.g., a unified cache). In one or more implementations, the system memory 108 is any of a variety of volatile memory, such as dynamic random access memory (DRAM). Additionally or alternatively, the system memory 108 is any of a variety of nonvolatile memory, such as resistive random access memory (e.g., memristors).
The system 100 is implementable in any of a variety of different types of computing devices that include or use virtual memory. For example, the system 100 is implementable in a smartphone or other wireless phone, a tablet or phablet computer, a notebook computer (e.g., netbook or ultrabook), a laptop computer, a wearable device (e.g., a smartwatch, an augmented reality headset or device, a virtual reality headset or device), an entertainment device (e.g., a gaming console, a portable gaming device, a streaming media player, a digital video recorder, a music or other audio playback device, a television), an Internet of Things (IOT) device, an automotive computer, a datacenter computer, a high-performance computing (HPC) super computer, and so forth.
In one or more implementations, the CPU cores 102 (1), . . . 102 (n) and the cache system 104 are also referred to as a core complex. The core complex is typically included on a physical die referred to as a CCD. A single device (e.g., a computer or any other computing device as discussed above) includes one or more CCDs.
The cache system 104 includes any number of cache levels. In one or more implementations, the cache system 104 includes multiple L2 caches, one L2 cache corresponding to each of the CPU cores 102 (1), . . . 102 (n). The cache system 104 also includes an L3 cache that is shared by the CPU cores 102 (1), . . . 102 (n). Each CPU core 102 (1), . . . 102 (n) also includes an internal level 1 (L1) cache (not shown). Although the system 100 is discussed herein as including three cache levels (L1, L2, and L3), it is to be appreciated that the system 100 includes any number of cache levels.
The cache level in the cache system 104 immediately preceding the system memory 108 is also referred to as the LLC. In one or more implementations, the LLC is an L3 cache and an immediately preceding level cache is an L2 cache. The immediately preceding level cache is also optionally referred to as LLC-1.
In one or more implementations, each immediately preceding level cache (e.g., each L2 cache) is a private cache whereas the LLC (e.g., an L3 cache) is a shared cache. A private cache refers to a cache that is assigned to and used by a single CPU core 102. The data stored in a private cache is also referred to as private data. A shared cache refers to a cache that is assigned to and used by multiple CPU cores 102 (1), . . . 102 (n). The data stored in a shared cache is also referred to as shared data.
In one or more implementations, the LLC is an exclusive cache, which refers to a cache that stores data that is not also stored in the immediately preceding level cache.
The shadow tag array 106 is associated with the LLC and includes metadata for each of multiple cache lines stored in each of the immediately preceding level caches. The metadata for a cache line indicates which of the immediately preceding level caches stores the cache line (e.g., the immediately preceding level cache corresponding to CPU core 102 (1), the immediately preceding level cache corresponding to CPU core 102 (n), and so forth). Typically, the shadow tag array 106 is used to quickly determine whether data requested by another core in a CCD is stored in any of the immediately preceding level caches. For example, if there are eight CPU cores and eight immediately preceding level caches, the shadow tag array 106 is checked to determine whether any of the eight immediately preceding level caches stores the requested data rather than having to access each of the eight individual immediately preceding level caches. The techniques discussed herein repurpose entries of the shadow tag array 106 to track pending LLC misses to data in an immediately preceding level cache (e.g., an L2 cache), as discussed in more detail below.
In one or more implementations, on a cache miss at cache levels preceding the LLC (e.g., L1 and L2 cache misses), an MSHR at that cache level is allocated. The MSHRs are allocated to track the status of the outstanding LLC cache misses, anticipating that the data coming with the response will be stored in the LLC, and capture shared data accesses from threads of different cores to avoid sending redundant requests to the system memory 108. However, using the techniques discussed herein, an MSHR need not be allocated at the LLC in certain situations in the event of a cache miss at the LLC, as discussed in more detail below.
In the example system 200, the LLC is the L3 cache 214 and the immediately preceding level cache is the L2 cache 208. It is to be appreciated, however, that the LLC is optionally any cache level (e.g., a level 4 (L4) cache) and the immediately preceding level cache is optionally a different cache level (e.g., an L3 cache if the LLC is an L4 cache). Additionally, the system 200 is discussed with reference to cache misses for data accesses issued by the CPU core 206. It is to be appreciated that requests issued by the CPU core 210, as well as any other cores (not shown) in the core complex 202 operate analogously. Furthermore, it is to be appreciated that references to data access include actual data accessed by memory load or store requests, code or instruction requests (e.g., to load code to executed), or a combination thereof.
In one or more implementations, L3 misses to private data are serviced as follows. Upon an L2 miss in response to a data access, an L2 MSHR is reserved, a victim line is evicted from the L2 cache, and a miss indication is issued to the L3 cache 214. A check is made as to whether an L3 cache 214 miss is detected (at 218) and whether a shadow tag array 216 miss is detected (at 220). If an L3 cache 214 miss is not detected (e.g., an L3 cache 214 hit is detected), the data access is satisfied by the L3 cache 214 (e.g., the requested data is returned to the L2 cache 208 or directly to the CPU core 206). If a shadow tag array 216 miss is not detected (e.g., a shadow tag array 216 hit is detected), the requested data is stored in the private cache of another core in the core complex 202 (e.g., the CPU core 210). Accordingly, the requested data is retrieved from the private cache of the other core in the core complex 202 and returned to the L2 cache 208 or directly to the CPU core 206.
If an L3 cache 214 miss and a shadow tag array 216 miss are detected (e.g., no other core has a pending miss to the same physical address so data is currently private), then the miss is recorded in the shadow tag array 216. The miss is recorded in the shadow tag array 216 in various manners, such as by setting a corresponding L3 pending miss bit and updating the L2 MSHR tag (also referred to as an L2 MSHR identifier). The data request is also sent to the system memory 204 (at 222) without reserving an L3 MSHR. Note that since the L2 cache 208 has already been updated, the shadow tag array 216 is updated to remain in synchronization with the L2 tags. When the response arrives from the system memory 204 (at 224), the shadow tag array 216 is checked. The entry in the shadow tag array 216 indicates that an L2 MSHR has already been reserved for the data (e.g., the entry in the shadow tag array 216 indicates which L2 MSHR is tracking the pending miss at the L2 cache). Accordingly, the entry in the shadow tag array 216 tracking the miss is updated (e.g., the corresponding L3 pending miss bit is cleared or reset) to mark that the miss on the CPU core 206 is no longer pending. The requested data is forwarded (at 226) to the L2 cache 208 or the CPU core 206 that requested the data using the L2 MSHR tag or identifier included in the shadow tag array 216 entry.
It is assumed that the data remained private while the miss was serviced by the system memory 204, so no other core, from the core complex 202, tried to access the same data. In one or more implementations, the shadow tag array 216 is indexed by the physical address of the data request and the response to the data request does not carry the physical address. This is in contrast to a conventional flow using L3 MSHRs where the response carries the L2 MSHR identifier that is stored in each L3 MSHR entry. Accordingly, a mechanism is provided to link the response with the corresponding miss request tracked in the shadow tag array 216. In one or more implementations, the set and way of the shadow tag array 216 are used instead of the L2 MSHR identifier. This set and way information is known at the time an L3 miss request is sent to the system memory 204 and is carried back over with the response. It should be noted that the response does not need to carry the L2 MSHR identifier because this identifier latter is stored in the shadow tag array 216 entry.
The tag information 302 indicates which cache line the shadow tag array entry 300 represents. The tag information 302 is, for example, an indication of the physical address data requested by the cache 214 miss.
The core bits 304 indicate which core's private cache (e.g., L2 cache), if any, stores the requested data. In one or more implementations, there are N bits, one bit corresponding to each CPU core in the core complex, that is set (e.g., to a value of 1) to indicate the private cache of the corresponding core stores the requested data, and cleared (e.g., to a value of 0) to indicate the private cache of the corresponding core does not store the requested data.
The pending LLC miss indicator 306 indicates that the shadow tag array entry 300 corresponds to a pending LLC miss. The shadow tag array entries typically mirror the state of the resident physical addresses in the LLC-1 caches of the core complex and act as a directory to filter external probe traffic and accelerate core-to-core transfers for cores within the core complex. The pending LLC miss indicator 306 (e.g., a single bit) differentiates shadow tag array entries tracking resident physical addresses from shadow tag array entries tracking pending misses.
The LLC-1 MSHR identifier 308 is an identifier of the LLC-1 MSHR that is tracking the pending miss at the LLC-1 cache.
The core identifier 310 is an identifier of the CPU core corresponding to the LLC-1 MSHR.
Returning to
Once the shadow tag array entry is updated, the request is dropped and without reserving an L3 MSHR. No request need be sent to the system memory 204 as the request has already been sent (in response to the request from the CPU core 210).
When the data response comes back from the system memory 204 (in response to the request from the CPU core 210), the same actions discussed above with reference to servicing L3 misses to private data are performed to service the pending miss from the CPU core 210. Additionally, the data is also returned to the L2 cache 208 or the CPU core 206 to service the pending miss from the CPU core 206. The shadow tag array entry is updated so that both pending miss bits and their corresponding MSHR tags are cleared. In the general case, these actions will be taken to track pending misses to shared data across all cores in the core complex. The response will forward the data along with their corresponding L2 MSHR tags to all cores that have registered a miss at the STA entry.
The LLC miss indicator 402 includes multiple bits each being analogous to the pending LLC miss indicator 306 of
Each LLC-1 MSHR identifier 404 is an identifier of the LLC-1 MSHR of the associated CPU core that is tracking the pending miss at the LLC-1 cache, analogous to the LLC-1 MSHR identifier 308 discussed above.
The LLC-1 MSHR tag (e.g., the LLC-1 MSHR id 404) in the same LLC STA entry is populated at the same time as the pending LLC miss indicator 402 to link the pending LLC miss tracked in the STA entry to the LLC-1 MSHR tracking the miss request at the LLC-1 cache. It should be noted that in this implementation the core identifier 310 need not be included in the shadow tag array entry because the relative position of the MSHR tag or the pending LLC miss bit uniquely identifies the core identifier in the core complex.
Additionally or alternatively, rather than having multiple pending LLC miss indicator 402 bits, the pending LLC miss indicator 402 is one bit indicating if the STA is tracking a pending LLC miss or the address is resident in the LLC-1 cache (as discussed above with reference to servicing L3 misses to private data). The N core bits 304 are instead used to track residency of addresses in the LLC-1 cache, to act as the pending miss N-bit vector when the pending miss bit per STA entry is set to 1. If the pending miss vector bit is set to 0, then the N-bit vector (the N core bits 304) tracks address residency in the N private LLC-1 caches.
Returning to
In the system 500, the SDMQ MSHRs track a number of pending shared data miss requests while the shadow tag array 216 is configured to hold all private data miss requests as well as some of the shared data miss requests. This is accomplished by reducing the number of pending miss request bits (e.g., indicators 402 of
For example, if the LLC shadow tag array supports only one pending miss and its associated LLC-1 MSHR tag, then all pending misses from other cores to the same physical address are tracked in the SDMQ MSHRs 504 and not in the shadow tag array 216. If the LLC shadow tag array supports 2 pending misses to the same physical address then it holds 2 pending miss bits and 2 LLC-1 MSHR tags from 2 different cores. Any remaining shared data pending misses from other cores of the same complex are tracked in the SDMQ and not in the shadow tag array.
Private data misses are handled in the same manner as discussed above with reference to servicing L3 misses to private data, except that each L3 miss request accesses both the shadow tag array 216 (at 220) and the SDMQ MSHR 504 pool (at 506) for a physical address hit. When a shared data access is detected (L3 cache 214 miss and shadow tag array 216 hit) then the cache controller (not shown) checks if there is available space in the shadow tag array 216 entry to record the shared data miss. If there is space, then the flow is the same as described above with the shadow tag array 216 entry recording the shared data miss. However, if there is no space in the shadow tag array 216 entry then the cache controller checks for available SDMQ MSHRs (e.g., SDMQ entries) in the SDMQ MSHRs 504. If there are no available SDMQ MSHRs then the L3 miss request is stalled and waits in an L3 request queue just like an L3 miss would in a conventional LLC design. If there is an available SDMQ MSHR, then the LLC miss is tracked in the SDMQ MSHRs 504 analogous to a conventional L3 cache 214 miss.
In one or more implementations, an SDMQ MSHR 504 operates analogous to a conventional L3 MSHR except that an SDMQ MSHR 504 need only track the physical address, the core identifier, and the LLC-1 MSHR tag. The shadow tag array 216 entry also holds a bit that indicates if there are SDMQ entries to the same physical address (e.g., an SDMQ bit). The SDMQ bit is set in the case where there is no space in the shadow tag array 216 entry and an SDMQ MSHR is reserved to track the shared data miss request.
When the data response comes back from the system memory 204, the shadow tag array 216 entry is checked first. The shadow tag array 216 forwards the data to the original requestor (e.g., CPU core 206) as well as all sharers registered in the shadow tag array 216 entry. If the SDMQ bit is set then the response uses the physical address from the shadow tag array 216 entry to search the SDMQ MSHRS 504 for matches. Multiple matches are possible, depending on the number of pending miss requests to the physical address from other cores. Upon a match, the cache controller sends the LLC-1 MSHR tag stored in the matching SDMQ MSHR 504 and the data received from the system memory 204 to the corresponding LLC-1 cache of the core identifier (at 508).
It should be noted that requests filling the L3 cache 214 (such as L3 cache 214 prefetches) need not be, and typically are not, tracked in the shadow tag array 216 because the shadow tag array 216 tracks addresses (pending or resident) in the L2 caches. When there is no SDMQ MSHRs 504, there are two options: (a) use dedicated L3 MSHRs to track such miss-fill L3 cache 214 requests, or (b) track the pending L3 cache 214 misses in the L3 cache 214 tags. If solution (a) is used then for every such L3 miss request, the L3 MSHR pool is searched for a physical address hit and in case of a physical address miss, then the L3 miss request reserves a new L3 MSHR and is sent to memory. The response carries the L3 MSHR tag and is tagged with a request type that identifies it as an L3 miss request that fills the L3 (and not the L2 cache). If solution (b) is used then then the pending miss is tracked in the L3 tags and the miss request carries the L3 index/way and is tagged with the same request type as in solution (a).
In this example, a physical address corresponding to a cache miss at an immediately preceding level cache is received at a current level cache (block 602). The current level cache is, for example, an L3 cache. The immediately preceding level cache is, for example, an L2 cache.
In response to a cache miss at the current level cache and a shadow tag array miss at the current level cache, the cache miss at the current level cache is recorded in the shadow tag array (block 604). Additionally, a request for data stored at the physical address is sent to a memory system (block 606).
In this example, a physical address corresponding to a cache miss at an immediately preceding level cache for a first processor core is received at a current level cache (block 702). The current level cache is, for example, an L3 cache. The immediately preceding level cache is, for example, an L2 cache.
In response to a cache miss at the current level cache and a shadow tag array hit at the current level cache, the cache miss is recorded at the current level cache for the first processor core in an entry of the shadow tag array (block 704). The entry of the shadow tag array already records a cache miss at the current level cache for a second processor core.
The various functional units illustrated in the figures and/or described herein (including, where appropriate, the CPU cores 102, the cache system 104, the system memory 108) are implemented in any of a variety of different manners such as hardware circuitry, software executing or firmware executing on a programmable processor, or any combination of two or more of hardware, software, and firmware. The methods provided are implemented in any of a variety of devices, such as a general purpose computer, a processor, or a processor core. Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a graphics processing unit (GPU), a parallel accelerated processor, a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), and/or a state machine.
In one or more implementations, the methods and procedures provided herein are implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium for execution by a general purpose computer or a processor. Examples of non-transitory computer-readable storage mediums include a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs).
Although the invention has been described in language specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed invention.
This invention was made with government support under contract number H98230-21-3-0001 awarded by the Department of Defense. The government has certain rights in the invention.
