Patent Application
20210081323
The present invention relates to caches and, more particularly, to a method of improving the L1 icache performance with large programs.
The chip area near the CPU processing pipeline, and overall power, are critical resources in any processor design. All hardware structures must be justified for inclusion in a design based on performance contribution and power usage. Features that only benefit selected modes of operation must be considered carefully and are often removed or minimized to save area and power.
Caches are a well-known approach to reducing the latency associated with the CPU accessing the instructions and data stored in the main memory of a computer system. Cache prefetching attempts to predict the information that the CPU will need, and then obtain and store the information before the CPU requests the information so that when the information is needed the CPU can access it quickly.
L1 caches are volatile memories located near the CPU processing pipeline, and have very short access times, typically requiring three to four clock cycles for the CPU to obtain an instruction. Many computer systems also include a L2 cache that lies between the L1 cache and the main memory. L2 caches have longer access times, typically requiring 12 to 25 clock cycles for the CPU to obtain the instruction. The main memory has significantly longer access times, typically requiring a 100 clock cycles or more for the CPU to obtain the instruction.
Virtually all CPU designs include a L1 instruction cache (icache) due to the improvement in CPU performance by significantly reducing the average access time required for a CPU to obtain a requested instruction. When successful, the memory latency can be reduced from 10 to 100 clock cycles, depending on where the information is stored, down to 3-4 clock cycles.
However, even a small L1 icache is a relatively large component of a CPU. As a result, L1 icaches have significant size limitations. One of the problems with utilizing small L1 icaches is that small L1 icaches are largely ineffective when it comes to working with programs that have a large code footprint.
The main reason for this is that large programs tend to have a high number of lukewarm instructions, which are instructions that occur infrequently but often enough to stay in the cache. As a result, when executing these large programs, the L1 icache incurs frequent misses (instructions looked for but not found in the L1 icache) and few hits (instructions looked for and found in the L1 icache).
In addition, the most effective instruction prefetching requires large tables for correlation prefetchers, filter tables and other improvements. However, due to the size limitations of L1 icaches, the most effective instruction prefetching is often size constrained. Thus, there is a need for an approach that increases the effectiveness and performance of the L1 icache with large programs without increasing the size of the L1 icache.
The present invention increases the hit rate of a L1 icache when operating with large programs without increasing the size of the L1 icache. A method of operating a computer system includes accessing a L1 icache that stores regular instructions and non-instruction information to obtain a regular instruction. The method also includes prefetching regular instructions to be stored in the L1 icache based on the non-instruction information stored in the L1 icache when the regular instruction is not present in the L1 icache.
The present invention also provides a computer system that includes a L1 icache that stores regular instructions and non-instruction information, and a processor that is coupled to the L1 icache. The processor accesses the L1 icache to obtain a regular instruction. In addition, the processor prefetches regular instructions to be stored in the L1 icache based on the non-instruction information stored in the L1 icache when the regular instruction is not present in the L1 icache.
The present invention further provides a non-transitory computer-readable medium that has computer-readable instructions stored thereon which, when executed by a processor, cause the processor to execute a method of cache prefetching. The method includes accessing a L1 icache that stores regular instructions and non-instruction information to obtain a regular instruction. The method also includes prefetching regular instructions to be stored in the L1 icache based on the non-instruction information stored in the L1 icache when the regular instruction is not present in the L1 icache.
A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description and accompanying drawings which set forth an illustrative embodiment in which the principals of the invention are utilized.
As described in greater detail below, in the present invention, limited additional information is stored after execution to improve future performance through directed prefetching that is only used the next time the instruction sequence starts execution. The present invention significantly increases the performance of a L1 icache when working with large programs by storing non-instruction information in a section of the L1 icache.
As shown in
Each 64-byte way is known as a cache line, which includes a cache line valid bit, a tag, and a number of instructions. The cache is accessed by a memory address which includes a tag, an index, and an offset. The tag distinguishes different addresses that can be saved in a set, while the index identifies the set. The offset identifies one of the bytes within the cache line. (The way is identified by checking every entry in the set in parallel. No bits are used to identify the ways.) Thus, seven index bits are required to identify one of 128 sets, while six offset bits are required to identify one of the 64 bytes in a set, thereby leaving a tag of 3 bits in a 16-bit (2 byte) memory address.
In accordance with the present invention, the L1 icache is arranged to have a section for regular instructions and a section for non-instruction information. Regular instructions are recently or frequently fetched instructions, while non-instruction information includes a prefetch correlation table.
For large programs, L1 icache sizes in the range of 64 KB experience a high number of misses due to lukewarm instructions, and provide diminishing performance improvement. However, by reallocating 4 KB-8 KB of a 64 KB L1 icache storage for a prefetch correlation table in accordance with the present invention, the L1 icache provides significantly better overall performance.
There are multiple methods of allocating a section of the L1 icache for the prefetch correlation table. In a multi-way set-associative cache, which compares the output of all of the ways simultaneously and identifies a hit from any way that matches the tag comparison, a number of ways in each set of a multi-way set-associative cache are reserved and put aside to store regular instructions, e.g., six ways in each set (way1-way6) can be put aside to store regular instructions, while a number of ways in each set are reserved and put aside to store the prefetch correlation table, e.g., two ways in each set (way7-way8) can be put aside to store the prefetch correlation table. Isolating a portion of the ways for a correlation table can be accomplished by enabling or disabling comparison for ways (way7-way8) associated with the non-instruction information.
Table 2 shows an example of the information stored in each of the first six ways of a set 0, which includes a data valid bit, a memory address that includes a tag, and a number of regular instructions.
Table 3 shows an example of the non-instruction information stored in the last two ways of a set 0, which includes a prefetch correlation table.
In a second example of a L1 icache, the regular instructions and the non-instruction information are intermingled and stored together in the ways of a set, except that an extra ID bit or bits are added to the tag to indicate whether the memory address belongs to a regular instruction or non-instruction information. (The extra ID bit or bits can alternately be added to any of the fields.)
Table 4 shows an example of the information stored in the 8 ways of a set 0, which includes a cache line valid bit, a memory address that includes a tag+ID bit/s, and a result. In this example, a “0” ID bit is added to the memory address to indicate a regular instruction, while a “1” ID bit added to the memory address to indicate non-instruction information.
When a L1 icache is accessed, a hit for one type of information requires a tag match plus a match of the extra bit or bits associated with the desired information. It is also possible for the division between different data types to be dynamic, where feedback such as the rate of cache misses causes the portion of the L1 icache allocated for prefetch data to be modified. If dynamic resizing is utilized, cache lines that are repurposed must be invalidated before reuse.
The prefetch correlation table is an array that contains non-sequential fetch addresses for prefetching assistance. Correlation tables are common for prefetchers but can become large. To reduce the size with minimal impact on performance, the correlation table stores an offset that is added to the lookup address to determine the correlated target address for the prefetch.
A tag in the correlation table is used to distinguish entries from each other, e.g. each 16-bit entry contains a 6-bit tag to match upper memory address bits and a 10-bit signed offset. In this way, 16 bits (2 bytes) represents a full 128-byte cache line of instructions, or a 64× information concentration in the L1 icache.
The correlation table can be addressed many ways, but a set associative organization has been shown to improve performance, e.g. 16 sets with 2 ways and 4-16 KB entries per way. Entries are replaced randomly within ways. A correlation table of 2 K, 16-bit entries or larger provides improved performance, representing 256 KB of instructions. This requires 4 KB of storage in the cache.
In addition, method 100 utilizes a filter table to avoid excessive prefetches that can add unnecessary, and detrimental, increased traffic to the memory subsystem. The filter table is an array of bits corresponding to cache lines. Bits are set as instructions retire (one bit per cache line). Bits are cleared randomly so the table does not remain completely filled with ones over time.
The filter table is used to filter potential prefetches so only addresses that were previously useful are prefetched. Since the program is much larger than the cache, many addresses are misses, but the filter table allows prefetches to focus on addresses that have been useful in the past. This reduces unnecessary and unwanted accesses to the L2 and larger memories.
A filter table using 1 bit per cache line maps the recent usefulness of a larger address space. Cache lines are 128 Bytes in this example. A 1 KB array maps to a 1 MB address region. A 256 Byte array maps to a 256 KB address region. The filter table is accessed prior to every prefetch, and does not need to be large to be effective.
Method 100 is executed by a processor that is pipelined for performance with an early pipeline stage that fetches instructions from the L1 icache. The early pipeline stage is followed by multiple pipeline stages that execute the instructions. In addition, a late pipeline stage includes instruction retirement where results are committed.
At reset time, the program counter in the instruction fetch stage is initialized to a preset or configured value to start program execution and fetch the first instruction. After each instruction is fetched and decoded, the program counter is updated, typically by an increment to the next sequential instruction, but branches can generate a non-sequential program counter update. At any time, program execution in later pipeline stages can also cause the program counter to be updated to redirect and restart the execution pipeline. The most common restart update is for a branch misprediction.
In the present invention, method 100 utilizes a next-line sequential prefetcher, a correlated prefetcher (which includes a prefetch correlation table), and a filter table. As the name suggests, the next-line sequential prefetcher increments the value of the program counter, and then fetches the instruction located at the address that corresponds to the incremented value of the program counter.
The correlated prefetcher tracks non-sequential accesses in a prefetch correlation table. In the present example, the prefetch correlation table maintains a mapping from the address of a missed memory address to a cache line offset that identifies the memory address of a predicted next memory address in the L1 icache.
Referring again to
When a read request has been received, method 200 moves to 212 to determine if the specific memory address is stored in the L1 icache, i.e., determine whether the current value of the program counter matches a valid line within the L1 icache. When the current value and a L1 icache line match, the match is referred to as a ‘hit,’ while when the current value does not match any of the valid cache lines, the non-match is referred to as a ‘miss.’
When there is a L1 icache hit, method 200 moves to 214 to output the instruction to the processor, and then to 216 to determine if the current value of the program counter is sequential or non-sequential. If the current value of the program counter is greater than the last value of the program counter by one, then the current value of the program counter is sequential. If the current value of the program counter is greater than the last value of the program counter by more than one, then the current value of the program counter is non-sequential.
When non-sequential, method 200 moves to 220 to access the correlation table, and then to 222 to determine whether a match was found in the correlation table. For example, method 200 can use 14 bits from the program counter to access a 4 KB correlation table in the following manner. A 4-way set associative organization would use the lower 9 bits from the program counter to select a group of four 16-bit entries where each entry contains an 11-bit signed offset and a 5-bit tag. The tags from all entries are compared against the remaining upper 5 bits of the program counter where a match indicates a hit and a mismatch indicates a miss. When there is a hit, method 200 moves to 224 to receive an 11-bit signed offset from the correlation table, which is then added to the program counter to generate a new, non-sequential target address.
After this, method 200 moves to 226 to access the filter table. For example, method 200 inputs the target address into the filter table, and receives a determination from the filter table as to whether the target address has been useful in the past. If the target address has been useful in the past, method 200 moves to 230 to determine if the target address is already stored in the L1 icache. When the target address is not stored in the L1 icache, method 200 moves to 232 to prefetch the target address.
Referring back to 216, when the current value of the program counter is sequential, method 200 moves to 240 to access the filter table. For example, method 200 inputs the specific address (current value of the program counter) into the filter table, and receives a determination from the filter table as to whether the specific address has been useful in the past. If the specific address has been useful in the past, method 200 moves to 242 to determine if the specific address is already stored in the L1 icache. When the specific address is not stored in the L1 icache, method 200 moves to 244 to prefetch the next sequential cache line.
Referring back to 212, when there is a L1 icache miss, method 200 moves to 260 to request the instruction from a larger memory, such as a L2 cache or the main memory, and then to 262 to determine if the current value of the program counter is sequential or non-sequential. When the current value of the program counter is non-sequential, method 200 moves to 264 to map the last value of the program counter to the current value of the program counter and store the mapping in the correlation table within the L1 icache.
Method 200 moves to 270 from 210 when there is no read request, from 222 when there is no match in the correlation table, from 226 when the filter table indicates that the target address is not useful, from 230 when the target address is already in the L1 icache, from 240 when the filter table indicates that the specific address is not useful, from 242 when the filter table indicates that the specific address is not useful, from 244 when the specific address is already in the L1 icache, from 262 when the memory address is sequential, and from 264 after the mapping has been added to the correlation table.
In 270, method 200 determines whether an instruction has been received from the larger memory (e.g., L2 cache). When no instruction has been received, method 200 returns to 210 to check for read requests. When an instruction has been received, method 200 moves to 272 where the L1 icache stores the memory address and instruction from the larger memory in the L1 icache. Next, method 200 moves to 274 where the L1 icache outputs the instruction to the processor. Method 200 then returns to 210.
One of the advantages of the present invention is that by storing the correlation table in a section of the L1 icache, a correlation prefetcher can be effectively used along with a next-line prefetcher to significantly improve the hit rate of a L1 icache when handling a large program (program with large code footprint).
Another advantage of the present invention is that by utilizing a next-line prefetcher after a predicted next memory addresses has been identified from the correlation table, the combined prefetcher is able to eliminate many of the accesses to the correlation table. As a result, in accordance with the present invention, the correlation table can be stored in the same array as the L1 icache.
The correlation table can be either statically or dynamically partitioned from the data section of the cache by setting a bit in the cache tags. By limiting the number of prefetches that are needed to access the correlation table, the contention between fetch and prefetch for bandwidth to the L1 icache is reduced.
In an alternate embodiment of the present invention, the non-instruction information can be a prefetch mask (prefetch hints identifying useful cache lines within a page) with color information. When a cache miss is detected, the next several instructions that are most likely to be executed are instructions from the same page as the missed instruction.
In accordance with the present invention, the memory addresses and associated instructions that have the same page number as the page number of the last instruction to be output by the L1 icache are prefetched from a larger memory (e.g., a L2 cache or main memory) and stored in the L1 icache in the instruction information ways (e.g., way1-way6).
As noted above, a L1 icache typically has a low L1 icache hit rate, and is largely ineffective, when working with large programs. However, since the next most likely instructions to be executed are from the same page as the last instruction to be executed, the page prefetching of the present invention substantially increases the hit rate and the effectiveness of the L1 icache when working with large programs.
In the event that a particular application has a large number of memory addresses and associated instructions that have the same page number, a color value can be associated with each memory address and associated instruction to limit the number of memory addresses and associated instructions that are prefetched. The color value represents a time range that includes the time that the instruction was fetched by a processor.
In operation, the color value functions as a filter. For example, rather than prefetching all of the memory addresses and associated instructions that have the same page number, instead only prefetch the memory addresses and associated instructions that have both the same page number and the same or a similar color value. Thus, the prefetched instructions are limited to those that were fetched by the processor around the same time.
As shown in
As further shown in
In the present example, L2 cache 416 and L3 cache 420 are also logically organized as an n-way set-associative device with a number of sets and a number of ways per set. In addition, when L2 cache 416 is coupled to a multi-core processor, where each core has separate L1 caches, L2 cache 416 stores information for each of the L1 caches.
Methods 100-300 can be implemented by processor 410, or by dedicated controllers associated with the L1 icache, or a combination of both. Methods 100-300 can also be stored on a non-transitory computer-readable medium that is operably coupled to the processor. The non-transitory computer-readable medium has computer-readable instructions stored thereon which, when accessed by the processor, cause the processor to execute methods 100-300.
Reference has been made in detail to several embodiments. While the subject matter has been described in conjunction with an alternative embodiment, it will be understood that they are not intended to limit the claimed subject matter to these embodiments. On the contrary, the claimed subject matter is intended to cover alternative, modifications, and equivalents, which may be included within the spirit and scope of the claimed subject matter as defined by the appended claims.
Furthermore, in the preceding detailed description, numerous specific details are set forth in order to provide a thorough understanding of the claimed subject matter. However, it will be recognized by one skilled in the art that embodiments may be practiced without these specific details or with equivalents thereof. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects and features of the subject matter.
Portions of the detailed description were presented and discussed in terms of a method. Although steps and sequencing thereof are disclosed in
Some portions of the detailed description are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits that can be performed on computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, computer-executed step, logic block, process, etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout, discussions utilizing terms such as “accessing,” “writing,” “including,” “storing,” “transmitting,” “traversing,” “associating,” “identifying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
In a typical configuration, a computing device includes one or more processors (CPU), input/output ports, network ports, and memory. The memory may include volatile memory, random-access memory (RAM), and/or non-volatile internal memory of the computer-readable media, such as the read-only memory (ROM), or flash memory (flash RAM). The memory is an example of the computer-readable media.
The computer-readable media include permanent medium and non-permanent medium, movable and immovable medium, and can realize information storage by any mode or technology. The information can be a computer-readable command, data structure, program module, or other data. The examples of computer storage media include, but are not limited to, phase change memory (PCM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other internal memory technology, compact disk read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, tape and cassette tape, magnetic tape or disk storage, or other magnetic storage device or any other non-transmission media which can be used for storing the information that can be accessed by the computing device. As defined in this article, the computer-readable media do not include transitory computer-readable media (transitory media), such as the modulation data signal and carrier wave.
It should also be noted that the terms “include,” “contain,” or any other variants are intended to cover the non-exclusive “containing” which makes the processes, methods, commodities, or devices having a series of elements include not only those elements, but also other elements not clearly set out, or the inherent elements of these processes, methods, commodities or devices. When there is no any further specification, the element specified by the sentence “including one . . . ” does not rule out that there are other identical elements in the processes, methods, commodities, or devices including the elements.
A person skilled in the art shall understand that an embodiment of the invention can be provided as a method, a system, or a computer program product. Therefore, the invention can adopt a form of a full hardware embodiment, full software embodiment, or embodiments combining software with hardware. Furthermore, the invention can also adopt a form of computer program products implemented on one or more computer-readable storage media (including, but not limited to, magnetic disk memory, CD-ROM, and optical memory) containing computer-readable program codes.
It should be understood that the above descriptions are examples of the present invention, and that various alternatives of the invention described herein may be employed in practicing the invention. Thus, it is intended that the following claims define the scope of the invention and that structures and methods within the scope of these claims and their equivalents be covered thereby.
