Patent Application
20170371561
In memory centric computing architectures the entire memory space is addressable by all computing nodes. For example, a computing node may include a processor, such as one that might be included within a System on a Chip (SoC). The computing node may also include memory. The computing node may be inserted into an enclosure that includes some number of additional computing nodes. The enclosure may be inserted into a rack that includes some number of additional enclosures. Each SoC may be able to access all memory, regardless of if the memory is on the same node as the SoC, is within the same enclosure as the SoC, or within the same rack as the SoC.
The ability to access memory anywhere in the system is enabled through the use of a memory fabric. All memory within the system is connected to the memory fabric. Each SoC is also connected to the fabric. The memory fabric may be globally addressable. In other words, any SoC can access any portion of the memory, regardless of where that memory resides, through the use of the global address of that memory being accessed. The fabric itself takes care of any necessary routing of the memory access requests.
The ability to access any fabric attached memory allows for a processor to access an amount of memory that might not be otherwise available to a processor. For example, the processor may be limited in the amount of memory that may be directly connected to the processor. By utilizing a memory fabric, the limits of the amount of memory that can be accessed by a processor can be reduced, if not eliminated.
However, the use of a fabric may introduce a new problem. When a processor is directly attached to memory, the latency for a memory access remains relatively constant because the processor is directly connected to the memory. In other words (ignoring any overload conditions) an access to directly attached memory may always have the same latency, because all memory is directly attached.
In a memory fabric, the latency of access to a memory location may be depend on where on the fabric the memory is located. Some memory locations may be located “close” to the processor that is requesting access, and thus may have a lower latency. Other memory locations may be located “far” from the processor that is requesting access, and thus may have a longer latency. In other words, memory access requests may belong to a memory access class, with each memory access class having a different latency. An example of a system with different memory access classes is described below.
This variable latency can introduce a problem when accessing memory with different latency. For example, a processor may have a queue that stores all memory requests that have not yet been completed. This queue may have a finite size. It is possible that memory access requests that have a long latency may cause the queue to become full, thus preventing memory access requests that have shorter latency from completing.
For example, assume that there are two memory access classes, one with a 100 ns latency (short), and another with a 500 ns latency (long). Also assume the queue is able to hold 10 outstanding memory requests at a time. Assume a steady flow of memory access requests from both access classes, and that those request arrive in round robin fashion (e.g. alternate short and long memory access classes). The queue may be initially empty, but under the previous assumptions, will immediately fill up with 5 long requests and 5 short. After 100 ns, all 5 of the short requests would complete. Using the round robin policy, in the best case, 3 short requests and 2 long requests are added. Thus, the queue would then have 7 long requests and 3 short requests. Now, after another 100 ns passes, the 3 short request in the queue complete. Again, using round robin scheduling, they might be replaced with 2 short requests, and 1 long request, resulting in a total of 2 short requests and 8 long requests in the queue.
As should be clear, the percentage of the queue that is occupied by the long requests continues to increase. At some point, the long latency requests may completely fill the queue, thus causing starvation of the short latency requests.
The techniques described herein avoid the problem of long latency memory accesses requests crowding out lower latency requests. The space in the memory pending queue is initially allocated between all possible memory access classes. Note, the space need not be allocated evenly. Some memory access classes may be initially allocated a larger amount of the queue than others.
Stall counters are established for each memory access class. A period of time may be defined in terms of a number of processor cycles. On each processor cycle, if a memory request for a certain class is received and is unable to be placed in the queue (because the queue is full), the stall counter for that memory access class is incremented. At the end of the period of time, the number of stall cycles for each memory class may be divided by the total number of cycles for the period of time. Thus, it can be determined for what percentage of the period of time memory access requests of a given class were stalled (i.e. for what percentage of cycles memory access requests could not be queued, because the queue was full).
If the percentage for a given memory access class exceeds a threshold, this may indicated the allocation of the queue for that particular access class is too low. Likewise, if the percentage does not exceed a threshold, this may indicate that the allocation to the particular memory access class is too high. The techniques described herein may then reallocate the queue by increasing the allocation of the queue to memory access classes that are stalled, while decreasing the allocation to memory access classes that are not stalled.
The SoC 110 may include a processor 112. The processor may be any type of electronic component capable of executing instructions stored on the medium 150. For example, the processor may be a central processing unit (CPU), graphics processing unit (GPU), field programmable gate array (FPGA), application specific integrated circuit (ASIC), or any other suitable processing resource. The SoC may also include a plurality of stall counters 114. The stall counters may be used to determine which classes of memory access are currently stalled. Operation of the stall counter is described in further detail below.
The SoC may also include a fabric stop 116. The fabric stop may be an interface between the SoC and the memory fabric. In other words, the processor 112 accesses memory by sending a request to the fabric stop 116. The fabric stop may then be responsible for interfacing with the memory fabric in order to fulfill the processor's memory request.
The fabric stop 116 may include a memory pending queue (MPQ) 118. The MPQ may be thought of as a queue that holds all pending memory requests. In other words, the MPQ holds all memory access requests from the processor that have not yet been fulfilled from the memory fabric. It should be understood that the MPQ has a finite size. In some cases, a memory request from a processor cannot be accepted because the MPQ is full. Such a condition may be referred to as a stall condition, or simply a stall. Stalls will be described in further detail below.
Although the description above described the memory fabric stop 116 as part of the SoC 110 the techniques described herein are not limited to such a structure. In other example implementations, the fabric stop may be external to the SoC, such as on a different integrated circuit. Likewise, the stall counters 114 may be included on the SoC, on the fabric stop, or elsewhere. What should be understood is that the techniques described herein are not limited to any particular placement or ordering of the components described above, but rather are operable regardless of the particular architecture of the node.
The SoC may be coupled to a non-transitory medium 150. The medium 150 may include thereon a set of instructions that when executed by the processor cause the processor to implement the techniques described herein. For example, the reallocation instructions 152 may be similar to the instructions described in
Each node 1-4 may also include a memory 1-4. Although the memory may be located on a node board, this does not mean that the memory on the node is only usable by that node. As explained above, there may be a memory fabric to which all memory is connected. Thus memory 1-4 is connected to the memory fabric, which is depicted in
In some implementations, multiple nodes may be aggregated together in an enclosure. For example, as shown nodes 1-2 are in enclosure 2, while nodes 3-4 are in enclosure 2. It should be understood that the particular number of nodes in an enclosure, or the number of enclosures is not important. Rather, it should be understood that there are different latencies for memory access, depending on where the memory is located. The structure described herein is merely for purposes of explanation.
As mentioned above, the latency for accesses to each class of memory may be different. For example, one memory access class may be a node accessing memory that is on the same node. For example, node 1 may access memory 1. Such a memory access class may have a low latency relative to other memory access classes described below.
Another memory access class may be when a node accesses memory that is located on a different node, but within the same enclosure. For example, if node 1 were to access memory 2. As should be clear, the latency of such an access may be greater than accessing memory that is on the same node because the request must pass through switch 1 to travel from node 1 to memory 2. A third memory access class may be when a node access memory that is on another node in a different enclosure. For example, node 1 may access memory 3. Again, it should be clear that such an access may have a relatively higher latency, as the request must pass through both switch 1 and 2 in order to reach memory 3.
Once again, it should be understood that the structure described above is merely for purposes of explanation. What should be understood is that there are different memory access classes, with each class having different amounts of latency. The particular layout of that memory is irrelevant. For example, in an alternate structure, a single node board may contain three different types of memory, which due to the physical characteristics of each type of memory, cause the latency of each type of memory to be different.
It should further be understood that the techniques described herein are not limited to three memory access classes, but rather can be reduced to two memory access classes, or extended to any number of memory access classes. Again, what should be understood is that there may be more than one memory access class, and that each class may have a different latency.
In operation, the total number of queue entries in the MPQ 116 may be allocated to each possible memory access class. In the example being described, there may be three memory access classes; same node; different node, same enclosure; and different node different enclosure. Each memory access class may have a different latency. The allocation of the MPQ to each class of memory access need not be uniform. In addition, the stall counters 114 for each memory access class may be reset.
The stall counters may be incremented for a memory access class each time a memory access request for that class is received but is unable to be added to the MPQ. For example, if on a given processor cycle an attempt is made to add a memory access request to the MPQ, but the MPQ is full, the stall counter for that memory access class may be incremented. This process may continue for a defined period of time. When the period of time concludes, an MPQ reallocation process may occur, as is described in further detail below.
At the end of the period of time, it may be determined if any of the memory access classes are stalled. A memory access class may be considered stalled when the stall counter for that memory access class exceeds a threshold for the period of time. The threshold may be set independently for each memory access class. In some implementations the memory access class may be considered stalled if the stall counter exceeds a defined threshold. In other implementations, the memory access class may be considered stalled when the total number of stall cycles for the class divided by the total number of processor cycles for the period of time exceeds a certain percentage.
Thus, at the end of the period of time, it may be determined which memory access classes are stalled and which ones are not. In one example implementation, only the memory access class with the largest stall counter (e.g. the memory access class that has spent been rejected from the MPQ more than any other memory access class) is considered stalled.
The system may then go through a MPQ reallocation process. At a high level, the process can be thought of as removing MPQ allocation from memory access classes that are not stalled, and granting them to those classes that are stalled. The process for performing this reallocation is described in further detail below. The stall counters may then be reset prior to the start of the next defined time period. This process may continue as long as the system is running, thus allowing for the allocation of the MPQ to be continuously adjusted based on the current memory access patterns of the workload running on the system.
Lines 1-3 set forth some basic definitions that will be useful further below. For example, assume that the system has three memory access classes, such as a node level (0), an enclosure level (1), and a rack level (2). The array s[i] may be a Boolean array that indicates if the particular indexed memory access class is currently stalled. For example, is s[1] equals TRUE, this may mean that the enclosure level memory access class is currently stalled. Likewise s[2] equals FALSE may indicate the rack level memory access class is not stalled.
In line 2, the array T[i] may indicate the allocation threshold for memory access class i. For example, if the MPQ is 1000 slots, and the current threshold for the node level access class is 100, then T[0] may equal 100. The variable m may be used to indicate the total number of possible memory access classes. As mentioned above, even though the description herein generally includes three levels of memory access classes (e.g. node, enclosure, and rack), the techniques described are in no way limited to 3 access class levels. As should be clear, the techniques are extensible to any number of memory access classes.
Line 5 sets forth a function, Change_MPQ_Threshold that may be used to change the threshold of the amount of the MPQ allocated to each memory access class. The function may take as inputs the s[i] array, the T[i] array, and the total number of memory access classes m, described above. In lines 6 and 7, several variables are initialized. Variable c is used to keep track of the available number of entries in the MPQ that are available for reallocation. The variables number_not_stalled and number_stalled may be used to keep track of the number of memory access classes that are not stalled or are stalled, respectively.
In lines 8 through 16, a loop is executed, with the control value being the variable m. In other words, lines 9-15 execute one time for each possible memory access class. At line 9, the s[ ] array for the current memory access class is checked to determine if the memory access class is stalled. As explained above, the memory access class may be considered stalled based on the stall cycle counter for the memory access class. If the result of the comparison is FALSE, it means that the memory access class is not stalled. As such, the number_not_stalled variable is incremented in line 10.
In line 11, the T[ ] array is checked to determine if reducing the allocation of the MPQ for the current memory access class will cause the allocation to drop below a minimum threshold. The allocation amount δ, which is an amount of MPQ capacity that might be removed for the current allocation, is subtracted from the current allocation. If that subtraction causes the allocation amount for the memory class to drop below the value ε (the minimum allocation threshold), no action is taken, and none of the allocation is removed from the current memory access class.
If subtracting δ from the current allocation does not cause the allocation to drop below the minimum threshold ε, then in line 12, an amount δ is subtracted from the allocation for the current memory access class. In line 13, the value δ is added to the current value of c, which keeps track of the amount of the MPQ that is available for reallocation.
In other words, in lines 8-16, each memory access class that is not stalled is examined to determine if it can lose an amount of MPQ allocation δ without falling below the minimum allocation threshold ε. If it can tolerate such a reduction, the allocation δ is added to a pool that is available for reallocation. If the memory access class cannot tolerate such a reduction, the allocation for that memory access class remains the same.
In line 18, it may be determined how many memory access classes are stalled by subtracting the number of classes determined to be stalled (determined above though iteration of line 10). In line 19, c (the total amount of MPQ available for reallocation) is divided by the total number of memory access classes that are stalled. This may determine the amount of allocation (i.e. incr) that will be added to each stalled access class. Due to the nature of computing systems, the amount of MPQ available for reallocation may not be evenly divisible by the number of stalled memory access classes. In line 20, a modulus operation is performed to determine the remainder (i.e. extra) from the operation in line 19.
In lines 21-26 a loop through all memory classes is performed. In line 22, the s[ ] array is checked for each memory access class to determine if the access class is stalled. If it is stalled, in line 23 the incr amount, which was determined in line 19, is added to the current allocation threshold for the class. The variable last is set to equal the memory access class for the current iteration. After the loop from lines 21-26 is complete, the variable last contains the index of the last memory access class that had its allocation increased. In one example implementation, at line 27, the extra allocation may be added to the last memory class. However, it should be understood that the extra could have been added to any memory access class that was having its allocation increased.
In block 320, it may be determined, at the memory stop of the memory fabric that there is at least one class of memory access that is not stalled. In other words, it may be determined that there is at least on class of memory access that is using an amount of its MPQ allocation such that the stall counter does not indicate that the memory access class is stalled. In such a case, it may be possible to remove MPW allocation from a memory access class that is not stalled without causing the memory access class to become stalled.
In block 330, at least a portion of a memory pending queue from the class of memory access that is not stalled may be reallocated to the class of memory access that is stalled. In other words, a portion of the MPQ that was allocated to a memory access class that is not stalled may be reallocated to one that is stalled.
In one implementation, the process may move to block 420. In block 420, it may be determined if the stall counter for each class of memory access exceeds a threshold for that class of memory access. As explained above, in some implementations, a memory access class is considered stalled if the stall counter for that class indicates that the class has been stalled for a number of cycles that exceeds the threshold for that class. In short, in block 420, it is determined which memory access classes are stalled.
In block 430, it may be determined, at the memory stop of the memory fabric that there are at least two classes of memory access that are stalled. In other words, in the example implementation of
In block 440, it may be determined at the memory stop of the memory fabric that there are at least two classes of memory access that are not stalled. In other words, in the example implementation of
In block 470, the at least a portion of the memory pending queue from all classes of memory access that are not stalled may be reallocated. As explained above, the memory access classes that are not stalled may have a portion of their MPQ allocation removed. This portion of MPQ allocation may then be available for reallocation to those memory access classes that are indicated as stalled. For example, the stalled memory access classes may be determined in block 420, 430 and the available MPQ allocation from non-stalled classes may be determined in block 440. In block 480, the stall counters may be reset after reallocating the at least a portion of the memory pending queue. As explained above, the allocations are not static. At the end of each allocation period, the stall counters are reset, in preparation for determining which memory access classes are stalled for the next period.
In an example of an alternate implementation, in block 450 it may be determine which memory access class has the largest stall counter. This may be indicative of the memory access class that is suffering from the greatest number of stalls. In some implementations, it is only the memory access class that is experiencing more stalls than all other memory access classes that is given a larger allocation of the MPQ.
In block 460 of this alternate implementation, it may be determined that the class of memory access with the largest stall counter is stalled and that all other classes of memory access are not stalled. In other words, the memory access class with the largest stall counter is indicated as stalled, but no other memory access class (even if it is actually stalled) is indicated as stalled. Thus, any reallocation that occurs will reallocate MPQ allocation to only the memory access class with the greatest stall counter.
In block 520, the portions of the memory pending queue may be reallocated when at least one class of memory access is stalled. As explained above, a memory access class may be considered stalled when the class experiences a level of stalls that exceeds a threshold. Exceeding the threshold may indicate that an insufficient portion of the MPQ has been allocated to the particular class that is stalled. In block 520, portions of the MPQ may be reallocated to attempt to alleviate the stall condition of stalled memory access classes.
In one example implementation, the process may move to block 620. In block 620, it may be determined that a class of memory access is stalled when a stall counter for the class of memory access is greater than the stall counter for any other class of memory access. In other words, the memory access class with the greatest number of stalls is considered as the only memory access class that is stalled.
In an alternate example implementation, the process moves to block 630. In block 630, it may be determined that a class of memory access is stalled when a stall counter exceeds a threshold. In other words, unlike the previous example implementation, multiple memory access classes may be stalled at the same time.
Regardless of the implementation, the process may move to block 640. In block 640, the amount of the memory pending queue allocated to each class of memory access that is not stalled may be reduced. However, the reduced amount cannot fall below a minimum threshold. In other words, each memory access class is guaranteed at least some minimum allocation of the MPQ. The allocation that is reduced from each non-stalled memory access class may create an available amount of memory pending queue.
In block 650, the amount of the memory pending queue allocated to each class of memory that is stalled may be increased. The increase may come from dividing the available amount of memory pending queue that was determined in block 640 amongst the classes of memory access that are stalled.
