Patent Grant
7000089
1. Technical Field
This invention relates generally to transactions, such as input/output (I/O) requests and their responses, and more particularly to serializing such transactions.
2. Description of the Prior Art
There are many different types of multi-processor computer systems. A symmetric multi-processor (SMP) system includes a number of processors that share a common memory. SMP systems provide scalability. As needs dictate, additional processors can be added. SMP systems usually range from two to 32 or more processors. One processor generally boots the system and loads the SMP operating system, which brings the other processors online. Without partitioning, there is only one instance of the operating system and one instance of the application in memory. The operating system uses the processors as a pool of processing resources, all executing simultaneously, where each processor either processes data or is in an idle loop waiting to perform a task. SMP systems increase in speed whenever processes can be overlapped.
A massively parallel processor (MPP) system can use thousands or more processors. MPP systems use a different programming paradigm than the more common SMP systems. In an MPP system, each processor contains its own memory and copy of the operating system and application. Each subsystem communicates with the others through a high-speed interconnect. To use an MPP system effectively, an information-processing problem should be breakable into pieces that can be solved simultaneously. For example, in scientific environments, certain simulations and mathematical problems can be split apart and each part processed at the same time.
A non-uniform memory access (NUMA) system is a multi-processing system in which memory is separated into distinct banks. NUMA systems are similar to SMP systems. In SMP systems, however, all processors access a common memory at the same speed. By comparison, in a NUMA system, memory on the same processor board, or in the same building block, as the processor is accessed faster than memory on other processor boards, or in other building blocks. That is, local memory is accessed faster than distant shared memory. NUMA systems generally scale better to higher numbers of processors than SMP systems.
Multi-processor systems usually include one or more memory controllers to manage memory transactions from the various processors. The memory controllers negotiate multiple read and write requests emanating from the processors, and also negotiate the responses back to these processors. Usually, a memory controller includes a pipeline, in which transactions, such as requests and responses, are input, and actions that can be performed relative to the memory for which the controller is responsible are output.
For transactions to be serviced correctly, usually they need to be serialized so that they are performed in the correct order. Serialization may occur within the pipeline of a memory controller, or prior to the transactions entering the pipeline. Transactions are commonly serialized by utilizing the cache addresses of memory lines to which they relate. This allows the serialization logic, for instance, to distinguish transactions from one another based on their addresses.
Typically, there is a serialization logic for each type of different transaction. For instance, non-coherent input/output (I/O)-related transactions may have one type of serialization logic, whereas coherent memory-related transactions may have another type of serialization logic. While this is a workable approach, it means that serialization logic must be developed for each type of different transaction, which can be time-consuming. Furthermore, space on an integrated circuit (IC) must be allocated for each developed serialization logic, which may be at a premium. For these and other reasons, therefore, there is a need for the present invention.
The invention relates to the assignment of an address to a transaction for serialization purposes. In a method of the invention, a simulated address is assigned to a transaction of a first type. The transaction is then serialized relative to other transactions of the first type, utilizing a serialization approach for transactions of a second type.
A system of the invention includes a plurality of processors, local random-access memory (RAM) for the plurality of processors, and at least one memory controller. The memory controller(s) manage transactions relative to the local RAM. Each controller assigns simulated addresses to those of the transactions that are of a first type, and serializes such transactions utilizing a serialization for those of the transactions that are of a second type.
A memory controller of the invention includes a pipeline having a number of stages to serialize and convert transactions to sets of actions to effect the transactions. Those of the transactions of a first type are assigned simulated addresses prior to serialization utilizing a serialization approach for those of the transactions of a second type. Other features, aspects, embodiments and advantages of the invention will become apparent from the following detailed description of the presently preferred embodiment of the invention, taken in conjunction with the accompanying drawings.
The drawings referenced herein form a part of the specification. Features shown in the drawing are meant as illustrative of only some embodiments of the invention, and not of all embodiments of the invention, unless otherwise explicitly indicated, and implications to the contrary are otherwise not to be made.
The method 100 first receives a transaction that is of a first type (102). The type of the transaction may be such that the transaction is an input/output (I/O)-related transaction, such as a memory-mapped I/O (MMIO) transaction. Such transactions are typically non-coherent, in that they are not cached, and thus do not have cache addresses to which they relate. The transaction may be a request for an action to be performed, or a response to a previous request for action. An example of a specific type of MMIO transaction is specifically a control status register (CSR) transaction, which relates to the CSR of a system.
A simulated address is assigned to the transaction (104). The simulated address is preferably a fake, or manufactured, address, that does not correspond or is otherwise non-representative of an actual utilizable address. That is, the simulated address does not refer to actual cache memory of the system. The simulated address is desirably unique as compared to any other simulated addresses that may have been previously assigned to transactions of the same (first) type, especially as to transactions that are still in the pipeline. This ensures that the transaction is uniquely identifiable by its simulated address, as compared to other transactions of the same type.
Once the transaction has been assigned a simulated address, it can then be serialized relative to other transactions that have been assigned other simulated addresses (106). Preferably, serialization is performed utilizing an existing serialization approach, or process, for transactions of a different, or second, type. For instance, the serialization approach may be that which already exists and already used for transactions that relate to cached memory.
Thus, the simulated address assigned to the transaction in 104 is used to serialize the transaction relative to other transactions of the first type in 106. That is, the serialization approach may be geared for transactions of the second type, such that transactions of the first type have simulated addresses assigned thereto that enable the same approach to be used to serialize the transactions of the first type, too. The simulated addresses that are assigned are such that they enable the transactions of the first type to be serialized as if they were transactions of the second type.
Finally, the transaction is effected (108). This means that processing occurs on the transaction so that it can be performed, or realized. For example, the pipeline may be used to convert the transaction into a set of concurrently performable actions.
The processors 306, memory bank 308, and secondary controller 314 constitute a first quad. Likewise, the processors 318, memory bank 320, and secondary controller 326 constitute a second quad. Each of these two quads shares the services of the controllers 310 and 322 and the caches 312 and 324 to form a node of eight processors with associated memory and caches. The memory controller 310 and the cache 312 service even addresses for both quads, and the memory controller 322 and the cache 324 service odd addresses for both quads.
Each quad accesses both even and odd addresses, but these accesses are segregated into even and odd for service by the respective memory controller and cache. The left part 302 has a left memory bank 308, whereas the right part 304 has a right memory bank 320. The memory banks 308 and 320 represent the respective random-access memory (RAM) local to the parts 302 and 306 respectively. The memory bank 308 contains all local memory for the first quad, and the memory bank 320 contains all local memory for the second quad.
The left memory controller 310 manages even address requests to and responses from both memory banks 308 and 320, whereas the right memory controller 322 manages odd address requests to and responses from both memory banks 308 and 320. Each of the controllers 310 and 322 may be an applications-specific integrated circuit (ASIC) in one embodiment, as well as another combination of software and hardware. To assist management of the banks 308 and 320, the controllers have caches 312 and 324, respectively. A left secondary controller 314 specifically interfaces the memory bank 308, the processors 306, and both memory controllers 310 and 322 with one another, and a right secondary controller 326 specifically interfaces the memory bank 320, the processors 318, and both memory controllers 310 and 322 with one another.
The left memory controller 310 is able to communicate directly with the right memory controller 322, as well as the secondary controller 326. Similarly, the right memory controller 322 is able to communicate directly with the left memory controller 310 as well as the secondary controller 314. Each of the memory controllers 310 and 322 is preferably directly connected to the interconnection network that connects all the nodes, such as the interconnection network 204 of
In a first, decode, pipeline stage, a transaction is decoded into an internal protocol evaluation (PE) command (402). The internal PE command is used by the method 400 to assist in determining the set of performable actions that may be concurrently performed to effect the transaction. In one embodiment, a look-up table (LUT) is used to retrieve the internal PE command, based on the transaction proffered. There may be more than one LUT, one for each different type of transaction. For instance, the method 400 may utilize a coherent request decode random-access memory (RAM) as the LUT for coherent memory requests, a non-coherent request decode RAM as the LUT for non-coherent memory requests, and a response decode RAM as the LUT for memory responses.
In a second, integration, pipeline stage, an entry within a PE RAM is selected based on the internal PE command (404). The PE RAM is the memory in which the performable actions are specifically stored or otherwise indicated. The entry within the PE RAM thus indicates the performable actions to be performed for the transaction, as converted to the internal PE command. In one embodiment, the PE command is first converted into a base address within the PE RAM, and an associated qualifier having a qualifier state, which is then used to select the appropriate PE RAM entry. Furthermore, the transaction may be arbitrated among other transactions within the second pipeline stage. That is, the transactions may be re-arbitrated within the second stage, such that the order in which the transactions had entered the pipeline may be changed.
In a third, evaluation, pipeline stage, the entry within the PE RAM is converted to a concurrent set of performable actions to effect the transaction (406). In one embodiment, this is accomplished by selecting the concurrent set of performable actions, based on the entry within the PE RAM, where the PE RAM stores or otherwise indicates the actions to be performed. Once the performable actions have been determined, the conversion of the transaction to the performable actions is complete. The actions may then be preferably concurrently dispatched for performance to effect the transaction relative to the memory of the multiple-processor system.
In one embodiment, the transaction may or may not have to be serialized. This can be indicated in the sixth bit, [6], of the command type attribute. If the bit is one, then the transaction is to be serialized, whereas if it is zero, then the transaction is not to be serialized. If the transaction is not to be serialized (602), then the method 600 proceeds to effect the transaction (108), as has been described. That is, the transaction is converted to a set of concurrently performable actions, which are then performed to effectuate the transaction.
However, if the transaction is to be serialized (602), then it is assigned a simulated address (104). In one embodiment, this includes first selecting a mask for constructing the simulated address (604). For example, there may be a number of different masks, where each mask corresponds to a different list of addresses from which the simulated address is selected, or determined. In one embodiment, the mask is selected based on bits [5:3] of the command type attribute. Because there are three such bits, the mask is thus selected from a total of 23, or eight, different masks. The mask has a set length desirably equal to the length of a cache address, such as 24 bits, or [23:0]. The highest bits of the mask are then used to construct the highest bits of the simulated address, such as the bits [23:7] of the mask.
The simulated address is constructed using the mask (606). That is, it can be said that the simulated address is selected from one of a list of addresses corresponding to the different masks. In one embodiment, the highest bits of the simulated address are determined by performing a logical OR operation on the highest bits of the mask with a number of bits determined by concatenating various bits of various attributes of the transaction. For instance, two zero bits may be concatenated with bits [5:0] of the command type attribute, bits [3:0] of the additional attribute, bits [3:0] of the source attribute, and the single bit [0] of the use-map attribute. The two zero bits are the highest bits of the resulting concatenation, and the single bit [0] of the use-map attribute is the lowest bit of the resulting concatenation. The resulting 17 bits are then logically OR'ed with the bits [23:7] of the mask to determine the highest 17 bits of the simulated address.
The lowest seven bits are determined by starting with zero, or 1 x0000000, and for each transaction that has the same highest 17 bits for a simulated, increasing by one thereafter. For instance, the first transaction having as its simulated address a given highest 17 bits has 1x0000000 as the lowest seven bits for its simulated address. The second transaction having these same highest 17 bits for its simulated address has 0x0000001 as the lowest seven bits for its simulated address, and so on. This effectively serializes subsequently received transactions that have the same highest 17 bits for their simulated addresses, in lists of addresses corresponding to the masks.
Once the simulated address has been determined, the transaction can be serialized (106). In one embodiment, this is accomplished by utilizing an already existing serialization scheme or approach that is used for serializing transactions of a different type that have addresses comparable to the simulated addresses. Finally, the transaction is effected (108), such as by conversion into a set of concurrently performable actions, performing these actions, and so on.
The simulated addresses for transactions that are to be serialized can be constructed in manners other than that which has been described in conjunction with the method 600 of
As another example, a number of lists of addresses may be employed without utilizing a mask, to construct the simulated addresses. The lists may be selected randomly, in a round-robin manner, or there may only be one list. As transactions arrive, they are assigned an address within one of the lists of addresses. Where there is only one list, it may start as a base address, and each successive transaction that needs to be serialized is assigned the base address, plus a counter, that is incremented after a transaction has been assigned an address. Still other approaches for assigning simulated addresses to transactions are also within the scope of the invention.
Embodiments of the invention allow for advantages over the prior art. By assigning simulated addresses to transactions of a first type, the transactions may be serialized utilizing a serialization approach already used for transactions of a different, second type. This means that no further code needs to be written, and take up space within the memory controller, for serializing transactions of the first type. Rather, the serialization approach already used for transactions of the second type is leveraged for use for transactions of the first type.
It will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of the invention. For instance, the system that has been described as amenable to implementations of embodiments of the invention has been indicated as having a non-uniform memory access (NUMA) architecture. However, the invention is amenable to implementation in conjunction with systems having other architectures as well. As another example, the system that has been described has two memory controllers. However, more or less memory controllers may also be used to implement a system in accordance with the invention. Accordingly, the scope of protection of this invention is limited only by the following claims and their equivalents.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5243699 | Nickolls et al. | Sep 1993 | A |
| 5649160 | Corry et al. | Jul 1997 | A |
| 6088800 | Jones et al. | Jul 2000 | A |
| 6128244 | Thompson et al. | Oct 2000 | A |
| 6154816 | Steely et al. | Nov 2000 | A |
| 6321303 | Hoy et al. | Nov 2001 | B1 |
| 6434699 | Jones et al. | Aug 2002 | B1 |
| Number | Date | Country | |
|---|---|---|---|
| 20040123015 A1 | Jun 2004 | US |
