Patent Application
20070282967
For a detailed description of illustrative embodiments of the invention, reference will now be made to the accompanying drawings in which:
Certain terms are used throughout the following description and claims to refer to particular system components. As one skilled in the art will appreciate, computer companies may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In the following discussion and in the claims, the terms “including” and “comprising” are used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to . . . .” Also, the term “couple” or “couples” is intended to mean either an indirect or direct electrical connection. Thus, if a first device couples to a second device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.
The following discussion is directed to various embodiments of the invention. Although one or more of these embodiments may be preferred, the embodiments disclosed should not be interpreted, or otherwise used, as limiting the scope of the disclosure. In addition, one skilled in the art will understand that the following description has broad application, and the discussion of any embodiment is meant only to be exemplary of that embodiment, and not intended to intimate that the scope of the disclosure is limited to that embodiment.
In accordance with some embodiments of the invention, each computer slice 10 comprises one or more processor elements, and as illustrated in
In accordance with some embodiments of the invention, at least one processor element from each computer slice 10 is logically grouped to form a logical processor. In the illustrative embodiments of
Inasmuch as there may be two or more processor elements within a logical processor executing the same application program, duplicate reads and writes are generated, such as reads and writes to network interfaces 23 and 24. In order to compare the reads and writes for purposes of fault detection, each logical processor has an associated synchronization logic. For example, logical processor 12 is associated with synchronization logic 18. Likewise, the processor elements PA2 and PB2 form a logical processor associated with synchronization logic 20. Thus, each computer slice 10 couples to each of the synchronization logics 18 and 20 by way of an interconnect 27. The interconnect 27 is a Peripheral Component Interconnected (PCI) bus, and in particular a serialized PCI bus, although any bus or network communication scheme may be equivalently used.
Each synchronization logic 18 and 20 comprises a voter logic unit, e.g., voter logic 22 of synchronization logic 18. The following discussion, while directed to voter logic 22 of synchronization logic 18, is equally applicable to the voter logic unit of the synchronization logic 20. Consider for purposes of explanation each processor element in logical processor 12 executing its copy of an application program, and that each processor element generates a read request to network interface 24. Each processor element of logical processor 12 sends its read request to the voter logic 22. The voter logic 22 receives each read request, compares the read requests, and (assuming the read requests agree) issues a single read request to the network interface 24. The read request in some embodiments is a series of instructions programming a direct memory access (DMA) engine of the network interface 24 to perform a particular task. The data is read from the network interface 24 and the returned data is replicated and passed to each of the processor elements of the logical processor by the synchronization logic 18. Likewise, for other input/output functions, such as writes and transfer of packet messages to other programs (possibly executing on other logical processors), the synchronization logic ensures that the requests match, and then forwards a single request to the appropriate location. In the event one of the processor elements in the logical processor does not function properly (e.g., fails to generate a request, fails to generate a request within a specified time, generates a non-matching request, or fails completely), the offending processor element is voted out and the overall user program continues based on requests of the remaining processor element or processor elements of the logical processor.
Each of the processor elements may couple to an I/O bridge and memory controller 26 (hereinafter I/O bridge 26) by way of a processor bus 28. The I/O bridge 26 couples the processor elements to one or more memory modules of a memory 30 by way of a memory bus. Thus, the I/O bridge 26 controls reads and writes to the memory area and also allows each of the processor elements to couple to synchronization logics 18 and 20.
Still referring to
Still referring to
In accordance with at least some embodiments, synchronization logic 38 couples the persistent memory of the computer slices to the communication network 36. For purposes of redundancy, a persistent memory may have two dedicated synchronization logics. Synchronization logic 38 is similar in form and in structure to synchronization logics 18 and 20, and synchronization logic 38 may also perform tasks associated with implementing the persistent memory. In particular, synchronization logic 38 has the ability to do direct memory accesses to the memory from each computer slice assigned to be persistent memory 34. When the persistent memory 34 is being accessed by remote direct memory access (RDMA) requests, the synchronization logic 38 receives a single RDMA request from the communications network 36, replicates the request, and applies the replicated requests one each to each memory 30. Thus, although the persistent memory in this illustrative case comprises two physical memories in different computer slices, the persistent memory 34 appears to accessing programs as a single persistent memory unit. This feature is a significant advantage over related-art systems where the writing device has to manage multiple independent persistent memory devices by, for example, duplicating write requests.
In computing systems utilizing two computer slices (dual-modular redundant), such as
In the specific case of RDMA writes to the persistent memory 34, the illustrative synchronization logic 38 duplicates those writes, checks that the accessing device is authorized to use the particular portion of the persistent memory (either internally, or possibly by message exchange with one or more of the I/O bridges 26). If the accessing device is authorized to access the particular portion of the persistent memory, the synchronization logic 38 forwards the direct memory access write to each physical memory 30 by way of their respective I/O bridge 26. The I/O bridge may be busy with other reads and/or writes when the direct memory access write arrives, and thus the write may be stored in buffers in the I/O bridge 26 and actually written to the memory at some later time. Regardless of whether the write to each memory takes place immediately, or after some delay, after forwarding to the I/O bridges the data exists in on different computer slices, and therefore in different fault zones. After forwarding the writes to the I/O bridges, the synchronization logic 38 sends an acknowledgement to the device which sent the DMA write over the communications network 36, which may be any currently available or later developed communications network having RDMA capability, such as ServerNet, GigaNet, Infiniband, or Virtual Interface architecture compliant system networks (SANS). In the illustrative case of a ServerNet communication network, the acknowledgement message sent by the synchronization logic 38, because it is sent after the data is placed in separate fault zones, may be viewed by the requesting device as an indication that the data is safely stored, and may take only on the order of 10 microseconds to generate. Thus, the acknowledgement is generated quickly, and there is no need to send a higher level acknowledgement when the data is actually written.
In the case of RDMA read requests received by the synchronization logic 38 from the communications network 36, the synchronization logic 38 preferably performs direct memory access reads to each physical memory of the persistent memory 34 and provides the requested data to the voter logic 40 within the synchronization logic 38. Much like the voter logic 22 in the illustrative synchronization logic 18, voter logic 40 in synchronization logic 38 compares the read request data from each portion of the persistent memory 34, and if the read request data matches, provides a single set of read request data to the requesting device by way of network interface 42 and communications network 36.
Still referring to
Once a particular application program has been assigned a portion of a persistent memory by the PMM, the PMM need not be contacted again by the application program unless the size of the assigned area needs to be changed, or the application program completes its operation with the persistent memory and wishes to release the memory area. Thereafter, the application program executing in each processor element in the illustrative logical processor 12 generates an RDMA request to the persistent memory, which RDMA requests are voted in the voter logic 22 of synchronization logic 18, and a single RDMA request sent across the communications network 36.
When being assigned portions of a persistent memory, in some embodiments the application program in the logical processor is assigned a virtual address in the persistent memory space. The virtual address of the assigned persistent memory space may be translated into a network virtual address, e.g., by way of network interface 42 associated with persistent memory 34. The application program thus need not know the virtual address in the persistent memory space, but only the network virtual address. The persistent memory request thus traverses the illustrative network 36 and arrives at the target network interface, such as network interface 42 in synchronization logic 38. The network interface 42 and/or the synchronization logic 38 translate the network virtual address into physical memory addresses within each computer slice. In accordance with these embodiments of the invention, the PMM, regardless of its location, programs the various synchronization logics with information such that the various translations may be completed. In alternative embodiments of the invention, application programming interfaces within the logical processor may be accessed to perform the various translations from virtual address to network virtual address and/or to physical memory address. While a particular application program or other requesters may be assigned a contiguous set of virtual addresses in the persistent memory, the locations within the physical memory 30 need not be contiguous.
The systems of
In accordance with embodiments of the invention, the various persistent memories are called “persistent” because the information contained in the persistent memory survives single points of failure. For example, in embodiments that implement persistent memory across multiple computer slices, the failure of a particular computer slice, because the information is still available in the non-failed computer slices, does not result in data loss. Moreover, for embodiments where the computer slices also utilize processor elements executing application programs, the persistent memory is not affected and the information that the application program wrote to the persistent memory 34 would still be available after restarting of the application program.
The persistent memory discussed to this point obtains some of its persistence in the form of duplication across multiple computer slices. In addition, or in place of, the persistence in the form of duplication across computer slices, the portion of a physical memory 30 that is assigned to the persistent memory may itself be non-volatile memory. Thus, some or all of the physical memory 30 assigned to a persistent memory within a computer slice may be magnetic random access memory (MRAM), magneto-resistive random access memory (MRRAM), polymer ferroelectric random access memory (PFRAM), ovonics unified memory (OUM), and flash memories of all kinds. Further still, the physical memory assigned to the persistent memory may be volatile in the sense that it loses data upon loss of power, but may be made to act as non-volatile by use of a battery-backed system.
Notwithstanding the persistence obtained by physical duplication and/or the use of non-volatile memories, each persistent memory may itself be made up of two or more partitions in each physical memory 30. In the case of a persistent memory comprising two partitions of a physical memory, each computer slice 10 maintains duplicate copies of the information in the persistent memory. Thus, in alternative embodiments of a persistent memory comprising the physical memory of two computer slices, with each physical memory having two partitions assigned to the persistent memory, four complete copies of the information in the persistent memory may be maintained.
An illustrative operation that may be performed by the processor element 46 is a memory “scrubbing” operation. Scrubbing may take many forms. In some embodiments, scrubbing may involve each processor element checking for memory faults identifiable by embedded error correction codes. Thus, these scrubbing operations are independent of the memory in other slices of the persistent memory. In alternative embodiments, scrubbing may involve comparisons of memory locations from computer slice to computer slice. In particular, in these alternative embodiments each processor element 46 may periodically or continuously scan the one or more partitions in the computer slice 10 of the persistent memory 34, and compare gathered information to that of the companion processor element or processor elements in other computer slices of the computing complex 1000. Thus, processor elements 46 associated with a persistent memory 34 may communicate to each other in a non-voted fashion using the synchronization logics. For example, voter logic 40 of synchronization logic 38, illustrative of all voter logics associated with persistent memories, comprises a plurality of registers 44. The processor elements 46 within the persistent memory may exchange messages with other processor elements associated with a persistent memory by writing data (in a non-voted fashion) to the registers 44, and then requesting that the synchronization logic 38 inform the other processor elements 46 of the presence of data by sending those other processor elements an interrupt (or by polling). Consider, for example, processor 46A performing a memory scrubbing operation that involves calculating the checksum of a predetermined block of memory. Processor element 46A may communicate this checksum to processor element 46B by writing the checksum to one or more of the registers 44, and then requesting that the voter logic 40 issue an interrupt to the target processor element. Processor element 46B, receiving the interrupt and decoding its type, reads the information from the one or more registers 44 in the voter logic 40. If additional processor elements associated with the persistent memory are present, these processor elements may also receive the interrupt and may also read the data. Processor element 46B, calculating the checksum on the same block of memory in its physical memory 30B, may thus compare the checksums and make a determination as to whether the physical memories match. Likewise, processor element 46B may send a checksum for the predetermined block of memory to processor element 46A to make a similar determination. Thus, the processor elements 46 may periodically or continuously scan the partitions associated with the persistent memory 34 to proactively identify locations where the memories, that should be duplicates of each other, differ. If differences are found, corrective action may be taken, such as copying a portion of a physical memory 30 assigned to be the persistent memory 34 to corresponding locations in the second computer slice. The discussion now turns to correcting faults in the persistent memory.
Returning again to
As mentioned above, when not reintegrating memory, the reintegration logics 32 are transparent to memory operations. When used for memory reintegration, however, the reintegration logics work in concert to duplicate memory writes bound for one memory, and apply them to the second memory. Consider for purposes of explanation that computer slice 10A of
In alternative embodiments, the synchronization logic associated with the persistent memory may perform the memory copy. The synchronization logic may read each memory location of the non-faulted portion, and write each corresponding location in the faulted portion. Any writes received by the synchronization logic across the communication network 36 would also be passed to both memories 30A and 30B, but reads would be only from the non-faulted portions. Once the memory is copied, the previously faulted partition is again utilized.
The various embodiments discussed to this point partition memory of a computer slice such that the persistent memory uses at least one partition, and an application program executing on a processor element uses another partition; however, a “computer slice” in accordance with embodiments of the invention need not have a processor element executing programs, and instead may have either no processor elements (and thus comprising a memory controller, possibly an integration logic, and possibly a reintegration logic), or a processor with relatively low capability which may perform memory scrubbing operations as discussed above and/or implement programs to perform memory copying.
The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
