Patent Grant
7900079
U.S. application Ser. No. 11/421,167 filed May 31, 2006, the subject matter of which application is incorporated herein by reference.
This invention relates to the movement of data bits between portions of computer systems. More particularly, this invention relates to alignment of data capture windows to synchronize the movement of data bits through parallel data paths.
The parallel transfer of the bits of data words between a processor and memory is along separate store and fetch paths. The store path proceeds from the controller of the memory to the central storage store buffers, to a set of dynamic random access memory chips (DRAMs) of the memory. The fetch path proceeds from the set of DRAMs to the central storage fetch buffers of the memory controller. The bit positions of the stored and fetched words are physically divided among different components that may have different latency characteristics (the time it takes to propagate data through the path). For example, a memory subsystem of a computer system can include multiple independent processor memory arrays (PMAs) that supply blocks of data in a plurality of parallel transfers of 64 bit wide data words. The bits of the data words come from different synchronous memory interface (SMI2) chips located on separate memory cards. The SMI2 chips communicate with the DRAMs of the memory subsystem and also with the main storage controller (MSC) chip. When the MSC chip receives a data word on the fetch data bus, it must capture all the 64 bits of a data word from the SMI2 chips within a capture window. Likewise, when the SMI2 chips receive data on the store data bus, they must also capture all 64 bits of a data word within a capture window. Both the SMI2 chips and the MSC chip have interface logic to capture data bits. However, there can be differences in the time it takes the bits in different bit positions of the data word to travel the paths between the SMI2 chips and the MSC chip because they travel through different paths with different latency characteristics. As a result, bits in the data words of the block can be misaligned with bits in one or more positions of one word arriving at their destination with bits in bit positions of other data words. The capture logic or windows are adjustable to accommodate different operating parameters in the chips. One method that is used to adjust these windows is a manual setting of the capture latches by trial and error. This method is time consuming and can require different adjustments of latches in different computers with the same model number.
In accordance with the present invention, the adjustment of the data capture window is accomplished dynamically for both the store and fetch paths from the Memory Controller based on use of a storage pattern providing a unique signature identifying different misalignment patterns.
Therefore, it is an object of the present invention to provide an improved method of adjustment of the capture window.
A further object of this invention is to provide an automatic method of adjustment of the capture windows that makes use of the self test capability of the memory.
Therefore it is another object of the invention to eliminate the manual adjustment of capture windows using trial and error methods.
The invention can best be understood by the following description while referring to the accompanying drawings of which:
When this misaligned block of data is read back a different set of data lines 106 are used for the fetch. These will not necessarily have the same latency for the store lines of the same bit position. As shown in
The memory subsystem 100 contains a plurality of independent processor memory arrays (PMAs) that supply blocks of data.
As pointed out above, when the MSC 209 chip receives data on the fetch data bus, it must capture all the data bits of each word from the 4 SMI2 chips within the same capture window of the data capture register 225. Likewise, when the SMI2 chips receive data on the store data bus, they must capture all the data bits of a word within the same capture window of the 4 SMI2 chips. These capture windows are adjustable in the MSC 209 in time by changing different operating parameters to accommodate for the above mentioned latency differences.
This adjustment of the capture windows makes use of a self test function of the memory subsystem performed in the MSC chip 209. The concurrent self test engine 214 is the core of the self test hardware in the MSC 209. Details of the concurrent self test engine can be found in the above mentioned U.S. patent application Ser. No. 11/421,167 (POU920060043) filed on May 31, 2006 which application is hereby incorporated by reference.
The self test engine functions to generate fetch and store patterns for the DRAMs in the dual inline memory modules (DIMMs) 208 in response to millicode commands relayed through dynamic capture millicode capture interface (MCI) controls 215 from the system processor. During test operations, the self test engine is used to generate rotating data patterns, using the data generator 216. These patterns are loaded into the DRAMs 208 through the store buffer, and data stored in the DRAMs are retrieved from the fetch buffer. Data in the fetch buffer are compared with the output of the data generator. The differences between the data generator 216 and fetch buffer 219 outputs are detected by comparator 220 and recorded by the bit error counters 222. The processor memory controller (PMC) 224 controls the store and fetch test operations receiving instructions from the CPU 100 through the MCI controls 215 and the output of the bit error counters 222 are provided to the MCI controls 215. The PMC also controls the adjustment of the timing of the fetched and stored data bits by controlling the latches in register 225.
During the adjustment of the latch timing in register 225, the self test function is used to generate two unique and continuous sets of data patterns shown in tables 1 and 2, one pattern for each of the words in two blocks of 32 data words. As shown in
When the self test function generates a fetch operation to obtain the data in the same central storage locations as the store, the “expected” stored word data patterns are regenerated for each transfer to the MSC chip. This data pattern from the fetch data buffer 219 is compared with regenerated data pattern from the data generator 216. Each data bit comparison is fed to a corresponding single bit error comparator 220. Each bit position of a pair of transfers sent to the error counter 222 which keeps track of any miscompares.
As shown in
This self test function to store another set of patterns is used to detect the alignment or misalignment of the store path window and fetch path window as observed at the fetch detection logic by unique signatures 400 of the self test detection logic. As pointed out above in verifying of the store path window, alignment is to store two unique data patterns into two consecutive DRAM addresses. The first store operation is done with a unique data pattern for first pair of transfer as before. The second store is done to the next consecutive address of the DRAMs after many cycles after the first store completes so there is no chance that the second store would overlay any part of the data window of the first store. The data block for both stores would contain unique data in all 64 data words. The next operations are 2 back-to-back fetches to the same central storage locations as the stores. The store data patterns are regenerated for each transfer in the MSC chip and compared to the returning data from the SMI2 chips for the 2 data blocks. Again, there are bit error counters for each bit position of a pair of transfers that keeps track of any miscompares. The counter 0 records 1 miscompare if either bit 0 in word 0 does not match the regenerated pattern or bit 0 in word 1 does not match or they both do not match.
With the store and fetch operation described in connection with
The following examples demonstrate how these unique data result generate by applying unique data patterns for the store path alignment and the fetch path alignment. In addition, these unique signatures also show the number of cycles which are early or late.
As pointed out previously, Tables 1 and 2 of the appendix show data patterns that are preloaded into the self test data generator for the store and fetch path error detection alignment. Only one bit of each alternate word of the entire two word data pattern has to be different than the rest of the data bits of the word. Each data pattern are rotated one bit position has to be different than the previous of the data transfer pattern. As table 1 shows, the “7FFF_FFFF_FFFF_FFFF” and “BFFF_FFFF_FFFF_FFFF” are the unique data pattern for the first two data transfers. The rest of the 32 pairs of data transfers are each 1 bit different from the previous data pattern and the next data pattern. These two 32 data transfers has to be stored into two consecutive DRAM addresses within a number of gaps to ensure no overlap on the store data. Following are examples of fetched outputs for each of the store and fetch error situation, above numbered.
Tables 3a and 3b. Store Data Capture Window is Early
This example shows the results when the data capture window is three cycles early by reading out the unique signature from these bit error counters. In the example, the expected data pattern is “7FFF FFFF FFFF FFFF” but the returning data pattern is “FFFF FFFF FFFF FFFF” on the first fetched word transfer. Therefore, one bit is miscompared for this transfer. The error counters for bit 0 is increment by ‘1’. After all transfers are completed the return, an unique signature is going to indicate the data capture window is early.
“2222—2222—2222—2222—2222—2222—2222—2111—2222—2222—2222—2222—2222—2222—2222—2222—2111”
Since the “111” occurs in the middle and end of the 2 word pattern, it indicates that store data capture is 3 cycles early. Therefore, the data capture window needs to be retarded by three cycles.
Tables 4a and 4b. Store Data Capture Window is Late
Tables 4a and 4b show the results when data capture window is late as a result of store path misalignment. The example shows the data capture window three cycles late by reading out the unique signature from these bit error counters. The expected data pattern is “7FFF FFFF FFFF FFFF” but the returning data pattern is “EFFF FFFF FFFF FFFF” on the first fetched word transfer. Therefore, two bits are miscompared for this transfer. The error counters for bit 0 and bit 3 are incremented by ‘1’. After the data transfers are completed the resultant unique signature is going to indicate the data capture window is late based on the result of these bit error counters. The unique signature is showed as the following:
“1112—2222—2222—2222—2222—2222—2222—2222—1112—2222—2222—2222—2222—2222—2222—2222”
Since “111” occurs in the beginning and middle of the 2 word transfer it indicates data reception is late by 3 cycles for proper store path alignment. Therefore, the data capture window needs to be moved up by three cycles.
Once the store capture windows are adjusted, the fetch capture windows can be adjusted using data of one of Tables 1 or 2.
Tables 5a and 5b. Fetch Data Capture Window is Early
Tables 5a and 5b show the fetched data pattern when the fetch data capture window is early. The expected data pattern is “7FFF FFFF FFFF FFFF” and the returning data pattern is “FFFF FFFF FFFF FFFF”. Therefore, the bit error counter for bit 0 is incremented by ‘1’ since the bit 0 is the only one bit is mis_compared. After all transfers are completed the return, the unique signature is generated and shows the data capture window is early by three cycles. These unique signature are based on the result of these bit error counters. The unique signature is the following:
“2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2111_”.
Since the “111” occurs only at the end of the data pattern, it indicates fetch data capture is 3 cycles early for proper fetch patch alignment. Therefore, the data capture window needs to set back by three cycles.
Table 6a and 6b. Fetch Data Capture Window is Late
Table 6 shows the data pattern from the DRAM. The expected data pattern is “7FFF FFFF FFFF FFFF” and the returning data pattern is “EFFF FFFF FFFF FFFF”. Therefore, the bit error counter for bit 0 and bit 3 are incremented by ‘1’ since these two bits are miscompared. After all data transfer are completed the return, the unique signature is generated and show the data Capture window is late by three cycles. The unique signature is showed as the following:
“1112—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222—2222_”.
When the “111” only occurs at the beginning of the dual word returns, it indicates data capture is 3 cycles too late for fetch patch alignment. Therefore, the data capture window needs to be moved up by three cycles.
Table 7. Store/Fetch Data Capture Window is Correct
Table 7 shows the data pattern return with the data capture words properly adjusted. The expected data pattern is “7FFFF FFFF FFFF FFFF” and the returning data pattern is also “7FFF FFFF FFFF FFFF”. The unique signature is therefore all zeroes.
Table 8a and 8b—Data Capture Window Late for Store and Fetch Commands
Table 8 shows the unique data pattern return for a combination of errors. Each combination of errors provides a unique signature that identifies the misalignment of the combination.
From the above you can see that the unique signature result 400 identifies each possible captured error and combinations thereof. The MCI provides the signature to the CPU 100 which identifies the correction to be made and instructs the PMCs 224 to adjust the appropriate fetch capture windows of the fetch capture register for fetch misalignments and/or the SMI2 controllers to adjust their appropriate store data capture windows for store misalignments.
Step 1. Load the initial unique Data Pattern of Tables 1 and 2 into the self test engine 214 and have self test engine run the store operation to load the data patterns into two consecutive memory block addresses.
Step 2. Have the self test engine perform a fetch operation after the store operation is complete.
Step 3. Read the bit error counter after the fetch operation is complete.
Step 4. Determine if the setting of all the capture windows is correct or not.
Step 5. If the bit error counters are all zeroes then the capture windows are correctly adjusted and proceeds to exit the operation.
Step 6. If the signature is not all zeroes, check the bit error counters to determine whether any store capture windows are late.
Step 7. If any store capture windows are late, adjust those windows to capture early. This may require several iterations to get the late storing windows properly adjusted. After each adjustment of step 7, proceed to steps 2 and perform steps 2, 3 and 4 over again until step 6 reads no store late capture windows.
Step 8. Check to see if any of the store capture windows capture early.
Step 9. Adjust any store capture windows that capture early to capture later repeating all iterations necessary to properly adjust the store capture windows.
Step 10. Once the store capture windows are indicated adjusted by step 9, determine if any fetch windows capture late.
Step 11. If any fetch capture windows are indicated as late by the bit error counter data, adjust them early repeating as many iterations as is necessary until step 10 indicates that all fetch late windows are properly adjusted. (In adjusting fetch windows the use of only one of the storage patterns of tables 1 and 2 is necessary.)
Step 12. With the adjustment of the late fetch windows indicated as complete, check to see if the storage patterns indicate if any fetch capture windows capture early.
Step 13. Adjust any capture windows indicated as adjusting early repeating the process as many times as necessary until step 4 indicates that all windows are properly adjusted.
Above we have described one embodiment of our invention. It should be understood that the invention is not limited to this embodiment. Many alterations of the concepts presented here are possible. For instance, the described embodiment relates to transfer of data between a memory and a CPU. The concepts apply equally as well to other transfers. Therefore, it should be understood that the invention should apply to all embodiments thereof falling within the spirit and scope of the following claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5325516 | Blomgren et al. | Jun 1994 | A |
| 6073253 | Nordstrom et al. | Jun 2000 | A |
| 6105144 | Wu | Aug 2000 | A |
| 6449727 | Toda | Sep 2002 | B1 |
| 6553527 | Shephard, III | Apr 2003 | B1 |
| 6625176 | Amann et al. | Sep 2003 | B1 |
| 6892314 | Chen | May 2005 | B2 |
| 6988231 | Barnett | Jan 2006 | B2 |
| 7035368 | Pickering et al. | Apr 2006 | B2 |
| 7069458 | Sardi et al. | Jun 2006 | B1 |
| 7089440 | Wu | Aug 2006 | B2 |
| 7321617 | Garlett et al. | Jan 2008 | B2 |
| 7334148 | Liu et al. | Feb 2008 | B2 |
| 7590879 | Kim et al. | Sep 2009 | B1 |
| 7633877 | Sharma et al. | Dec 2009 | B2 |
| 7680179 | Garlett et al. | Mar 2010 | B2 |
| 20010054135 | Matsuda | Dec 2001 | A1 |
| 20070140397 | Jaussi et al. | Jun 2007 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20080126664 A1 | May 2008 | US |
