Patent Application
20230141595
Modern dynamic random-access memory (DRAM) provides high memory bandwidth by increasing the speed of data transmission on the bus connecting the DRAM and one or more data processors, such as graphics processing units (GPUs), central processing units (CPUs), and the like. In one example, graphics double data rate (GDDR) memory has pushed the boundaries of data transmission rates to accommodate the high bandwidth needed for graphics applications. In order to ensure the correct reception of data, modern GDDR memories have required extensive training prior to operation to make sure that the receiving circuit can correctly capture the data. Over time, however, GDDR data transmission systems experience voltage and temperature (VT) drift, which cause the optimum points for the delays to change such that re-training must be performed periodically, which causes the system to have to stall operation while performing the retraining.
In the following description, the use of the same reference numerals in different drawings indicates similar or identical items. Unless otherwise noted, the word “coupled” and its associated verb forms include both direct connection and indirect electrical connection by means known in the art, and unless otherwise noted any description of direct connection implies alternate embodiments using suitable forms of indirect electrical connection as well.
A data processing system includes a data processor coupled to a memory. The data processor includes a reference clock generation circuit for providing a reference clock signal, a first delay circuit for delaying the reference clock signal by a first amount to provide a command and address signal, a second delay circuit for delaying the reference clock signal by a second amount to provide a read data signal, a calibration circuit for determining current values of the first and second amounts, and a compensation circuit for calculating drifts in the first and second amounts based on a measured temperature change, at least one voltage sensitivity coefficient, and at least one temperature sensitivity coefficient, and for updating the first and second amounts according to the drifts.
A data processor adapted to be coupled to a memory includes a reference clock generation circuit, a first delay circuit, a second delay circuit, a calibration circuit, and a compensation circuit. The reference clock generation circuit provides a reference clock signal. The first delay circuit delays the reference clock signal by a first amount to provide a command and address signal. The second delay circuit delays the reference clock signal by a second amount to provide a read data signal. The calibration circuit determines current values of the first and second amounts. The compensation circuit calculates drifts in the first and second amounts based on a measured temperature change, at least one voltage sensitivity coefficient, and at least one temperature sensitivity coefficient, and for updating the first and second amounts according to the drifts.
A method for a data processor to update timing values for accessing a memory to compensate for voltage and temperature (VT) drift during operation without performing link retraining includes generating a reference clock signal. The reference clock signal is delayed by a first amount using a first delay circuit to provide a command and address signal. The reference clock signal is delayed by a second amount using a second delay circuit to provide a read data signal. Current values of said first and second amounts are determined using a calibration circuit. Drifts in said first and second amounts are calculating based on a measured temperature change, at least one voltage sensitivity coefficient, and at least one temperature sensitivity coefficient using a compensation circuit.
GPU 110 is a discrete graphics processor that has extremely high performance for optimized graphics processing, rendering, and display, but requires a high memory bandwidth for performing these tasks. GPU 110 includes generally a set of command processors 111, a graphics single instruction, multiple data (SIMD) core 112, a set of caches 113, a memory controller 114, a DDR physical interface circuit (DDR PHY) 117, and a GDDR PHY 118.
Command processors 111 are used to interpret high-level graphics instructions such as those specified in the OpenGL programming language. Command processors 111 have a bidirectional connection to memory controller 114 for receiving high-level graphics instructions such as OpenGL instructions, a bidirectional connection to caches 113, and a bidirectional connection to graphics SIMD core 112. In response to receiving the high-level instructions, command processors issue low-level instructions for rendering, geometric processing, shading, and rasterizing of data, such as frame data, using caches 113 as temporary storage. In response to the graphics instructions, graphics SIMD core 112 performs low-level instructions on a large data set in a massively parallel fashion. Command processors 111 and caches 113 are used for temporary storage of input data and output (e.g., rendered and rasterized) data. Caches 113 also have a bidirectional connection to graphics SIMD core 112, and a bidirectional connection to memory controller 114.
Memory controller 114 has a first upstream port connected to command processors 111, a second upstream port connected to caches 113, a first downstream bidirectional port to DDR PHY 117, and a second downstream bidirectional port to GDDR PHY 118. As used herein, “upstream” ports are on a side of a circuit toward a data processor and away from a memory, and “downstream” ports are in a direction away from the data processor and toward a memory. Memory controller 114 controls the timing and sequencing of data transfers to and from DDR memory 130 and GDDR memory 140. DDR and GDDR memory have asymmetric accesses, that is, accesses to open pages in the memory are faster than accesses to closed pages. Memory controller 114 stores memory access commands and processes them out-of-order for efficiency by, e.g., favoring accesses to open pages, while observing certain quality-of-service objectives.
DDR PHY 117 has an upstream port connected to the first downstream port of memory controller 114, and a downstream port bidirectionally connected to DDR memory 130. DDR PHY 117 meets all specified timing parameters of the version of DDR memory 130, such as DDR version five (DDR5), and performs timing calibration operations at the direction of memory controller 114. Likewise, GDDR PHY 118 has an upstream port connected to the second downstream port of memory controller 114, and a downstream port bidirectionally connected to GDDR memory 140. GDDR PHY 118 meets all specified timing parameters of the version of GDDR memory 140, and performs timing calibration operations at the direction of memory controller 114.
The interface timing to DDR memory 130 and GDDR memory 140 are susceptible to VT drift. Known techniques for compensation for VT drift center around periodic retraining of the link. However, retraining causes all operations in the system to be stalled while performing the retraining, which may hurt performance and cause jumps and stalls in graphics workloads, diminishing user experience.
In order to overcome the burden of periodic retraining, the inventors have developed various methods for reducing system link sensitivity to VT-induced phase drift. The disclosed VT drift compensation methods reduce, and in some cases eliminate, the need for periodic high-speed link phase retraining. In the exemplary embodiment, the techniques are applied to a GDDR memory interface but they are not restricted to only GDDR memory nor only to memory interfaces.
As shown in
Calibration controller 115 is a circuit that controls calibration of timing parameters for DDR PHY and 117 and GDDR PHY 118. On system startup, the link between DDR PHY 117 and DDR memory 130 has to be trained, and the link between GDDR PHY 118 and GDDR memory 140 is trained. Training generally includes determining the value of a reference voltage used by the memory and PHY to capture input data, the timing relationship between the command clock and data clock(s), and the timing relationship between data and the clock at the sender so that it can be reliably captured by the receiver. Techniques for performing these calibrations are well known and vary based on the DDR and GDDR versions. Moreover, a de facto industry standard for the interface between the memory controller and the memory PHY known as the “DFI” standard has been developed to specify the signaling and characteristics of the interface between the memory controller and the PHY. One of the features of recent versions of the DFI standard is the definition of certain lower-level training features such that most of the calibration functions performed automatically by the PHY, while the overall calibration flow is directed by the memory controller.
In accordance with various embodiments disclosed herein, compensation circuit 116 leverages these capabilities of the PHY circuit such as GDDR PHY 118 to adjust for VT drift without having to do a recalibration operation using calibration controller 115 and GDDR PHY 118. Compensation circuit 116 calculates drifts in timing parameters that are used to control delays in GDDR PHY 118. In one particular embodiment, compensation circuit 116 calculates drifts based on a measured temperature change, at least one voltage sensitivity coefficient, and at least one temperature sensitivity coefficient, and compensates for the timing changes based on these parameters by updating delay amounts of GDDR PHY 118.
GDDR memory 140 includes a set of mode registers 141 and a temperature sensor 142. Mode registers 141 provide a programming interface to control the operation of GDDR memory 140 in the data processing system. As will be explained further below, mode registers 141 store at least one voltage sensitivity coefficient and at least one temperature sensitivity coefficient that are used in VT drift compensation. GDDR memory 140 also includes a temperature sensor for measuring the temperature of GDDR memory 140. In one form, the temperature sensor 142 provides temperature data to compensation circuit 116 in GPU 110 during a refresh operation that ensures that compensation circuit 116 receives updated temperature information periodically.
The inventors have discovered that certain calibrated timing parameters can be adjusted based on measured temperature and voltage differences alone without the need for a performance-impacting recalibration during normal operation. Accordingly, this disclosure describes various methods for reducing system link sensitivity to VT-induced phase drift. The disclosed VT drift compensation methods reduce, and in some cases eliminate, the need for periodic high-speed link phase retraining. This disclosure is presented with respect to a graphics DDR memory interface but is not restricted to only GDDR memory nor only to memory interfaces.
For some GDDR, version 6 (GDDR6) physical layer interface (PHY) systems, voltage and temperature (VT) drift of a parameter known as “WCK2DQI” VT drift direction and magnitude was successfully inferred by monitoring the VT phase drift of an error detection and correction (EDC) lane (WCK2DQO) with respect to a PHY reference clock. As used herein, WSK2DQI means write clock (WCK) to data in delay, and WCK2DQO means WCK_to data-out delay. The PHY reference clock was a branched clock source shared with the error detection and correction (EDC) lane. This basic relationship can be expressed as shown in Equation [1]:
WCK2DQI_drift=WCK2DQO_drift*α [1]
in which α is a scaling factor derived from a hardware evaluation.
Even though many products in use today leverage this WCK2DQI drift correlation to WCK2DQO phase drift, it is not a perfect solution and does not work with all DRAM vendors and applications, and there are several limitations or drawbacks of this method. The inventors herein propose methods to better leverage drift tracking to reduce or eliminate periodic training overhead for high-speed link interfaces, including parameters in GDDR interfaces.
There are two main limitations of the simple model of temperature drift correlation expressed in Equation [1]. First, Equation [1] assumes little to no process variation among DRAM devices. Second, Equation [1] assumes WCK2DQO VT drift symmetrically scales to WCK2DQI for both voltage and temperature sensitivity. In other words, the a scalar must be equivalent for both temperature and voltage, or as expressed in Equation (2):
α_temp=α_volt=WCK2DQI_drift/WCK2QO_drift
The inventors have found that in fact some DRAM devices do not have symmetric correlation between WCK2DQO and WCK2DQI VT drifts. As an example, TABLE I shows VT drift coefficients for write clock to DQ for one such DRAM device:
In which ps represents time in picoseconds, V represents voltage in Volts, and Tc represents temperature in degrees Celsius. Note that VDD represents the memory's typical internal power supply voltage at the worst-case processing corner, VDDQ represents the memory's typical input/output power supply voltage at the worst-case processing corner, and Tc represents temperature at the worst-case processing corner.
On the other hand, the measurements are different for VT drift coefficients for write clock to DQ in from the same DRAM vendor, as shown in TABLE II below:
As can be seen from TABLES I and II above, Equation (2) does not hold true for this DRAM vendor. The variations for this specific example are described by the following equations:
α_temp=0.7/1.1=0.636
α_volt=−30/−180=0.166
α_avg=(alpha_temp+alpha_volt)/2 Equation (1):
Any determination of WCK2DQI VT drift based on WCK2DQO VT drift using this conventional technique would result in a significant phase tracking error, defined by Equation (4):
Phase tracking error=α_error*WCK2DQ_drift [4]
wherein α_error=abs(α_temp±α_volt)*0.5 and WCK2DQ_drift is the total phase drift observed. The 0.5 multiplier used to derive a error assumes that the asymmetry between voltage and temperature alpha factors are averaged.
So, for example, if there is an observed drift of 100 picoseconds (ps) from WCK2DQO, this drift will result in a phase tracking error on WCK2DQI of 100 ps*0.47/2, which results in an error of 23 ps. This amount of phase tracking error is a significant amount and limits the accuracy and therefore the usefulness of existing phase tracking techniques based on Equation [2]. Moreover, this amount was computed without process mismatch terms for different DRAMs of the same vendor product line being considered.
The inventors of the present disclosure have developed new methods and apparatus to overcome these aforementioned limitations. The source of these limitations will be described with respect to a typical GDDR memory PHY to GDDR memory link, which will now be described.
GPU 110 includes a phase locked loop (PLL) 210, a command and address (“C/A”) circuit 220, a read clock circuit 230, a data circuit 240, and a write clock circuit 250. These circuits form part of GDDR PHY 118 of GPU 110.
Phase locked loop 210 operates as a reference clock generation circuit and has an input for receiving an input clock signal labelled “CKIN”, and an output.
C/A circuit 220 includes a delay element 221, a selector 222, and a transmit buffer 223 labelled “TX”. Delay element 221 has an input connected to the output of PLL 210, and an output, and has a variable delay controlled by an input, not specifically shown in
Read clock circuit 230 include a receive buffer 231 labelled “RX”, and a selector 232. Receive buffer 231 has an input connected to a corresponding integrated circuit terminal for receiving a signal labelled “RCK”, and an output. Receive clock selector 232 has a first input for connected to the output of PLL 210, a second input connected to the output of receive buffer 231, an output, and a control input for receiving a mode signal, not shown in
Data circuit 240 includes a receive buffer 241, a latch 242, delay elements 243 and 244, a serializer 245, and a transmit buffer 246. Receive buffer 241 has a first input connected to an integrated circuit terminal that receives a data signal labelled generically as “DQ”, a second input for receiving a reference voltage labelled “VREF”, and an output. Latch 242 is a D-type latch having an input labelled “D” connected to the output of receive buffer 241, a clock input, and an output labelled “Q” for providing an output data signal. The interface between GDDR PHY 118 and GDDR memory 140 implements a four-level, pulse amplitude modulation data signaling system known as “PAM-4”, which encodes two data bits into one of four nominal voltage levels. Thus, receive buffer 241 discriminates which of the four levels is indicated by the input voltage, and outputs two data bits to represent the state in response. For example, receive buffer 241 could generate three slicing levels based on VREF defining four ranges of voltages, and use three comparators to determine which range the received data signal falls in. Data circuit 240 includes latches which latch the two data bits and is replicated for each bit position. Delay element 243 has an input connected to the output of selector 232, and an output connected to the clock input of latch 242. Delay element 244 has an input connected to the output of PLL 210, and an output. Serializer 245 has inputs for receiving a first data value of a given bit position and a second data value of the given bit position, the first and second data values corresponding to sequential cycles of a burst, a control input connected to the output of delay element 244, and an output connected to the corresponding DR terminal. Each data byte of the data bus has a set of data circuits like data circuit 240 for each bit of the byte. This replication allows different data bytes that have different routing on the printed circuit board to have different delay values.
Write clock circuit 250 includes a delay element 251, a selector 252, and a transmit buffer 253. Delay element 251 has an input connected to the output of PLL 210, and an output. Selector 252 has a first input for receiving a first clock state signal, a second input for receiving a second clock voltage, a control input connected to the output of delay element 251, and an output. Transmit buffer 253 has an input connected to the output of selector 252, and an output a first output connected to a corresponding integrated circuit terminal for providing a true write clock signal labelled “WCK_t” thereto, and a second output connected to a corresponding integrated circuit terminal for providing a complement write clock signal labelled “WCK_c” thereto.
GDDR memory 140 includes generally a write clock receiver 270, a command/address receiver 280, and a data path transceiver 290. Write clock receiver 270 includes a receive buffer 271, a buffer 272, a divider 273, a buffer/tree 274, and a divider 275. Receive buffer 271 has a first input connected to an integrated circuit terminal of GDDR memory 140 that receives the WCK_t signal, a second input connected to an integrated circuit terminal of GDDR memory 140 that receives the WCK_c signal, and an output. In the example shown in
Command/address receiver 280 includes a receive buffer 281 and a slicer 282. Receive buffer 281 has a first input connected to a corresponding integrated circuit terminal of GDDR memory 140 that receives the C/A signal, a second input for receiving VREF, and an output. The C/A input signal is received as a normal binary signal having two logic states levels and is considered a non-return-to-zero (NRZ) signal encoding. Slicer 282 has a set of two data latches each having a D input connected to the output of receive buffer 281, a clock input for receiving a corresponding one of the output of divider 275, and a Q output for providing a corresponding C/A signal.
Data path transceiver 290 includes a serializer 291, a transmitter 292, a serializer 293, a transmitter 294, a receive buffer 295, and a slicer 296. Serializer 291 has an input for receiving a first read clock level, a second input for receiving a second read clock level, a select input connected to the output of buffer/tree 274, and an output. Transmitter 292 has an input connected to the output of serializer 293, and an output connected to the RCK_terminal of GDDR memory 140. Serializer 293 has an input for receiving a first read data value, a second input for receiving a second data value, a select input connected to the output of buffer/tree 274, and an output. Transmitter 294 has an input connected to the output of serializer 293, and an output connected to the corresponding DQ terminal of GDDR memory 140. Receive buffer 295 has a first input connected to the corresponding DQ terminal of GDDR memory 140, a second input for receiving the VREF value, and an output. Slicer 296 has a set of four data latches each having a D input connected to the output of receive buffer 295, a clock input connected to the output of buffer/tree 274, and a Q output for providing a corresponding DQ signal.
Interface 260 includes a set of physical connections that are routed between a bond pad of the GPU 110 die, through a package impedance to a package terminal, through a trace on a printed circuit board, to a package terminal of GDDR memory 140, through a package impedance, and to a bond pad of the GDDR memory 140 die.
In operation, data processing system can be used as a graphics card or accelerator because of the high bandwidth graphics processing performed by graphics SIMD core 112. Host CPU 120, running an operating system or an application program, sends graphics processing commands to CPU 110 through DDR memory 130, which serves as a unified memory for GPU 110 and host CPU 120. It may send the commands using, for example, as OpenGL commands, or through any other host CPU to GPU interface. OpenGL was developed by the Khronos Group, and is a cross-language, cross-platform application programming interface for rendering 2D and 3D vector graphics. Host CPU 120 uses an application programming interface (API) to interact with GPU 110 to provide hardware-accelerated rendering.
Data processing system 100 uses two types of memory. The first type of memory is DDR memory 130, and is accessible by both GPU 110 and host CPU 120. As part of the high performance of graphics SIMD core 112, GPU 110 uses a high-speed graphics double data rate (GDDR) memory.
In high-speed DDR memories, read or write data can have variable transmission path delays that change with respect to the clock signal that is used to latch the data elements. Moreover, the JEDEC committee has specified that the processor will calibrate the link such that the data elements can be properly transferred between the data processor and the memory to perform the series of data elements delays between GPU 110 and GDDR memory 140. The various signal processing paths lengths inject skew into the system such that as VT change during operation, the drifts in various signal paths do not track each other such that a simple temperature scaling adjustment shown in Equation [2] does not produce accurate compensated calibration values. This property will now be described.
A timing path 310 shows the path of the write clock formed by differential signals WCK_t and WCK_c to the capture of input (write) data in slicer 296. Timing path 310 shows the received write clock flows through the DRAM package, receive buffer 271, buffer 272, divider 273, and buffer/tree 274 before it arrives at the clock input of slicer 296. A timing path 350 shows the path of the data input signal during a write cycle and shows the received data flows through the DRAM package impedance, and receive buffer 283 to the input of slicer 284. Timing path 310 goes through more circuitry than timing path 350 and changes in VT affect it more than changes in timing path 350. These path delays affect the timing parameter known as WCK2DQI.
A timing path 320 shows the path of the write clock to the output of the read clock RCK. Timing path 320 shows the received write clock flows through the DRAM package resistance receive buffer 271, buffer 272, divider 273, buffer/tree 274, divider 275, serializer 291, transmit buffer 292, and the package impedance to form the read clock. This path delay determines the timing parameter known as WCK2RCK.
A timing path 330 shows the path of the write clock to the capture of the command/address signals in slicer 296. Timing path 320 shows the received write clock flows through the DRAM package impedance, receive buffer 271, buffer 272, divider 273, buffer/tree 274, and divider 275 before it arrives at the clock input of slicer 282. A timing path 340 shows the path of the C/A input signal during a command cycle and shows the received data flows through the DRAM package, and receive buffer 281 to the input of slicer 282. This path affects the timing parameter known as WCK2CA. Timing path 330 goes through more circuitry than timing path 340, and changes in VT affect it more than changes in timing path 340. These path delays affect the timing parameter known as WCK2CA.
These representative circuit diagrams illustrate that VT drift will affect each of these paths differently. For example, propagation time through a package routing path would be affected by temperature but not by the memory's power supply voltages. On the other hand, propagation time through active circuitry would be affected not only by temperature but also by power supply voltage.
For example, the mode register set command is a command that writes to particular bits of a particular mode register of GDDR4 memory 140 to invoke a temperature readout operation. GDDR memory 140 provides the temperature readout derived from temperature sensor 142 on DQ pins 7:0. GDDR memory 140 keeps the DQ pins stable for an extended period of time to allow the temperature to be read before initial timing calibration. In the illustrated embodiment, GDDR memory 140 provides a Binary Temperature Readout within a maximum of a time t
According to some embodiments, calibration controller 115 in memory controller 114 has the flexibility to leverage drift tracking information from WCK2RCK in combination with other VT compensation methods. For example, if the phase drift registered from the WCK2RCK drift exceeds a threshold, then calibration controller 115 could optionally trigger a full Write/Read/CA calibration. This full calibration could be used to update VT sensitivity coefficients. To facilitate this technique, calibration controller 115 may extract multiple voltage and temperature drift magnitudes throughout device operation and update one or more offsets to better predict VT behavior in the future.
To fully leverage the VT sensitivity information stored in mode registers 141, an allowed error tolerance is set as well as maximum drift thresholds that cause the UMC to issue full link-retraining when necessary. An exemplary a set of parameters that can be used for this process is shown in TABLEs III-V.
TABLE III corresponds to GDDR DRAM Reference RX (Write) operations at 32 Gbps transfer speeds:
TABLE IV corresponds to GDDR DRAM Reference TX (Read) Operations at 32 Gbps transfer speeds:
TABLE V corresponds to GDDR DRAM C/A Timing Reference Operations at 32 Gbps transfer speeds:
A data processing system or portions thereof described herein can be embodied one or more integrated circuits, any of which may be described or represented by a computer accessible data structure in the form of a database or other data structure which can be read by a program and used, directly or indirectly, to fabricate integrated circuits. For example, this data structure may be a behavioral-level description or register-transfer level (RTL) description of the hardware functionality in a high-level design language (HDL) such as Verilog or VHDL. The description may be read by a synthesis tool which may synthesize the description to produce a netlist including a list of gates from a synthesis library. The netlist includes a set of gates that also represent the functionality of the hardware including integrated circuits. The netlist may then be placed and routed to produce a data set describing geometric shapes to be applied to masks. The masks may then be used in various semiconductor fabrication steps to produce the integrated circuits. Alternatively, the database on the computer accessible storage medium may be the netlist (with or without the synthesis library) or the data set, as desired, or Graphic Data System (GDS) II data.
While particular embodiments have been described, various modifications to these embodiments will be apparent to those skilled in the art. For example, the embodiments have been described with reference to a graphics double data rate (GDDR) DRAM, but can also be applied to other memory types including non-graphics DDR memory, high-bandwidth memory (HBM), and the like. Moreover while they have been described with reference to a data processing system having a discrete GPU for very high performance graphics operations, they can also be applied to a data processing system with an accelerated processing unit (APU) in which the CPU and GPU are incorporated together on a single integrated circuit chip. The use differential signaling or single-ended signaling, and NRZ data signaling or PAM-4 signaling, can also vary in different embodiments.
Accordingly, it is intended by the appended claims to cover all modifications of the disclosed embodiments that fall within the scope of the disclosed embodiments.
This application claims priority to provisional application U.S. 63/276,950 filed Nov. 8, 2021, the entire contents of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 63276950 | Nov 2021 | US |
