Patent Grant
9594647
The present invention relates generally to computer memory, and more specifically, to out-of-synchronization detection and out-of-order detection in a memory system.
Contemporary high performance computing main memory systems are generally composed of one or more memory devices, which are connected to one or more memory controllers and/or processors via one or more memory interface elements such as buffers, hubs, bus-to-bus converters, etc. The memory devices are generally located on a memory subsystem such as a memory card or memory module and are often connected via a pluggable interconnection system (e.g., one or more connectors) to a system board (e.g., a PC motherboard).
Overall computer system performance is affected by each of the key elements of the computer structure, including the performance/structure of the processor(s), any memory cache(s), the input/output (I/O) subsystem(s), the efficiency of the memory control function(s), the performance of the main memory devices(s) and any associated memory interface elements, and the type and structure of the memory interconnect interface(s).
Extensive research and development efforts are invested by the industry, on an ongoing basis, to create improved and/or innovative solutions to maximizing overall system performance and density by improving the memory system/subsystem design and/or structure. High-availability systems present further challenges as related to overall system reliability due to customer expectations that new computer systems will markedly surpass existing systems in regard to mean-time-between-failure (MTBF), in addition to offering additional functions, increased performance, increased storage, lower operating costs, etc. Other frequent customer requirements further exacerbate the memory system design challenges, and include such items as ease of upgrade and reduced system environmental impact (such as space, power and cooling). In addition, customers are requiring the ability to access an increasing number of higher density memory devices (e.g., DDR3 and DDR4 SDRAMs) at faster and faster access speeds.
In a high-availability memory subsystem, a memory controller typically controls multiple memory channels, where each memory channel has one or more dual in-line memory modules (DIMMs) that include dynamic random access memory (DRAM) devices and in some instances a memory buffer chip. The memory buffer chip typically acts as a slave device to the memory controller, reacting to commands provided by the memory controller. The memory subsystem can be configured as a redundant array of independent memory (RAIM) system to support recovery from failures of either DRAM devices or an entire channel. In RAIM, data blocks are striped across the channels along with check bit symbols and redundancy information. Examples of RAIM systems may be found, for instance, in U.S. Patent Publication Number 2011/0320918 titled “RAIM System Using Decoding of Virtual ECC”, filed on Jun. 24, 2010, the contents of which are hereby incorporated by reference in its entirety, and in U.S. Patent Publication Number 2011/0320914 titled “Error Correction and Detection in a Redundant Memory System”, filed on Jun. 24, 2010, the contents of which are hereby incorporated by reference in its entirety.
In one example of a RAIM system, a memory controller interfaces with multiple channels, where each channel includes at least one buffered DIMM with an option of cascading one or more additional buffered DIMMs per channel. In the RAIM system, data from all channels must be perfectly aligned to a same upstream cycle at the memory controller. Since wire latencies to each of the channels and cascades could be drastically different, all round-trip controls and data are tightly aligned. The memory buffer chips perform a learning process to establish ‘deskewing’ of all the data and controls in order to obtain lock-step, synchronous behavior. However, deskewing logic for a pipelined memory operation can be expensive in area as well as power. In another example of a RAIM system, data tagging outside the memory buffer also solves the deskewing problem while keeping staging latches (area and power) minimized. In these synchronous memory subsystems that have a greater tolerance for synchronization alignment between channels, it is possible to support out-of-order processing and a degree of synchronization variation; however, performance can be reduced if the memory subsystem is allowed to deviate too far out of order or out of synchronization over a period of time.
Embodiments include a method, system, and computer program product for out-of-synchronization detection and out-of-order detection in a memory system. A system for out-of-synchronization and out-of-order detection in a memory system includes a plurality of channels, each providing communication with a memory buffer chip and a plurality of memory devices. A memory control unit is coupled to the plurality of channels. The memory control unit is configured to perform a method that includes receiving frames on two or more of the channels. The memory control unit identifies alignment logic input in each of the received frames and generates a summarized input to alignment logic for each of the channels of the received frames based on the alignment logic input. The memory control unit adjusts a timing alignment of the summarized input for each of the channels of the received frames based on a skew value per channel. Each of the timing adjusted summarized inputs is compared. Based on a mismatch between at least two of the timing adjusted summarized inputs, a miscompare signal is asserted by the memory control unit.
A computer implemented method for out-of-synchronization and out-of-order detection in a memory system includes receiving frames on two or more channels of the memory system at a memory control unit. The memory control unit identifies alignment logic input in each of the received frames and generates a summarized input to alignment logic for each of the channels of the received frames based on the alignment logic input. The memory control unit adjusts a timing alignment of the summarized input for each of the channels of the received frames based on a skew value per channel. Each of the timing adjusted summarized inputs is compared. Based on a mismatch between at least two of the timing adjusted summarized inputs, a miscompare signal is asserted by the memory control unit.
A computer program product for out-of-synchronization and out-of-order detection in a memory system is provided. The computer program product includes a tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method. The method includes receiving frames on two or more channels of the memory system at a memory control unit. The memory control unit identifies alignment logic input in each of the received frames and generates a summarized input to alignment logic for each of the channels of the received frames based on the alignment logic input. The memory control unit adjusts a timing alignment of the summarized input for each of the channels of the received frames based on a skew value per channel. Each of the timing adjusted summarized inputs is compared. Based on a mismatch between at least two of the timing adjusted summarized inputs, a miscompare signal is asserted by the memory control unit.
The subject matter which is regarded as embodiments is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The forgoing and other features, and advantages of the embodiments are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
Exemplary embodiments provide out-of-synchronization detection and out-of-order detection in a memory system. The memory system includes a processing subsystem that communicates synchronously with a memory subsystem in a nest domain. The memory subsystem also includes a memory domain that can be run synchronously or asynchronously relative to the nest domain. The memory system includes a memory controller that interfaces with the memory subsystem having multiple memory channels. The memory controller includes an out-of-sync/out-of-order detector to detect out-of-synchronization and out-of-order conditions between the memory channels while tolerating a degree of skew within the memory system. Upon detecting misalignment beyond a threshold amount, actions can be taken to re-align and synchronize the memory channels.
Each of the channels 110 is a synchronous channel which includes a downstream bus 114 and an upstream bus 116. Each downstream bus 114 of a given channel 110 may include a different number of lanes or links than a corresponding upstream bus 116. In the example of
The CP system 102 may also include a cache subsystem 118 that interfaces with the processor 104. A cache subsystem interface 122 of the CP system 102 provides a communication interface to the cache subsystem 118. The cache subsystem interface 122 may receive data from the memory subsystem 112 via the MCU 106 to store in the cache subsystem 118.
The MBAs 212a and 212b include memory control logic for managing accesses to the DRAM devices 204, as well as controlling timing, refresh, calibration, and the like. The MBAs 212a and 212b can be operated in parallel, such that an operation on DDR port 210a or 210b can be performed in parallel with an operation on DDR port 210c or 210d.
The memory buffer chip 202 also includes an interface 214 configured to communicate with a corresponding interface 216 of the MCU 106 via the channel 110. Synchronous communication is established between the interfaces 214 and 216. As such, a portion of the memory buffer chip 202 including a memory buffer unit (MBU) 218 operates in a nest domain 220 which is synchronous with the MCS 108 of the CP system 102. A boundary layer 222 divides the nest domain 220 from a memory domain 224. The MBAs 212a and 212b and the DDR ports 210a-210d, as well as the DRAM devices 204 are in the memory domain 224. A timing relationship between the nest domain 220 and the memory domain 224 is configurable, such that the memory domain 224 can operate asynchronously relative to the nest domain 220, or the memory domain 224 can operate synchronously relative to the nest domain 220. The boundary layer 222 is configurable to operate in a synchronous transfer mode and an asynchronous transfer mode between the nest and memory domains 220, 224. The memory buffer chip 202 may also include one or more multiple-input shift-registers (MISRs) 226, as further described herein. For example, the MBA 212a can include one or more MISR 226a, and the MBA 212b can include one or more MISR 226b. Other instances of MISRs 226 can be included elsewhere within the memory system 100. As a further example, one or more MISRs 226 can be positioned individually or in a hierarchy that spans the MBU 218 and MBAs 212a and 212b and/or in the MCU 106.
The boundary layer 222 is an asynchronous interface that permits different DIMMs 208 or DRAM devices 204 of varying frequencies to be installed into the memory domain 224 without the need to alter the frequency of the nest domain 220. This allows the CP system 102 to remain intact during memory installs or upgrades, thereby permitting greater flexibility in custom configurations. In the asynchronous transfer mode, a handshake protocol can be used to pass commands and data across the boundary layer 222 between the nest and memory domains 220, 224. In the synchronous transfer mode, timing of the memory domain 224 is phase adjusted to align with the nest domain 220 such that a periodic alignment of the nest and memory domains 220, 224 occurs at an alignment cycle in which commands and data can cross the boundary layer 222.
The nest domain 220 is mainly responsible for reconstructing and decoding the source synchronous channel packets, applying any necessary addressing translations, performing coherency actions, such as directory look-ups and cache accesses, and dispatching memory operations to the memory domain 224. The memory domain 224 may include queues, a scheduler, dynamic power management controls, hardware engines for calibrating the DDR ports 210a-210d, and maintenance, diagnostic, and test engines for discovery and management of correctable and uncorrectable errors. There may be other functions in the nest or memory domain. For instance, there may be a cache of embedded DRAM (eDRAM) memory with a corresponding directory. If the cache is created for some applications and other instances do not use it, there may be power savings by connecting a special array voltage (e.g., VCS) to ground. These functions may be incorporated within the MBU 218 or located elsewhere within the nest domain 220. The MBAs 212a and 212b within the memory domain 224 may also include logic to initiate autonomic memory operations for the DRAM devices 204, such as refresh and periodic calibration sequences in order to maintain proper data and signal integrity.
As in the example of
The memory subsystem 112b may also include power management logic 310 that provides a voltage source for a voltage rail 312. The voltage rail 312 is a local cache voltage rail to power a memory buffer cache 314. The memory buffer cache 314 may be part of the MBU 218. A power selector 316 can be used to determine whether the voltage rail 312 is sourced by the power management logic 310 or tied to ground 318. The voltage rail 312 may be tied to ground 318 when the memory buffer cache 314 is not used, thereby reducing power consumption. When the memory buffer cache 314 is used, the power selector 316 ties the voltage rail 312 to a voltage supply of the power management logic 310. Fencing and clock gating can also be used to better isolate voltage and clock domains.
As can be seen in reference to
As in the examples of
The PLL 402 is a memory controller PLL configured to provide a master clock 408 to the MCS 108 and the interfaces 216a-216n in the MCU 106 of the CP system 102. The PLL 404 is a nest domain PLL that is coupled to the MBU 218 and the interface 214 of the memory buffer chip 202a to provide a plurality of nest domain clocks 405. The PLL 406 is a memory domain PLL coupled the MBAs 212a and 212b and to the DDR ports 210a-210d to provide a plurality of memory domain clocks 407. The PLL 402 is driven by a reference clock 410 to establish the master clock 408. The PLL 404 has a reference clock 408 for synchronizing to the master clock 405 in the nest domain 220. The PLL 406 can use a separate reference clock 414 or an output 416 of the PLL 404 to provide a reference clock 418. The separate reference clock 414 operates independent of the PLL 404.
A mode selector 420 determines the source of the reference clock 418 based on an operating mode 422 to enable the memory domain 224 to run either asynchronous or synchronous relative to the nest domain 220. When the operating mode 422 is an asynchronous operating mode, the reference clock 418 is based on the reference clock 414 as a reference clock source such that the PLL 406 is driven by separate reference clock and 414. When the operating mode 422 is a synchronous operating mode, the reference clock 418 is based on the output 416 of an FSYNC block 492 which employs PLL 404 as a reference clock source for synchronous clock alignment. This ensures that the PLLs 404 and 406 have related clock sources based on the reference clock 408. Even though the PLLs 404 and 406 can be synchronized in the synchronous operating mode, the PLLs 404 and 406 may be configured to operate at different frequencies relative to each other. Additional frequency multiples and derivatives, such as double rate, half rate, quarter rate, etc., can be generated based on each of the multiplier and divider settings in each of the PLLs 402, 404, and 406. For example, the nest domain clocks 405 can include multiples of a first frequency of the PLL 404, while the memory domain clocks 407 can include multiples of a second frequency of the PLL 406.
In an asynchronous mode of operation each memory buffer chip 202a-202n is assigned to an independent channel 110. All data for an individual cache line may be self-contained within the DRAM devices 204 of
When implemented as a RAIM system, the memory buffer chips 202a-202n can be configured in the synchronous mode of operation. In a RAIM configuration, memory data is striped across multiple physical memory channels 110, e.g., five channels 110, which can act as the single virtual channel 111 of
To support and maintain synchronous operation, the MCU 106 can detect situations where one channel 110 becomes temporarily or permanently incapacitated, thereby resulting in a situation wherein the channel 110 is operating out of sync with respect to the other channels 110. In many cases the underlying situation is recoverable, such as intermittent transmission errors on one of the interfaces 216a-216n and/or interface 214 of one of more of the memory buffer chips 202a-202n. Communication on the channels 110 may utilize a robust cyclic redundancy code (CRC) on transmissions, where a detected CRC error triggers a recovery retransmission sequence. There are cases where the retransmission requires some intervention or delay between the detection and retransmission. A replay system including replay buffers for each of the channels can be used to support a recovery retransmission sequence for a faulty channel 110. Portions of the replay system may be suspended for a programmable period of time to ensure that source data to be stored in the replay buffer has been stored prior to initiating automated recovery. The period of time while replay is suspended can also be used to make adjustments to other subsystems, such as voltage controls, clocks, tuning logic, power controls, and the like, which may assist in preventing a recurrence of an error condition that led to the fault. Suspending replay may also remove the need for the MCU 106 to reissue a remaining portion of a store on the failing channel 110 and may increase the potential of success upon the replay.
Although the recovery retransmission sequence can eventually restore a faulty channel 110 to fully operational status, the overall memory subsystem 112 remains available during a recovery period. Tolerating a temporary out of sync condition allows memory operations to continue by using the remaining good (i.e., non-faulty) channels 110 until the recovery sequence is complete. For instance, if data has already started to transfer back to the cache subsystem 118 of
To support timing alignment issues that may otherwise be handled using deskewing logic, the MCU 106 and the memory buffer chip 202 may support the use of tags. Command completion and data destination routing information can be stored in a tag directory 424 which is accessed using a received tag. Mechanisms for error recovery, including retrying of read or write commands, may be implemented in the memory buffer chips 202 for each individual channel 110. Each command that is issued by the MCU 106 to the memory buffer chips 202 may be assigned a command tag in the MCU 106, and the assigned command tag sent with the command to the memory buffer chips 202 in the various channels 110. The various channels 110 send back response tags that comprise data tags or done tags. Data tags corresponding to the assigned command tag are returned from the buffer chip in each channel to correlate read data that is returned from the various channels 110 to an original read command. Done tags corresponding to the assigned command tag are also returned from the memory buffer chip 202 in each channel 110 to indicate read or write command completion.
The tag directory 424, also associated with tag tables which can include a data tag table and a done tag table, may be maintained in the MCU 106 to record and check the returned data and done tags. It is determined based on the tag tables when all of the currently functioning channels in communication with the MCU 106 return the tags corresponding to a particular command. For data tags corresponding to a read command, the read data is considered available for delivery to the cache subsystem 118 of
Timing and signal adjustments to support high-speed synchronous communications are also managed at the interface level for the channels 110.
In interface 216, commands and/or data are stored in a transmit first-in-first-out (FIFO) buffer 518 to transmit as frames 520. The frames 520 are serialized by serializer 522 and transmitted by the driver buffers 504 as groups 508 of serial data on the lanes 502 to interface 214. In interface 214, serial data received at receiver buffers 506 are deserialized by deserializer 524 and captured in a receive FIFO buffer 526, where received frames 528 can be analyzed and reconstructed. When sending data from interface 214 back to interface 216, frames 530 to be transmitted are stored in a transmit FIFO buffer 532 of the interface 214, serialized by serializer 534, and transmitted by the driver buffers 514 as groups 510 of serial data on the lanes 512 to interface 216. In interface 216, serial data received at receiver buffers 516 are deserialized by deserializer 536 and captured in a receive FIFO buffer 538, where received frames 540 can be analyzed and reconstructed.
The interfaces 214 and 216 may each include respective instances of training logic 544 and 546 to configure the interfaces 214 and 216. The training logic 544 and 546 train both the downstream bus 114 and the upstream bus 116 to properly align a source synchronous clock to transmissions on the lanes 502 and 512. The training logic 544 and 546 also establish a sufficient data eye to ensure successful data capture. Further details are described in reference to process 600 of
At block 602, the lanes 502 and 512 of each channel 110 are initialized and calibrated. The training logic 544 and 546 can perform impedance calibration on the driver buffers 504 and 514. The training logic 544 and 546 may also perform static offset calibration of the receiver buffers 506 and 516 and/or sampling latches (not depicted) followed by a wire test to detect permanent defects in the transmission media of channel 110. Wire testing may be performed by sending a slow pattern that checks wire continuity of both sides of the clock and data lane differential pairs for the lanes 502 and 512. The wire testing may include driving a simple repeating pattern to set a phase rotator sampling point, synchronize the serializer 522 with the deserializer 524 and the serializer 534 with the deserializer 536, and perform lane-based deskewing. Data eye optimization may also be performed by sending a more complex training pattern that also acts as a functional data scrambling pattern.
Training logic 544 and 546 can use complex training patterns to optimize various parameters such as a final receiver offset, a final receiver gain, peaking amplitude, decision feedback equalization, final phase rotator adjustment, final offset calibration, scrambler and descrambler synchronization, and load-to-unload delay adjustments for FIFOs 518, 526, 532, and 538.
Upon detecting any non-functional lanes in the lanes 502 and 512, a dynamic sparing process is invoked to replace the non-functional/broken lane with an available spare lane of the corresponding downstream bus 114 or upstream bus 116. A final adjustment may be made to read data FIFO unload pointers of the receive FIFO buffers 526 and 538 to ensure sufficient timing margin.
At block 604, a frame transmission protocol is established based on a calculated frame round trip latency. Once a channel 110 is capable of reliably transmitting frames in both directions, a reference starting point is established for decoding frames. To establish synchronization with a common reference between the nest clock 405 and the master clock 408, a frame lock sequence is performed by the training logic 546 and 544. The training logic 546 may initiate the frame lock sequence by sending a frame including a fixed pattern, such as all ones, to the training logic 544 on the downstream bus 114. The training logic 544 locks on to the fixed pattern frame received on the downstream bus 114. The training logic 544 then sends the fixed pattern frame to the training logic 546 on the upstream bus 116. The training logic 546 locks on to the fixed pattern frame received on the upstream bus 116. The training logic 546 and 544 continuously generate the frame beats. Upon completion of the frame lock sequence, the detected frame start reference point is used as an alignment marker for all subsequent internal clock domains.
A positive acknowledgement frame protocol may be used where the training logic 544 and 546 acknowledge receipt of every frame back to the transmitting side. This can be accomplished through the use of sequential transaction identifiers assigned to every transmitted frame. In order for the sending side to accurately predict the returning acknowledgment, another training sequence referred to as frame round trip latency (FRTL) can be performed to account for the propagation delay in the transmission medium of the channel 110.
In an exemplary embodiment, the training logic 546 issues a null packet downstream and starts a downstream frame timer. The training logic 544 responds with an upstream acknowledge frame and simultaneously starts an upstream round-trip timer. The training logic 546 sets a downstream round-trip latency value, when the first upstream acknowledge frame is received from the training logic 544. The training logic 546 sends a downstream acknowledge frame on the downstream bus 114 in response to the upstream acknowledge frame from the training logic 544. The training logic 544 sets an upstream round-trip delay value when the downstream acknowledge frame is detected. The training logic 544 issues a second upstream acknowledge frame to close the loop. At this time the training logic 544 goes into a channel interlock state. The training logic 544 starts to issue idle frames until a positive acknowledgement is received for the first idle frame transmitted by the training logic 544. The training logic 546 detects the second upstream acknowledge frame and enters into a channel interlock state. The training logic 546 starts to issue idle frames until a positive acknowledgement is received for the first idle frame transmitted by the training logic 546. Upon receipt of the positive acknowledgement, the training logic 546 completes channel interlock and normal traffic is allowed to flow through the channel 110.
At block 606, a common synchronization reference is established for multiple memory buffer chips 202a-202n. In the case of a fully synchronous multi-channel structure, a relative synchronization point is established to ensure that operations initiated from the CP system 102 are executed in the same manner on the memory buffer chips 202a-202n, even when the memory buffer chips 202a-202n are also generating their own autonomous refresh and calibration operations. Synchronization can be accomplished by locking into a fixed frequency ratio between the nest and memory domains 220 and 224 within each memory buffer chip 202. In exemplary embodiments, the PLLs 404 and 406 from both the nest and memory domains 220 and 224 are interlocked such that they have a fixed repeating relationship. This ensures both domains have a same-edge aligned boundary (e.g., rising edge aligned) at repeated intervals, which is also aligned to underlying clocks used for the high speed source synchronous interface 214 as well as frame decode and execution logic of the MBU 218. A common rising edge across all the underlying clock domains is referred to as the alignment or “golden” reference cycle.
Multi-channel operational synchronization is achieved by using the alignment reference cycle to govern all execution and arbitration decisions within the memory buffer chips 202a-202n. Since all of the memory buffer chips 202a-202n in the same virtual channel 111 have the same relative alignment reference cycle, all of their queues and arbiters (not depicted) remain logically in lock step. This results in the same order of operations across all of the channels 110. Even though the channels 110 can have inherent physical skew, and each memory buffer chip 202 performs a given operation at different absolute times with respect to the other memory buffer chips 202, the common alignment reference cycle provides an opportunity for channel operations to transit the boundary layer 222 between the nest and memory domains 220 and 224 with guaranteed timing closure and equivalent arbitration among internally generated refresh and calibration operations.
As previously described in reference to
At block 702, the nest domain clocks 405 are aligned with a frame start signal from a previous frame lock of block 604. The nest domain 220 can use multiple clock frequencies for the nest domain clocks 405, for example, to save power. A frame start may be defined using a higher speed clock, and as such, the possibility exists that the frame start could fall in a later phase of a slower-speed nest domain clock 405. This would create a situation where frame decoding would not be performed on an alignment cycle. In order to avoid this, the frame start signal may be delayed by one or more cycles, if necessary, such that it always aligns with the slower-speed nest domain clock 405, thereby edge aligning the frame start with the nest domain clocks 405. Clock alignment for the nest domain clocks 405 can be managed by the PLL 404 and/or additional circuitry (not depicted). At block 704, the memory domain clocks 407 are turned off and the memory domain PLL 406 is placed into bypass mode.
At block 706, the MCS 108 issues a super synchronize (“SuperSync”) command using a normal frame protocol to all memory buffer chips 202a-202n. The MCS 108 may employ a modulo counter matching an established frequency ratio such that it will only issue any type of synchronization command at a fixed period. This establishes the master reference point for the entire memory subsystem 112 from the MCS 108 perspective. Even though the SuperSync command can arrive at the memory buffer chips 202a-202n at different absolute times, each memory buffer chip 202 can use a nest cycle upon which this command is decoded as an internal alignment cycle. Since skew among the memory buffer chips 202a-202n is fixed, the alignment cycle on each of the memory buffer chips 202a-202n will have the same fixed skew. This skew translates into a fixed operational skew under error free conditions.
At block 708, sync logic of the memory buffer chip 202, which may be part of the mode selector 420, uses the SuperSync decode as a reference to trigger realignment of the reference clock 418 that drives the memory domain PLL 406. The SuperSync decode is translated into a one cycle pulse signal 494, synchronous with the nest domain clock 405 that resets to zero a modulo counter 496 in the FSYNC block 492. The period of this counter 496 within the FSYNC block 492 is set to be the least common multiple of all memory and nest clock frequencies with the rising edge marking the sync-point corresponding to the reference point previously established by the MCS 108. The rising edge of FSYNC clock 416 becomes the reference clock of PLL 406 to create the memory domain clocks. By bringing the lower-frequency output of PLL 406 back into the external feedback port, the nest clock 405 and memory clock 407 all have a common clock edge aligned to the master reference point. Thus, the FSYNC block 492 provides synchronous clock alignment logic.
At block 710, the memory domain PLL 406 is taken out of bypass mode in order to lock into the new reference clock 418 based on the output 416 of the PLL 404 rather than reference clock 414. At block 712, the memory domain clocks 407 are turned back on. The memory domain clocks 407 are now edge aligned to the same alignment reference cycle as the nest domain clocks 405.
At block 714, a regular subsequent sync command is sent by the MCS 108 on the alignment cycle. This sync command may be used to reset the various counters, timers and MISRs 226 that govern internal memory operation command generation, execution and arbitration. By performing a reset on the alignment cycle, all of the memory buffer chips 202a-202n start their respective internal timers and counters with the same logical reference point. If an arbiter on one memory buffer chip 202 identifies a request from both a processor initiated memory operation and an internally initiated command on a particular alignment cycle, the corresponding arbiter on the remaining memory buffer chips 202 will also see the same requests on the same relative alignment cycle. Thus, all memory buffer chips 202a-202n will make the same arbitration decisions and maintain the same order of operations.
Embodiments may provide internally generated commands at memory buffer chip 202 to include DRAM refresh commands, DDR calibration operations, dynamic power management, error recovery, memory diagnostics, and the like. Anytime one of these operations is needed, it must cross into the nest domain 220 and go through the same arbitration as synchronous operations initiated by the MCS 108. Arbitration is performed on the golden cycle to ensure all the memory buffer chips 202 observe the same arbitration queues and generate the same result. The result is dispatched across boundary layer 222 on the golden cycle which ensures timing and process variations in each memory buffer chip 202 is nullified.
Under normal error free conditions, the order of operations will be maintained across all of the memory buffer chips 202a-202n. However, there are situations where one channel 110 can get out of sync with the other channels 110. One such occurrence is the presence of intermittent transmission errors on one or more of the interfaces 214 and 216. Exemplary embodiments include a hardware based recovery mechanism where all frames transmitted on a channel 110 are kept in a replay buffer for a prescribed period of time. This time covers a window long enough to guarantee that the frame has arrived at the receiving side, has been checked for errors, and a positive acknowledgement indicating error free transmission has been returned to the sender. Once this is confirmed, the frame is retired from the replay buffer. However, in the case of an erroneous transmission, the frame is automatically retransmitted, or replayed, along with a number of subsequent frames in case the error was a one-time event. In many cases, the replay is sufficient and normal operation can resume. In certain cases, the transmission medium of the channel 110 has become corrupted to the point that a dynamic repair is instituted to replace a defective lane with a spare lane from lanes 502 or 512. Upon completion of the repair procedure, the replay of the original frames is sent and again normal operation can resume.
Another less common occurrence can be an on-chip disturbance manifesting as a latch upset which results in an internal error within the memory buffer chip 202. This can lead to a situation where one memory buffer chip 202 executes its operations differently from the remaining memory buffer chips 202. Although the memory system 100 continues to operate correctly, there can be significant performance degradation if the channels 110 do not operate in step with each other. In exemplary embodiments, the MISRs 226 monitor for and detect such a situation. The MISRs 226 receive inputs derived from key timers and counters that govern the synchronous operation of the memory buffer chip 202, such as refresh starts, DDR calibration timers, power throttling, and the like. The inputs to the MISRs 226 are received as a combination of bits that collectively form a signature. One or more of the bits of the MISRs 226 are continually transmitted as part of an upstream frame payload to the MCU 106, which monitors the bits received from the MISRs 226 of the memory buffer chips 202a-202n. The presence of physical skew between the channels 110 results in the bits from the MISRs 226 arriving at different absolute times across the channels 110. Therefore, a learning process is incorporated to calibrate checking of the MISRs 226 to the wire delays in the channels 110.
In exemplary embodiments, MISR detection in the MCU 106 incorporates two distinct aspects in order to monitor the synchronicity of the channels 110. First, the MCU 106 monitors the MISR bits received on the upstream bus 116 from each of the memory buffer chips 202a-202n and any difference seen in the MISR bit stream indicates an out-of-sync condition. Although this does not pose any risk of a data integrity issue, it can negatively impact performance, as the MCU 106 may incur additional latency waiting for an entire cache line access to complete across the channels 110. Another aspect is monitoring transaction sequence identifiers (i.e., tags) associated with each memory operation and comparing associated “data” tags or “done” tags as the operations complete. Once again, skew of the channels 110 is taken into account in order to perform an accurate comparison. In one example, this skew can manifest in as many as 30 cycles of difference between the fastest and slowest channel 110. If the tags are 7-bits wide, with five channels 110, and a maximum 30-cycle difference across channels 110, this would typically require 5×7×30=1050 latches to perform a simplistic compare. There may be some cases that equate to about 40 bit-times which is about 4 cycles of deskew after aligning to a frame. To further reduce the number of latches, a MISR can be incorporated within the MCU 106 to encode the tag into a bit stream, which is then pipelined to eliminate the skew. By comparing the output of the MISR of the MCU 106 across all of the channels 110, a detected difference indicates an out-of-order processing condition.
In either of these situations, the afflicted channel 110 can at least temporarily operate out of sync or out of order with respect to the other channels 110. Continuous availability of the memory subsystem 112 may be provided through various recovery and self-healing mechanisms. Data tags can be used such that in the event of an out-of-order or out-of-sync condition, the MCU 106 continues to function. Each read command may include an associated data tag that allows the MCS 108 to handle data transfers received from different channels 110 at different times or even in different order. This allows proper functioning even in situations when the channels 110 go out of sync.
For out-of-sync conditions, a group of hierarchical MISRs 226 can be used accumulate a signature for any sync-related event. Examples of sync-related events include a memory refresh start, a periodic driver (ZQ) calibration start, periodic memory calibration start, power management window start, and other events that run off a synchronized counter. One or more bits from calibration timers, refresh timers, and the like can serve as inputs to the MISRs 226 to provide a time varying signature which may assist in verifying cross-channel synchronization at the MCU 106. Hierarchical MISRs 226 can be inserted wherever there is a need for speed matching of data. For example, speed matching may be needed between MBA 212a and the MBU 218, between the MBA 212b and the MBU 218, between the MBU 218 and the upstream bus 116, and between the interfaces 216a-216n and the MCS 108.
For out-of-order conditions, staging each of the tags received in frames from each channel 110 can be used to deskew the wire delays and compare them. A MISR per channel 110 can be used to create a signature bit stream from the tags received at the MCU 106 and perform tag/signature-based deskewing rather than hardware latch-based deskewing. Based on the previous example of 7-bit wide tags, with five channels 110, and a maximum 30-cycle difference across channels 110, the use of MISRs reduces the 1050 latches to about 7×5+30×5=185 latches, plus the additional support latches.
To minimize performance impacts, the MCS 108 tries to keep all channels 110 in lockstep, which implies that all commands are executed in the same order. When read commands are executed, an associated data tag is used to determine which data correspond to which command. This approach also allows the commands to be reordered based on resource availability or timing dependencies and to get better performance. Commands may be reordered while keeping all channels 110 in lockstep such that the reordering is the same across different channels 110. In this case, tags can be used to match the data to the requester of the data from memory regardless of the fact that the command order changed while the data request was processed.
Marking a channel 110 in error may be performed when transfers have already started and to wait for recovery for cases where transfers have not yet occurred. Data blocks from the memory subsystem 112 can be delivered to the cache subsystem interface 122 of
If the data error is detected before the block data transfer has begun to the cache subsystem 118, the transfer is stalled until all frames have been checked for any data errors. Assuming errors are infrequent, the performance impact is minimal. This reduces the use of channel redundancy and may result in avoidance of possible uncorrectable errors in the presence of previously existing errors in the DRAM devices 204.
The MCU 106 may also include configurable delay functions on a per-command type or destination basis to delay data block transfer to upstream elements, such as caches, until data error detection is completed for the block. Command or destination information is available for making such selections as inputs to the tag directory. This can selectively increase system reliability and simplify error handling, while minimizing performance impacts.
To support other synchronization issues, the MCU 106 can re-establish synchronization across multiple channels 110 in the event of a channel failure without having control of an underlying recovery mechanism used on the failed channel. A programmable quiesce sequence incrementally attempts to restore channel synchronization by stopping stores and other downstream commands over a programmable time interval. The quiesce sequence may wait for completion indications from the memory buffer chips 202a-202n and inject synchronization commands across all channels 110 to reset underlying counters, timers, MISRs 226, and other time-sensitive circuitry to the alignment reference cycle. If a failed channel 110 remains out of synchronization, the quiesce sequence can be retried under programmatic control. In many circumstances, the underlying root cause of the disturbance can be self healed, thereby resulting in the previously failed channel 110 being reactivated and resynchronized with the remaining channels 110. Under extreme error conditions the quiesce and recovery sequence fails to restore the failed channel 110, and the failed channel 110 is permanently taken off line. In a RAIM architecture that includes five channels 110, the failure of one channel 110 permits the remaining four channels 110 to operate with a reduced level of protection.
A repeating pattern of clock cycles is depicted in
In
The MCU 106 includes a respective buffer control block 908A-E for each channel 110. These buffer control blocks 908A-E hold data that are returned on read data buses 906 from memory control unit interfaces 905 until the data may be sent to the cache subsystem 118 of
Memory control unit interfaces 905 send the tag information via tag buses 907 to both the buffer control blocks 908A-E and channel control block 909. Channel control block 909 tracks received tags using the data tag table 910 and done tag table 911, each of which are shown in further detail in
In the example of
Each QL of data and check bytes (or corresponding channel redundancy information in the case of the fifth channel 110) is contained in a frame that is transmitted across a channel 110. This frame contains CRC information to detect errors with the data in the frame along with attributes contained in the frame and associated with the data. One of these attributes is a data tag which is used to match incoming frame data with previously sent fetch commands to memory. Another attribute is a done tag which is used to indicate command completion in a memory buffer chip 202. Each fetch command is directly associated with a 256-byte data block fetch through an assigned data tag. In this example, a 256-byte fetch requires four QL's or frames per channel. These four frames are sent consecutively by each channel 110 assuming no errors are present.
Data arrives independently from each of the five channels 110 and are matched up with data from other channels 110 using the previously mentioned data tag. The channel control block 909 tracks the received tags using the data tag table 910 and the done tag table 911. The channel control block 909 sets a bit in the data tag table 910 based on receiving a data tag, and sets a bit in the done tag table 911 based on receiving a done tag. Embodiments of data tag table 910 and done tag table 911 are shown in further detail in
A controller 914, which may be part of the channel control block 909, controls the transfer of data through the buffer control blocks 908A-E to the data buffer read control and dataflow logic 912. The data buffer read control and dataflow logic 912 can read the buffer control blocks 908A-E asynchronously relative to the memory control unit interfaces 905 populating the buffer control blocks 908A-E with data. The data buffer read control and dataflow logic 912 operates in the system domain 916, while the memory control unit interfaces 905 operate in the memory controller nest domain 918, where the system domain 916 and the memory controller nest domain 918 are asynchronous clock domains that may have a variable frequency relationship between domains. Accordingly, the buffer control blocks 908A-E form an asynchronous boundary layer between the system domain 916 and the memory controller nest domain 918. In an exemplary embodiment, the controller 914 is in the memory controller nest domain 918 and sends signals to the data buffer read control and dataflow logic 912 via an asynchronous interface 920.
The MCU 106 also includes an out-of-sync/out-of-order (OOS/OOO) detector 922 that may be incorporated in the memory control unit interfaces 905. The OOS/OOO detector 922 compares values received across multiple channels 110 for out-of-synchronization conditions and/or out-of-order conditions. Data tags and done tags received in frames on the channels 110 can be used in combination with data from the MISRs 226 of
The MISR alignment logic 1002 may include a combination of latches and switches to adjust the relative skew of the tag MISR inputs 1006 and/or the high-level memory buffer chip MISR inputs 1008 based on the skew values 1004 per channel 110. The alignment may be relative to a frame boundary or an alignment reference cycle. In an exemplary embodiment, latency counters 1010a, 1010b, 1010c, 1010d, and 1010e can be used for determining skew values 1004. In another exemplary embodiment, rather than including 30 or more latches per channel in the MISR alignment logic 1002, a multi-cycle skew can be handled primarily by programming an associated latency counter 1010 to the desired value to assist in the alignment.
A 5-way compare 1012 is used to check for misalignment between the skew adjusted inputs to the MISR alignment logic 1002. To avoid including failed channels in the 5-way compare 1012, channel marks 1014 are used to mask failed channels and remove them from inclusion in the comparison. Based on a mismatch detected by the 5-way compare 1012, a miscompare signal 1016 is asserted and output from the 5-way compare 1012, which can trigger a quiesce sequence as part of resynchronization. The miscompare signal 1016 can be used to isolate an out-of-sync channel from the remaining channels 110. Out-of-order detection can be used to trigger a degraded operation mode and/or to initiate a recovery action. In an embodiment, a miscompare based on the tag MISR inputs 1006 indicates an out-of-order condition, and a miscompare based on the high-level memory buffer chip MISR inputs 1008 indicates an out-of-sync condition.
In an exemplary embodiment, the tag MISR inputs 1006 each receive an output of a tag MISR 1018 per channel 110. The example of
Similar to the tag MISR 1018, the high-level memory buffer chip MISR inputs 1008 each receive an output of a high-level memory buffer chip MISR 1036 per channel 110. The example of
In the example of
The MISR alignment logic 1102 may include a combination of latches and switches to alter the relative skew of the tag inputs 1106 and/or the memory buffer chip MISR inputs 1108 based on the skew value sets 1104. The alignment may be relative to a frame boundary or an alignment reference cycle. To support a larger number of cycles of skew and reduce the number of latches in the alignment logic 1102, latency counters 1110a, 1110b, 1110c, 1110d, 1110e, 1110f, 1110g, 1110h, 1110i, and 1110j can be used for larger skew value sets 1104. For example, rather than including 30 or more latches per channel in the alignment logic 1102, a multi-cycle skew can be handled primarily by programming an associated latency counter 1110 to the desired value. In an embodiment, latency counters 1110a-1110e are associated with the tag inputs 1106, and latency counters 1110f-1110j are associated with the memory buffer chip MISR inputs 1108.
A pair of 5-way compare blocks 1112a and 1112b can be used to check for misalignment between the skew adjusted inputs to the alignment logic 1102. In an embodiment, the 5-way compare block 1112a is associated with the latency counters 1110a-1110e and tag inputs 1106, and the 5-way compare block 1112b is associated with the latency counters 1110f-1110j and the memory buffer chip MISR inputs 1108. To avoid including failed channels in the 5-way compare 1112a and the 5-way compare 1112b, channel marks 1114 are used to mask failed channels and remove them from inclusion in the comparisons. Based on a mismatch detected by the 5-way compare 1112a for data tags or done tags, a miscompare signal 1116a is asserted and output from the 5-way compare 1112a, which can trigger a quiesce sequence as part of resynchronization. Similarly, based on a mismatch detected by the 5-way compare 1112b for memory buffer chip MISRs, a miscompare signal 1116b is asserted and output from the 5-way compare 1112b, which can trigger a quiesce sequence as part of resynchronization.
In an exemplary embodiment, the tag inputs 1106 each receive an output of a bit-wise AND 1118 per channel 110. A mask 1130 in combination with the bit-wise AND 1118 can be used to select specific fields from tag inputs 1128 to form the tag inputs 1106. The tag inputs 1128 can include a combination of a data tag and a done tag received in a frame from a memory buffer chip 202 of
The memory buffer chip MISR inputs 1108 each receive an output of a bit-wise AND 1136 per channel 110. The example of
The MISR inputs 1312 enter a bit-wise AND 1304 which performs the AND, bit-by-bit of the MISR inputs 1312 and a MISR mask 1314 for the purpose of being able to block certain MISR inputs 1312 from entering the MISR register 1306. In order to maintain a partial history of the MISR register 1306, the output of the MISR register 1306 is shifted and enters a bit-wise XOR 1302 with the output of the bit-wise AND 1304. The purpose of the performing an XOR is so that history from the MISR register 1306 and new inputs from the MISR inputs 1312 can be combined and compressed as independent events. An XOR, unlike an AND or an OR function, has an output that may observe a change whenever any of its inputs are changed.
To further illustrate, consider an event that is expected on a memory buffer chip 202. Normally, assume that a refresh pulse is supposed to appear on cycle x. If a refresh pulse/timer is delayed by one cycle, the refresh pulse will not show up on cycle x but will instead show up on cycle x+1. Thus, this appears as two incorrect events, one on cycle x, where the refresh is missing, and another on cycle x+1, where an unexpected refresh is observed. In order to keep the x and the x+1 cycle events from nullifying each other, the MISR logic 1300 performs a shift by one bit each cycle. This is done by taking all but the most significant bit of the MISR register 1306, shifting it left by one bit, and feeding the shifted bus into the bitwise XOR 1302. The least significant bit 1308 can shift in a zero. In an exemplary embodiment, a feedback of bit 0 of the MISR register 1306 can be fed back to the least significant bit 1308, thus accumulating a compressed history signature indefinitely. This feedback option can be dynamically programmed as needed.
For a refresh example, assume a normal memory buffer chip 202 that is running as designed in sync. On the first cycle x, the MISR inputs 1312, bit 2 should be a 1. Assuming the previous state of the MISR register 1306 is zeros, this means that after the first cycle, the MISR register 1306 will have a value of “00100000000” (0:10, zero being on the left). On the second cycle, the refresh timer is zero again (since it is typically a one-cycle pulse). When the MISR logic 1300 accumulates another cycle of history (during x+1), the ‘1’ that was in bit 2 of the MISR register 1306 shifts over to bit 1 while the new MISR input 1312 bit 2, having a new value of “0” is XORed with bit 3 of the MISR register 1306, having a value of “0” to give a new MISR register value of “01000000000” after it receives a clock at the end of cycle x+1.
For a memory buffer chip 202 that is not in sync, assume the refresh counter came up a cycle later. Following the same example but with the refresh timer delayed by one cycle, the MISR register 1306 will have a value of “00000000000” at the end of cycle x and will have a value of “00100000000” at the end of cycle x+1.
At block 1406, the MCU 106 generates a summarized input to alignment logic, such as MISR alignment logic 1002 of
At block 1408, the MCU 106 adjusts timing alignment of the summarized input for each of the channels 110 of the received frames based on a skew value per channel 110. For example, the MISR alignment logic 1002 can adjust relative timing of the tag MISR inputs 1006 and the high-level memory buffer chip MISR inputs 1008 based on the skew values 1004 per channel 110. In the embodiment of
At block 1410, the MCU 106 compares each of the timing adjusted summarized inputs. The comparing may be performed by the 5-way compare 1012 of
At block 1412, based on a mismatch between at least two of the timing adjusted summarized inputs, a miscompare signal 1016, 1116a, or 1116b is asserted by the MCU 106. The miscompare signals 1016, 1116a, 1116b can be used to take further actions for quiescing and resynchronizing one or more of the channels 110 or running in an impaired operating state. Where the mismatch of at least two of the timing adjusted summarized inputs is based on one or more of the data tag and the done tag, the mismatch may be detected as an out-of-order condition. Where the mismatch of at least two of the timing adjusted summarized inputs is based on the MISRs 226 from the memory buffer chips 202 of
As will be appreciated by one skilled in the art, one or more aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, one or more aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system”. Furthermore, one or more aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
Referring now to
Program code, when created and stored on a tangible medium (including but not limited to electronic memory modules (RAM), flash memory, Compact Discs (CDs), DVDs, Magnetic Tape and the like is often referred to as a “computer program product”. The computer program product medium is typically readable by a processing circuit preferably in a computer system for execution by the processing circuit. Such program code may be created using a compiler or assembler for example, to assemble instructions, that, when executed perform aspects of the invention.
Technical effects and benefits include detecting misalignment between synchronized memory channels in a memory subsystem. Misalignment can be an out-of-synchronization condition and/or an out-of-order condition. Misalignment checks can be performed on a frame basis by extracting and summarizing various received values as alignment logic inputs. Skew adjustments can be made to the summarized inputs based on skew values per channel that may be determined during system initialization and training. Failed channels can be masked from the comparison process.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of embodiments have been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the embodiments. The embodiments were chosen and described in order to best explain the principles and the practical application, and to enable others of ordinary skill in the art to understand the embodiments with various modifications as are suited to the particular use contemplated.
Computer program code for carrying out operations for aspects of the embodiments may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of embodiments are described above with reference to flowchart illustrations and/or schematic diagrams of methods, apparatus (systems) and computer program products according to embodiments. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
This application is a Continuation of legally related U.S. application Ser. No. 15/073,699 filed Mar. 18, 2016, which is a Continuation of U.S. patent application Ser. No. 13/835,485, filed on Mar. 15, 2013, and issued as U.S. Pat. No. 9,430,418, the entire contents of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6771670 | Pfahler | Aug 2004 | B1 |
| 8041990 | O'Connor et al. | Oct 2011 | B2 |
| 8793544 | Johnson et al. | Jul 2014 | B2 |
| 20120166894 | Jang | Jun 2012 | A1 |
| 20120198309 | Alves | Aug 2012 | A1 |
| 20120284576 | Housty | Nov 2012 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20160371159 A1 | Dec 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15073699 | Mar 2016 | US |
| Child | 15262111 | US | |
| Parent | 13835485 | Mar 2013 | US |
| Child | 15073699 | US |
