Expandable asymmetric-channel memory system

Abstract

An expandable memory system that enables a fixed signaling bandwidth to be configurably re-allocated among dedicated memory channels. Memory channels having progressively reduced widths are dedicated to respective memory sockets, thus enabling point-to-point signaling with respect to each memory socket without signal-compromising traversal of unloaded sockets or costly replication of a full-width memory channel for each socket.

Description

TECHNICAL FIELD

The present invention relates generally to the field of data processing and more particularly to high-bandwidth memory systems.

BACKGROUND

Modular memories are commonly used in computers and other electronic systems that may require after-market storage-capacity expansion. For example, multiple memory-module sockets may be coupled to a memory controller via a shared memory channel, with each socket being coupled to the memory channel via a stub path. Memory modules are initially installed in fewer than all the sockets so that one or more additional modules may be inserted later to increase (expand) the system storage capacity. Unfortunately, the stub path coupling each socket to the shared memory channel may present a substantial source of undesired noise (especially signal reflections), and thus limit signaling rates and therefore peak memory bandwidth.

Although separate, replicated memory channels could be coupled point-to-point between the controller and respective memory-module sockets, that approach multiplies the number of controller interface pins and external signal lines by the number of memory-module sockets, driving up cost and limiting the number of expansion sockets that can be supported.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 illustrates an embodiment of a modular, expandable memory system having asymmetric memory channels coupled between a memory interface device and four memory sockets;

FIG. 2 illustrates an embodiment of a generalized input/output (I/O) steering circuit that may be used to support switchable interconnections between an n-bit wide internal data path and a generalized number of module sockets;

FIGS. 3A and 3B illustrate a more detailed embodiment of an I/O steering circuit that switchably couples the lanes of an internal data path to asymmetric memory channels of a four-socket memory system in response to a switch-control value;

FIG. 4 illustrates various alternative dispositions (or locations) of steering circuitry within a memory system;

FIG. 5 illustrates exemplary data width adjustments performed to conform the data interface of a memory module to the memory channel width in an asymmetric-channel memory system; and

FIG. 6 illustrates an exemplary sequence of operations that may be carried out within an asymmetric-channel memory system to determine the socket loading configuration and program corresponding configuration values within a memory interface IC and, if present, programmable-width memory devices.

DETAILED DESCRIPTION

An expandable memory system and components thereof that enable signaling bandwidth to be configurably re-allocated among dedicated memory channels are disclosed in various embodiments. In one embodiment, for example, memory channels having progressively reduced widths are dedicated to respective memory sockets, thus enabling point-to-point signaling with respect to each memory socket without signal-compromising traversal of unloaded sockets or costly replication of a (full-width) memory channel for each socket. When one or more memory modules are installed in previously unoccupied sockets (i.e., to expand the system storage capacity), a fraction of the signaling bandwidth allocated to the previously loaded socket(s) is reallocated to the newly loaded (occupied) sockets. Thus, the net signaling bandwidth may remain fixed as new memory modules are added, but the bandwidth share allocated to each memory module is diluted (reduced) to account for the increased memory module count. Further, recognizing that the bandwidth share drops progressively as the module population grows, and by establishing a predefined socket loading order, progressively narrower memory channels (i.e., signaling paths for transfer of read and write data, at least, and possibly other signals) may be coupled to later-loaded sockets.

As an example, because the bandwidth share allocated to each socket (assuming an even distribution) is halved for each doubling of the memory module population, an expansion from a single module population to a two-module population may be effected by splitting the net signaling bandwidth (B) allocated to the first-loaded memory socket (i.e., the “base socket”) in two, retaining half of the signaling bandwidth (B/2) for the base socket and re-directing the other half of the signaling bandwidth to the first expansion socket. Because the net signaling bandwidth corresponds to a fixed number of signaling links, L, (i.e., L=B/R, where R is the peak signaling rate of each link), it follows that the first expansion socket need only be coupled to half as many signaling links as the base socket. Thus, memory channels having different numbers of constituent signaling links (i.e., asymmetric memory channels) are coupled to the two memory sockets—L signaling links to the base socket and L/2 signaling links to the first expansion socket. By this arrangement, when a memory module is installed in the base socket only (i.e., single module configuration), the L links that constitute the full-width memory channel to the base socket are all used to convey information-bearing signals and are thus referred to herein as “live” links, while the L/2 links that constitute the half-width memory channel to the first expansion socket remain unused and are said to be “dormant.”

By contrast, when a memory module is installed in the first expansion socket (expanding the memory system capacity), half of the available signaling bandwidth is reallocated from the base socket to the first expansion socket to establish effective half-width channels to each socket. More specifically, the L/2 links that form the half-width memory channel to the expansion socket (i.e., the “half-width expansion channel”) are made live, and half of the L links that constitute the full-width memory channel to the base socket (the “base channel”) are rendered dormant, thereby maintaining a fixed number of live links, L (and therefore a fixed signaling bandwidth), but with increased storage capacity. As a matter of terminology, the half-width expansion channel is said to be activated (i.e., constituent links transitioned from dormant to live status), and half of the base channel deactivated, with the remaining L/2 live links of the base channel forming a half-width “logical channel.”

FIG. 1 illustrates an embodiment of a modular, expandable memory system 200 having asymmetric memory channels 211₀-211₃ coupled between a memory interface IC (integrated circuit) 201 and four memory sockets 215. In the example shown and embodiments that follow, each of the memory sockets is assumed to be a memory-module socket (or “module socket”) that enables removable insertion of a memory module such as a single or dual in-line memory module (SIMM or DIMM); a small form-factor circuit board substrate having multiple integrated circuit memory devices mounted to one face or both faces and contacts that mate to counterpart contacts within the module socket. In more general embodiments, however, a “memory socket” or “socket” may be any structure that permits post-production installation (and/or removal for replacement or repair purposes) of memory modules or any other socket-able memory element including, without limitation, a memory module having virtually any practicable form factor, an integrated circuit memory package that encapsulates or otherwise includes one or more integrated circuit memory dice, or even a bare integrated circuit die. Also, the expression “memory interface IC” refers to any integrated circuit component or device having an interface that enables communication of memory control commands (or requests) and corresponding transfer of read and write data with respect to one or more memory modules and/or memory devices. Examples of a memory interface IC include, without limitation, a memory controller IC (which may be an IC die dedicated to a memory control function, or an IC such as a microprocessor or application-specific integrated circuit (ASIC) having memory controller circuitry integrated onto the IC die), a memory buffer IC (sometimes referred to as a “buffer-on-board” or “BoB”) to be coupled between a memory controller IC and one or more memory modules, or a system on chip (SoC) that incorporates multiple functional blocks along with a memory control function, and so forth.

In the embodiment of FIG. 1, the memory interface IC 201 includes core logic 203 to carry out the core function(s) of the memory interface IC (e.g., at least a memory control function in a memory controller IC, and at least memory-side-to-controller-side signaling exchange in a memory buffer IC), and input/output (I/O) steering circuitry 205 that forms a signaling interface for the four module sockets. The core logic 203 and I/O steering circuitry 205 are coupled to one another via internal signaling path 208 formed by a number (n) of internal signaling links. For simplicity of explanation, the internal signaling path 208 is assumed to be a data path for bi-directional conveyance of read and write data between the core logic and the I/O steering circuitry, and one or more internal control paths (not shown) are used to convey memory control information (commands/requests, addresses, timing signals, masking signals, module/rank select signals, configuration information, etc.) for managing operations within any socketed (i.e., installed) memory modules. Similarly, the asymmetric memory channels themselves (collectively, 211) are assumed to be data channels for bidirectional conveyance of read and write data between the memory interface IC and the module sockets, and one or more control channels (not shown) are used to convey memory control information to the socketed memory modules. These assumptions are carried forward in other embodiments discussed below, though in all cases the control information may be transmitted via the data path (e.g., via time-multiplexed operation) and the separate internal control path and external control channel omitted. Also, while each of the asymmetric memory channels is shown as a point-to-point signaling link extending between the memory interface IC 201 and a single memory-module socket, each of the asymmetric memory channels in the embodiment of FIG. 1 and any of the embodiments discussed below could alternatively be coupled in multi-drop fashion to multiple memory-module sockets, or to multiple memory sockets, each of any type.

Still referring to FIG. 1, the constituent signaling links of the internal data path 208 are referred to herein as “data lanes” to distinguish them from external “DQ” signaling links that constitute each memory channel. Also, for clarity of explanation, the data lane signaling rate (i.e., the on-chip signaling rate) is assumed to match the DQ signaling rate (i.e., the on-chip and off-chip data signaling rates match), so that the bandwidth of the n-lane internal data path 208 is matched by a live set of n DQ links within one or more of the asymmetric memory channels 211. In alternative embodiments, signaling rates on the internal data path 208 and asymmetric memory channels 211 may be different, with the higher signaling-rate path being proportionally narrower to equalize the net internal and external signaling bandwidths.

As shown, the asymmetric memory channels have respective widths configured according to the expected loading order of the module sockets to which they are connected. More specifically, a full-width memory channel 211₀ (i.e., n DQ links) is coupled to base socket, S0, to allocate the full system bandwidth to a memory module installed in that socket in a single-module configuration (i.e., all other sockets empty—also referred to herein as the base configuration, or x1 configuration). By contrast, a half-width memory channel (n/2 DQ links) 211₁ is coupled to the first expansion socket, S1, to enable half the system bandwidth to be reallocated from the base socket to the first expansion socket when the latter is loaded with a memory module. That is, when the system capacity is expanded from the base configuration to a two-module configuration (x2 configuration) through module installation in S1, half of the memory bandwidth is reallocated from the originally inserted memory module to the newly inserted memory module. Accordingly, in the two-module configuration, only half the DQ links of base memory channel 211₀ are live, and the other half dormant. Following that same approach, quarter-width memory channels (n/4 DQ links each) 211₂, 211₃ are coupled to the second and third expansion slots, S2 and S3, to enable respective one-fourth shares of the system bandwidth to be reallocated from sockets S0 and S1 to sockets S2 and S3 when those sockets are populated.

The bandwidth reallocation operation may be better appreciated by considering that each of the memory channels 211₀-211₃ and the internal signaling path 208 to be formed by one or more sets of signaling links, referred to herein as signal-link groups, with each signal-link group corresponding to the minimum bandwidth share to be allocated to any one of the memory channels 211—that is, a (1/M)th share of the signaling bandwidth, where M is the number of module sockets, and thus n/M signaling links. Accordingly, in the four-socket example of FIG. 1, each signal-link group comprises n/4 signaling links so that the internal data path includes four signal-link groups (each including n/4 constituent data lanes) as shown at 215. Similarly, as shown at 217, the base memory channel 211₀ includes four signal-link groups, the first expansion channel 211₁ (i.e., to socket S1) includes two signal-link groups and the second and third expansion channels 211₂, 211₃ (i.e., to sockets S2 and S3) each include a single signal-link group. The I/O steering logic 205, switchably couples the signal-link groups of the internal data path to respective signal-link groups within one or more of the memory channels in accordance with the socket loading configuration, thus steering data signals between the internal data lanes 208 and respective external signaling links, and vice-versa. More specifically, as shown at 224, in the single-module configuration 225 (i.e., base socket loaded only), the four signal-link groups of the internal path 208 are coupled respectively to the four signal-link groups of the base memory channel 211₀, thus allocating the full memory system bandwidth to the lone memory module in the system. When expanded to the two-module population shown at 227, two of the four signal-link groups (i.e., half) of the internal path are reallocated from the base socket S0 to the first expansion socket S1 to establish a half-width memory channel to each installed memory module.

When expanded further to the three-module population shown at 229, the signal-link groups are allocated unevenly between the three populated sockets (S0, S1, S2), with the base module in socket S0 maintaining 50% share of the system bandwidth, and the remaining 50% share being split evenly among the modules in the first and second expansion sockets (S1 and S2). In alternative embodiments, particularly where the total system bandwidth permits an even three-way split (e.g., internal data path width evenly divisible by 3), an even distribution of the signaling bandwidth among the three installed memory modules may be supported.

Finally, when expanded to the four-module population shown at 231, the signaling bandwidth is again distributed evenly, with each of the four signal-link groups of the internal path being coupled to a signal-link group within a respective one of the four memory channels 2110-2113, thereby allocating a one-fourth bandwidth share to each of the four installed memory modules.

Reflecting on the bandwidth allocation options shown at 224, the numbers of live and dormant signaling links remains constant, regardless of the socket loading configuration. Thus, when a single memory module is installed, the four signal-link groups that constitute base memory channel 211₀ are live, while remaining four signal-link groups that constitute memory channels 211₁-211₃ (i.e., two signal-link groups to S1 and one each to S2 and S3) are dormant. When two memory modules are installed, two of the signal-link groups in the base memory channel 211₀ are deactivated and the same number activated in the S1 memory channel 211₁, thereby maintaining a fixed memory bandwidth. Similarly, when expanding from two installed modules to three, one of the signal-link groups in the first expansion channel 211₁ (i.e., to S1) is deactivated and the sole constituent signal-link group of the second expansion channel 211₂ is activated. Finally, in the four module configuration, three of the four signal-link groups of base memory channel 211₀ are deactivated, freeing the corresponding three signal-link groups of the internal data path 208 to be distributed one apiece among the other three memory channels 211₁-211₃ and thus maintaining the same total memory bandwidth as other socket loading configurations, but distributing that bandwidth among the four memory channels to establish a quarter-bandwidth connection to each of the memory modules (225, 227, 229 and 231).

FIG. 2 illustrates an embodiment of a generalized I/O steering circuit 241 that may implemented on an IC die (e.g., of a memory interface IC or buffer IC) and used to support switchable interconnections between an n-bit wide internal data path 208 and a generalized number (m) of module sockets. Again, the individual signaling links of the internal data path and each memory channel are conceptually divided into signal-link groups of n/m links each. The m signal-link groups of the internal data path 208, one for each of the m module slots, are referred to herein as “lane groups” to distinguish them from the counterpart DQ signal-link groups or “DQ groups” (DQ_G) that constitute the external memory channels. The I/O steering circuit itself includes a physical signaling interface 242 referred to herein alternatively as the input/output (I/O) circuit or PHY (physical-layer signal drivers and receivers), as well as a steering switch 243 that switchably couples lane groups LG₁-LG_m-1 to the DQ groups. In the embodiment shown, the physical-layer signal drivers and receivers are paired (i.e., each pair including an output driver and receiver) to form transceivers 250 that are coupled respectively to interface nodes 251. In one embodiment, the interface nodes 251 are implemented by contacts disposed on the IC die 240 (i.e., together with the internal data path 208 and the remainder of the I/O steering circuitry 241) to enable electrical connection of IC die 240 to the external signaling links that form the asymmetric memory channels. More generally, the interface nodes 251 may be implemented by any structure that enables the signal drivers and/or receivers within IC die 240 to be coupled directly or indirectly to the external signaling links that form the asymmetric memory channels including, for example and without limitation, bond pads, contacts, pins, inductive or capacitive interconnects and so forth. In the single-ended, bidirectional signaling arrangement shown, each interface node 251 is coupled between a respective signal transceiver 254 and external signal link 255. In a differential signaling embodiment, two interface nodes are provided per external signal link to enable transmission and reception of a differential data signal (i.e., two complementary signal components per information-bearing signal) on the pair of signal lines that constitute the differential external signaling link.

Continuing with the single-ended embodiment shown, because the base socket (S0) will receive, at minimum, (1/M)th of the signal bandwidth in any configuration and thus will always be coupled to at least one live DQ group, one lane group may be dedicated (i.e., coupled via a non-switched and therefore permanent connection) to a corresponding DQ group for the base socket as shown by permanent coupling 245 between lane group LG₀ and the set of interface nodes 251 coupled to a base channel DQ group. The constituent data lanes of the permanently connected lane group are thus referred to herein as non-switched lanes. By contrast, each of the other lane groups, LG₁-LG_m-1, may need to be reallocated from the base channel to one of the expansion memory channels (i.e., to sockets S1-Sm−1) and is thus switchably coupled via steering switch 243 to the set of interface nodes for either a base-channel DQ group or an expansion-channel DQ group. Accordingly, the constituent data lanes of these steered or switched lane groups are referred to as “switched lanes.”

In one embodiment, steering switch 243 includes switching elements 247₁-247_m-1 to switchably couple the data lanes of each switched lane group to one of N sets of interface nodes for corresponding DQ lanes, where N varies according to the number of different memory channels to which a given lane group may be allocated. Because each different allocation option results in the signals of the switched lane group being steered to/from a different DQ group (i.e., steered to/from a different memory channel and thus to/from a different socket), the allocation options are referred to herein as “steering options.” Further, in an embodiment in which the number of links allocated to higher-numbered expansion sockets is halved for each doubling of the socket count (i.e., base socket S0 coupled to L signaling links, first expansion socket S1 coupled to L/2 signaling links, expansion sockets S2-S3 coupled to L/4 signaling links, expansion sockets S4-S7 coupled to L/8 signaling links, etc.), the final lane group is subject to at least log₂m steering options (where ‘m’ is the number of module sockets). For example, if there are eight module sockets total (base socket, S0, plus seven expansion sockets, S1-S7), then one of the corresponding eight lane groups, LG₀) is permanently coupled to one of the sets of the interface nodes for the S0 memory channel (i.e., the memory channel dedicated to socket S0), the next lane group, LG₁, may be switchably coupled to either a set of interface nodes for the S0 memory channel or to a set of interface nodes for the S1 memory channel, the next lane group (not specifically shown) may be switchably coupled to a set of interface nodes for the S0 memory channel, the S1 memory channel or the S2 memory channel, and so forth with the final lane group being switchably coupled to a set of interface nodes for the S0, S1, S3 or S7 memory channels. Although numerous different connection arrangements are possible (including those which permit a more efficient steering switch implementation as discussed below), the following table presents one possible combination of bandwidth allocations for each of the loading configurations in an eight socket system:

	TABLE 1
	Lane	Number of Installed Modules
	Group	1	2	3	4	5	6	7	8
	LG0	S0	S0	S0	S0	S0	S0	S0	S0
	LG1	S0	S1	S0	S1	S1	S1	S1	S1
	LG2	S0	S0	S2	S2	S2	S2	S2	S2
	LG3	S0	S1	S3	S3	S3	S3	S3	S3
	LG4	S0	S0	S0	S0	S0	S0	S0	S4
	LG5	S0	S1	S0	S1	S5	S5	S5	S5
	LG6	S0	S0	S2	S2	S2	S6	S6	S6
	LG7	S0	S1	S3	S3	S3	S3	S7	S7

Reflecting on the bandwidth allocation options of Table 1, each of the switched lane groups is switched between as few as two and at least as many as log₂ m+1 different steering options. Lane group LG1, for example, is switchably coupled to either socket S0 or socket S1 (two steering options) and thus may be steered by a 1:2 switch element 247₁ as shown. By contrast, lane group LG7 is switchably coupled to one of four different sockets (i.e., log₂8+1=four steering options) so that, in the 8-socket example of Table 1, switch element 247_m-1 may be implemented by a 1:4 switch), including socket S0 in the single-module (x1) configuration, socket S1 in the x2 configuration, socket S3 in the x3-x6 configurations and socket S7 in the x7 and x8 configurations. Thus, steering switch may be formed by m−1 switching elements (one for each of the switched lane groups), including switches having as few as two steering options (switch selections) per data lane, and switches having at least as many as log₂ m+1 steering options. Also, because the number of required interface nodes increases by half the base number (n) for each doubling of the number of module sockets, the total number of interface nodes is given by n+0.5*n*log₂m (or n+½*n*(SO−1), where ‘*’ denotes multiplication. That is, the number of interface nodes is proportional to the log of the socket count (not the socket count itself as in the replicated-channel system discussed above). Further, the number of interface nodes required per memory socket (and thus the memory channel width and maximum allocable fraction of the system bandwidth for that socket) halves with each doubling of the number of memory sockets, with the sockets in the last-to-be-loaded half of the system having half the minimum number of interface nodes provided to the sockets in the first-to-be-loaded half of the system. Analytically, the number of interface nodes per socket in such an embodiment may be expressed as: n/(2*roundup(log₂X)), where ‘*’ denotes multiplication, ‘X’ is the socket number (an integer between 1 and m that corresponds to the order in which the socket is loaded, with socket number 1 being loaded first), and “roundup( )” is a function that rounds its argument up to the nearest integer. For example, in an eight-socket system in accordance with the foregoing, the number of interface nodes per socket may be as follows (more or fewer sockets may be provided in alternative embodiments):

	TABLE 2
	Socket	Socket	Interface Nodes
	Position	No.	(memory channel width)
	S0	1	n
	S1	2	n/2
	S2	3	n/4
	S3	4	n/4
	S4	5	n/8
	S5	6	n/8
	S6	7	n/8
	S7	8	n/8

FIG. 3A illustrates a more detailed embodiment of an I/O steering circuit 280 that switchably couples an n-lane internal data path to asymmetric memory channels of a four-socket memory system in response to a switch-control value (SC). As shown, the number of external signaling links per memory channel 211 follow the progressively halved allocation described above in reference to Table 2 (i.e., full-width channel 211₀ to socket S0, half-width channel 211₁ to socket S1, and quarter-width channels 211₂ and 211₃ to sockets S2 and S3, respectively). Referring to detail view 281, the internal data path may be viewed as including multiple 4-bit lane fragments, LF₀, LF_(n/4)−1, each supplied to a respective one of 4-bit switching cells 283₀-283_(n/4)−1 for switched coupling to a counterpart set of external signaling links that constitutes (i) a respective fragment of the overall set of external signaling links (i.e., one of DQ fragments, DQF₀-DQF_(n/4)−1) and (ii) a respective fragment of each of the asymmetric memory channels. That is, when the switch-control value indicates a single-module configuration, each of the 4-bit switching cells 283 couples a respective one of the 4-bit lane fragments to the corresponding 4-bit fragment of the base memory channel (i.e., the four links to socket S0), thereby coupling all n lanes of the internal data path 208 to counterpart signaling links within the full-width (n-bit wide) base memory channel. When an additional module is installed in socket 51, the switch-control value is changed accordingly to enable each of the switching cells 283 to reallocate two data lanes within a respective one of the four-bit lane fragments from the S0 memory channel to the corresponding two-bit fragment of the S1 memory channel, thereby splitting the internal data path bandwidth between the half-width S1 memory channel and half the links of the full-width base memory channel (deactivating the other half of the links within the base memory channel). When additional modules are installed in sockets S2 and S3 (thus fully loading the memory system), the configuration value is changed to a corresponding state to enable each of the switching cells 283 to reallocate three data lanes per lane fragment from the S0 memory channel to the 1-bit fragments of the S2 and S3 memory channels and to one half (i.e., one link) of the 2-bit fragment of the S1 memory channel, thus establishing an even 4-way split of the internal data path bandwidth between the four memory channels.

In the three-module configuration, the internal bandwidth is unevenly allocated between the base memory channel and the first and second expansion channels. In the particular implementation shown, for example, the first expansion channel is allocated half the signaling bandwidth (and thus is coupled to two data lanes per lane fragment), while the base channel and the second expansion channel are each allocated a respective quarter of the signaling bandwidth (and thus are each coupled to a single data lane per lane fragment). Note that this configuration involves allocating more bandwidth to an expansion socket than the base socket, but provides the advantage of reduced switching circuitry as shown in cell detail view 287. As can be seen, data lane 0 of each four-lane fragment is dedicated (coupled without switching and thus permanently) to link 0 within the counterpart 4-link fragment of the base memory channel (S0_DQ0), while the remaining three data lanes are coupled via respective switching elements 291, 293, 295 to one of at least two signaling links. More specifically, data lane 1 is switchably coupled to either S0_DQ1 (link DQ1 within the memory channel to socket S0) or S1_DQ0 (link DQ0 within the expansion channel to socket S1) via switching element 291, data lane 2 is switchably coupled to either S0_DQ2 or S2_DQ0 (the latter being a link within the memory channel to socket S2) via switching element 293, and data lane 3 is switchably coupled to either S0_DQ3 (the fourth of the four links of the S0 channel fragment), S1_DQ1 (the second of the two links of the S1 channel fragment) or S3_DQ0 (a link within the memory channel to socket S3) via switching element 295. Though depicted as multiplexer/demultiplexer circuits in FIG. 3A, the switching elements 291, 293, 295 may be implemented by analog switches (e.g., pass gates) or any other practicable switching circuits in alternative embodiments. Also, though shown as a passive, hardwired connection, the permanent coupling between lane 0 and S0_DQ0 may be established through one or more delay matching elements, including permanently-configured switching elements. The overall switching arrangement described above for each of the four settings of a two-bit switch control value, SC[1:0], is shown in FIG. 3B. While a particular socket loading order is presented in FIGS. 3A and 3B (i.e., S0→S0/S1→S0/S1/S2→S0/S1/S2/S3), different socket loading orders may be dictated (e.g., S0→S0/S1→S0/S2/S3→S0/S1/S2/S3) through changes in the implementation of switching cells 283. The socket allocated a larger share of the bandwidth than others in uneven allocation arrangements (e.g., the three-module allocation arrangement shown in FIG. 3B in which a larger bandwidth share is allocated to socket S1 than to sockets S0 or S2) may similarly be changed through changes in the implementation of switching cells 283.

FIG. 4 illustrates various alternative dispositions of the above-described steering circuitry within a memory system. More specifically, the steering circuitry may be disposed upstream or downstream from the physical I/O circuitry to establish the different memory control IC implementations shown at 301 and 303, respectively. The steering circuitry may also be omitted from the memory control IC entirely as shown at 305 and instead be disposed externally to the memory control IC, for example within a buffer IC or entirely off-chip, as shown at 307. In the off-chip instance, the steering circuitry may be formed, for example and without limitation, by switching elements formed directly on a printed circuit board, such as a motherboard, backplane, line card or the like.

The fundamental function of the steering circuitry remains unchanged whether disposed upstream of the I/O circuitry as in memory control IC 301 (i.e., steering circuit 321 disposed between I/O circuitry 323 and controller core logic 311) or downstream of the I/O circuitry as in memory control IC 303 (i.e., I/O circuit 341 disposed between controller core logic 311 and steering circuit 343), though implementation details may change. In the former arrangement (steering circuitry disposed between controller core and I/O circuitry), separate unidirectional steering circuits 325 and 327 may be provided for incoming and outgoing data flow directions as shown. In that case, the n “logic-side” output data lanes (DL_LO) are switchably coupled via outbound steering circuit 327 to a live subset of a larger number of “PHY-side” output data lanes (DL_PO) which are then driven, via respective interface nodes, onto a corresponding subset of live external signaling links (DQ) by output drivers 331. Inbound steering circuit 325 performs the reverse function, steering signals received via the live subset of DQ links, receiver circuits 329 and PHY-side input data links (DL_PI) to the full-set of logic-side input data links (DL_LI). In the four-channel/four-socket example shown, the number of PHY-side data lanes (DL_PI, DL_PO) is twice the number of logic-side data lanes (DL_LI, DL_LO), so half the PHY-side input data lanes and half the PHY-side output data lanes are dormant in any loading configuration. The signal drivers and/or receivers coupled to dormant data lanes may remain unpowered to reduce power consumption and thermal loading.

Detail view 330 illustrates an example of a digital demultiplexer element 371 that may be replicated as necessary within steering circuits 325 and 327 to provide a unidirectional 1:2 signal steering function. As shown, an incoming data lane (“Input”) is coupled to the inputs of two logic AND gates 373, 375, with gate 373 having an inverting enable-input coupled to switch-control line (SC) and gate 375 having a non-inverting enable-input coupled to the switch-control line. By this arrangement, a logic-low switch-control signal enables AND gate 373 to pass the incoming data signal to output A (“OutA”) and disables AND gate 375 from passing the data signal to output B (“OutB”). Conversely, a logic-high switch control signal enables AND gate 375 to pass the incoming data signal to output B and disables AND gate 373 from passing the data signal to output A. Demultiplexer element 371 may be used as a building block (or the logic therein extended) to provide a generalized 1:N or N:1 steering function within steering circuits 325 and 327.

When disposed downstream of the I/O circuitry as in memory control IC 303, the steering circuitry 343 steers signals between a set of n internal signaling links, DQ_I, and a live subset of the larger number external signaling links, DQ_X. While this approach obviates separate steering circuits for different data flow directions as in IC 301, a degree of signal attenuation may result from placement of the steering circuitry on the output side of the signal drivers (and in front of the inputs to the signal receivers). Also, because link-to-link timing compensation may be provided within the signal output drivers, any link-to-link delay skew resulting from, for example timing differences in switched and non-switched signal paths) may reduce system timing margin (though this may alternatively be compensated by receiver-side timing calibration).

Detail view 350 illustrates an example of a switching element that may be replicated as necessary within steering circuit 343 (or within steering circuits 325 and/or 327) to provide a bidirectional 1:2 (and 2:1) or unidirectional signal steering function. As shown, an internal signal link, DQ_I is switchably coupled to each of two external signaling links DQ_XA and DQ_XB via respective pass gates 383 and 385. Each of the pass gates is implemented by a pair of parallel-coupled transistors that is switched on in response to opposite polarity gate voltages (e.g., PMOS and NMOS transistors in the example shown). A switch control signal is provided to the gate terminals of each pass-gate 383 and 385 in a reverse polarity orientation such that one of the pass gates is switched to a conducting state (i.e., switched on) and the other to a substantially non-conducting state (i.e., switched off), depending on the state of the switch control signal. More specifically, in the embodiment shown, a logic high switch control signal switches on pass gate 383 (through application of the logic high voltage to the NMOS transistor and a logic low voltage, generated by inverted 387, to the PMOS transistor) and switches off pass gate 385 to switchably couple the internal data link, DQ_I[i] exclusively and bidirectionally to external link DQ_XA[i]. Conversely, a logic low switch-control signal switches on pass gate 385 (logic low signal applied to PMOS transistor and logic high signal, generated by inverter 387, applied to NMOS transistor) and switches off pass gate 383 to switchably couple DQ_I[i] exclusively and bidirectionally to external link DQ_XB[i]. Analog switching element may be used as a building block (or the number of pass gates therein increased) to provide a generalized 1:N or N:1 steering function.

As mentioned, the steering circuitry may also be disposed in a buffer IC 307, thus enabling use of memory control IC that lacks an internal steering function (e.g., the buffer IC may provide for capacity expansion while maintaining a compatible system integration with legacy systems that utilize memory controllers lacking an internal steering function). In the embodiment shown, for example, buffer IC 307 is coupled between a set of asymmetric memory channels (shown as four channels having widths n, n/2, n/4 and n/4) and a memory control IC 305 having controller core logic 313 and physical I/O circuitry (drivers 365 and receivers 367), but lacking the above-described signal steering function. More specifically, the buffer IC 307 includes a steering circuit 361 and memory-side interface (indicated conceptually by links DQ_M) to the asymmetric memory channels, as well as a controller-side interface (indicated by links DQ_C) to the memory control IC 305. The buffer IC may be mounted to a motherboard or other substrate (thus forming a buffer-on-board) or together with the memory control IC and/or other IC's in an integrated circuit package (e.g., a system-in-package). With regard to internal organization, the steering circuit 361 may be disposed on either side of the physical memory-side signaling interface (e.g., as in memory IC 301 or memory IC 303) and thus implemented using either of the multiplexing or switching elements 371 and 381.

The steering circuit may also be implemented entirely off-chip, for example, by a collection of zero-ohm resistors, mechanical switches or other conductive interconnects that do not significantly compromise signal integrity (i.e., negligible signal radiation, impedance discontinuity, etc.) and that may be configured at production time or post-production time to permit memory capacity selection and/or expansion.

Still referring to FIG. 4, the detail view labeled “Off-Chip” illustrates an example of an off-chip switching element 391 formed by a pair of conductive interconnects 393 and 395 and that may be replicated as necessary within an off-chip implementation of steering circuit 361 to provide a bidirectional or unidirectional signal steering function. As shown, when interconnect 393 is shorted (i.e., controller-side and memory-side terminals electrically coupled to one another) and interconnect 395 opened, a controller-side signaling link, DQ_C is switchably coupled to memory side signaling link DQ_MA, and switchably decoupled (isolated) from memory side signaling link DQ_MB. Conversely, when interconnect 395 is shorted (i.e., controller-side and memory-side terminals electrically coupled to one another) and interconnect 393 opened, a controller-side signaling link, DQ_C is switchably coupled to memory side signaling link DQ_MA, and switchably decoupled (isolated) from memory side signaling link DQ_MB. Off-chip switching element 391 may be used as a building block (or the number of conductive interconnects therein increased) to provide a generalized 1:N or N:1 steering function.

As discussed in reference to FIG. 1, as bandwidth is reallocated to newly loaded module sockets, the bandwidth allocated to the previously loaded sockets is reduced (i.e., diluted). More specifically, as shown by the different steering configurations at 224, the effective width of the memory channel (i.e., the number of live links therein) to one or more of the previously loaded sockets is reduced, which in turn dictates a reduction in the number of live links coupled to each of the integrated circuit memory devices (e.g., dynamic random access memory (DRAM) ICs, static RAM ICs, flash memory ICs, etc) disposed on the corresponding memory module. That is, the effective width (or logical width) of the data path coupled to each of the memory ICs disposed on a previously loaded memory module is reduced as bandwidth is reallocated from that module to another memory module. Accordingly, in one embodiment, whenever a change in the socket loading configuration results in a decrease or increase in the effective width of the channel coupled to a given memory module (i.e., effective width of channel to a given socket is narrowed or widened), the data signaling interfaces of the individual memory ICs disposed on that memory module are correspondingly narrowed or widened. FIG. 5 illustrates an example of this operation in the context of an eight-socket memory system 400 having a memory interface IC 401 with an on-chip I/O steering circuit 403 that operates generally as described above (and which may be, for example, a memory control IC or a buffer IC). The eight memory channels 405 exhibit progressively halved widths as described in reference to table 2 above so that the minimum physical width of any of the channels and the minimum logical width (effective width) of any of the memory channels is n/8. In the particular example shown, the width of the internal data path 402 is assumed to be 64 bits, yielding memory channels 405 having widths of 64 bits (S0), 32 bits (S1), 16 bits (S2, S3) and 8 bits (S4, S5, S6, S7), as shown.

Referring to detail view 410 of FIG. 5, a memory module 411 that may be installed within each of the memory sockets 407 is assumed for purposes of example to include eight memory ICs 415₀-415₇ having respective 8-bit wide data interfaces coupled to corresponding 8-bit slices of a 64-bit module data interface 412. The eight memory ICs (collectively, 415) are operated as a memory rank (i.e., selected as a unit for read/write purposes) and each includes, in addition to the 8-bit wide data interface 419, a command interface (not shown), memory core 423 and control logic 421 to access the memory core in response to commands received via the command interface. More specifically, the control logic coordinates data retrieval and storage operations in response to memory read and write commands, outputting read data and receiving write data via the data interface. The control logic 421 may also include a configuration register, “creg” (or other configuration control circuit) that may be programmed in response to incoming commands to establish operating characteristics and modes of the memory device, including adjusting the effective (logical) width of the data interface 419 to a number of different settings by disabling one or more of the constituent signal driver/receiver circuits 425 within the data interface and corresponding internal circuitry. Thus, in a base system configuration in which a memory module is installed in socket S0 only (all other sockets empty), each memory device 415 of the installed memory module may be programmed to employ the full physical width of its respective data interface 419 (i.e., logical width=physical width=8 bits), thus establishing a 64-bit module data width to match the 64-bit wide full-width memory channel allocated to socket S0 as shown at 430. When, the memory system is expanded to a two-module configuration (memory modules in slots S0 and S1), the S0 memory module may be reprogrammed (i.e., by issuing a programming command and corresponding control register value) to change the data interface 419 width from x8 to x4 (i.e., disabling half the data I/O circuits as shown in dashed outline in view 432) to establish a 32-bit module interface width and thus the match half-width (32-bit) memory channel retained to the S0 memory channel. Similarly, when the logical width of the S0 channel is halved again from 32 bits to 16 bits (e.g., as in the four-module configuration), the memory devices of the S0 module may be reprogrammed to have a x2 data interface width as shown at 434 and thus establish a 16-bit module data width to match the new channel width. Finally, when the logical width of the S0 channel is halved yet again from 16 bits to eight bits (e.g., as in the eight-module configuration), the memory devices of the S0 module may again be reprogrammed to have a x1 data interface width as shown at 436 and thus establish an 8-bit module data width to match the 8-bit memory channel width. Note that the modules inserted into the expansion slots may also be programmable width modules and thus programmed according to the width of the memory channel allocated to their respective sockets. Alternatively, any or all of those memory modules may have fixed widths as required by the sockets into which they are inserted. In that regard, none of the memory modules in the system need have programmable widths, with those having widths larger than the available channel widths for a given module population removed from the system in favor of narrower-width modules.

Reflecting on FIG. 5, it should be noted that numerous details may be varied in alternative embodiments. For example, different interface widths, memory devices per module, memory channel widths, and so forth may be employed. Also, the specific I/O circuits (or pins) that are disabled for a given width configuration may vary. For example, in the x4 memory device configuration (432), the disabled I/O circuits may be disposed adjacent one another (e.g., top four I/O circuits or bottom four I/O circuits disabled) instead of being interleaved with enabled I/O circuits as shown. Further, memory module 411 or any of the memory modules discussed herein may be buffered modules (e.g., buffer IC disposed on memory module and coupled between the on-module memory devices and the module interconnects as in a fully-buffered DIMM (FBDIMM)) and the programmable width feature as descried above in relation to individual memory devices could alternatively be implemented within the on-module buffer IC.

FIG. 6 illustrates an exemplary sequence of operations that may be carried out within an asymmetric-channel memory system to determine the socket loading configuration and program corresponding configuration values within a memory interface IC and, if present, programmable-width memory devices. At 451, the characteristics of the module population are determined, for example, by attempting to read a serial-presence-detect (SPD) component (452a) or other non-volatile memory included on the memory module to record module characteristics such as the quantity/capacity/capability of memory devices, number of ranks, etc.; attempting to write-and-read back operations to ascertain module presence (452b); querying another source of configuration information (452c), and so forth.

At 453, the possible steering configuration (there may be more than one, particularly where the modules installed in a given socket are unable to support the bandwidth allocable to that socket) are determined based on the characteristics of the module population. At 455, a steering configuration is selected (e.g., in response to user selection (456a), by looking up a previously recorded selection by a user or system designer (e.g., in a hardware look-up table or stored as part of a software program) (456b), by applying the default steering configuration corresponding to the module population characteristics (456c), etc.). At 457, the steering logic is configured in accordance with the selected steering configuration for example, by programming a register within the memory interface IC (or establishing the state of off-chip switch elements if such configurability is supported) in accordance with the selected steering configuration. At 459, programmable width memory devices, if present within the module population, may be programmed in accordance with the selected steering configuration to establish matching logical channel widths and on-module data interface widths.

The foregoing operations may be effectuated, in whole or part, through execution of machine-executable program code. The program code (i.e., sequences of instructions and related data) may be recorded on one or more computer-readable media for later retrieval and execution within one or more processors of a special purpose or general purpose computer system or consumer electronic device or appliance. Computer-readable media in which such instructions and data may be embodied include, but are not limited to, machine-readable storage media in various forms. Examples of such machine-readable media include, for example and without limitation, optical, magnetic or semiconductor storage media such as floppy disk, CD, DVD, active RAM within a computer, or any other type of physical device from which the instructions may be extracted under full or partial control of a machine.

The various circuits disclosed herein may be described using computer aided design tools and expressed (or represented), as data and/or instructions embodied in various computer-readable media, in terms of their behavioral, register transfer, logic component, transistor, layout geometries, and/or other characteristics. Formats of files and other objects in which such circuit expressions may be implemented include, but are not limited to, formats supporting behavioral languages such as C, Verilog, and VHDL, formats supporting register level description languages like RTL, and formats supporting geometry description languages such as GDSII, GDSIII, GDSIV, CIF, MEBES and any other suitable formats and languages. Computer-readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, computer storage media in various forms (e.g., optical, magnetic or semiconductor storage media, whether independently distributed in that manner, or stored “in situ” in an operating system).

When received within a computer system via one or more computer-readable media, such data and/or instruction-based expressions of the above described circuits may be processed by a processing entity (e.g., one or more processors) within the computer system in conjunction with execution of one or more other computer programs including, without limitation, net-list generation programs, place and route programs and the like, to generate a representation or image of a physical manifestation of such circuits. Such representation or image may thereafter be used in device fabrication, for example, by enabling generation of one or more masks that are used to form various components of the circuits in a device fabrication process (i.e., implementing the circuits in an integrated circuit die or dice).

In the foregoing description and in the accompanying drawings, specific terminology and drawing symbols have been set forth to provide a thorough understanding of the present invention. In some instances, the terminology and symbols may imply specific details that are not required to practice the invention. For example, any of the specific numbers of bits, signal path widths, signaling or operating frequencies, component circuits or devices and the like may be different from those described above in alternative embodiments. Additionally, the various signaling links and other interconnections between circuit elements or blocks may be shown as buses or as single signal lines. Each of the buses may alternatively be a single signal line, and each of the single signal lines may alternatively be buses. Signals and signaling paths shown or described as being single-ended may also be differential, and vice-versa. A signal driving circuit is said to “output” a signal to a signal receiving circuit when the signal driving circuit asserts (or deasserts, if explicitly stated or indicated by context) the signal on a signal line coupled between the signal driving and signal receiving circuits. The expression “timing signal” is used herein to refer to a signal that controls the timing of one or more actions within an integrated circuit device and includes clock signals, strobe signals and the like. The term “coupled” is used herein to express a direct connection as well as a connection through one or more intervening circuits or structures. Integrated circuit device “programming” may include, for example and without limitation, loading a control value into a register or other storage circuit within the device in response to a host instruction and thus controlling an operational aspect of the device, establishing a device configuration or controlling an operational aspect of the device through a one-time programming operation (e.g., blowing fuses within a configuration circuit during device production), and/or connecting one or more selected pins or other contact structures of the device to reference voltage lines (also referred to as strapping) to establish a particular device configuration or operation aspect of the device. The terms “exemplary” and “embodiment” are used to express an example, not a preference or requirement.

While the invention has been described with reference to specific embodiments thereof, it will be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope. For example, features or aspects of any of the embodiments may be applied, at least where practicable, in combination with any other of the embodiments or in place of counterpart features or aspects thereof. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. An integrated circuit device for use in a memory system comprising memory sockets, the integrated circuit device comprising: an internal data path formed by a first number of internal signaling links that convey data to be output from the integrated circuit device;a signaling interface having a second number of interface nodes to be coupled to the memory sockets via respective external signaling links, the second number being greater than the first number; andswitch circuitry to switchably couple the internal data path to one of a plurality of different subsets of the interface nodes in accordance with occupancy of the memory sockets, each of the different subsets of the interface nodes being constituted by the same number of the interface nodes as the others of the different subsets of the interface nodes.

2. The integrated circuit device of claim 1 wherein each of the different subsets of the interface nodes within the signaling interface comprises the first number of the interface nodes.

3. The integrated circuit device of claim 1 further comprising an integrated circuit die on which the internal data path and switch circuitry are located, wherein the interface nodes comprise contacts disposed on the integrated circuit die to enable electrical connection of the integrated circuit die to the external signaling links.

4. The integrated circuit device of claim 1 wherein a first portion of the internal signaling links is permanently coupled to interface nodes that constitute a fraction of the interface nodes within each of the different subsets.

5. The integrated circuit device of claim 1 wherein the second number of interface nodes comprises a plurality of socket-specific groups of the interface nodes, each socket-specific group of the interface nodes to be coupled exclusively to a respective one of the memory sockets via a respective set of the external signaling links.

6. The integrated circuit device of claim 5 wherein the plurality of socket-specific groups of the interface nodes comprises a first socket-specific group of the interface nodes to be coupled exclusively to a first one of the memory sockets, the first socket-specific group of the interface nodes including more interface nodes than any others of the socket-specific groups of the interface nodes.

7. The integrated circuit device of claim 6 wherein the first socket-specific group of the interface nodes includes at least twice as many interface nodes as any others of the socket-specific groups of the interface nodes.

8. The integrated circuit device of claim 1 wherein the second number of interface nodes comprises N interface nodes to be coupled exclusively to a first one of M memory sockets within the memory system, and wherein the second number is less than the product of M and N.

9. The integrated circuit device of claim 1 wherein the second number (N) of interface nodes is given by the expression: N=L+L*(0.5*log2M), in which ‘L’ is the first number of internal signaling links, ‘M’ is the number of memory sockets within the memory system, and ‘*’ denotes multiplication.

10. The integrated circuit device of claim 1 wherein each of the memory sockets has a unique socket number (X) which is an integer in the range from 1 to the number of memory sockets (M) and wherein the second number of interface nodes comprises M groups of interface nodes each to be coupled to a respective one of the M memory sockets, and wherein the number of interface nodes within each of the M groups is given by the expression: L/(2roundup(log2X), where ‘L’ is the first number of internal signaling links, “roundup( )” is a function that rounds its argument up to the nearest integer, and ‘/’ denotes a division operation.

11. The integrated circuit device of claim 1 wherein a first subset of the different subsets of the interface nodes is to be coupled exclusively to a first one of the memory sockets and a second subset of the different subsets of the interface nodes is to be coupled in part to the first one of the memory sockets and in part to a second one of the memory sockets.

12. The integrated circuit device of claim 11 wherein a third subset of the different subsets of the interface nodes is to be coupled in part to the first one of the memory sockets, in part to a second one of the memory sockets and in part to a third one of the memory sockets.

13. The integrated circuit device of claim 12 wherein a first portion of the interface nodes within the first subset are included within the second subset, and a second portion of the interface nodes within the first subset are not included within the second subset.

14. The integrated circuit device of claim 1 further comprising a programmable register to store a switch-control value that indicates occupancy of the memory sockets and to output the switch-control value to the switch circuitry.

15. The integrated circuit device of claim 1 further comprising a central processing unit.

16. The integrated circuit device of claim 1 wherein the signaling interface comprises a memory interface to receive signals from and to output signals to one or more occupied ones of the memory sockets, the integrated circuit device further comprising a control interface to receive signals from and output signals to an external memory controller circuit.

17. A method of operation within an integrated circuit device having, within a signaling interface, a first number of interface nodes coupled to memory sockets via respective external signaling links, the method comprising: switchably coupling an internal signaling path to selected one of a plurality of different subsets of the interface nodes in accordance with occupancy of the memory sockets, each of the different subsets of the interface nodes being constituted by the same number of the interface nodes as the others of the different subsets of the interface nodes; andtransferring signals between the internal signaling path and the one of the subsets of the interface nodes.

18. The method of claim 17 wherein the internal signaling path comprises a fewer number of internal signaling links than the first number of interface nodes, and wherein each of the different subsets of the interface nodes comprises a number of the interface nodes that matches the number of the internal signaling links.

19. The method of claim 18 wherein a first portion of the internal signaling links is permanently coupled to interface nodes that constitute a fraction of the interface nodes within each of the different subsets.

20. The method of claim 17 further comprising programming a switch-control value within a programmable register of the integrated-circuit device, wherein the switch-control value is indicative of the occupancy of the memory sockets.

21. The method of claim 17 further comprising receiving a memory access request and wherein transferring signals between the internal signaling path and the one of the subsets of the interface nodes comprises transferring data signals between the internal signaling path and the one of the subsets of the interface nodes in response to the memory access request.

22. The method of claim 17 wherein the integrated circuit device is a buffer integrated circuit device comprising a controller interface coupled to a memory controller and wherein the signaling interface comprises at least part of a memory interface coupled to one or more of the memory sockets.

23. An integrated circuit device for use in a memory system comprising memory sockets, the integrated circuit device comprising: an internal data path formed by a first number of internal signaling links that convey data to be output from the integrated circuit device;a signaling interface having a second number of interface nodes to be coupled to the memory sockets via respective external signaling links, the second number being greater than the first number; andmeans for switchably coupling the internal data path to one of a plurality of different subsets of the interface nodes in accordance with occupancy of the memory sockets, each of the different subsets of the interface nodes being constituted by the same number of the interface nodes as the others of the different subsets of the interface nodes.