Patent Grant
9575270
Field of the Invention
The present invention relates to techniques for communicating signals in semiconductor dies and multi-chip systems. More specifically, the present invention relates to a multi-chip system which includes semiconductor dies that communicate signals using on-chip photonics.
Related Art
Advances in semiconductor technology have led to significant increases in the performance of integrated circuits, such as microprocessors. For example, the clock frequency of a typical microprocessor has increased from 16 MHz in 1985 to 3600 MHz by 2005, an improvement of 230 times, which represents an annual growth rate of nearly 30%. This performance trend has allowed computers to perform increasingly complicated functions across a wide variety of applications.
Unfortunately, several issues are beginning to constrain further increases in the clock frequency. In particular, the latency of global on-chip wires increases as technological trends reduces their thickness but not their length. In contrast, many local wires do not suffer from this delay penalty because their lengths shrink along with their thickness.
Moreover, as integration densities and clock frequencies continue to increase, the power consumption of high-performance microprocessors also increases. Consequently, many existing microprocessors consume over 100 W of power, which strains the capabilities of air cooling systems. In fact, many microprocessors have become power-limited, which means they could be operated at higher clock frequencies at the cost of significant increases in power consumption, and thus, in required cooling.
These design constraints have led designers to change the architecture of modern microprocessors. In particular, many microprocessors now include multiple processor cores. These processor cores keep computation and associated communication operations local, which reduces global delays in critical paths. Additionally, individual processor cores can be selectively enabled, thereby allowing unused processor cores to be put into a sleep mode to conserve power and then awakened when they are needed. Moreover, the use of smaller processor cores with shared logical blocks reduces the cost of developing and debugging microprocessors.
Furthermore, many multiple-core microprocessors support chip multi-threading (CMT). This technique helps address the increasing gap between microprocessor performance and the latency associated with fetching instructions and data from main memory, which has grown from a few clock cycles to hundreds of clock cycles over the past two decades. This gap often limits system performance because the microprocessor spends an increasing amount of time waiting for memory accesses instead of executing code. In a microprocessor that uses CMT, a thread can be quickly swapped in and out of execution. This rapid thread switching improves overall system throughput because instead of waiting for a memory request to return when the current thread accesses memory, the microprocessor can put the current thread to sleep and reactivate another thread. Consequently, utilization and throughput in such multi-threaded microprocessors is much higher than in single-threaded microprocessors.
However, microprocessors that include multiple cores and support multiple threads executing on each core have significantly higher communication requirements than single-core, single-threaded microprocessors. In particular, these microprocessors use high-bandwidth communication to: maintain coherence; pass messages; and/or perform simultaneous memory accesses. Moreover, as microprocessor throughput continues to increase, corresponding bandwidth requirements are expected to increase to terabits-per-second and beyond. Given the aforementioned latency problems, it may be difficult to meet these requirements using conventional conductive wires.
Hence, what is needed is a method and an apparatus which provide improved intra-chip and inter-chip communication without the problems listed above.
One embodiment of the present invention provides a system that includes an array of chip modules (CMs), where a given CM in the array includes a semiconductor die that is configured to communicate data signals with one or more adjacent CMs through electromagnetic proximity communication using proximity connectors. Note that the proximity connectors are proximate to a surface of the semiconductor die. Moreover, the given CM is configured to communicate optical signals with other CMs through an optical signal path using optical communication, and the optical signals are encoded using wavelength-division multiplexing (WDM).
In some embodiments, the optical communication between CMs includes communication of optical signals between two CMs that pass through an intermediate CM without intermediate processing of electrical signals associated with the optical signals.
In some embodiments, the electromagnetically coupled proximity communication includes: capacitive proximity communication, inductive proximity communication, conductive proximity communication, and/or optical proximity communication.
In some embodiments, the proximity connectors include optical coupling elements. Moreover, the optical coupling elements include: a lens, a diffraction grating, an arrayed-waveguide gating, and/or a mirror. In some embodiments, the optical proximity communication includes evanescent coupling.
In some embodiments, the array includes a 1-dimensional chain of CMs or a 2-dimensional array of CMs.
In some embodiments, the optical signal path facilitates communication in a first direction and a second optical signal path facilitates communication in a second direction. For example, the first direction and the second direction may be approximately perpendicular. Alternatively, the first direction and the second direction may be approximately parallel but oriented in approximately opposite directions.
In some embodiments, the optical signal path is configured for bidirectional communication of information between the CMs. Moreover, the optical signal path may include one or more waveguides.
In some embodiments, the array includes N subsets of CMs, and the given CM communicates with CMs in a given subset using optical signals in a sub-channel on the optical signal path, which is associated with a given wavelength in a set of N wavelengths.
In some embodiments, the array includes N subsets of CMs, each of which includes M CMs, and the given CM is coupled to CMs in a given subset by a first optical signal path and is coupled to other subsets by a second optical signal path. Moreover, within the given subset CMs communicate with the given CM using optical signals on a sub-channel in the first optical signal path, which is associated with a given wavelength in a set of M wavelengths. Additionally, the given CM communicates with another subset using optical signals in another sub-channel on the second optical signal path, which is associated with a given wavelength in a set of N wavelengths.
In some embodiments, the array includes N subsets of CMs, each of which includes M CMs, and the given CM is coupled to CMs in a given subset by a first optical signal path and is coupled to other subsets by a second optical signal path. Moreover, CMs in the array communicate with the given CM using optical signals in a sub-channel on the first optical signal path or on the second optical signal path, which is associated with a given wavelength in a set of N×M wavelengths.
In some embodiments, the array includes N subsets of CMs, each of which includes M CMs, and CMs within a given subset are coupled by a separate optical signal path. Moreover, the given CM communicates with a particular CM in the given subset using optical signals in a sub-channel on the separate optical signal path, which is associated with a given wavelength in a set of M wavelengths.
In some embodiments, a surface of a first CM in the array faces a surface of a second CM in the array, so that proximity connectors on the first CM are coupled to proximity connectors on the second CM, thereby facilitating the electromagnetic proximity communication between the first CM and the second CM.
In some embodiments, proximity connectors on a first CM in the array are coupled to proximity connectors on a second CM in the array via a bridge component, thereby facilitating the electromagnetic proximity communication between the first CM and the second CM.
Another embodiment of the present invention provides a computer system that includes the array.
Another embodiment of the present invention provides a method for communicating data signals in an array of CMs. During this method, a second CM receives optical signals from a first CM via an optical signal path. Next, a third CM receives the optical signals from the second CM via the optical signal path, where adjacent CMs, such as the first CM and the second CM or the second CM and the third CM, are configured to communicate with each other using electromagnetic proximity communication through electromagnetically coupled proximity connectors. Moreover, the optical signals are encoded using wavelength-division multiplexing (WDM).
Note that like reference numerals refer to corresponding parts throughout the drawings.
The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
Embodiments of a method, a multi-chip module (MCM) (such as a switch or a processor), and systems that include the MCM are described. This MCM includes an array of chip modules (CMs) or single-chip modules (SCMs), and a given SCM includes at least one semiconductor die. Note that the MCM is sometimes referred to as a ‘macro-chip.’ Furthermore, the semiconductor die communicates with other semiconductor dies, CMs, SCMs, and/or devices in the MCM using proximity communication of electromagnetically coupled signals (which is referred to as ‘electromagnetic proximity communication’), such as capacitively coupled signals and/or proximity communication of optical signals (which are, respectively, referred to as ‘electrical proximity communication’ and ‘optical proximity communication’). In some embodiments, the electromagnetic proximity communication includes inductively coupled signals and/or conductively coupled signals.
In some embodiments, a first component (such as the semiconductor die) in the MCM is situated face-to-face with another component (for example, a bridge chip or another semiconductor die) such that proximity pads or connectors on the first component are capacitively coupled and/or optically coupled with proximity pads or connectors on the other component. Note that electromagnetic proximity connectors allow communication of electromagnetic signals between the components without having to route the signals through intervening signal lines within a printed circuit board. Moreover, optically coupled proximity connectors allow communication of optical signals between the components without having to route the signals through a fiber-optic cable or a separate optical interconnect. Thus, the proximity connectors can provide high bandwidth intra-chip and inter-chip communication.
Moreover, in some embodiments optical signals are optically communicated between CMs in the MCM in an optical signal path using optical communication, where the optical signals are encoded using wavelength-division multiplexing (WDM). This optical signal path may include a waveguide for low latency intra-chip and inter-chip communication. For example, the optical signal path may include optical communication between a first CM and a second CM, optical communication through the second CM, and optical communication between the second CM and a third CM. Note that the optical communication through the second CM may take place without intermediate processing of electrical signals associated with the optical signals. In an exemplary embodiment, the optical signal path is an all-optical bypass channel.
By allowing optical signals to pass through the second CM without detection or conversion, the MCM may also provide reduced latency inter-chip communication by removing the delays associated with conversion(s) from the optical to the electrical domain (and vice versa). Moreover, the use of an all-optical bypass channel may reduce power consumption and the expense of the MCM. Consequently, these techniques may facilitate the creation of arbitrarily large arrays with very high performance.
Embodiments of the MCM may be used in a variety of applications, including: VLSI circuits, telephony, storage area networks, data centers, networks (such as local area networks), and/or computer systems (such as multiple-core processor computer systems). For example, the MCM may be included in a backplane that is coupled to multiple processor blades, or the MCM may couple different types of components (such as processors, memory, I/O devices, and/or peripheral devices). In some embodiments, the MCM performs the functions of: a switch, a hub, a bridge, and/or a router. Note that the electrical signals and/or optical signals may include analog signals, data packets, and/or data streams. These signals may include binary symbols, un-encoded signals, and/or encoded symbols (for example, using multiple-pulse amplitude modulation and/or WDM, where different sub-channels are used to communicate information). Moreover, communication in the optical signal path may be: unidirectional or bidirectional (including half-duplex and/or full-duplex operation).
We now describe embodiments of a macro-chip or MCM.
In an exemplary embodiment, MCM 100 includes a computer system, including one or more processors and/or memories. In some embodiments, MCM 100 includes a switch. In these embodiments, one or more of the semiconductor dies 110 and 112 implement some or all of the functionality of a switch. Such semiconductor dies are sometimes referred to as ‘switch chips’ or ‘logic chips.’ Therefore, one or more of the semiconductor dies 110 and 112 may include I/O ports to communicate input signals and output signals, as well as multiple switching elements that selectively couple the input and output ports. Furthermore, in some embodiments at least one of the semiconductor dies 110 and 112 includes a flow-control mechanism or logic that provides flow-control information that configures the switching elements or determines routing of data. This flow-control information may be communicated in the electrical domain and/or in the optical domain, and may be communicated along with the data and/or separately from the data (for example, in a separate flow-control communication channel, i.e., out-of-band control). Moreover, flow-control logic may be internal or external to MCM 100 (or MCM 130 in
Note that the bridge chips 116 may include one or more waveguides and/or multiple signal lines that couple signals from receive proximity connectors on one end of the bridge chips 116 to transmit proximity connectors on the other end. Such communication across the bridge chips 116 may be synchronous and/or asynchronous. In some embodiments bridge chips 116 include active electronics and/or optical components to transmit and receive signals, to amplify signals, and/or to resynchronize phases of signals on at least two of the signal lines. For example, the bridge chips can include a pipeline circuit that includes flip-flops and/or synchronization circuitry that are latched by clock signals to correct for an accumulated phase error. In these embodiments, the bridge chips 116 may be implemented using semiconductor dies, however, these dies may include different circuits and functionality than the semiconductor dies 114. In other embodiments, the bridge chips 116 are manufactured from a material other than a semiconductor.
As described below with reference to
In an exemplary embodiment, MCM 130 includes a 6×6 array of semiconductor dies 114 and a 5×5 array of bridge chips 116. In another exemplary embodiment, MCM 130 includes a 4×4 array of semiconductor dies 114 and a 3×3 array of bridge chips 116. Furthermore, electric and optical information in MCMs 100 (
In another exemplary embodiment of MCMs 100 (
Note that because of the use of proximity communication in the MCMs 100 (
In some embodiments, proximity communication transfers information between components in the MCMs 100 (
Optical routing (in an optical control path and/or an optical signal path) on a given component in MCMs 100 (
Although MCMs 100 (
We now described embodiments of devices (such as SCMs and MCMs) and systems that include proximity communication.
In one embodiment, the proximity connectors 212 may be on or proximate to at least one surface of the semiconductor die 210, the SCM and/or the MCM. In other embodiments, the semiconductor die 210, the SCM and/or the MCM may be coupled to the proximity connectors 212. In an exemplary embodiment, the plurality of proximity connectors 212 are substantially located at or near one or more corners (proximity connectors 212-1 and 212-2) and/or edges (proximity connectors 212-3) of the semiconductor die 210. In other embodiments, proximity connectors 212 may be situated at one or more arbitrary locations on, or proximate to, the surface of the semiconductor die 210.
As illustrated for the proximity connectors 212-1, there is a first pitch 214-1 between adjacent connectors or pads in a first direction (X) 216 of the surface and a second pitch 214-2 between adjacent connectors or pads in a second direction (Y) 218 of the surface. In some embodiments, the first pitch 214-1 and the second pitch 214-2 are approximately equal.
In order to communicate data signals using proximity communication, transmit and receive proximity connectors 212 on adjacent semiconductor dies 210 may have, at worst, only limited misalignment, i.e., substantially accurate alignment. For densely packed proximity connectors, i.e., proximity connectors 212 having a small spacing or pitch 214 (
In some embodiments, the proximity connectors 212 may be aligned in six degrees of freedom, including: the first direction (X) 216 (
In some embodiments, allowed misalignment in the first direction (X) 216 (
Solutions, such as self-alignment and/or self-adjustment of relative positions of the proximity connectors 212 on adjacent semiconductor dies 210 and/or in a component (such as one of the bridge chips 116 in
Reducing or eliminating the misalignment 314, in turn, may lead to at least partial overlap of one or more proximity connectors 212 on the adjacent semiconductor dies 210 and increase a magnitude of the capacitively coupled data signals. In addition, the solutions may reduce misalignment in the first plane, i.e., the plane including at least some of the proximity connectors 212, when used in conjunction with techniques such as electronic steering (where data signals are routed to respective proximity connectors 212 based on the alignment in the first plane). Consequently, these solutions may facilitate proximity communication between the semiconductor dies 210, SCMs and/or MCMs. The solutions may also reduce and/or eliminate a need for narrow tolerances, precise manufacturing, and/or precise assembly of the semiconductor dies 210, the SCM and/or the MCM. In addition, improved alignment of the proximity connectors 212 may reduce power consumption in MCMs 100 (
In the embodiments described above and below, the proximity connectors 212 on the adjacent semiconductor dies 210 utilize capacitive coupling and/or optical coupling for inter-chip communication. In other embodiments, different connectors may be overlapped on adjacent semiconductor dies 210. For example, one embodiment of the present invention uses inductive proximity connectors and/or magnetic proximity connectors, where data signals are communicated inductively and/or magnetically between terminals on closely adjacent semiconductor dies 210. Another embodiment conductively couples connectors in adjacent semiconductor dies 210 using an array of solder balls.
While the device 200 (
We now describe embodiments of optical components on semiconductor dies that may be included in the MCMs 100 (
In general, long wire latencies do not restrict the operating frequency in a system. Instead, these latencies may increase the cost of communication within the system. For example, when there are long wire latencies, memories that maintain coherency by snooping on other memories or that operate by casting out scatter or gather requests may be restricted to use a limited set of neighboring memories. In another example, architectures that pipeline communication requests may maintain queues of outstanding requests until these queues are fulfilled. Unfortunately, as communication distances and delays increase, the queues may become larger and larger, increasing the size, the power consumption, and the complexity of the system.
In some embodiments, these problems may be reduced and/or eliminated through the use of on-chip and/or inter-chip optical communication. For example, as discussed previously semiconductor dies 110, 112, and 114 (
Note that the speed of light in an optical waveguide is determined by the effective refractive index of the waveguide that is used to confine and communicate the light across the chip or a series of chips. In particular, the speed of light is inversely proportional to this effective refractive index and is essentially c/n, where c is the speed of light in vacuum and n is the effective refractive index of the medium. In general, n has real and imaginary components that depend on wavelength, but for many materials n is a real, non-negative number. For example, for silicon n is approximately 3.5.
In some embodiments, the effective index of a waveguide is reduced by using a so-called ‘slotted’ waveguide, while still strongly confining and guiding the light. Consequently, the speed of optical signal transmission in these waveguides is in the range of 1-1.8×108 cm/s or up to 10 times the speed of corresponding electrical signals. Furthermore, in some embodiments the delay or latency associated with the waveguides is further reduced using photonic-crystal techniques.
Therefore, in this embodiment proximity communication has been generalized to communication of arbitrary electro-magnetic waves across the gap between the semiconductor dies 410. By increasing the frequency of the electromagnetic waves, optical frequencies are reached, and the electro-magnetic waves are none other than light. Thus, optical proximity communication may be used to communicate information between the semiconductor dies 410, and optical transmission across the semiconductor dies 410 may occur via the waveguides 412. Note that MCM 400 includes an optical signal path or optical control path that spans multiple semiconductor dies 410 without optical-to-electrical conversions or electrical-to-optical conversions at intermediate points.
As noted previously, it may be difficult to perform logical computations (such as determining local flow-control information) in the optical domain. Therefore, in some embodiments the semiconductor dies 410 include conversion elements 414, which convert optical signals to electrical signals (and vice versa). These electrical signals may be coupled to logic circuits 416, such as a local flow-control mechanism or logic that determines local flow-control information based on information from adjacent semiconductor dies 410 and/or flow-control information communicated within a switch. After determining the local flow-control information, the resulting electrical signals may be converted into optical signals using a conversion element (such as conversion element 414-1) and the optical signals may be communicated to other semiconductor dies 410. Alternatively, optical signals corresponding to flow-control information may be received and converted into electrical signals by the conversion element 414-1 prior to setting switching elements on semiconductor die 410-1. Note that the conversion elements 414 may include wavelength combiners and splitters, or may simply tap a percentage of the light as it passes by in the waveguides 412, thereby facilitating conversion from the optical to the electrical domain.
While MCM 400 uses optical proximity communication between the semiconductor dies 410, in other embodiments electrical proximity communication is used in conjunction with on-chip optical communication. This is shown in
Note that MCM 400 (
Note that the conversion element 510 may include fewer components or additional components. Moreover, two or more components may be combined into a single component and/or a position of one or more components may be changed.
Note that in addition to carrying data and/or flow-control information the semiconductor dies 610 and 640 may also include circuits 612 and 642. For example, these circuits may include a switch or a computer, including a computation core (such as a processor) and/or a storage core (such as memory). Consequently, the photonic networks on each of the semiconductor dies 610 and 640 may also interact with routing points or conversion elements 616 to convert signals from the optical domain to the electrical domain prior to coupling to the circuits 612 and 642 (via signal lines 618), and back again after processing and/or storage.
In some embodiments, the on-chip photonic networks carry one or more synchronization signals in addition to data and/or flow-control information. For example, clock signals may be communicated in a MCM (such as MCM 100 in
In an exemplary embodiment, data, flow-control information, and/or clock signals are communicated on the waveguides 614 using WDM signaling. Such signaling may be accomplished using optical multiplexers and demultiplexers that are integrated on chip using silicon-on-insulator technology. In addition, the semiconductor dies 610 and 640 may include: arrayed waveguide gratings, ring resonators, and/or Mach-Zander phase modulators.
Note that the semiconductor dies 610 and 640 may include fewer components or additional components. For example, additional components that provide at least some of the previously described functionality may be coupled to semiconductor dies 610 and 640 using flip-chip binding. Moreover, two or more components may be combined into a single component and/or a position of one or more components may be changed.
We now described embodiments of devices (such as SCMs and MCMs) and systems with reduced inter-chip communication latency.
A given semiconductor die in MCM 700, such as semiconductor die 714-1, may include a conversion element (such as a laser or a diode) to convert electrical signals to optical signals, which are transmitted on the optical-bypass waveguides. Similarly, the given semiconductor die may include another conversion element (such as a detector or a diode) to convert the optical signals to the electrical signals (if the given semiconductor die is the destination of the optical signals). More generally, the given semiconductor die may include one or more add-drop elements (such as add or drop waveguides) which are configured: to insert the optical signals onto the optical waveguide(s) 712 (for example, using the conversion element); to remove the optical signals from the optical waveguide(s) 712 (for example, using the other conversion element), and/or to allow the optical signals to pass through the given semiconductor die on the optical waveguide(s) 712.
In some embodiments, the one or more add-drop elements convert a portion of the optical signals in the waveguides 712 to determine if the given semiconductor die is the destination of the optical signals (for example, based on routing information or a header associated with data packets in the optical signals). If yes, the remainder of the optical signals may be converted to electrical signals, and if no, the remainder of the optical signals may pass through the given semiconductor die without processing.
However, in some embodiments the optical signals are encoded (for example, using frequency-division multiplexing and/or WDM) and one or more sub-channels correspond to the given semiconductor die. In these embodiments, the one or more add-drop elements convert the optical signals corresponding to the one or more sub-channels (such as those associated with one or more carrier wavelengths) to electrical signals and the remainder of the optical signals pass through the given semiconductor die without processing.
By using all-optical bypass channels, MCM 700 may facilitate low-latency optical communication between semiconductor dies 714. In particular, signal propagation in optical waveguides may be 5-10× faster than signal propagation on existing on-chip metal wires. Moreover, optical waveguides may not require repeaters (i.e., optical-to-electrical signal conversion and back) on the semiconductor dies 714 and at every coupling between the semiconductor dies 714. Instead, electrically driven optical amplifiers and/or erbium amplifiers may be used to compensate for optical losses without conversion to the electrical signal domain. Moreover, multiple optical signals traversing one or more of the waveguides 712 on separate wavelengths may be simultaneously amplified so long as the wavelengths fall within the amplification band of the amplifier(s). In an exemplary embodiment, such amplifiers occur every few cm along the length of the waveguides 712 (or an associated optical signal path) and/or at every few couplings between semiconductor dies 714.
In some embodiments, optical signals on the all-optical bypass channels are encoded using WDM. Consequently, in some embodiments, a single waveguide provides bandwidth densities that are an order of magnitude greater than in existing electrical interconnects. In an exemplary embodiment, a single waveguide carries up to Terabits per second of bandwidth and the use of all-optical bypass channels between semiconductor dies 714 facilitates inter-chip communication at these bandwidths with: reduced power dissipation, reduced area utilization, and reduced latency.
Note that MCM 700 may include fewer components or additional components. For example, in addition to the full-row waveguides MCM 700 may include full-column waveguides, and these waveguides may support bidirectional communication In some embodiments, semiconductor dies 714 are interconnected by linear or mesh-like arrays of all-optical bypass channels. Furthermore, in some embodiments, the MCM 700 includes a 1-dimensional chain of semiconductor dies 714 (or CMs) and/or a 2-dimensional array of semiconductor dies 714 (or CMs). Note that two or more components in MCM 700 may be combined into a single component and/or a position of one or more components may be changed.
Note that MCM 800 may include fewer components or additional components. Moreover, two or more components may be combined into a single component and/or a position of one or more components may be changed.
Note that MCM 900 may include fewer components or additional components. Moreover, two or more components may be combined into a single component and/or a position of one or more components may be changed.
Note that MCM 1000 may include fewer components or additional components. Moreover, two or more components may be combined into a single component and/or a position of one or more components may be changed.
We now described embodiments of one or more optical signal paths, and in particular, optical signal paths that communicate optical signals which are encoded using WDM. These techniques may facilitate low latency inter-chip communication using a limited number of waveguides, as well as a limited number of sources (such as lasers), and detectors, thereby reducing power consumption.
Coding, such as WDM, may be used to determine where an optical signal has come from and/or where an optical signal is going to, thereby eliminating electrical processing of flow-control information or packet headers at intermediate chips in an optical signal path. For example, a given column (or subset) of the chips may be associated with a given sub-channel (such as a range of wavelengths centered on a carrier wavelength or a group of wavelengths each centered on associated carrier wavelengths) and chips within this column may communicate using this sub-channel. Thus, in MCM 1100 a set of N carrier wavelengths may be used, and a given column (such as chips 1110) may utilize a sub-channel associated with a given carrier wavelength in this set.
In an exemplary embodiment, N is 6, i.e., the set of carrier wavelengths includes λ1, λ2, λ3, λ4, λ5 and λ6. An optical signal may be first communicated in a horizontal direction via waveguide 1122 to reach a target column within the MCM 1100. Then, the optical signal may be sent to its destination(s) within the target column. In particular, because a different carrier wavelength is used to send optical signals to different columns, splitters or filters at each chip along the waveguide 1122 can remove optical signals having the appropriate wavelengths while other optical signals are unaffected. Thus, if a transmission targeting chips 1110 uses carrier wavelength λ1, the corresponding optical signals can be tapped out of the waveguide 1122 at chip 1110-1. Similarly, if transmissions targeting chips 1112 use carrier wavelength λ2, the corresponding optical signals can be tapped out at chip 1112-1.
Note that if the linear dimension of the array exceeds the number of carrier wavelengths that can be used in waveguide 1122, multiple waveguides may be used. For example, if a single waveguide can concurrently carry eight carrier wavelengths, a 16×16 array may use at least two horizontal waveguides to couple the columns or subsets of chips.
An extension of this approach allows multiple chips to be specified during transmission while using a common waveguide. This is shown in
In an exemplary embodiment, N is 6, i.e., the set of carrier wavelengths includes λ1, λ2, λ3, λ4, λ5 and λ6. Moreover, chips in the MCM 1200 may be associated with carrier wavelengths based on the following array of permutations
Thus, in row 1 or column 1, chip 1110-1 may be uniquely specified using carrier wavelength λ1, and in row 2 or column 3 chip 1114-2 may be uniquely specified using carrier wavelength λ2, etc.
More generally, an array of chips (such as the MCM 1200) may include N subsets, each of which includes M chips. For a given chip in the MCM 1200, a first optical signal path (such as a waveguide) may be coupled to the other chips within a given subset. Communication with a particular one of these chips may utilize a carrier wavelength in a set of M carrier wavelengths. Moreover, the given chip may also be coupled to other subsets via a second optical signal path (such as another waveguide). Communication with a particular one of these subsets may utilize a carrier wavelength in a set of N carrier wavelengths.
Further generalizing this approach, each chip in an array may be specified by a unique carrier wavelength. Thus, each row and each column may have a unique set of carrier wavelengths. This is shown in
In a variation on this embodiment, each column in MCM 1300 may be coupled to each row by a separate waveguide. This may reduce the total number of carrier wavelengths at the cost of additional waveguides. For example, a waveguide may couple column 1 with row 4. Similar, but independent waveguides may couple optical signals from column 1 to the other rows. In general, in this embodiment, for an N×N array, a given chip has N outgoing waveguides each communicating optical signals using N carrier wavelengths. The specific choice of waveguide and carrier wavelength, respectively, determines the row and column of the recipient chip. In some embodiments, there are also N incoming waveguides to the given chip.
In a further variation, a given chip in the array is coupled to different subsets of chips using separate waveguides. This is shown in
In an exemplary embodiment, chip 1110-1 communicates optical signals to other chips in row 1 using carrier wavelength λ1 and communicates optical signals to other chips in row 2 using carrier wavelength λ2, etc. Thus, optical signals to chip 1112-1 may be communicated using a first waveguide in the waveguides 1410 via a sub-channel associated with carrier wavelength λ1. Similarly, optical signals to chip 1114-1 may be communicated using a second waveguide in the waveguides 1410 via the sub-channel associated with carrier wavelength λ1.
In the general case, an array includes N subsets of chips, each of which includes M chips. Chips within a given subset are coupled by a separate optical signal path, such as a waveguide. Moreover, a give chip communicates with a particular chip in the given subset using optical signals in a sub-channel on the separate optical signal path, which is associated with a given wavelength in a set of M wavelengths.
Inter-chip communication using optical signal paths to communicate optical signals that are, at least in part, encoded using WDM have several advantages. In particular, these techniques facilitate minimum-distance, point-to-point communication between chips with a manageable number of optical signal paths (such as waveguides) per chip. For example, there may be N outgoing and/or N coming waveguides for each chip in a N×N array, and N carrier wavelengths may be sent and/or received at each chip. By using such dedicated optical signal paths to communicate optical signals between any pair of chips, flow-control and collision-detection control logic may be reduced or eliminated because the sender and/or destination chip is determined by the waveguide and/or carrier wavelength used. This arrangement may ensure that the incoming/outgoing data bandwidth is balanced and that an array of chips is isomorphic (symmetric) at any location, which may reduce the number of components in the array. Note that these techniques and configurations can coexist with other communication techniques and/or networks in the same MCM, for example, a tree network can be embedded in the array of chips to support global operations, such as broadcasts or barriers.
Note that MCM 1100 (
Optical signals that are encoded using WDM may be multiplexed onto and/or de-multiplexed from the optical signal paths (such as the waveguides) using a variety of techniques and optical coupling elements, including: a lens, a diffraction grating, a mirror, and/or an arrayed-waveguide grating. Arrayed-waveguide gratings are capable of precise multiplexing/de-multiplexing of multiple sub-channels traveling in a waveguide with relatively low losses. For example, the multiple sub-channels can be de-multiplexed into separate waveguides, each of which carries a sub-channel associated with a single carrier wavelength.
However, if an input wavelength is detuned from this central carrier wavelength, a phase changes will occur in the different waveguides 1514. Because of the constant path length difference between adjacent waveguides in the waveguides 1514, this phase change will increase linearly from the inner waveguide to the outer waveguide in the waveguides 1541. Consequently, the resulting wavefront will be tilted at the output aperture and the focal point in the image plane will be shifted away from the center. By appropriately positioning output waveguides 1516 in the image plane, different carrier wavelengths (or spatial frequencies) can be spatially separated. Note that by reversing the direction of propagation of the light, i.e., by reversing the input and output, the arrayed-waveguide grating 1500 may be used as a multiplexer.
Arrayed-waveguide grating have additional properties that may be of use in the interconnect topologies described above. For example, if additional wavelengths are available for use in an MCM, then the wrap-around property of the arrayed-waveguide grating can be used. In particular, the free spectral range of an arrayed-waveguide grating is defined as the frequency shift for which the phase-shift, ΔΦ, equals 2π. Two frequencies separated by the free spectral range that are input into an arrayed-waveguide-grating de-multiplexer will focus and leave though the same output waveguide, because their phase at the output waveguides 1516 is the same. Thus, if additional sub-channels are available in an MCM, multiple carrier wavelengths separated by the free spectral range can be assigned to the same waveguide destined for a particular chip. Moreover, because these carrier wavelengths are widely spaced, it may be easier to distinguish between these carrier wavelengths on the particular chip with less sensitivity to crosstalk and/or temperature.
Another useful property of arrayed-waveguide gratings is the ability to automatically shuffle the carrier wavelengths, as described above with reference to
Note arrayed-waveguide grating 1500 may include fewer components or additional components. For example, in embodiments that utilize dense wavelength-division multiplexing, there may be a temperature controller that stabilizes the temperature of the arrayed-waveguide grating 1500 to within a few C. Moreover, two or more components may be combined into a single component and/or a position of one or more components may be changed.
In exemplary embodiments, combinations of encoding techniques and communication channels may be used to achieve a target bandwidth capacity. Each configuration may have unique advantages for a given application. For example, 2.5 Gbps channel data rates can be extended to 40 Gbps using time-domain multiplexing (such as encoding using time-division-multiple-access) of electrical signals and data rates greater than 100 Gbps may be achieved using time-domain multiplexing of optical signals. As noted previously, data rates greater than 1 Tbps may be achieved by wavelength multiplexing several such channels using dense wavelength-division multiplexing and/or coarse wavelength-division multiplexing. Alternatively, a data rate of 1 Tbps may be achieved by multiplexing 1 Gbps time-domain multiplexed channels to a moderate number of SCM channels (e.g. 32) and then either multiplexing these channels onto parallel fibers (or waveguides) or using dense wavelength-division multiplexing.
To match the high bandwidth of an MCM that uses proximity communication, a high bandwidth I/O technology may be used to communicate data (such as data units) to and/or from the MCM. Fiber optic links are one such technology.
In some embodiments, one or more of the optical transceivers 1614 may convert optical signals received using one or more of the optical links 1616 into electrical signals. These signals may be communicated within the MCM 1600 using electrical proximity communication. In addition, one or more of the optical transceivers 1614 may convert electrical signals from the MCM 1600 into optical signals that are transmitted on one or more of the optical links 1616. Note that the transceivers 1614 may include conversion elements, such as conversion elements 440 (
In some embodiments, an MCM includes a semiconductor die and one or more optical transceivers. This shown in
Note that MCMs 1600 (
We now describe embodiments of methods for communicating signals.
In some embodiments of the process 1800 there may be additional or fewer operations. Moreover, the order of the operations may be changed, and two or more operations may be combined into a single operation.
Note that the present invention may include systems that contain one or more MCMs and/or switches, which include components (such as one or more semiconductor dies) that communicate signals using electromagnetically coupled proximity connectors, such as capacitively coupled proximity connectors and/or optical proximity connectors. For example,
Computer system 1900 may include memory 1924, which may include high speed random access memory and/or non-volatile memory. More specifically, memory 1924 may include: ROM, RAM, EPROM, EEPROM, FLASH, one or more smart cards, one or more magnetic disc storage devices, and/or one or more optical storage devices. Memory 1924 may store an operating system 1926, such as SOLARIS, LINUX, UNIX, OS X, or WINDOWS, that includes procedures (or a set of instructions) for handling various basic system services for performing hardware dependent tasks. Memory 1924 may also store procedures (or a set of instructions) in a communication module 1928. The communication procedures may be used for communicating with one or more computers and/or servers, including computers and/or servers that are remotely located with respect to the computer system 1900.
Memory 1924 may also include the one or more program modules (of sets of instructions) 1930. Instructions in the program modules 1930 in the memory 1924 may be implemented in a high-level procedural language, an object-oriented programming language, and/or in an assembly or machine language. The programming language may be compiled or interpreted, i.e., configurable or configured to be executed by the one or more processing units 1910.
Computer system 1900 may include one or more macro-chips 1908 (such as one or more MCMs) that include electrical proximity communication and/or optical proximity communication as described in the previous embodiments. While not shown in the computer system 1900, in some embodiments the one or more macro-chips 1908 may be coupled to one or more network interface circuits (such as one or more optical transceivers).
Computer system 1900 may include fewer components or additional components. Moreover, two or more components may be combined into a single component and/or a position of one or more components may be changed. In some embodiments, the functionality of the computer system 1900 may be implemented more in hardware and less in software, or less in hardware and more in software, as is known in the art.
Although the computer system 1900 is illustrated as having a number of discrete items,
The foregoing descriptions of embodiments of the present invention have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.
This application is a continuation application of, and hereby claims priority under 35 U.S.C. §120 to, pending U.S. patent application Ser. No. 11/863,815, entitled “Wavelength-Division Multiplexing for Use in Multi-Chip Systems,” by inventors Brian W. O'Krafka, Ashok V. Krishnamoorthy, John E. Cunningham, Xuehze Zheng, Ilya A. Sharapov and Ronald Ho filed on 28 Sep. 2007.
This invention was made with United States Government support under Contract No. NBCH3039002 awarded by the Defense Advanced Research Projects Administration. The United States Government has certain rights in the invention
| Number | Name | Date | Kind |
|---|---|---|---|
| 5093879 | Bregman | Mar 1992 | A |
| 6304125 | Sutherland | Oct 2001 | B1 |
| 6429955 | Suemura | Aug 2002 | B1 |
| 6876793 | Ling | Apr 2005 | B1 |
| 7356213 | Cunningham | Apr 2008 | B1 |
| 7496712 | O'Krafka | Feb 2009 | B1 |
| 7525199 | Lauterbach | Apr 2009 | B1 |
| 7693424 | Krishnamoorthy | Apr 2010 | B1 |
| 7817880 | Drost | Oct 2010 | B1 |
| 7961990 | Krishnamoorthy | Jun 2011 | B2 |
| 20010040929 | Shepston | Nov 2001 | A1 |
| 20030035169 | Byers | Feb 2003 | A1 |
| 20040197046 | Drost | Oct 2004 | A1 |
| 20050078902 | Beausoleil | Apr 2005 | A1 |
| 20050285214 | Krishnamoorthy | Dec 2005 | A1 |
| 20060251421 | Arnon | Nov 2006 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20130071121 A1 | Mar 2013 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11863815 | Sep 2007 | US |
| Child | 13657722 | US |
