Patent Grant
9879926
As is known, operating electronic components, such as electronic devices, produce heat. This heat should be removed from the devices in order to maintain device junction temperatures within desirable limits, with failure to remove heat effectively resulting in increased device temperatures, and potentially leading to thermal runaway conditions. Several trends in the electronics industry have combined to increase the importance of thermal management, including heat removal for electronic devices, including technologies where thermal management has traditionally been less of a concern, such as CMOS. In particular, the need for faster and more densely packed circuits has had a direct impact on the importance of thermal management. For example, power dissipation, and therefore heat production, increases as device operating frequencies increase. Also, increased operating frequencies may be possible at lower device junction temperatures. Further, as more and more devices are packed onto a single chip, heat flux (Watts/cm2) increases, resulting in the need to remove more power from a given size chip or module. These trends have combined to create applications where it is no longer desirable to remove heat from modern devices, and electronic system containing such devices, solely by traditional air cooling methods, such as by using air cooled heat sinks with heat pipes or vapor chambers. Such air cooling techniques are inherently limited in their ability to extract heat from electronic components with moderate to high power density.
In one aspect, the shortcomings of the prior art are overcome and additional advantages are provided through the provision of a method of controlling cooling of an electronic system. The method includes: automatically determining at least one adjusted control setting for at least one adjustable cooling component of a cooling system cooling the electronic system, the automatically determining being based, at least in part, on power being consumed by the cooling system and temperature of a heat sink to which heat extracted by the cooling system is rejected; and wherein the automatically determining operates to reduce power consumption of at least one of the cooling system or the electronic system, while ensuring that at least one targeted temperature associated with at least one of the cooling system or the electronic system is within a desired range.
In another aspect, a control system is provided for controlling operation of a cooling system cooling an electronic system. The control system includes a memory and a processor coupled to the memory. The processor and the memory are configured to facilitate performing a method which comprises: automatically determining at least one adjusted control setting of at least one adjustable cooling component of a cooling system cooling the electronic system, the automatically determining being based, at least in part, on power being consumed by the cooling system and temperature of a heat sink to which heat extracted by the cooling system is rejected; and wherein the automatically determining operates to reduce power consumption of at least one of the cooling system or the electronic system, while ensuring that at least one targeted temperature associated with at least one of the cooling system or the electronic system is within a desired range.
In a further aspect, a computer program product for controlling cooling of an electronic system is provided. The computer program product includes a computer-readable storage medium readable by a processor and storing instructions for execution by the processor for performing a method. The method includes: automatically determining at least one adjusted control setting for at least one adjustable cooling component of a cooling system cooling the electronic system, the automatically determining being based, at least in part, on power being consumed by the cooling system and temperature of a heat sink to which the heat extracted by the cooling system is rejected; and wherein the automatically determining operates to reduce power consumption of at least one of the cooling system or the electronic system while ensuring that at least one targeted temperature associated with at least one of the cooling system or the electronic system is within a desired range.
Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention.
One or more aspects of the present invention are particularly pointed out and distinctly claimed as examples in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
As used herein, the terms “electronics rack”, and “rack unit” are used interchangeably, and unless otherwise specified include any housing, frame, rack, compartment, blade server system, etc., having one or more heat-generating components of a computer system or electronic system, and may be, for example, a stand-alone computer processor having high, mid or low end processing capability. In one embodiment, an electronics rack may comprise a portion of an electronic system, a single electronic system or multiple electronic systems, for example, in one or more sub-housings, blades, books, drawers, nodes, compartments, etc., having one or more heat-generating electronic components disposed therein. An electronic system(s) may be movable or fixed, for example, relative to an electronics rack, with rack-mounted electronic drawers and blades of a blade center system being two examples of electronic systems (or subsystems) of an electronics rack to be cooled.
“Electronic component” refers to any heat-generating electronic component of, for example, a computer system or other electronic system requiring cooling. By way of example, an electronic component may comprise one or more integrated circuit dies, and/or other electronic devices to be cooled, such as one or more electronics cards. In one implementation, an electronics card may comprise a plurality of memory modules (such as one or more dual in-line memory modules (DIMMs)).
Further, as used herein, the terms “coolant-cooled structure”, “coolant-cooled cold plate” and “coolant-cooled cold rail” refer to structures having one or more channels (or passageways) formed therein or passing therethrough, which facilitate the flow of coolant (such as liquid coolant) through the structure. A coolant-cooled structure may be, for example, a coolant-cooled cold plate, or a coolant-cooled cold rail, or a coolant manifold. In one example, tubing is provided extending through the coolant-cooled structure. An “air-to-coolant heat exchanger” or “air-to-coolant heat exchange assembly” means any heat exchange mechanism characterized as described herein through which coolant can circulate; and includes, one or more discrete air-to-coolant heat exchangers coupled either in series or in parallel. An air-to-coolant heat exchanger may comprise, for example, one or more coolant flow paths, formed of thermally conductive tubing (such as copper or other tubing) in thermal or mechanical contact with a plurality of air-cooled cooling fins. Size, configuration and construction of the air-to-coolant heat exchanger can vary without departing from the scope of the invention disclosed. Still further, “data center” refers to a computer installation containing one or more electronics racks to be cooled. As a specific example, a data center may comprise one or more rows of rack-mounted computer units, such as server units.
One example of coolant used within the cooling systems and cooled electronic systems disclosed herein is water. However, the concepts presented are readily adapted to use with other types of coolant. For example, the coolant may comprise a brine, a glycol mixture, a fluorocarbon liquid, or other coolant, or refrigerant, while still maintaining the advantages and unique features of the present invention.
Reference is made below to the drawings, which are not drawn to scale for ease of understanding, wherein the same reference numbers used throughout different figures designate the same or similar components.
Due to ever-increasing air flow requirements through electronics racks, and the limits of air distribution within a typical data center installation, liquid-coolant-based cooling is being combined with conventional air-cooling.
In the embodiment illustrated, system coolant supply manifold 350 provides system coolant to cooling apparatuses disposed within the electronic systems or subsystems (for example, to coolant-cooled cold plates or cold rails) via flexible hose connections 351, which are disposed between the supply manifold and the respective electronic systems within the rack. Similarly, system coolant return manifold 360 is coupled to the electronic systems via flexible hose connections 361. Quick connect couplings may be employed at the interface between flexible hoses 351, 361 and the individual electronic systems. By way of example, these quick connect couplings may comprise various types of commercially available quick connect/disconnect couplings.
Although not shown, electronics rack 110 may also include an air-to-coolant heat exchanger, for example, disposed at an air outlet side thereof, which also receives system coolant from the system coolant supply manifold 350 and returns system coolant to the system coolant return manifold 360.
As illustrated, coolant flowing through warm-liquid coolant loop 420, after circulating through air-to-coolant heat exchanger 415, flows via coolant supply plenum 430 to one or more electronic systems of electronics rack 400, and in particular, one or more cold plates and/or cold rails 435 associated with the electronic systems, before returning via coolant return manifold 431 to warm-liquid coolant loop 420, and subsequently to a cooling unit 440 disposed (for example) outdoors from the data center. In the embodiment illustrated, cooling unit 440 includes a filter 441 for filtering the circulating coolant, a condenser (or air-to-coolant heat exchanger) 442 for removing heat from the coolant, and a pump 443 for returning the coolant through warm-liquid coolant loop 420 to air-to-coolant heat exchanger 415, and subsequently to the coolant-cooled electronics rack 400. By way of example, hose barb fittings 450 and quick disconnect couplings 455 may be employed to facilitate assembly or disassembly of warm-liquid coolant loop 420.
In one example of the warm coolant-cooling approach of
The illustrated coolant-based cooling approach further includes multiple coolant-carrying tubes connecting in fluid communication coolant-cooled cold plates 620 and coolant-cooled cold rails 625. These coolant-carrying tubes comprise (for example), a coolant supply tube 640, multiple bridge tubes 641, and a coolant return tube 642. In the embodiment illustrated, bridge tubes 641 connect one coolant-cooled cold rail 625 in series between the two coolant-cooled cold plates 620, and connect in parallel two additional coolant-cooled cold rails 625 between the second coolant-cooled cold plate 620 and the coolant return tube 642. Note that this configuration is provided by way of example only. The concepts disclosed herein may be readily adapted to use with various configurations of cooled electronic system layouts. Note also, that as depicted herein, the coolant-cooled cold rails are elongate, thermally conductive structures comprising one or more channels through which liquid coolant passes, for example, via one or more tubes extending through the structures. The coolant-cooled cold rails are disposed, in the embodiment illustrated, at the ends of the two arrays (or banks) 631, 632 of electronics cards 630, and multiple thermal spreaders are provided coupling in thermal communication electronics cards 630 and coolant-cooled cold rails 625.
By way of further enhancement, disclosed hereinbelow with reference to
For instance, disclosed herein below is a method of controlling cooling of an electronic system, which includes automatically determining at least one adjusted control setting for at least one adjustable cooling component of a cooling system cooling the electronic system. The automatically determining is based, at least in part, on power being consumed by the cooling system and a temperature of a heat sink to which heat extracted by the cooling system is rejected. The automatically determining operates to reduce power consumption of at last one of the cooling system or the electronic system, while ensuring that at least one targeted or controlled temperature associated with at least one of the cooling system or the electronic system is within a desired range. By way of example, the targeted temperature may be a coolant temperature, for example, at the inlet to the electronic system (such as an electronics rack).
The cooled electronic system depicted in
In the depicted embodiment, cooling system 710 includes a liquid-to-liquid heat exchanger 720 and a liquid-to-air heat exchanger 730. First coolant loop 721 couples in fluid communication with liquid-to-liquid heat exchanger 720, as does a second coolant loop 731 connecting liquid-to-liquid heat exchanger 720 to liquid-to-air heat exchanger 730. In this embodiment, a first coolant pump 722 pumps coolant through first coolant loop 721, and a second coolant pump 732 pumps coolant through second coolant loop 731. In addition, an air-moving device, such as a fan 733, facilitates air movement across liquid-to-air heat exchanger 730, and a recirculation valve 734 is provided, which may be a controllable valve with multiple valve settings between an open position and a closed position. A controller 740, such as a programmable logic controller or a computer, implements (in one embodiment) the control system processing described herein. Controller 740 is coupled to control, for instance, one or more of first coolant pump 722, second coolant pump 732, fan 733 and recirculation valve 734. In operation, controller 740 senses or receives the power and/or speed (or revolutions per minute (RPMs)) of first coolant pump 722, second coolant pump 732, and fan 733. Controller 740 further senses a targeted or control temperature (Ttarget) associated with, for example, the electronic system or electronics rack, as well as power consumed by the electronic system (e.g., IT power).
Previously determined models may then used to determine new or adjusted control settings (or states) for the cooling system, including (by way of example) the coolant pump speed(s), fan speed(s) and valve positions by calculating partial derivatives (δTtarget/δPtotal) (in one embodiment) for each adjustable cooling component of the cooling system over a series of incremental steps and evaluations 780. This modeling and partial derivatives approach can provide an energy efficient path to meeting desired thermal specifications, as discussed further below. The new control settings are then applied to one or more of the coolant pumps, fans or valve of the cooling system 785, after which the control system exits the control loop and waits for a next control initiation event 790.
A controller 840 is provided coupled to one or more adjustable cooling components of cooling system 810, such as first coolant pump 822, second coolant pump 832, fan 833 and recirculation valve 834. In addition, controller 840 may be coupled to sense or obtain a variety of temperature, power and pressure states, including, for example, electronic system power measurements, electronic component temperatures and airflow temperatures across electronic components, airflow relative humidity and dew point temperature, for example, at the inlet and outlet of air-to-coolant heat exchanger 802, pressure drop across the electronic system (or electronics rack), coolant inlet temperature to an electronic system (or rack), coolant outlet temperature, heat exchanger pump power being consumed, temperature and pressure of coolant within the second coolant loop before and after the liquid-to-liquid heat exchanger 820 or buffer, coolant pressure within the second coolant loop, ambient airflow temperature and relative humidity, air exhaust temperature from the liquid-to-air dry cooler, power consumed by and speeds of the coolant pumps and fans, post dry cooler coolant temperature within the second coolant loop, etc. Additionally, controller 840 is configured with control processing to reduce, or optimize, power consumption of the cooling system while providing a desired targeted temperature, such as liquid coolant temperature (Tliq), to the cooled electronic system.
A variety of control process embodiments may be implemented by the control system, depending for example, on the target or control temperature selected, and whether cooling system power is considered alone or whether total power consumed is considered, including the cooling system power and electronic system power loads. For example, control of coolant inlet temperature (Tliq) to an electronics rack may be desired while minimizing cooling system power (Pc) consumption employing a single adjustable cooling component of the cooling system. In the example of
In another embodiment, the control system controls cooling and energy consumption, again for a targeted temperature variable, such as coolant inlet temperature to the electronic system or rack, while minimizing coolant system power consumption using multiple adjustable cooling components of the cooling system. In this approach, partial derivatives (δTliq/δPc)i are determined for each adjustable cooling component (i) of the multiple adjustable cooling components of the cooling system. Note that Tliq is the targeted temperature in this example, and Pc is the power consumption of the cooling system. As discussed further below, the partial derivatives may be analytically or numerically determined by one skilled in the art. These derivatives advantageously facilitate determining a best adjustable cooling component to engage at a particular sub-step of the control process. The partial derivative(s) employed herein may be, for instance, the rate of change of the targeted temperature due to a rate of change of power to the particular adjustable coolant component for which the partial derivative is being calculated. The control system essentially determines what happens to the targeted temperature when power is changed to a particular cooling component of the cooling system, while keeping any other cooling components constant.
In a further embodiment, control may be provided with reference to a targeted temperature variable (such as the coolant inlet temperature to the electronic system), while minimizing total power consumed (Pt) by the electronic system and the cooling system using multiple adjustable cooling components of the cooling system. The total power (Pt) consumed equals the power consumed by the electronic system (PIT), as well as the power consumed by the cooling system (Pc). Partial derivatives (δTt/δPt)i may be determined, either analytically or numerically, as discussed herein. Note that a numerical determination may be employed where an analytical determination results in equations too complex to be readily solved. Since control is provided (in this embodiment) to the total power usage of the cooled electronic system, a more energy efficient solution may be obtained using this approach.
In still another embodiment, control may be provided with reference to, for example, a targeted temperature variable which comprises a temperature associated with an electronic component, such as a processor of the electronic system, or a dual-in-line memory module (DIMM) or array of the electronic system, or (for example) to air temperature across the electronic system, such as a rack inlet air temperature, while minimizing total power (Pt) using, for example, multiple adjustable cooling components of the cooling system providing coolant to the electronic system. Note that in this embodiment, the models employed may be more complex, potentially utilizing derived knowledge of component temperatures as a function of air and liquid temperatures, workloads and flow rates. However, this approach might provide the finest level of temperature control with potentially optimal energy use reduction/control. The external control platform or system may tie into the electronic system diagnostics if instrumentation is unavailable to provide the temperature and power values desired.
To facilitate the following further explanation, the below-listed variables are defined:
By way of specific example, the targeted temperature (Tt) can be modeled as follows:
Tt=f(Si,S2, . . . ,Sm,PIT)=Tamb+(Σz=1→w(Az,0×(S1)Az,1×(S2)Az,2× . . . ×(Sm)Az,m))×PIT
where:
For example, in
By way of further example, with Tt=Tliq, and with two heat exchange steps and three pieces of cooling equipment, as shown in
Tt=Tamb+PIT[(76.0×Sbuffer0.334×Spump−1.071×Sfan0)+(13.4×Sbuffer0×Spump0.378×Sfan−1.225)]
For more specific calculations: PIT=12.3+0.037×Tliq, may be employed with temperature Tliq being solved iteratively.
The power consumed (Py) may be modeled as set forth below. Note that Py=power consumed, and Py is either i) cooling equipment power (Pc) or ii) total power (Pt) (i.e., IT power (PIT) plus cooling equipment power (Pc)).
Using the above-noted example of
Pc=f(S1,S2, . . . ,Sm)=Σi=1
where:
By way of example, in
Pt=Pc+PIT=Pc+C0+C1×Tliq
By way of further example, with two heat exchange steps and three pieces of cooling components (as shown in
As noted, the models employed may utilize partial derivatives for the targeted temperature variable and the cooling power or total power consumed given a fractional change in speed of a cooling component i. For example, a partial derivative (δTliq/δPc)pump represents the ratio of the fractional change in the coolant inlet temperature (Tliq) with the cooling equipment power (Pc) for a fractional change in the pump operating speed (Spump). The partial derivatives (δTt/δPy)i can be determined by one skilled in the art either analytically or numerically:
Referring to
In a second control state 910, defined between the specified control temperature (Tt,spec) and a critical temperature (Tt,crit), valves of the cooling system may be closed incrementally, and cooling system components may have speeds adjusted to reduce the targeted temperature (Tt) in a most energy efficient manner, such as described below. In a third control state 920, the control temperature is above the critical target temperature. In this state, the control system automatically closes any recirculation valves, and the adjustable control components are set to full speed within the cooling system to attempt to bring the targeted temperature below the critical temperature. In a fourth control state 930, the target temperature is below the dew point temperature (Tdp), or below the dew point temperature plus a temperature tolerance ε4. In this control state, any recirculation valves are opened incrementally, and the speeds of the adjustable cooling components, such as pumps and fans, are adjusted to increase the targeted temperature (Tt) in a most energy efficient manner, such as described below.
As noted,
Assuming that the cooling system is not to be switched off 1004, or if so, that the current electronic system power dissipation is greater than the minimum acceptable power dissipation for switching the system off 1012, then processing determines an adjusted set of control settings for the cooling equipment's speed and valve positions based on current sensor inputs and the characterized system functions 1006. One embodiment of this process is depicted in
Assuming that one or more adjusted component settings are to be determined, then from
Assuming that the control temperature (Tt) has converged, then processing determines (in one embodiment) partial derivatives (δTt/δPy)i for the given inputs and the power and thermal relationships, for all controllable cooling components of the cooling system 1032.
Continuing with
As shown in
Continuing with
Assuming that the partial derivative of each cooling component does not exceed the derivative tolerance, or if so, that the speed of each component having a derivative exceeding the tolerance is set to a minimum, then processing determines whether the partial derivative is already less than a negative tolerance ε3. If “yes”, then processing determines whether the speed of each cooling component having a partial derivative less than −ε3 is set to a maximum 1058, and if so, a warning is posted 1062 and processing returns to the process of
Assuming that the speed of each cooling component meeting the partial derivative criteria of 1056 is not set to maximum, then processing increases the speed by x % of the cooling component with the largest negative derivative 1060, before continuing with the simulation of
Referring to
Assuming that the partial derivatives are larger than or equal to −ε3, or that the speed of each component having a partial derivative meeting the criteria is at a minimum, then processing determines whether the recirculation valve(s) is full open 1072, and if “no”, opens the valve by k % 1074, before continuing with the simulation processing of
If each valve is full open, processing determines whether the partial derivative (δTt/δPy)i is greater than ε2 1076, and if “yes” determines whether, for any component meeting the criteria, its speed is set to a maximum 1078. If “no”, then the speed (Si) of the component with the largest positive derivative is increased by x % 1080, after which processing continues with the simulation of
Assuming that the targeted temperature (Tt) is in the first control state 900 (
Assuming that each partial derivative is not less than negative ε3 tolerance, or if so, it is already set to a minimum, processing returns to the control flow of
As will be appreciated by one skilled in the art, one or more aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, one or more aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system”. Furthermore, one or more aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
Referring now to
Program code embodied on a computer readable medium may be transmitted using an appropriate medium, including but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for one or more aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language, such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language, assembler or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
One or more aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of one or more aspects of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition to the above, one or more aspects of the present invention may be provided, offered, deployed, managed, serviced, etc. by a service provider who offers management of customer environments. For instance, the service provider can create, maintain, support, etc. computer code and/or a computer infrastructure that performs one or more aspects of the present invention for one or more customers. In return, the service provider may receive payment from the customer under a subscription and/or fee agreement, as examples. Additionally or alternatively, the service provider may receive payment from the sale of advertising content to one or more third parties.
In one aspect of the present invention, an application may be deployed for performing one or more aspects of the present invention. As one example, the deploying of an application comprises providing computer infrastructure operable to perform one or more aspects of the present invention.
As a further aspect of the present invention, a computing infrastructure may be deployed comprising integrating computer readable code into a computing system, in which the code in combination with the computing system is capable of performing one or more aspects of the present invention.
As yet a further aspect of the present invention, a process for integrating computing infrastructure comprising integrating computer readable code into a computer system may be provided. The computer system comprises a computer readable medium, in which the computer medium comprises one or more aspects of the present invention. The code in combination with the computer system is capable of performing one or more aspects of the present invention.
Although various embodiments are described above, these are only examples. Further, other types of computing environments can benefit from one or more aspects of the present invention.
As a further example, a data processing system suitable for storing and/or executing program code is usable that includes at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements include, for instance, local memory employed during actual execution of the program code, bulk storage, and cache memory which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.
Input/Output or I/O devices (including, but not limited to, keyboards, displays, pointing devices, DASD, tape, CDs, DVDs, thumb drives and other memory media, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the available types of network adapters.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprise” (and any form of comprise, such as “comprises” and “comprising”), “have” (and any form of have, such as “has” and “having”), “include” (and any form of include, such as “includes” and “including”), and “contain” (and any form contain, such as “contains” and “containing”) are open-ended linking verbs. As a result, a method or device that “comprises”, “has”, “includes” or “contains” one or more steps or elements possesses those one or more steps or elements, but is not limited to possessing only those one or more steps or elements. Likewise, a step of a method or an element of a device that “comprises”, “has”, “includes” or “contains” one or more features possesses those one or more features, but is not limited to possessing only those one or more features. Furthermore, a device or structure that is configured in a certain way is configured in at least that way, but may also be configured in ways that are not listed.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below, if any, are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of one or more aspects of the invention and the practical application, and to enable others of ordinary skill in the art to understand one or more aspects of the invention for various embodiments with various modifications as are suited to the particular use contemplated.
This invention was made with Government support under Contract No. DE-EE0002894, awarded by the Department of Energy. Accordingly, the U.S. Government has certain rights in the invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3807493 | Stewart | Apr 1974 | A |
| 5168921 | Meyer | Dec 1992 | A |
| 5202837 | Coe et al. | Apr 1993 | A |
| 5371665 | Quisenberry et al. | Dec 1994 | A |
| 5414591 | Kumura et al. | May 1995 | A |
| 5596507 | Jones et al. | Jan 1997 | A |
| 5794454 | Harris et al. | Aug 1998 | A |
| 6006168 | Schumann et al. | Dec 1999 | A |
| 6011371 | Van Brocklin et al. | Jan 2000 | A |
| 6263959 | Ikeda et al. | Jul 2001 | B1 |
| 6400045 | Hosokawa et al. | Jun 2002 | B1 |
| 6411512 | Mankaruse et al. | Jun 2002 | B1 |
| 6434000 | Pandolfi et al. | Aug 2002 | B1 |
| 6446026 | Dean et al. | Sep 2002 | B1 |
| 6462949 | Parish et al. | Oct 2002 | B1 |
| 6480014 | Li et al. | Nov 2002 | B1 |
| 6557354 | Chu et al. | May 2003 | B1 |
| 6698218 | Goth et al. | Mar 2004 | B2 |
| 6775137 | Chu et al. | Aug 2004 | B2 |
| 6924981 | Chu et al. | Aug 2005 | B2 |
| 6934118 | Hidaka et al. | Aug 2005 | B2 |
| 6968709 | Goth et al. | Nov 2005 | B2 |
| 7002799 | Malone et al. | Feb 2006 | B2 |
| 7042726 | Cader et al. | May 2006 | B2 |
| 7103509 | Shah et al. | Sep 2006 | B2 |
| 7151668 | Stathakis | Dec 2006 | B1 |
| 7218129 | Beaman et al. | May 2007 | B2 |
| 7233501 | Ingalz | Jun 2007 | B1 |
| 7273090 | Crocker et al. | Sep 2007 | B2 |
| 7286355 | Cheon | Oct 2007 | B2 |
| 7310737 | Patel et al. | Dec 2007 | B2 |
| 7346468 | Bashor et al. | Mar 2008 | B2 |
| 7365973 | Rasmussen et al. | Apr 2008 | B2 |
| 7365985 | Ni | Apr 2008 | B1 |
| 7385810 | Chu et al. | Jun 2008 | B2 |
| 7388749 | Feroli et al. | Jun 2008 | B1 |
| 7395851 | Lee et al. | Jul 2008 | B2 |
| 7436663 | Matsushima et al. | Oct 2008 | B2 |
| 7440278 | Cheng | Oct 2008 | B2 |
| 7443672 | Peng et al. | Oct 2008 | B2 |
| 7450385 | Campbell et al. | Nov 2008 | B1 |
| 7469551 | Tilton et al. | Dec 2008 | B2 |
| 7573714 | Ali | Aug 2009 | B2 |
| 7583043 | Chung et al. | Sep 2009 | B2 |
| 7595550 | Cady et al. | Sep 2009 | B2 |
| 7626820 | Konshak et al. | Dec 2009 | B1 |
| 7660109 | Iyengar et al. | Feb 2010 | B2 |
| 7684196 | Eckberg et al. | Mar 2010 | B2 |
| 7701714 | Shabany | Apr 2010 | B2 |
| 7703291 | Bushnik et al. | Apr 2010 | B2 |
| 7746634 | Hom et al. | Jun 2010 | B2 |
| 7791882 | Chu et al. | Sep 2010 | B2 |
| 7796393 | Lengen et al. | Sep 2010 | B2 |
| 7796399 | Clayton et al. | Sep 2010 | B2 |
| 7817412 | Sullivan | Oct 2010 | B2 |
| 7826216 | Moss | Nov 2010 | B2 |
| 7830657 | Chu et al. | Nov 2010 | B2 |
| 2907406 | Campbell et al. | Mar 2011 | A1 |
| 8027162 | Campbell et al. | Sep 2011 | B2 |
| 8583290 | Campbell | Nov 2013 | B2 |
| 20040060305 | Singh et al. | Apr 2004 | A1 |
| 20040109288 | Beitelmal et al. | Jun 2004 | A1 |
| 20040221604 | Ota et al. | Nov 2004 | A1 |
| 20050049729 | Culbert et al. | Mar 2005 | A1 |
| 20050109049 | Chan | May 2005 | A1 |
| 20060146497 | Gauche et al. | Jul 2006 | A1 |
| 20060176186 | Larson et al. | Aug 2006 | A1 |
| 20060224254 | Rumi et al. | Oct 2006 | A1 |
| 20060225433 | Jammu et al. | Oct 2006 | A1 |
| 20060250772 | Salmonson et al. | Nov 2006 | A1 |
| 20070044493 | Kearney et al. | Mar 2007 | A1 |
| 20070211430 | Bechtolsheim | Sep 2007 | A1 |
| 20070300084 | Goodrum et al. | Dec 2007 | A1 |
| 20080112075 | Farquhar et al. | May 2008 | A1 |
| 20080239564 | Farquhar et al. | Oct 2008 | A1 |
| 20080245063 | Tutunoglu et al. | Oct 2008 | A1 |
| 20080259566 | Fried | Oct 2008 | A1 |
| 20080310105 | Cheng | Dec 2008 | A1 |
| 20080313492 | Hansen | Dec 2008 | A1 |
| 20090002951 | Legen et al. | Jan 2009 | A1 |
| 20090080151 | Kalms et al. | Mar 2009 | A1 |
| 20090086432 | Campbell et al. | Apr 2009 | A1 |
| 20090120607 | Cheon et al. | May 2009 | A1 |
| 20090133866 | Campbell et al. | May 2009 | A1 |
| 20090177334 | Artman et al. | Jul 2009 | A1 |
| 20090219687 | Lin | Sep 2009 | A1 |
| 20090259347 | Gross et al. | Oct 2009 | A1 |
| 20090268404 | Chu et al. | Oct 2009 | A1 |
| 20090277616 | Cipolla et al. | Nov 2009 | A1 |
| 20090284924 | Konshak et al. | Nov 2009 | A1 |
| 20100023787 | Ho | Jan 2010 | A1 |
| 20100002510 | Cipolla et al. | Feb 2010 | A1 |
| 20100027220 | Hughes et al. | Feb 2010 | A1 |
| 20100076607 | Ahmed | Mar 2010 | A1 |
| 20100085712 | Hrehor et al. | Apr 2010 | A1 |
| 20100094593 | Reddy et al. | Apr 2010 | A1 |
| 20100126696 | Novotny et al. | May 2010 | A1 |
| 20100142147 | Chang et al. | Jun 2010 | A1 |
| 20100175866 | Tani et al. | Jul 2010 | A1 |
| 20100027452 | Sri-Jayantha et al. | Oct 2010 | A1 |
| 20100252234 | Campbell et al. | Oct 2010 | A1 |
| 20100306598 | Ackaret et al. | Dec 2010 | A1 |
| 20100312415 | Loucks | Dec 2010 | A1 |
| 20100313590 | Campbell et al. | Dec 2010 | A1 |
| 20110051372 | Barringer et al. | Mar 2011 | A1 |
| 20110057803 | Yamaoka et al. | Mar 2011 | A1 |
| 20110060470 | Campbell et al. | Mar 2011 | A1 |
| 20110060472 | Ocondi et al. | Mar 2011 | A1 |
| 20110069454 | Campbell et al. | Mar 2011 | A1 |
| 20110112694 | Bash et al. | May 2011 | A1 |
| 20110197612 | Campbell et al. | Aug 2011 | A1 |
| 20110224947 | Kriss | Sep 2011 | A1 |
| 20110277967 | Fried et al. | Nov 2011 | A1 |
| 20110315343 | Campbell et al. | Dec 2011 | A1 |
| 20110315344 | Campbell et al. | Dec 2011 | A1 |
| 20110315353 | Campbell et al. | Dec 2011 | A1 |
| 20110315355 | Campbell et al. | Dec 2011 | A1 |
| 20110317367 | Campbell et al. | Dec 2011 | A1 |
| 20120120603 | Campbell et al. | May 2012 | A1 |
| 20130340994 | David et al. | Dec 2013 | A1 |
| 20130340995 | David et al. | Dec 2013 | A1 |
| 20130340996 | David et al. | Dec 2013 | A1 |
| 20130345893 | David et al. | Dec 2013 | A1 |
| 20140123493 | Campbell et al. | May 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 201222836 | Apr 2009 | CN |
| 19845821 | Apr 2000 | DE |
| 2436669 | May 2011 | GB |
| 01077199 | Mar 1989 | JP |
| 2010040886 | Feb 2010 | JP |
| 323643 | Dec 2007 | TW |
| 200937175 | Sep 2009 | TW |
| WO 97047167 | Dec 1997 | WO |
| WO 02037918 | May 2002 | WO |
| WO 2007140110 | Dec 2007 | WO |
| WO 2010075358 | Jul 2010 | WO |
| Entry |
|---|
| Simons et al., “A Survey of Vapor Phase Cooling Systems”, EDN Magazine, vol. 14, No. 1, pp. 53-56 (Jan. 1969). |
| Crispell et al., “Use of Liquid Metal to Promote Boiling on Immersed Integrated Circuit Chips”, IBM Technical Disclosure Bulletin, IPCOM000058250D, TDB 09-88, pp. 365-366 (Sep. 1, 1988). |
| Delia et al., “System Cooling Design for the Water-Cooled IBM Enterprise System/900 Processors”, IBM Journal of Research and Development, vol. 36, No. 4, pp. 791-803 (Jul. 1992). |
| Mudawar, et al., “Immersion-Cooled Standard Electronic Clamshell Module: A Building Block for Future High-Flux Avionic Systems”, Journal of Electronic Packaging, vol. 116, pp. 116-125 (Jun. 1994). |
| Nelson et al. “Thermal Performance of an Integral Immersion Cooled Multichip Module Package”, IEEE on Components, Packages, and Manufacturing Technology—Part A, vol. 17, No. 3, pp. 405-412 (Sep. 1994). |
| Arent, “Liquid Coolant Distribution Unit with Convertible Air or Water Cooling”, IBM Technical Disclosure Bulletin, IP Prior Art Database, ip.com No. IPCOM000044828D (Feb. 6, 2005). |
| Morgan, “IBM Hints at Triple Redundancy in Power6”, (http://www.itjungle.com/breaking/bn032906-story-1.html) (Mar. 29, 2006). |
| Ellsworth, Jr. et al., “The Evolution of Water Cooling for IBM Larger Server Systems: Back to the Future”, IEEE, Piscataway, NJ, 2008 11th Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITHERM '08), pp. 266-274 (May 2008). |
| IBM Technical Disclosure, “Highly-Efficient Copper Rail Design for Optimized Cooling of DRAM Modules”, IPCOM000184053D (Jun. 9, 2009). |
| Geisler et al., “Passive Immersion Cooling of 3-D Stacked Dies”, IEEE Transactions on Components and Packaging Technologies, vol. 32, No. 3, pp. 557-565 (Sep. 2009). |
| Forrest et al., “Pressure Effects on the Pool of Boiling of the Fluorinated Ketone”, Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), pp. 1-9 (2010) (No further date information is available). |
| Rubenstein et al., “Hypbrid Cooled Data Center Using Above Ambient Liquid Cooling”, IEEE (2010) (No further date information is available). |
| Chainer et al., “Thermoelectric-Enhanced, Liquid-Based Cooling of a Multi-Component Electronic System”, U.S. Appl. No. 13/102,195, filed May 6, 2011. |
| Eckberg et al., “Multi-Rack, Door-Mounted Heat Exchanger”, U.S. Appl. No. 13/115,404, filed May 25, 2011. |
| Campbell et al., “Multi-Fluid, Two-Phase Cooling of Electronic Component(s)”, U.S. Appl. No. 13/281,945, filed Oct. 26, 2011. |
| Chainer et al., “Directly Connected Heat Exchanger Tube Section and Coolant-Cooled Structure”, U.S. Appl. No. 13/283,933, filed Oct. 28, 2011. |
| Campbell et al., “Multi-Rack Assembly With Shared Cooling Unit”, U.S. Appl. No. 13/285,116, filed Oct. 31, 2011. |
| Chainer et al., “Dynamically Limiting Energy Consumed by Cooling Apparatus”, U.S. Appl. No. 13/305,967, filed Nov. 29, 2011. |
| David et al., Office Action for U.S. Appl. No. 13/527,909, filed Jun. 20, 2012 (U.S. Patent Publication No. 2013/0345893 A1), dated Nov. 10, 2014 (13 pages). |
| David et al., Office Action for U.S. Appl. No. 13/775,545, filed Feb. 25, 2013 (U.S. Patent Publication No. 2013/0340996 A1), dated Oct. 20, 2015 (10 pages). |
| David et al., Office Action for U.S. Appl. No. 13/775,529, filed Feb. 25, 2013 (U.S. Patent Publication No. 2013/0340995 A1), dated Sep. 21, 2015 (17 pages). |
| David et al., Final Office Action for U.S. Appl. No. 13/775,529, filed Feb. 25, 2013 (U.S. Patent Publication No. 2013/0340995 A1), dated Feb. 17, 2016 (17 pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20130340994 A1 | Dec 2013 | US |
