Densification in data centers is becoming so extreme that the power density of the systems in the center is growing at a rate unmatched by technological developments in data center heating, ventilation, and air-conditioning (HVAC) designs. Current servers and disk storage systems, for example, generate thousands of watts per square meter of footprint. Telecommunication equipment generates two to three times the heat of the servers and disk storage systems.
Computer designers are continuing to invent methods that extend the air-cooling limits of individual racks of computers (or other electronic heat-generating devices) that are air-cooled. High heat capacity racks, however, require extraordinary amounts of air to remove the heat dissipated by the racks and use expensive and large air handling equipment.
Some electrical devices, such as liquid-cooled mainframe computers, do use liquid cooling. In some situations, liquid cooling provides significant improvements over air-cooled systems. For instance, liquid cooling can more effectively remove large amounts of heat from data centers or even single servers.
Prior liquid cooling systems, however, are not fault tolerant. In other words, a server or data center can unexpectedly shutdown if a failure occurs with the cooling system. For example, if the cooling line in a building fails, then a single server or an entire data center will not be sufficiently cooled. As such, the server or entire data center can overheat and shutdown. As another example, if a non-fault tolerant cooling system needs serviced, then all servers or data centers on this cooling system must be temporarily shutdown while the system is repaired.
The data center 104 is situated in a building or room that has a floor 110 above a floor slab 112. A network of pipes 114 extends between the floor 110 and floor slab 112. The pipes carry a cooling fluid to and from the liquid cooling unit 103 and the electronic device 102. By way of example, the cooling fluids include, but are not limited to, water, refrigerant, single phase fluids, two phase fluids, etc. Further, although the pipes are shown within the floor, they can be located in various places, such as, but not limited to, the ceiling, the walls, on top of the floor, underground, etc.
As shown, fluid initially enters the liquid cooling unit 103 along a supply line 116 and exits the liquid cooling unit along a return line 118. Specifically, the fluid passes into a heat exchanger 120, such as a liquid-to-liquid heat exchanger. This heat exchanger 120 is connected to a liquid cooling loop 122 that extends between the liquid cooling unit 103 and the electronic device 102. A pump 123 pumps the fluid along supply line 124 from the heat exchanger 120 to heat generating components or electronics 128. After cooling the electronics 128, the fluid is pumped along return line 130 back to the heat exchanger 120.
The electronics 128 generate heat that is removed by the fluid and transferred away through the return line 130. In turn, the heat exchanger 120 removes, dissipates, and/or exchanges this heat so cooled fluid pumped along the supply line 124 can remove heat from the electronics 128. Embodiments in accordance with the present invention are not limited to any particular type of heat exchanger 128. Various types of heat exchangers, now known or developed in the future, are applicable with embodiments of the invention. By way of example, the heat exchanger 128 can use one or more of thermal dissipation devices, heat pipes, heat spreaders, refrigerants, heat sinks, liquid cold plates or thermal-stiffener plates, evaporators, refrigerators, thermal pads, air flows, and/or other devices adapted to remove or dissipate heat.
In one exemplary embodiment, the cooling system 100 is fault tolerant. For example, if one or more of the supply or return lines 116 or 118 breaks, fails, needs serviced, or otherwise shuts-down, then the electronic device 102 (example, servers or racks in data center 104) will not immediately or contemporaneously overheat and shutdown. In the event of such a failure or servicing in the cooling system 100, the liquid cooling loop 122 extending between the liquid cooling unit 103 and the electronic device 102 continues to cool the electronics 128 for a period of time, for example, until the liquid heats up. During this time, data can be stored to the electronic device or the system can gradually shutdown without overheating. Alternatively, the system can be repaired or serviced before the electronic device is required to shutdown.
One exemplary embodiment uses a coolant storage device 150 located in the liquid cooling unit 103 to provide fault tolerance for cooling system 100. The coolant storage device 150 is a storage tank that stores extra fluid volume (i.e., more than is due, normal, or needed for operations) so the cooling capacity increases for the fluid in the liquid cooling loop 122 and provides additional time for operation during cooling loss to the electronic devices 102 or data center 104. For example, if lines 116 or 118 break or if service is disrupted or disconnected, then the system continues to operate for an extended period of time due to the added fluid capacity in the storage tank. Thus, extra fluid stored in the coolant storage device 150 provides added fluid carrying capacity for servers, racks, and other electronic devices in the date center 104. This extra fluid is used to prolong the runtime of the electronic device 102 so it is not required to be shutdown when, for example, the heat exchanger fails or lines 116 and 118 fail.
Embodiments in accordance with the present invention are not limited to any particular amount of extra fluid stored in the coolant storage device 150. For example, 5%, 10%, 15%, etc. extra fluid can be stored in the coolant storage device 150. On one hand, more extra fluid increases the cooling capacity to the electronics 128 and increases the time period before such electronics overheat or otherwise shutdown. On the other hand, more extra fluid also increases volume and weight to the cooling system 100.
In one exemplary embodiment, the coolant storage device 150 is a porous mass with a high cooling capacity. For instance, this porous mass includes copper or other materials to absorb or dissipate heat. As an example, a copper matrix or other porous medium is placed in a tank or reservoir (example, coolant storage device 150). The porous or added mass provides additional mass so that the cooling capacity increases for the fluid in the liquid cooling loop 122 and provides additional time for operation during cooling loss to the electronic devices 102 or data center 104. Thus, a porous mass in the coolant storage device 150 provides added cooling capacity for servers, racks, and other electronic devices in the date center 104.
Embodiments in accordance with the present invention are not limited to any particular type or amount of porous material in the coolant storage device 150. On one hand, more porous material increases the cooling capacity to the electronics 128 and increases the time period before such electronics overheat or otherwise shutdown. On the other hand, more porous material also increases volume and weight to the cooling system 100.
In yet another exemplary embodiment, the coolant storage device 150 includes a phase change material. Phase change materials exhibit a phase change and utilize energy to change phase. Such materials or compounds melt and/or solidify at certain temperatures in order to store or release energy. As an example, a thermal phase change material (example, ice to water, waxes and paraffin) or phase change matrix or combination is placed in a tank or reservoir (example, coolant storage device 150). The phase change material increases the cooling capacity for the fluid in the internal liquid cooling loop 122 and provides additional time for operation during cooling loss to the electronic devices 102 or data center 104. Thus, a phase change material in the coolant storage device 150 provides added cooling capacity for servers, racks, and other electronic devices in the date center 104.
Embodiments in accordance with the present invention are not limited to any particular type or amount of phase change material in the coolant storage device 150. Different phase change materials offer different degrees of thermal storage and increases in the time period before such electronics overheat or otherwise shutdown. Typically, large amounts of energy are required for phase change; thus, embodiments in accordance with the present invention provide an increases thermal capacity at a higher density.
Thus the coolant storage device 150 provides continued cooling of electronics even after the building coolant discontinues or stops. The cooling system continues to operate as long as the integrity of internal coolant loop is intact and functioning and as long as the stored coolant is not used up.
In
In one exemplary embodiment, the liquid cooling unit 206 is modular and a self-contained unit. For example, the liquid cooling unit is removable, serviceable, and replaceable into the server 202.
As shown in
As noted, the liquid cooling unit can be modular. As such, the rack 302 can continue to operate while the liquid cooling unit 310 is serviced, replaced, or otherwise repaired. For example, if the primary pump 320 or heat exchanger 324 is temporary shutdown or otherwise fails, the electronics 304 are continued to be cooled within the secondary-liquid cooling loop 330 that extends through the electronics 304, pump 308, and coolant storage device 306. The pump 308 pumps fluid through the electronics within the secondary-liquid cooling loop. The coolant storage device 306 provides the extra coolant, phase change material, or mass needed coolant or fluid necessary to continuously cool the electronics 304 while the liquid cooling unit 310 is serviced, for example.
A modular liquid cooling unit 340 includes a heat exchanger 342 and a primary pump 344. As shown, the liquid supply and return lines 346 connect to both the rack 302 and liquid cooling unit 340. A liquid cooling loop 345 includes a supply line 347 and a return line 348 that circulate fluid to a second rack 360.
The rack 360 includes a coolant storage device 370, liquid cooled electronics 372, and a backup pump 374 all connected via a secondary-liquid cooling loop 376. Plural valves 380 and couplings 382 are used to isolate fluid flow to the electronics 372 along the secondary-liquid cooling loop 376.
In one exemplary embodiment, the primary pump 344 pumps fluid to cool the electronics 372 during normal operations. When servicing or repair is required, dripless couplings 382 are disconnected. Valves 380 are opened and backup pump 374 is activated. Coolant or fluid is circulated through the secondary-liquid cooling loop 376. In turn, the electronics 372 are cooled with the coolant storage device until the stored coolant is used-up. The coolant can store the temperature using, for example, volume of fluid, mass, and/or phase change material as discussed in connection with
As shown in
Exemplary embodiments in accordance with the present invention provide various redundant cooling schemes for computers and computer systems (such as data centers). Some embodiments provide a redundant infrastructure that uses a building's liquid or fluid (i.e., coolant) to the computers or computer systems.
The coolant converters 410A and 410B can utilize a redundant power supply, such as a dual grid power system coupled to the server 400. As shown, the server uses two independent power supplies, a bulk power supply A (430A) and bulk power supply B (430B). Specifically, power supply 430A couples to coolant converter 410A, and power supply 430B couples to coolant converter 410B. Each power supply has an independent power source, shown as alternating current AC source A for power supply 430A and AC source B for power supply 430B. The electronics and pumps of coolant converter 410A are powered from power supply 430A, while the electronics and pumps of coolant converter 410B are powered from power supply 430B.
Thus, the server 400 has both redundant power supplies and redundant cooling systems. In one exemplary embodiment, the coolant converters 410A and 410B are identical. In another exemplary embodiment, the coolant converters are different (example, one is a primary coolant converter and one is a backup coolant converter). Further, one coolant converter is a liquid converter and one coolant converter is or utilizes air cooling. Further, since coolants A and B are independent and input separately, these coolants can be the same (example, both water or both refrigerants) or different.
As shown, coolant from heat generating components (HGCs, such as computers, computer systems, data centers, etc.) is pumped with pump 510 along line 512. The coolant enters unit 513 at an evaporator 514 and circulates through a compressor 515 and a condenser 516. An optional heater 520 is provided along line 522 as the coolant is returned to the heat generating components. Control electronics 530 control power to the coolant converter 500. These electronics detect failures and monitor and control temperatures and flow rates. Data (such as current readings, measurements, status reports, etc.) are viewable on a display or sent (such as a report) to a server manageability system. Further, as shown, an optional pump 532 is provided to pump coolant along a building coolant supply line 534 and building coolant return line 536.
In an alternate embodiment, the coolant converter 500 includes a heat exchanger (shown in place of unit 513). In this alternate embodiment, if supply temperatures are below the desired internal temperatures, the temperatures can still be accurately controlled. Further in some embodiments, a heat exchanger is more efficient than a refrigeration system. Modulating the supply flow rate is an effective way to control server coolant temperatures as well. As before, the optional heater(s) provide additional control if necessary, and the pumps are located on both inlet and outlet sides.
As shown, air from heat generating components (HGCs, such as computers, computer systems, data centers) is drawn with a fan 610 along flow path 612. The air enters unit 613 at an evaporator 614 and circulates through a compressor 615 and a condenser 616. An optional heater 620 is provided along flow path 622 as the cooled air is returned to the heat generating components. Control electronics 630 control power to the coolant converter 600. These electronics detect failures and monitor and control temperatures and flow rates. Data (such as current readings, measurements, status reports, etc.) are viewable on a display or sent (such as a report) to a server manageability system. Further, as shown, an optional pump 632 is provided to pump coolant along a building coolant supply line 634 and building coolant return line 636.
In an alternate embodiment, the coolant converter 600 includes a heat exchanger (shown in place of unit 613). In this alternate embodiment, if supply temperatures are below the desired ambient temperatures, the internal temperatures can still be accurately controlled. Further in some embodiments, a heat exchanger is more efficient than a refrigeration system. Modulating the supply flow rate is an effective way to control server temperatures as well. As before, the optional heater(s) provide additional control if necessary, and the pumps are located on both inlet and outlet sides.
Embodiments in accordance with the invention are connectable to a building coolant system in a variety of ways.
A second option shows a server 710B connected to have a redundant building coolant supply. The server 710B includes two coolant converters 742A and 742B and two power supplies 744A and 744B (such as described in connection with
A third option shows a server 710C connected to have a redundant building coolant supply. The server 710C includes two coolant converters 762A and 762B and two power supplies 764A and 764B (such as described in connection with
Embodiments in accordance with the present invention include many different topologies for providing computers, computer systems, and data centers with redundant coolant, such as coolant provided from the building. One such embodiment uses two coolant loops to chill cold plates for heat generating components in an electronic device (example, a server). These cold plates may be a single cold plate with two isolated cooling loops in the assembly. As an alternative, the cold plate may be two separate cold plates that are in intimate contact with each other or the heat generating component.
Further, embodiments in accordance with the present invention are connectable to a building or coolant system in a variety of ways that are not limited to the three options discussed in connection with
Any one or more of the embodiments in accordance with the present invention can be combined for yet additional embodiments. Further, other cooling techniques can be utilized with embodiments described herein to provide high performance cooling, such as, but not limited to, spray cooling and/or immersion cooling.
Embodiments in accordance with the present invention enable components (example, coolant converters) to be hot swappable. Thus, a rack or server is not required to be turned off or shut down during servicing or repair. Further in some embodiments, couplings are designed to blindmate so that coolant converters simply slide in and slide out when making internal coolant connections (such as connections with a server). The coolant converters are also adapted to be modular and either internal or external to a server. Internal coolant converters require no additional floor space (i.e., in addition to the server or computer itself). Further, by providing dual building coolant supply as well as dual internal coolant loops, no single point of failure exists in the cooling systems.
The electronics or heat-generating components include any electronic component that generates heat during operation (such as, but not limited to, computers, servers, racks, data centers, etc.). For example, heat-generating components include, but are not limited to, one or more of electronic power circuits, integrated circuits (ICs) or chips, digital memory chips, application specific integrated circuits (ASICs), processors (such as a central processing unit (CPU) or digital signal processor (DSP)), discrete electronic devices (such as field effect transistors (FETs)), other types of transistors, resistors, capacitors, transistors, diodes, memories, and/or electronic devices that require heat to be thermally dissipated from the device for the device to operate properly or within a specified temperature range.
Further, although embodiments in accordance with the present invention are generally directed to liquid cooling systems, such systems can also use or combine airflow for cooling. For example, active heatsinks include one or more fans to assist in cooling.
Further as noted, the liquid cooling units or coolant converters can be modular and replaceable. In some embodiments, each unit or converter is an independently-operable unit or module that can be constructed with standardized units or dimensions for flexibility and replaceability for use in the electronic devices. As such, the units or converters can be connected to or removed from the electronic devices (example, a server) without connecting, removing, or replacing other components in the electronic device (example, the heat-generating components, other liquid cooling units, other coolant converters, heat exchangers, etc.). As such, the liquid cooling units or coolant converters can be serviced (example, replaced or repaired) without shutting down or turning off the respective electronic device (example, server housing the unit or converter).
As used herein, the term “module” means a unit, package, or functional assembly of electronic components for use with other electronic assemblies or electronic components. A module may be an independently-operable unit that is part of a total or larger electronic structure or device. Further, the module may be independently connectable and independently removable from the total or larger electronic structure (such as liquid cooling units or coolant converters being modules and connectable to servers in data centers).
While the invention has been disclosed with respect to a limited number of embodiments, those skilled in the art will appreciate, upon reading this disclosure, numerous modifications and variations. It is intended that the appended claims cover such modifications and variations and fall within the true spirit and scope of the invention.