Patent Application
20230309262
With the increased performance and power consumption of high performance computing environments (such as data centers), system designers are continually seeking ways to improve the cooling technology of the underlying electronic components that generate heat.
An immersion bath chamber 103 is fluidically coupled to a coolant distribution unit (CDU) 104 that includes a pump 105 and heat exchanger 106. During continued operation of the electronic components, the liquid’s temperature will rise as a consequence of the heat it receives from the operating electronics. The pump 105 draws the warmed liquid 102 from the immersion bath chamber 103 to the heat exchanger 106. The heat exchanger 106 transfers heat from the warmed fluid to a secondary liquid within a secondary cooling loop 107 that is fluidically coupled to a cooling tower and/or chilling unit 108. The removal of the heat from the liquid 102 by the heat exchanger 106 reduces the temperature of the liquid which is then returned to the chamber 103 as cooled liquid.
In a high computing environment, such as a data center, the respective CDUs of multiple immersion bath chambers are coupled to the secondary loop 107, and, the cooling tower and/or chilling unit 108 removes the heat generated by the electronics within the multiple immersion bath chambers from the data center.
With the increasing performance and corresponding heat dissipation of the electronic components within the immersion bath 102, engineers and technicians are continually seeking ways to improve the efficiency of the thermal transfer from the circuit boards’ respective electronic components to the immersion bath liquid 102.
Ideally, the immersion bath liquid 202 exhibits a high rate of fluid flow through the heat sink fins 212 so that the large amounts of heat generated by the one or more semiconductor chips that are operating within the underlying chip package can be efficiently transferred to the immersion bath 202. Unfortunately, referring to
Firstly, horizontal flow of cooled fluid along the x axis is essentially blocked by the circuit boards 201. Secondly, with the chamber walls, the circuit boards 201 and the circuit boards’ respective electronics and packaging introducing large surface areas that the fluid 202 is to flow over, the fluid 202 experiences viscosity forces that resist its flow throughout the chamber 203, generally (the viscosity forces are proportional to the surface areas of the chamber walls, the circuit boards 201 and the circuit boards’ respective electronics and packaging).
Thirdly, to the extent attempts have been made to induce high velocity currents that run horizontally along the y axis or along the vertical z axis, such attempts have placed various structures and/or components (e.g., baffles, jets) in peripheral regions 213_1, 213_2 outside the circuit boards 201. Unfortunately, with the currents being directed toward the heat sinks 211 from the periphery 213_1, 213_2, the heat sinks 211 can impose flow impedances that cause the currents to flow around the heat sinks 211 rather than through their fins 212.
Finally, the immersion bath liquid 202 has appreciable density and corresponding mass that causes the fluid 202 to experience downward (-z) gravitational forces that act against upward (buoyant) fluid flow in the + z direction. The lack of upward fluid flow, combined with the aforementioned viscous forces, causes flow stagnations and re-circulations within the chamber 203, which, in turn, result in insufficient fluid flow (upward or otherwise) through the heat sink fins 212.
A solution, referring to
Notably, as compared to previous solutions that only direct or increase fluid flow generally over a multitude of components, e.g., from the periphery 213 of an electronic circuit board (as described above with respect to
For ease of illustration, again, only a single heat sink 311 is depicted on the circuit board 301 of
Examples of such pluggable units include a blade server, a CPU unit, an accelerator unit, a memory unit, storage unit, etc. Here, the function of the pluggable electronic unit largely corresponds to the functions that the assembly’s printed circuit board 401 are designed to perform or support (e.g., a computing system in the case of a blade server, one or more multicore processor chips in the case of a CPU unit, one or more accelerators (e.g., one or more neural network chips, artificial intelligence machine learning chips, artificial intelligence inference engine chips, graphics processing unit (GPU) chips, etc.) in the case of an accelerator unit, multiple memory chips (e.g., multiple dual in-line memory modules (DIMMs)) in the case of a memory unit, multiple storage devices (e.g., multiple solid state drives (SSDs)) in the case of a storage unit, etc.).
As observed in
The frame 431 mechanically supports the printed board 401 whereas the cover 432 mechanically protects the printed circuit board 401 and its electronic components from mechanical shocks/blows that can be imparted to the assembly 400, e.g., during insertion/removal of the assembly to/from the immersion liquid. Notably, the frame 431 and cover 432 have various perforations or other openings that allow fluid to enter the space where the printed circuit board 401 resides so that the electronic components can be cooled by fluidic flow through the assembly 400.
As observed in the particular embodiment of
In still other embodiments, as observed in the side view of
In still other embodiments, the flow enhancement structure is formed from a combination of elements that are formed in the cover 432 and individual elements that are mounted to any/all of the cover 432, printed circuit board 401 and frame 431.
In various embodiments, the immersion bath includes a framework having multiple slots that respective pluggable electronic units can plug into. In the particular embodiment of
During installation of a pluggable unit, the pluggable unit is entered into the slot from the top of the immersion bath and pressed downward in the -z direction along the guide rails 546_1, 546_2 until it is plugged into the backplane 541. As part of the installation into the slot, the first fluidic connector 542 is connected to the second fluidic connector 548 thereby fluidically coupling the channel 545 that emanates from the flow enhancement structure to CDU return line 543.
During operation of the particular embodiment of
The fluid continues to flow 544_1, 544_2 upward above the duct 521. Due to suction from the CDU pump and/or gravity, the immersion coolant flows 544_1, 544_2 are drawn into the intake duct opening 547 of the flow enhancement structure 521. The fluid 544_1, 544_2 then flows through the flow enhancement structure 521 (ideally, with a high fluidic velocity).
Here, the flow enhancement structure 521 is designed as a kind of housing that encompasses the fins of the heat sink of a high performance chip package 511 that resides directly beneath the flow enhancement structure 521. With the immersion coolant 544_1, 544_2 flowing at a high velocity through the flow enhancement structure 521, and with the flow enhancement structure 521 confining the currents 544_1, 544_1 to flow through the space between the fins of the heat sink, heat is transferred from the heat sink fins to the immersion bath flows 544_1, 544_1 with high efficiency (low thermal resistance).
Notably, because the high performance semiconductor chip(s) within the chip package beneath the heat sink 511 generate most of the pluggable unit’s heat, as the currents 544_1, 544_2 flow around the flow enhancement structure 521 immediately after injection into the bath, they do not capture significant heat because they flow through/across electronics that generate significantly less heat than the chip(s) in the high performance chip package beneath the heat sink 511. As such, the temperature of the fluid as it enters the intake duct 547 should be relatively cool (the temperature is only slightly warmed than the temperature of the fluid that enters the immersion bath from the CDU).
As discussed above, higher thermal transfer efficiencies from the heat sink fins to the fluid within the flow enhancement structure 521 is achieved with increasing fluid flow velocity through the structure 521. In order to increase the flow rate through the enhancement structure 521, the draw/suction from the CDU pump can be increased.
In other approaches, however, e.g., to avoid excessively powerful/expensive CDU pump equipment, the flow rate through the flow enhancement structure 521 is increased by establishing a sufficiently large height difference 553 between the upper surface of the coolant within the immersion bath chamber 503 (upper liquid free surface 551) and an opening 552 in the CDU return line 543 (lower liquid free surface 552).
Specifically, as observed in
Importantly, the opening 552 in the return line 543 and an opening in the plenum 549 physically connects the lower liquid free surface 552 to the same ambient as the upper liquid free surface 551. With this arrangement, gravity will cause the rate of fluid flow through the enhancement structure 521 to increase as the height difference 553 between the upper and lower liquid free surfaces 551, 552 increases. As such, for example, extremely high flow rates through the flow enhancement structure 521 can be achieved (e.g., without expensive/powerful CDU pump equipment) by setting the opening 552 in the CDU return line 543 and the plenum 549 sufficiently below the immersion bath chamber 503.
It is pertinent to point out that the specific flow patterns 544_1, 544_2 and intake duct arrangement 547 of
Note that electro-mechanical interfaces can also exist at the backplane 541, and/or, can be coupled to the pluggable unit through the upper surface of the immersion bath 551. In various embodiments (as suggested by
Consistent with the discussion of
In various embodiments, one or more traditional CDU return lines are also coupled to the chamber 503 so that less than all of the CDU return flows through the chamber’s flow enhancement structure(s).
Although the embodiments of
Specifically, once generated, the bubbles immediately begin to rise or otherwise flow 651 at a high velocity through the space between the heat sink fins 612. Importantly, gaseous pressure within a bubble increases as the diameter/size of the bubble decreases. Thus, small bubbles (e.g., having a diameter within a range 0.1 mm to 10 mm inclusive) having high internal pressures are essentially “non-deformable” within the liquid. The non-deformable property of the smaller bubbles causes them to exhibit an effective “hardness” that creates a wake behind the bubbles as they flow 651 through the liquid. The wake creates a low pressure region behind the bubbles’ that is filled by surrounding liquid 652. Thus, as the bubbles rise upward, liquid is drawn in behind them, which, in turn, effectively increases buoyant fluid flow through the heat sink fins 612.
Additionally, the bubbles can create buoyancy forces near the surfaces of the fins that effectively thin the viscous (momentum) boundary layer and create a more uniform flow through the fin region (increasing flow rate). Where the bubbles are located in the fin region, they effectively decrease the density of the fluid there and significantly increase the buoyance induced flow. As mentioned above, the bubbles will entrain cold fluid to be nearer to the fin surfaces which significantly reduces the thermal resistance (increases the thermal transport/efficiency). These dynamics can exist not only along fin surfaces but also at the root of the fins which can significantly increase the thermal transport.
Notably, the bubble generation device 621 can be an active device that requires electrical power to operate. As such, there can be power wiring that runs from the device 621 to the printed circuit board 601 that the heat sink’s semiconductor chip package is mounted to. In various embodiments, a first mechanical connector is mounted to the printed circuit board 601 and the wiring is a feature of the bubble generation device 621. The wiring includes a second mechanical connector that is mated to the first mechanical connector. Alternatively, a reverse approach is applied and the wiring is a feature of the printed circuit board 601.
The bubble generation device 621 can be mounted to any of the printed circuit board 601 (e.g., with a standoff), the pluggable unit’s frame, the pluggable unit’s cover, or any combination of these.
In another or combined approach, as observed in
In either of the approaches of
A flow enhancement structure that includes a bubble generation device as described above with respect to
The design and operation of the bubble generation device 621 can vary from embodiment to embodiment. According to one approach, the bubble generation device is a frit device that creates bubbles by injecting air through a roughed three dimensional structure (e.g., small carbon rocks as with a typical aquarium filter).
According to another approach, the bubble generation device 621 is a chamber having holes that receives compressed air. The compressed air flows out of the holes forming a stream of bubbles from each of the holes. According to a further approach, the holes are further defined as the hollowed-out inner region of hollow needles that emanate from the chamber. The bubbles emanate from the openings in the needles (akin to smoke from a plurality of chimneys).
In either of these approaches, bubble generation can be promoted by lining the exposed hole surfaces with a coating that affects the ability of the coolant liquid to wet to these surfaces. For example, if bubbles are created at least in part through the flow of coolant liquid into a hole or needle, the hole surfaces (including the exposed hole surfaces within the needles) can be coated with a material that promotes the wetting of the coolant liquid to the hole surfaces. By contrast, if bubbles are created at least in part by preventing the flow of coolant liquid into a hole or needle, the hole surfaces (including the exposed hole surfaces within the needles) can be coated with a material that inhibits the wetting of the coolant liquid to the hole surfaces.
In yet another bubble generation approach, the bubble generation device 621 is peristaltic pump. For example, the bubble generation device 621 includes a hollow chamber covered by a porous membrane. Within the chamber is a cam that continually rotates. The irregular shape of the cam causes air to be compressed against the inner surface of the membrane during a first portion of the cam’s rotation. The compression of the air against the inner surface of the porous membrane causes bubbles to be emitted from the outer surface of the membrane into the surrounding liquid. During a second portion of the cam’s rotation, a new intake of air rushes to the membrane’s inner surface. The process then repeats. In further embodiments, the membrane is elastic. During the second rotation described above, the cam stretches the membrane outward to enhance the air rush intake draw to the inner surface of the membrane.
In another, similar bubble generation approach, bubbles are generated through mechanical agitation (e.g., bending, vibration, etc.). Here, the agitation causes two phases: a first phase in which air is compressed within a chamber and emitted as bubbles from holes in the chamber, and, a second phase in which air is drawn into the chamber. The different phases can correspond, for example, to opposite amplitudes of a periodic mechanical motion (e.g., the chamber is bent in a first direction in the first phase and bent in a second, opposite direction in the second phase). The source of the mechanical agitation can be, e.g., a piezo-electric device, an electro-magnetic device (e.g., a speaker), etc.
In another approach, a multiphase (two phase) flow of air and fluid is created. Then, the multiphase flow is passed through a structure that chops the air of the flow into bubbles that are released into the coolant liquid. For example, a first fluidic channel that is coupled to the coolant fluid and a second fluidic channel that is coupled to a source of external air are joined before a pump which draws flows from both fluidic channels to create the multiphase flow. As the multiphase flow flows through the pump, the pumping action of the pump “chops” the air into bubbles. The flow exits the pump into a chamber with holes. The bubbles pass through the holes into the liquid coolant.
In an alternate approach, the multiphase flow is created by pumping only the coolant liquid through the pump. A venturi device is coupled downstream from the pump output. A venturi device is a fluidic channel having a first section with a wider flow cross-sectional area than a second section with a narrower flow cross-sectional area. In the second section, the fluid’s velocity increases and pressure decreases. An air intake is coupled to the second section, which, having low pressure draws the air into the second section to create a multiphase flow. The high velocity of the fluid in the second section also chops the air into small bubbles. The multiphase flow from the venturi device then passes into a chamber with holes as described above.
In yet another approach, the bubble generation device includes a heat generator that nucleates bubbles by heating the coolant liquid above its boiling point.
With the bubble generation device 621 being an active component, its use can be modulated as part of a larger power management scheme for the pluggable unit it is a component of, and/or, the larger immersion bath that the device 621 is immersed within. For example, to the extent that the rate at which bubbles are generated can be adjusted (e.g., by compressing more or less air into a chamber, adjusting the rotation action of a cam or the pumping action of a pump, adjusting the heat used to nucleate bubbles), adjustments in bubble rate are made, e.g., in view of the power consumption and/or usage of the semiconductor chip(s) within the chip package whose heat sink the bubble generation device 621 is positioned to generate bubbles for the fins of.
Thus, for instance, at one extreme when the chip(s) are in a sleep mode, there is no bubble generation activity, at another extreme when the chip(s) are in a maximum power/usage mode there is maximum bubble generation activity, and, when the chip(s) are operating midway between these two extremes there is bubble generation activity that is less than maximum bubble generation activity. Thus, the aforementioned wiring of the bubble generation device can include or be responsive to control software programs that are executing, e.g., on a processor on the pluggable unit’s circuit board, or some other computer or CPU system.
In the teachings above, note that the heat sink fins have been drawn as needle-like structures that emanate upward from the base of the heat sink. See, for example, fins 312 of
Networked based computer services, such as those provided by cloud services and/or large enterprise data centers, commonly execute application software programs for remote clients. Here, the application software programs typically execute a specific (e.g., “business”) end-function (e.g., customer servicing, purchasing, supply-chain management, email, etc.).
Remote clients invoke/use these applications through temporary network sessions/connections that are established by the data center between the clients and the applications. A recent trend is to strip down the functionality of at least some of the applications into more finer grained, atomic functions (“micro-services”) that are called by client programs as needed. Micro-services typically strive to charge the client/customers based on their actual usage (function call invocations) of the micro-service application.
In order to support the network sessions and/or the applications’ functionality, however, certain underlying computationally intensive and/or trafficking intensive functions (“infrastructure” functions) are performed.
Examples of infrastructure functions include encryption/decryption for secure network connections, compression/decompression for smaller footprint data storage and/or network communications, virtual networking between clients and applications and/or between applications, packet processing, ingress/egress queuing of the networking traffic between clients and applications and/or between applications, ingress/egress queueing of the command/response traffic between the applications and mass storage devices, error checking (including checksum calculations to ensure data integrity), distributed computing remote memory access functions, etc.
Traditionally, these infrastructure functions have been performed by the CPU units “beneath” their end-function applications. However, the intensity of the infrastructure functions has begun to affect the ability of the CPUs to perform their end-function applications in a timely manner relative to the expectations of the clients, and/or, perform their end-functions in a power efficient manner relative to the expectations of data center operators. Moreover, the CPUs, which are typically complex instruction set (CISC) processors, are better utilized executing the processes of a wide variety of different application software programs than the more mundane and/or more focused infrastructure processes.
As such, as observed in
As observed in
Notably, each pool 701, 702, 703 has an IPU 707_1, 707_2, 707_3 on its front end or network side. Here, each IPU 707 performs pre-configured infrastructure functions on the inbound (request) packets it receives from the network 704 before delivering the requests to its respective pool’s end function (e.g., executing software in the case of the CPU pool 701, memory in the case of memory pool 702 and storage in the case of mass storage pool 703). As the end functions send certain communications into the network 704, the IPU 707 performs pre-configured infrastructure functions on the outbound communications before transmitting them into the network 704.
Depending on implementation, one or more CPU pools 701, memory pools 702, mass storage pools 703 and network 704 can exist within a single chassis, e.g., as a traditional computing system (e.g., server computer). In a disaggregated computing system implementation, one or more CPU pools 301, memory pools 702, and mass storage pools 703 are, e.g., separate pluggable electronic units (e.g., pluggable CPU units, pluggable memory units (M), pluggable mass storage units (S)). Although not depicted in
Notably, a traditional computing system and/or any of the above mentioned pluggable units can be mechanically configured to be immersed in an immersion bath where the mechanical configuration is designed to increase fluid flow in the vicinity of a heat sink and/or a heat sink’s fins as described at length above.
In various embodiments, the software platform on which the applications 705 are executed include a virtual machine monitor (VMM), or hypervisor, that instantiates multiple virtual machines (VMs). Operating system (OS) instances respectively execute on the VMs and the applications execute on the OS instances. Alternatively or combined, container engines (e.g., Kubernetes container engines) respectively execute on the OS instances. The container engines provide virtualized OS instances and containers respectively execute on the virtualized OS instances. The containers provide isolated execution environment for a suite of applications which can include, applications for micro-services.
Embodiments of the invention may include various processes as set forth above. The processes may be embodied in program code (e.g., machine-executable instructions). The program code, when processed, causes a general-purpose or special-purpose processor to perform the program code’s processes. Alternatively, these processes may be performed by specific/custom hardware components that contain hard wired interconnected logic circuitry (e.g., application specific integrated circuit (ASIC) logic circuitry) or programmable logic circuitry (e.g., field programmable gate array (FPGA) logic circuitry, programmable logic device (PLD) logic circuitry) for performing the processes, or by any combination of program code and logic circuitry.
Elements of the present invention may also be provided as a machine-readable medium for storing the program code. The machine-readable medium can include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, FLASH memory, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards or other type of media/machine-readable medium suitable for storing electronic instructions.
In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
| Number | Date | Country | Kind |
|---|---|---|---|
| PCT/CN2023/093755 | May 2023 | WO | international |
The present application claims the benefit of priority to Patent Cooperation Treaty (PCT) Application No. PCT/CN2023/093755, filed May 12, 2023, the entire content of which is incorporated herein by reference.
