Patent Application
20230375279
This invention relates to passive coolant distribution units (pCDU), and in particular apparatus, devices, systems, and methods for providing a passive coolant distributions unit (pCDU) to promote refrigerant flow circulation, manage and monitor refrigerant inventory in a closed loop system.
Anthropogenic gas emissions are shifting the Earth's global energy balance, thereby resulting in climate change. By 2050, global warming is estimated to result in one-third of the animal and plant species to extinction, while inevitably shaving 11-14% off global economic output, which equates to approximately 23 trillion dollars. See Swiss Re Institute April 2021 report. https://www.swissre.com/dam/jcr:b257cfe9-68e8-4116-b232-a87949982f7c/nr20210421-ecc-publication-en.pdf
To combat these grim projections and to avoid irreversible economic and environmental consequences, countries have introduced carbon credit to incentivize enterprises to reduce or capture carbon emissions.
Despite strong economic incentives to lower carbon emission, most data centers and telecom central offices are designed and still operate with air-cooling technologies, and their energy consumption to cool hardware equipment is on average 40% of the total energy consumed in the facility. Furthermore, the waste heat is in the form of hot air with low energy density, which requires costly air handling systems and infrastructure upgrades to utilize that heat for other applications.
With growing awareness of social responsibility to combat climate change, there is a global consensus for action to improve energy efficiency and reduce carbon emissions across all industries. Hence, it is extremely important to develop novel high-efficiency cooling technologies along with amenable heat capture/reuse technologies, which satisfy sustainability mandates as well as market adoption.
Moving away from conventional air-based cooling technology, liquid/refrigerant-based cooling technology, which has superior cooling performance compared to that of conventional air-based technology, is gaining traction. Herein, we disclose methods, systems, and apparatus to optimize novel cooling systems to achieve next-level thermal performance while reaching sustainability goals through smart energy management solutions.
Heat Exchangers in Coolant Distribution Unit (PDU) for Liquid Cooling
A heat exchanger is part of a liquid-based electronics cooling system. In implementations, a heat exchanger is enclosed together with a pump unit that is fluidically coupled with a heat exchanger. A pump unit that drives coolant throughout a cooling system and removes heat from electronic equipment. Depending on the system implementation, the heat exchangers can reject the heat to air or liquid.
For example, U.S. Pat. No. 9,668,382 to Steinke et al., which is incorporated by reference, discloses a coolant distribution unit for a multi-node chassis. In this implementation, an air-to-coolant heat exchanger is in fluid communication between the inlet conduit and the outlet conduit, and a pump circulates a coolant throughout the system.
Similarly, U.S. Pat. No. 8,297,069 to Novotny et al., which is incorporated by reference discloses a coolant distribution unit comprising a liquid-to-liquid heat exchanger and a pump that drives a coolant in the system.
Invariably a heat exchanger for liquid cooling system requires at least one pump to drive a coolant. Features that enable pumpless coolant distribution unit however are not disclosed.
IoT System/Subsystem Implemented for Managing Infrastructure and Improving Overall Efficiency
IoT devices are becoming a technology of choice to help enterprises reduce carbon emissions by adding intelligence to optimize operations. Increasingly important to sustainable industrial practice, the global IoT market is projected to reach 3.9 trillion to 11.1 trillion by 2025. See https://www.mckinsey.com/˜/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/The%20Internet%20of%20Things%20How%20to%20capture%20the%20value%20of%20IoT/How-to-capture-the-value-of-IoT.pdf
In essence, IoT devices are network-enabled hardware that can transmit/receive data over the internet, and they can be embedded into a wide range of industrial equipment, sensors, actuators, or appliances to optimize industrial processes. Recent advances in cloud-based technology and machine learning (AI/ML) are being seamlessly integrated with IoT devices to execute autonomous tasks and provide intelligent analytics. Along with advances in 5G technology, IoT devices will lead the paradigm shift towards sustainable Industry 4.0 ecosystem, which encompasses holistic smart manufacturing, supply chain management based on big data to achieve sustainability targets.
Liu and others (See Q Liu, et. al., Green data center with IoT sensing and cloud-assisted smart temperature control system, Computer Networks, Volume 101, 2016) disclosed a method to build green data centers utilizing remote sensor networks (
Similarly, Evans et al. disclosed methods, systems, and apparatus (See U.S. Pat. No. 10,643,121 to Evans et al., which is incorporated by reference) for improving operational efficiency within a data center by modelling data center performance and predicting power usage efficiency (
A group from Chengdu Qinchuan Technology Development filed a patent application (See China Patent 102496209 to Zehua Shao titled Intelligent object network heat meter and management system.
The patent application discloses methods, systems and apparatus that are simple in structure, functionally realized based on the technique of internet of things, suitable for being mounted and used in various areas, and wide in application range. In essence, this is an IoT-enabled heat meter comprising a CPU controller, temperature, and flow measurement apparatus.
U.S. Pat. No. 8,706,308 to Reichmuth et al., which is incorporated by reference, disclosed methods, systems, and apparatus comprising one or more energy load meters coupled with a facility's energy supply system. The invention enables accurate modelling of past and future use, measuring current use and savings, diagnosing current problems, and making adjustments where sensible, and monetizing energy savings.
U.S. Pat. No. 10,328,408 to Victor et al., which is incorporated by reference, disclosed methods, systems, and apparatus comprising a heat exchanger along with many other components in chemical/petrochemical plants and one or more sensors associated with the heat exchanger.
A plant or refinery may include equipment such as reactors, heaters, heat exchangers, regenerators, separators, and others. Types of heat exchangers include shell and tube, plate and shell, plate fin, air cooled, wetted-surface air cooled, and others. Operating methods may impact deterioration in equipment condition, prolong equipment life, extend production operating time, or provide other benefits. Mechanical or digital sensors may be used for monitoring equipment, and sensor data may be programmatically analyzed to identify developing problems. For example, sensors may be used in conjunction with one or more system components to detect and correct maldistribution, cross-leakage, strain, pre-leakage, thermal stresses, fouling, vibration, problems in liquid lifting, conditions that can affect air-cooled exchangers, conditions that can affect a wetted-surface air-cooled heat exchanger, etc. An operating condition or mode may be adjusted to prolong equipment life or avoid equipment failure.
The use of IoT devices, cloud technology, AI and machine learning for system-level monitoring and optimization is pervasive in open literature. For example, U.S. Pat. No. 11,416,739 to Qin, which is incorporated by reference, discloses a simulation model based on conditions associated with a building infrastructure to optimize the overall energy efficiency.
None of the above prior art discloses enabling features to build a coolant distribution system supporting active and passive two-phase, and liquid-base, cooling systems. Also, none of the above prior art discloses a system that provides information related to the amount of waste thermal energy being captured from ICT systems from a coolant distribution system. Furthermore, none of the prior art discloses methods, systems, and apparatus to obtain data from multiple cooling systems collectively to enable holistic understanding of the overall system. These enabling features will incentivize data centers and telecom operators to adapt the disclosed invention to achieve sustainability targets.
The prior art does not disclose hardware and software design features that are critical to build a passive coolant distribution unit (pCDU), which is extremely challenging without pumps that drive flow circulation in a thermosyphon loop. Implementing passive coolant distribution unit requires numerous novel features.
Thus, the need exists for solutions to the above problems with the prior art.
A primary objective of the present invention is to provide a passive coolant distribution unit (pCDU), and in particular apparatus, devices, systems, and methods for providing a passive coolant distributions unit (pCDU) to promote refrigerant flow circulation, manage and monitor refrigerant inventory in a closed loop system.
A secondary objective of the present invention is to provide passive coolant distribution unit (pCDU) that is designed to promote refrigerant flow circulation, manage and monitor refrigerant inventory in the closed loop system, reject heat to the facility service lines, such as chilled water or any other air/liquid loop, and collect data using IoT-based sensor technology and analytics.
The invention provides a promising thermal energy management approach is the extension of passive two-phase heat transfer to rack scale. Thermosyphon loops (TL) coupled with high-performance evaporators and condensers enable a scalable rack cooling system that operates with high thermal performance. This is achieved without the need for active pumping/control and operates within a hybrid cooling paradigm complimenting the significant investment in current air-cooling technology. This cooling technology provides key differentiators with quantifiable advantages including higher revenue for datacenter/telecom operators, significant cooling energy reduction, lower carbon footprint, higher reliability, easy serviceability, zero-maintenance using eco-friendly, low-cost cooling fluids, extended product life cycle and ease of implementation into existing and new datacenter/telecom sites.
Furthermore, passive two-phase cooling technologies, using zero ozone depleting potential, low global warping potential refrigerants as working fluids, provide excellent key metrics (e.g., heat density, efficiency, reliability, sustainability etc.) and represent a viable, long-term solution with regards to hardware densifications and thermal performance required by next-generation telecommunications and computing environments. It is also extremely important that cooling technology is highly efficient, which will address positive environmental & sustainability (ESG) requirements, for wider technology adoption.
Two-phase cooling systems can be implemented in passive mode (e.g., thermosyphons where gravity/buoyancy is the driving force) or in active mode with mechanical drivers (e.g., pump(s) allowing for fluid flow circulation). Here, we disclose features for a (semi)passive coolant distribution unit, which can be adapted for both active and passive two-phase cooling systems.
This patent application discloses a passive coolant distribution unit (pCDU) that is designed to promote refrigerant flow circulation, manage and monitor refrigerant inventory in the closed loop system, reject heat to the facility service lines, such as chilled water or any other air/liquid loop, and collect data using IoT-based sensor technology and analytics. Data stream from pCDU can assist end users to monitor, manage, optimize, and detect anomalies during the operation of thermal management systems. In one implementation, users can quantify energy flows from the hardware being cooled to the secondary heat rejection side to address sustainability needs by tracking capture efficiency, heat rejection temperature and CO2 footprint. F-gas accounting system whereby the system collects and stores data regarding the system refrigerant charge. System interlock that requires the servicing technician to input charge mass during initial charging and any subsequent re-filling.
The prior art does not disclose hardware and software design features that are critical to build a passive coolant distribution unit (pCDU), which is extremely challenging without pumps that drive flow circulation in a thermosyphon loop. Implementing passive coolant distribution unit requires numerous novel features.
An embodiment of a passive coolant distribution unit (pCDU) system, that includes: a heat exchanger, a coolant distribution manifold, a fluid inventory management system between the heat exchanger and the coolant distribution manifold, for allowing condensed liquid to accumulate in a fluid inventory storage unit, a supplementary positive displacement pump downstream to the fluid inventory storage unit, and sensors selected from at least one temperature sensor and at least one pressure sensor, the sensors providing a feedback loop for maintaining a liquid level in the fluid inventory storage unit.
The system can further include a chassis for housing the pCDU system, the chassis having detachable brackets for installing the chassis adjacent to a computer server rack.
The system can further include a second coolant distribution manifold, and quick connects for adding the second coolant distribution manifold to the passive coolant distribution unit (pCDU) system.
The system can further include a pump unit for providing a thermosyphon loop in the system.
The pump unit can be installed fluidically in parallel to a main liquid coolant flow path in the system.
The system can further include a backflow preventer to stop pumped liquid from returning to the fluid inventory storage unit.
The fluid inventory management system can include a level sensor that ensures coolant level is maintained at a selected level, to prevent fluid cavitation in the pump unit.
The fluid inventory management system can include a modular and stackable liquid accumulator.
The system can further include a reserve tank fluidically connected to the fluid inventory management system, and a valve for allowing coolant fluid to be transported to a main accumulator in the fluid inventory management system.
The system can further include a leak detection monitor, for detecting sudden drop in temperature from temperature sensors in the system, which constitutes a major leak, and a safety isolation valve triggered by the leak detection monitor to contain coolant from leaking out.
The system can further include a leak detection monitor for detecting data from chemical sensors for detecting coolant leak from the system, the chemical sensors, selected from at least one of metal oxide, Infrared, and MEMS-based sensors, the chemical sensors are useful to detect minor leaks which include small leaks (mostly due to diffusion of coolant molecules through polymer seals) that does not immediately affect overall system performance.
The system can further include a control module for actuating an isolation valve when a major leak, which includes a sudden drop in temperature is detected in the system to contain coolant from leaking out, and an emergency alert generated from the control module to inform system operators of the major leak detection to provide a shut down of the system.
The system can further include an air ingress management control for separating air ingresses into the system, and accumulating the separated air ingresses into a second fluid inventory storage unit.
The system can further include data collected from a PDU (Power Distribution Unit) and the heat exchanger to calculate rack-level heat capture efficiency, and storage of the data in a database for storing energy in an out of the system for an end-user to monitor and account for carbon offset credit (sustainability accounting).
The system can further include server start-up detection and heater block starter feature
The system can further include flow stabilization methods during start-up (with PCDU)
The system can further include load balancing in heat reuse applications, where the demand load fluctuates in time, to ensure sufficient heat rejection from the target equipment to be cooled, e.g., IT equipment, servers.
The system can further include a water leak detection monitor, for detecting accidental water leaks in or around a heat exchanger.
Further objects and advantages of this invention will be apparent from the following detailed description of the presently preferred embodiments which are illustrated schematically in the accompanying drawings.
The drawing figures depict one or more implementations in accord with the present concepts, by way of example only, not by way of limitations. In the figures, like reference numerals refer to the same or similar elements.
Before explaining the disclosed embodiments of the present invention in detail it is to be understood that the invention is not limited in its applications to the details of the particular arrangements shown since the invention is capable of other embodiments. Also, the terminology used herein is for the purpose of description and not of limitation.
In the Summary above and in the Detailed Description of Preferred Embodiments and in the accompanying drawings, reference is made to particular features (including method steps) of the invention. It is to be understood that the disclosure of the invention in this specification does not include all possible combinations of such particular features. For example, where a particular feature is disclosed in the context of a particular aspect or embodiment of the invention, that feature can also be used, to the extent possible, in combination with and/or in the context of other particular aspects and embodiments of the invention, and in the invention generally.
In this section, some embodiments of the invention will be described more fully with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout, and prime notation is used to indicate similar elements in alternative embodiments.
Other technical advantages may become readily apparent to one of ordinary skill in the art after review of the following figures and description.
It should be understood at the outset that, although exemplary embodiments are illustrated in the figures and described below, the principles of the present disclosure may be implemented using any number of techniques, whether currently known or not. The present disclosure should in no way be limited to the exemplary implementations and techniques illustrated in the drawings and described below
Unless otherwise specifically noted, articles depicted in the drawings are not necessarily drawn to scale.
In a typical implementation, a passive coolant distribution unit consists of multiple sub-components, such as a heat exchanger with sensors for energy metering, two or more pairs of quick connects for making fluidic connections, fluid inventory management system, optional pump to supplement gravitational force for coolant circulation for coolant storage, IoT-system comprising sensors, microcontroller, firmware and network module for data processing and data input/output through the network, and back-end software to provide additional software features, such as data center monitoring and optimization.
A list of the components will be described.
A list of abbreviations in the application will now be defined.
A list of component numbers in the figures will now be described.
The main function of a heat exchanger 805 is to transfer heat from IT equipment to facility-level cooling infrastructure. Heat exchanger technology can include, plate heat exchangers, tube-in-tube heat exchangers, shall and tube heat exchangers or any other type of heat exchanger.
A passive coolant distribution unit requires extra care when sizing the heat exchanger 805 to ensure pressure drop is not prohibitively high for passive coolant circulation.
When forming a fluidic connection between the heat exchanger 805 with a flow distribution unit (i.e., manifold) 802, pressure drop across connecting pipes/tubes should be carefully designed in a passive thermosyphon system to ensure excessive pressure drop does not negatively impact the flow circulation.
To ensure proper flow circulation with desired direction, tubing/hose diameter with different sizes can be used to make fluidic connection between the heat exchanger 805 and rest of the system. A heat exchanger 805 can be equipped with additional sensors 808, such as temperature measuring probes and flow meters to measure amount of energy flow through the heat exchanger.
Referring to
A port (e.g., Schrader valve) for charging or evacuating coolant could be introduced in any location along the tubing or hose within or outside of an enclosure. Quick connects for easily making or breaking fluidic connection can be installed on or outside of an enclosure for easy installation. In a typical implementation, there would be a pair of quick connectors for coolant side and a second pair of quick connectors for facility coolant side (e.g., chilled water line). Optionally, additional pairs of quick connects can be implemented to expand the cooling system in a modular fashion.
A cubical, rectangular, or irregular shaped container with mechanical specifications that allows storage of coolant storage is installed inside a pCDU enclosure. In a typical implementation, a fluid inventory management system 804 is installed between a heat exchanger 805 and a coolant distribution manifold 802 in such a way that condensed liquid accumulates in the fluid inventory storage unit instead of stagnating inside a heat exchanger 805.
A pump unit, such as a positive displacement pump, can be optionally added to the system inside an enclosure of pCDU system. The pump unit is installed downstream of a liquid accumulator to ensure the inlet side of the pump unit is always in liquid phase. In a typical implementation, a pump unit is fluidically connected in parallel to the rest of the system, and mechanical displacement of a coolant supplements gravitational driving force that drives coolant circulation. Additional features may include passive/active hydraulic components to prevent backflow in the system. During operation, a pump unit can stay dormant to save energy, and a pump is only activated when supplemental driving force is encouraged to improve the overall performance of a cooling system. A control loop mechanism may be implemented to further optimize the system operation.
A microcontroller is a central component that orchestrates a data streaming process. In a typical implementation, several modules, such as power module, network modules, energy metering modules, and analog/digital circuits enabling temperature sensors and mass flow meters are populated on a same PCB with a microcontroller chip.
The PCB can have multiple I/O ports for sensor heads/probes that can provide accurate measurements, which in turn can be processed in the microcontroller and exposed for end users via standard data transfer protocol, such as HTTP API or SNMP.
In a typical implementation the pCDU system can be equipped with a network module that enables wired (e.g., Ethernet) and/or wireless connectivity (e.g., WIFI or BLE) to the local/remote network. The microcontroller can be flashed with a firmware that can be updated with manual programming or updated remotely Over-the-Air (OTA). One of major functions of the firmware is hosting a webserver that allows end users to provision the pCDU.
Backend software for pCDU could provide a system monitoring platform, which can include a GUI based dashboard for visualizing real-time data streams. The backend database can communicate with a machine learning module to provide further analytic insights, such as projections, estimates, or predictions, all of which can be derived based on sensor measurements from single or multiple pCDU units.
It is important to monitor hydraulic and thermal performance of a liquid cooling system for electronics cooling application. In a mission critical infrastructure (i.e., data centers), real-time monitoring and anomaly detection is critical to minimize system down time and early maintenance. It is even more critical in a passive coolant distribution system (i.e., gravity driven thermosyphon loop) to have real-time telemetry to ensure robust coolant circulation within the system without a pump unit.
In a representative implementation, a pCDU hardware will have a unique identifier in such a way that each pCDU hardware can become a node in a local area network.
Through this local network, sensor readings and other industry standard data stream sources, such as Redfish, IPMI or SNMP can be ingested for backend software features. Optionally, these datasets can be pre-processed and packaged to payloads in JSON format, which in turn is wirelessly broadcast via standard IoT protocols (e.g., MQTT, AMQP, WebSocket etc.).
Payloads ultimately reaches a local or cloud server for further data consumption and analytics. Light-weight analytics could be done on a microcontroller-layer (e.g., embedded AI/ML Artificial intelligence and Machine Learning), while more complex analysis would be typically carried out at the backend.
The backend software ingests data stream from pCDU as well as other IT equipment to analyze the operational status of a data center, such as temperature or energy usage.
Software features based on real-time telemetry data can be integrated with other software platforms such as data center infrastructure management (DCIM) software. Furthermore, a software feature can analyze the data and update operational parameters in a data center accordingly to maximize the overall energy efficiency, or any other relevant figure of merit (FoM) of the system.
Detachable Universal Installation Hardware
To take advantage of gravitational force for driving coolant circulation, a passive coolant distribution unit (pCDU) is installed above IT equipment.
A chassis including various sub-components such as a heat exchanger, sensors, microcontroller, metering modules, and quick connectors, can be mounted across various locations above IT equipment.
For example, a chassis can be installed inside a server rack (rack-mount), on top of a server rack (top of the rack), on a structural ceiling (above the rack), inside plenum space (remote installation), or even on a building roof.
For rackmount deployment, a pCDU 907 can be installed at the highest point within the rack 903. Placing the chassis outside of the rack 903 would enable IT operators to add more computing units to maximize the compute density in the server rack 903. Also moving the coolant distribution unit outside of the IT envelope removes facility side chilled water to a remote location. Above the rack installation 904 is attached to a server rack 900.
A remote pCDU 905 can be installed on a suspended ceiling structure, such as Unistrut 906 to utilize the space above server racks 900. A rooftop pCDU 907 can be placed outside of a data center building 908.
To ensure a pCDU chassis is pre-designed to accommodate flexible deployment scenarios, threaded holes and mounting accessories can be mounted across multiple locations throughout the chassis.
The invention can make use of detachable universal hardware for flexible deployment and installation of pCDU (1000). A pCDU chassis 1000 can have multiple tapped screw holes 1001 for detachable mounting hardware 1002, 1003, and 1004.
For rack-mount installation, mounting brackets 1002 can be installed on the side of a pCDU chassis. For installation using suspended ceiling structure, a pCDU can hang on a rigid structure via a bracket 1004. For above the rack or top of the roof installation, a bracket 1003 can be installed.
IT equipment in data centers continuously undergoes refresh cycles to upgrade computational capability. A liquid based cooling system would have to be flexible to accommodate these continuous changes, while minimizing server downtime arising from refrigerant recovery, charging, and rearranging fluidic connections.
One of the strategies to minimize such downtime is to avoid having to recover refrigerants when only a subcomponent, such as a coolant distribution manifold, needs to be replaced due to increased power output requirements. This can be achieved by having a second set of quick connect couplings on a pCDU.
A pCDU 1100 can be provisioned with multiple servers 1101, a coolant distribution unit 1102 and a pair of quick disconnectors 1104 all of which are fluidically connected. For ease of maintenance and second pair of quick connectors. A redundant manifold can be installed during server operation. Once all servers are connected to a new coolant distribution manifold, the old manifold can be removed. This procedure minimizes server downtime during this manifold replacement operation.
Similarly, when the system requires a heat exchanger with a bigger capacity, a second pCDU can be added through the second pair of quick connect to increase the capacity.
A pCDU 1200 can be fluidically connected with a coolant distribution manifold 1201 and multiple servers 1202. The first pair of quick connectors 1203 are used for making connection between a pCDU 120 and a coolant distribution manifold 1201. A redundant pair of quick connectors 1204) can be used for adding a second pCDU 1205 for increasing cooling capacity or increasing internal volume for increased coolant storage capacity in the thermosyphon loop system. Multiple pCDU units can be daisy-chained.
With reference to
In a typical implementation, a liquid accumulator installed at the outlet side of a heat exchanger, where excess liquid is stored and maintained for providing hydraulic static pressure. This hydrostatic pressure is primarily responsible for driving the coolant circulation inside a thermosyphon loop based system. Having a liquid accumulator also prevents entrainment of gas-phase coolant from entering a pump unit that has negative impact.
A positive displacement pump can be installed in parallel at the outlet of liquid accumulator, which will ensure that a pump will always receive liquid with substantial inlet pressure to prevent cavitation.
When a positive displacement pump is not active, the hydraulic path through the pump is closed, and all liquid will flow through a bypass. When a pump is inactive, coolant circulation can be solely driven by gravity in a thermosyphon based system (“gravity only mode”). Once a pump is turned on and becomes active, liquid flow will be driven by gravity as well as mechanical force. In a typical implementation, the liquid bypass requires a backflow preventing mechanism to stop pumped liquid from returning to the liquid accumulator and the pump suction side. A passive component, such as a check-valve, or an active component, such as a three-way valve or solenoid valve can be deployed to achieve the required functionality.
Referring to
A branching point, which can be a simple tee or a 3-way valve, 1305 can be added to the downcomer to create a parallel path 1306. A pump unit, for example a positive displacement pump unit, 1307 can be installed in-line with the parallel fluidic path 1306.
When pump unit 1307 is turned off, a parallel fluidic path along 1306, 1307 is blocked and coolant is driven only by gravity. When a pump becomes active, coolant flows through the pump via pump-assisted gravity driven flow.
To prevent undesired backflow, a backflow preventing device 1308 can be added to the system. The back flow preventing device can be an active on/off valve, or passive device, such as check valve with low activation pressure differential. Alternatively, a branching point 1305 can be a 3-way valve that can be actuated with external trigger.
An additional control system can be implemented to maximize energy efficiency by turning off a pump unit when not needed. Also, the pump speed can be optimized based on input data from sensors. In a typical implementation, temperature and/or pressure sensors can be deployed to obtain necessary information which can be used for constructing a feedback loop. For example, liquid subcooling, i.e., the temperature difference between the coolant inlet and outlet temperatures at the heat exchanger, can be used for feedback to optimize the volumetric flow rate of the system via controlling the pump speed. When subcooling is small the pump operates at low speed
Referring to
The pressure and/or temperature data can be transmitted to a controller 1402 which determines the ideal pump speed of a positive displacement pump 1403 and sends control signal. If a branching point 1404 and back-flow preventing device 1405 are implemented with actuatable components, a controller unit 1402 also sends control signal to these components.
A pump control system that optimizes pump speed based on predicted coolant mass flow rate, which is derived from machine learning model using input features from at least one of IT equipment and a PDU (Power Distribution Unit).
Coolant mass flow rate can be approximated using CPU power consumption as an input. CPU power consumption can be measured or predicted.
In a typical implementation, absolute pressure or subcooling temperature or both can be maintained throughout the thermosyphon operation by adjusting the pump speed to ensure the onset of coolant boiling is optimized. A control system is also utilized to ensure pump unit can rest as much as possible to extend the lifetime of the pump.
In an alternative implementation, a pump with an impeller rotating in a concentric channel can be installed in-line with a downcomer. For example, a peripheral pump, also known as regenerative turbine pump, can be implemented.
Since a regenerative turbine pump does not block the coolant flow when turned off, the system implementation can be simplified without a parallel flow path for installing a pump unit. When the system operates under “gravity-mode” the pump is turned off and coolant freely flows through a pump.
When pump is in operation, the control system will evaluate if the pump can be turned off, based on the system performance, to achieve maximize energy efficiency. If a thermosyphon loop can sustain robust coolant circulation without pump, the pump will be turned off to save energy and extend lifetime of the pump. Also, the pump speed can be optimized based on input data from sensors. In a typical implementation, temperature and/or pressure sensors can be deployed to obtain necessary information which can be used for constructing a feedback loop. For example, a subcooling temperature could be used as a proxy to optimize the volumetric flow rate of the system via controlling the pump speed.
When the system runs on “gravity-mode” the pump unit is turned off, while coolant freely flows through the pump unit via gravity. When additional static pressure is necessary to improve the thermosyphon system performance, a pump unit can be activated to add additional static pressure to increase the overall coolant mass flow rate in the system.
Coolant Inventory Management
Optimum amount of coolant must be maintained in a passive thermosyphon system to ensure proper operation of the system. A coolant inventory management consists of a reservoir (“liquid accumulator”) that is placed above an evaporator to ensure enough gravity driven force proportional to the height difference is generated.
Also, the reservoir is located below the outlet of a heat exchanger in such a way that condensing coolant from heat exchanger is drained properly into the reservoir. When a pump unit is implemented, the liquid accumulator is positioned above the pump to ensure that liquid is always flowing through the inlet port of a pump unit to prevent fluid cavitation.
In a typical implementation, a heat exchanger/condenser 1601 and one or more evaporators 1602 are fluidically connected with a downcomer pipe 1603 and a riser pipe 1604. When heat is added to evaporator 1602, liquid coolant is displaced and could reach a heat exchanger, leading to non-ideal operating condition with lower heat transfer coefficient.
To prevent the condenser from “flooding”, a component with substantial internal volume, or “liquid accumulator” 1605 can be installed in-line with a downcomer pipe 1603. To ensure proper drainage of condensing liquid to a liquid accumulator, the liquid accumulator can be positioned lower than the outlet port of a heat exchanger, yet ensuring enough distance between evaporator 1602 and liquid accumulator 1605 to have enough static pressure to drive the coolant from within the thermosyphon system.
Throughout the thermosyphon-loop operation, liquid level could fluctuate from low level 1605 under low heat load to high level 1606 under high heat load. The liquid accumulator is sized properly in such a way that the highest liquid level does not reach the heat exchanger. Optional pump unit 1608 can be installed below the liquid accumulator to ensure coolant entering the pump unit is always in liquid phase.
Rack-mount equipment has to be always optimized for minimizing the overall height, since increased height (rack unit) will take up highly valuable rack space that can be otherwise used for adding more computational resources. Therefore, it would be ideal to have a liquid accumulator with an overall form factor that resembles a “pizza box”, with minimum height while ensuring enough internal volume can be provided for storing coolant. Depending on the overall system implementation, the required internal volume can vary, therefore a modular liquid accumulator can be envisioned to minimize manufacturing cost. For example, a modular liquid accumulator can be constructed by brazing or welding parts pre-defined with connector ports.
Referring to
In a typical implementation, one of the top ports 1706 can be fluidically connected to an outlet port of heat exchanger, whereas the second top port 1707m can be fluidically connected with an inlet port of heat exchanger through a capillary tube.
Elevated pressure from liquid vapor mixture can act as a pressure source to ensure the liquid coolant is flushed out properly during thermosyphon operation. If a single fluid distribution manifold is deployed, a bottom port 1708 can be fluidically connected to a downcomer pipe, while the second bottom port 1709 can be capped.
If two fluid distribution manifolds are implemented, a second bottom port 1709 can be fluidically connected to the second manifold. Alternatively, a single part with port and recessed surface can be used to further simplify the manufacturing process.
In typical thermosyphon-based electronics cooling application, non-hermetic seals are introduced to the system, which can potentially lead to minor coolant loss over time. Therefore, it would be advantageous to be able to monitor the coolant inventory within the system.
A minor coolant leak (minor coolant loss) refers to a minor coolant leak by coolant molecule diffusion across non-metallic seals and materials over a long period of time.
A liquid level sensor such as a capacitive sensor can be implemented inside a liquid accumulator to monitor the amount of coolant inside the reservoir. Since the level of coolant inside the reservoir tank dynamically changes depending on the heat load, the ideal level can be determined at different heat loads into the system. The amount of heat load can be determined directly or indirectly from IT equipment via Baseboard Management Controller (BMC) or power distribution unit (PDU). If the liquid level falls below the ideal level, coolant will have to be added to the system.
To determine if the liquid level is properly maintained, additional information such as the amount of heat load from IT equipment should be monitored as well. A controller unit 1804 collects data from IT equipment 1800 or power distribution unit 1805 and compares the actual measurement against expected ideal level. If liquid level drops below a pre-determined safety threshold level, a proper maintenance protocol would have to be executed.
Referring to
The reservoir is located at the highest elevation, above a heat exchanger 1906 to ensure liquid is drained into the system properly when the valve 1904 is opened. A port 1907 is provisioned on a reservoir tank for charging or discharging. Instead of a fully automated maintenance approach, a manual isolation valve can be implemented instead.
Coolant can leak out of the non-hermetic thermosyphon loop into the atmosphere over time. This would result in lower fluid charging ratio, and therefore decreased thermal performance. Therefore, it is important to monitor any leaks in the system. As part of the leak monitoring system, multiple sensors can be installed throughout the system.
In a typical implementation, live data stream from temperature or chemical sensors (e.g., metal oxide, Infrared, or MEMS-based sensors) can be monitored throughout the operation.
The system can use a refrigerant leak detection system such as the system shown and described in U.S. Pat. No. 6,772,598 to Rinehart, which is incorporated by reference.
The chemical sensors can be used to detect anomalies in the system. Although highly unlikely, if there is a catastrophic failure (i.e., pipe rupture), liquid phase coolant will rapidly leak out to the ambient and start to evaporate. This will result in sudden drop in temperature. Temperature data collected throughout the system can be monitored to capture this behavior to detect accidental coolant leaks.
Refrigerant based cooling system should be designed to contain coolant throughout the operation. However, due to mechanical defects, such as cracks or polymer seal failure, system may lose coolant from the system.
RAnomaly detection algorithms, such as isolation forest, local outlier factor, robust covariance, support vector, or other machine learning methods, could be applied to the data to identify anomalous behavior in the system. If in case any anomalies are detected, such as sudden drop in temperature associated with rapid evaporation of coolant into ambient, maintenance alerts can be triggered to notify system operators.
If any of the temperature sensors detect sudden drop in temperature due to rapid evaporation of coolant in to ambient, which indicates a major leak, a safety isolation valve can be triggered to contain coolant from leaking into ambient, while IT equipment undergoes through a graceful shutdown sequence.
A major leak can include catastrophic rupture or polymer seal failure that forces the system to be shut down. Major coolant leak will lead to sudden temperature drop that can be used as a proxy for detecting such accident.
A minor leak can include Refrigerant diffusion through polymer seal could potentially lead to slow but steady loss of refrigerant. Despite a minor leak, a cooling system would perform as expected until the amount of coolant in the system drops below required minimum amount.
The use of certain flammable coolants requires chemical sensors (leak detectors) by law for safety reason.
The system can use a water leak detection monitor, for detecting accidental water leaks in or around a heat exchanger. See U.S. Pat. No. 4,835,522 to Andrejasich et al., which is incorporated by reference.
Studies have shown ambient air diffusion into the system could negatively impact the overall cooling performance. For example, air intrusion could cause an increase in thermal resistance or decrease in mass flow rate in the thermosyphon loop (TSL) at a given heat flux. The non-hermetic nature of the thermosyphon loop could cause air to diffuse into the system through potential leak paths such as elastomeric sealants in pipe fittings.
Based on real-time data stream, the software can analyze, monitor, and detect air intrusion into a coolant loop. Anomaly detection algorithms, such as isolation forest, local outlier factor, robust covariance, support vector, or other machine learning methods, could be applied to the data.
To remove diffused air from the system, a small component with a defined internal volume can be positioned at or near the top of the TSL that can act as an air trap. This component can selectively accumulate air over refrigerant vapor. A non-intrusive sensor, such as ultrasonic time-of-flight (Time of Flight) sensor or temperature sensors, can be equipped around the air trap to evaluate and monitor the amount of air trapped. As part of the maintenance procedure, the collected air could be removed by a release mechanism.
During scheduled maintenance, the air reservoir can be evacuated via an upper access valve 2204 while the lower valve 2205 is closed to prevent loss of working fluid.
Modern IT equipment supports detailed real time telemetry. For example, the latest server motherboards are equipped with BMC (baseboard management controller), which exposes variety of data related to computing hardware, such as CPU temperature and fan speed, for consumption through industry standard data transfer protocols such as IPMI or Redfish.
Real-time telemetry data provides essential information for data center operators to plan facility upgrades or optimization. However, in colocation facilities, where access to data from IT equipment is not viable, insights derived from such data are lost.
Limited access to IT equipment also poses a challenge to a pCDU which could otherwise ingest real-time data and run predictive analytics to provide end users with operational insights related to the cooling system performance. For example, heat load from CPU/GPU and temperature measurements can be used to predict coolant mass flow rate without costly flow measurement devices.
Coolant mass flow rate can be approximated using CPU power consumption as an input. CPU power consumption can be measured or predicted.
In the absence of heat load information, CPU/GPU heat load can be predicted using machine learning based models using several input features, such as CPU/GPU specifications and data related detailed power consumption, which could be obtained from a “smart” PDU. The machine learning model can be deployed in pCDU installed in co-location data centers to provide operational insights, despite limited access to data from IT equipment.
The system can use prediction models such as the model shown and described in U.S. Pat. No. 10,581,974 to Sustaeta et al., which is incorporated by reference.
A prediction controller can be used to estimate CPU (Central Processing Unit) heat load based on a PDU (Power Distribution Unit), wherein the prediction controller is utilized when data from IT equipment is not accessible. A machine learning model can be used to predict CPU power consumption using time series data from PDU as an input.
Machine learning model will correlate real time power consumption from individual servers to CPU power consumption.
In a data center environment 2300 where access to real time telemetry data from IT equipment is possible, training datasets can be readily obtained 2301. In an environment where data access is limited, such as colocation data centers 2302, a prediction model can be deployed to predict data related to IT hardware.
A Colocation data center is a facility where IT equipment is owned by individual tenants and the data center only provides cooling/power as a service. Since IT equipment is owned by tenants, data center operators do not have operational visibility in terms of what is going on inside IT equipment.
Consequently, predicted CPU/GPU heat load or actual CPU/GPU heat load based on telemetry data can be further utilized to derive coolant mass flow rate within a thermosyphon loop or vapor quality. These parameters, in turn, can be used to determine if a supplementary pump should be turned on to switch from gravity-driven coolant flow to pump-assisted gravity driven coolant flow to ensure adequate amount of coolant is being circulated to cool computer chips.
Unlike air-cooling based systems, liquid/refrigerant cooling systems enable accurate measurement of the amount of waste heat captured in hot water.
A pCDU is provisioned with industry standard energy metering unit (BTU meter), which not only could provide accurate amount of heat being captured in facility-side coolant loop, but also provide the waste heat capture efficiency, based on the heat load to IT equipment through BMC or PDU.
For example, the amount of heat captured can be calculated based on a simple energy balance calculation, whereas the input energy can be extracted from the CPU/GPU heat load or derived from the PDU. The heat capture efficiency can be monitored over time to improve operational energy efficiency. In addition, significant decrease in heat capture efficiency inside a heat exchanger can be used as a proxy to detect heat exchanger fouling.
Heat generated from IT equipment is transferred to facility-side coolant loop inside a heat exchanger 2501. For example, chilled water enters 2506 and warm water exits 250) from a pCDU. Temperature sensors 2508, 2509 and flow meter 2510 inside a BTU meter can calculate the amount of heat transferred to the facility-side coolant loop using the energy balance equation.
The input energy into the heat exchanger can be directly extracted from the heat load from CPU/GPU or derived from a PDU. Monitoring the heat capture efficiency can help data center operators access quantitative information to improve overall operational energy efficiency. In addition, the heat capture efficiency can be used as a proxy to detect heat exchanger fouling.
Data stream related to sustainability, such as heat capture efficiency, can be coupled with an auditable or immutable database, which is an append-only database that cannot be deleted or modified. Coupled with a standards-based metering system from pCDU hardware, the database could be managed, certified, and validated by a third party.
The end-user of the system can develop carbon offset projects based on this end-to-end platform to produce permanent, additional, verified, enforceable, real carbon offsets. The carbon offset can be monetized in the form of carbon tax credit. Software-as-a-Service (SaaS) platform or license-based subscription model can orchestrate the end-to-end process from data generation to streamlined filing carbon credit claims. Metered heat collection and energy capture quantification from computer racks/telecoms shelves for carbon tax credit claims.
Data collected from a PDU (Power Distribution Unit) and the heat exchanger can be used to calculate rack-level heat capture efficiency.
And storage of the data in a database for storing energy in an out of the system for an end-user to monitor and account for carbon offset credit (sustainability accounting). Archiving energy efficiency and carbon offset information can use the technology used in U.S. Pat. No. 8,000,938 to Mcconnell et al., which is incorporated by reference.
To overcome challenges associated with flow instability during a cold start of a two-phase thermosyphon loop (e.g., new deployment or rack-level maintenance), the start-up process may include an active pump to establish a robust flow. However, in case of gravity driven passive thermosyphon loop without a pump, establishing a robust flow under low heat could become quite challenging.
In a typical thermosyphon loop, mass flow rate of low boiling point refrigerants tends to increase as the amount of heat being introduced to the system increases. When a server rack equipped with a two-phase thermosyphon loop-based cooling system is first turned on, heat from IT equipment during cold start may not be substantial enough to reach the ideal condition for the two-phase flow. Moreover, there is the potential for temperature overshoots prior to robust coolant circulation being established. This challenge stems in part from the lack of control of the IT equipment power that is typically defined independently at the server-level.
To mitigate the challenges associated with a cold start-up scenario, it would be advantageous to establish circulation in the thermosyphon loop in a controlled way independent of the thermal power delivered by the servers. Resistive heating elements (localized or distributed) located on the riser(s) and controlled by the pCDU can provide precise amounts of heat into the thermosyphon loop during the system operation, but it is envisioned to be utilized primarily during the initial start-up. When IT equipment turns on and start to provide substantial power beyond the threshold amount of heat, the additional heating element can be turned off automatically. It is worth mentioning that IT equipment at idling state still draws significant amount of power that is well beyond the required amount heat to establish two-phase flow, therefore the resistive element would draw power only during initial start-up or maintenance.
Power delivery to the heaters is controlled by a pCDU using, for example solid state relays packaged into the pCDU and powered by AC or DC power delivered from the rack PDU. During start-up, the heat input to the riser(s) will initiate a stable recirculation of two-phase refrigerant mixture within the thermosyphon loop, while avoiding uncontrolled and unstable flow patterns in the presence of the uncontrolled heat source, e.g., servers/IT hardware.
Once the control system, e.g., pCDU, detects robust two-phase flow and the servers are operating, the heater turns off automatically to improve energy efficiency of the system. A specific example of an implementation is shown in a figure below. Here the starter heaters are located at the bottom of the riser where the static pressure is the greatest.
Each server can be fluidically connected with a heat exchanger via a downcomer pipe 2606 and a riser pipe 2607 forming a thermosyphon loop. A heating element 2608 can be installed at the bottom of the riser. When servers are first turned on with little heat load into the thermosyphon loop, flow instability can occur.
PDU power draw can be monitored 2609 to detect the moment when servers are initially turned on. The control module 2602 could activate the heating element 2608 through a control signal 2610 to provide sufficient coolant driving force by generating liquid vapor mixture in the riser column. When the system establishes robust coolant circulation, the heater can be turned off.
In addition, it would be advantageous to establish circulation in the thermosyphon loop in a controlled way by precisely controlling the start-up sequence of servers in a rack. In a thermosyphon loop-based cooling system, a server at the bottommost position would have the greatest static pressure head to aid the stable flow within the thermosyphon loop, and therefore is relatively more advantageous than any servers above to initiate the two-phase flow.
As the mass flow rate slowly increases during the early start-up, a microcontroller in a pCDU could orchestrate the start-up process by communicating with the power distribution unit to power on servers starting from bottom and progressing upwards to the top, while taking threshold mass flow rate as an input to trigger sequence of events precisely.
Once a stable coolant flow is established, the overall system becomes resilient to external perturbations, such as fluctuations in heat flux caused by changing computational demands. This process can be executed with or without Data Center Infrastructure Management (DCIM) software or Data Center Management (DCM) software.
A resistive heating element (2608) provides thermal energy to circulate coolant in a thermosyphon loop.
The system can use a server start-up detection and heater block starter feature. When a server rack is turned on, computer servers start to draw electrical power from a PDU (power distribution unit). This initial “start-up” can be detected by simply monitoring the PDU usage. A control module can detect “start-up” and trigger a heater block to turn on. The heater block provides thermal energy to a thermosyphon loop to generate coolant circulation. Once control module detects a substantial coolant flow circulation, the heater block is turned off as thermal energy from computer chips provide adequate heat to sustain the flow.
The bottom server group 2705 which has the greatest static pressure head can aid the onset of stable annular two-phase flow in a thermosyphon loop.
As mass flow rate increases and further making the passive two-phase thermosyphon loop robust, other power banks 2703 can be turned on to provide power to a middle server group 2706. Finally, the last server group 2707 can be turned on.
The system can use flow stabilization methods during start-up (with PDU)
In addition, it would be advantageous to establish circulation in the thermosyphon loop in a controlled way by precisely controlling the start-up sequence of servers in a rack. In a thermosyphon loop-based cooling system, a server at the bottommost position would have the greatest static pressure head to aid the stable flow within the thermosyphon loop, and therefore is relatively more advantageous than any servers above to initiate the two-phase flow. As the mass flow rate slowly increases during the early start-up, a microcontroller in a pCDU could orchestrate the start-up process by communicating with the power distribution unit to power on servers starting from bottom and progressing upwards to the top, while taking threshold mass flow rate as an input to trigger sequence of events precisely. Once a stable coolant flow is established, the overall system becomes resilient to external perturbations, such as fluctuations in heat flux caused by changing computational demands. This process can be executed with or without Data Center Infrastructure Management (DCIM) software or Data Center Management (DCM) software.
In heat reuse applications, the thermal load from the IT equipment and the heat reuse demand may not always be well matched. For example, rejecting all the heat from the IT equipment to a space heating application may only be feasible under certain environmental conditions, e.g., in winter, but not summer.
However, deploying IT equipment based on the baseline heat requirements may be too restrictive for deployments. Moreover, the heat reuse application may require a well-controlled supply temperature.
To mitigate this issue, it is advantageous to have control of the secondary loop working fluid supply temperature and the secondary side fluid flow rate through the condenser. The supply temperature from the heat reuse application to the IT equipment racks is set to a maximum value at the design stage.
However, when the heat reuse load is small, the supply temperature could exceed the specified maximum value. By provisioning a centralized liquid/liquid heat exchanger, heat can be shunted to the facility loop (e.g., dry cooler or chiller) to maintain the supply temperature within spec. At the same time, server utilization and, thus heat load, on a rack basis can fluctuate in time.
In order to ensure the design specified return temperature to the heat reuse application, a fail-open automated valve controlled by the pCDU can be provided at the inlet to the condenser on the secondary side to reduce the flow rate through the condenser when server utilization and thus heat load is low.
The IT equipment racks 2801 incorporate a primary cooling loop 2802 that interfaces to a facility level secondary loop manifold 2803. An automated valve 2804 controlled by the pCDU adjusts the flow of the secondary flow through the condenser to ensure the temperature leaving the condenser on the secondary side is consistent with the heat reuse application 2805 requirements.
A liquid/liquid heat exchanger 2806 is provided for supplemental heat rejection to, for example, a variable speed pumped dry cooler loop 2807 with sufficient capacity to reject the design IT heat load that is controlled to achieve a specified secondary side supply temperature to the IT equipment racks.
Although specific advantages have been enumerated above, various embodiments may include some, none, or all of the enumerated advantages.
Modifications, additions, or omissions may be made to the systems, apparatuses, and methods described herein without departing from the scope of the disclosure. For example, the components of the systems and apparatuses may be integrated or separated. Moreover, the operations of the systems and apparatuses disclosed herein may be performed by more, fewer, or other components and the methods described may include more, fewer, or other steps. Additionally, steps may be performed in any suitable order. As used in this document, “each” refers to each member of a set or each member of a subset of a set.
To aid the Patent Office and any readers of any patent issued on this application in interpreting the claims appended hereto, applicants wish to note that they do not intend any of the appended claims or claim elements to invoke 35 U.S.C. 112(f) unless the words “means for” or “step for” are explicitly used in the particular claim.
The term “approximately” is similar to the term “about” and can be +/−10% of the amount referenced. Additionally, preferred amounts and ranges can include the amounts and ranges referenced without the prefix of being approximately.
While the invention has been described, disclosed, illustrated and shown in various terms of certain embodiments or modifications which it has presumed in practice, the scope of the invention is not intended to be, nor should it be deemed to be, limited thereby and such other modifications or embodiments as may be suggested by the teachings herein are particularly reserved especially as they fall within the breadth and scope of the claims here appended.
This application is a Continuation-In-Part of U.S. patent application Ser. No. 18/198,522 filed May 17, 2023 which claims the benefit of priority to U.S. Provisional Application Ser. No. 63/344,291 filed May 20, 2022, and this application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 63/353,224 filed Jun. 17, 2022. The entire disclosure of each of the applications listed in this paragraph are incorporated herein by specific reference thereto.
| Number | Date | Country | |
|---|---|---|---|
| 63344291 | May 2022 | US | |
| 63352224 | Jun 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18198522 | May 2023 | US |
| Child | 18209752 | US |
