Patent Application
20090326884
The present invention relates to data centers, and more particularly, to techniques for monitoring and predicting environmental operating conditions in a data center.
The continuing rise of energy consumption of computing hardware systems installed in data centers combined with the cooling infrastructure required to provide operable conditions for this hardware has emerged as a key limitation for future growth in the field of information technology (IT). As a result, undesirable tradeoffs commonly must be made between energy/cooling requirements, space limitations and cost constraints on the one hand and increased IT performance on the other.
A key parameter used to assess environmental conditions in a data center is ambient temperature distribution. Ideally, the ambient temperature would be continuously monitored throughout the entire data center. However, due to the complex physical composition of a data center, current monitoring and/or complete computational fluid dynamics (CFD) modeling techniques can only either provide a one-time snapshot of the entire data center or continuous measurements in a few locations. Making continuous measurements or continuous modeling of an entire data center using conventional techniques is impractical, if at all possible, from both an efficiency and cost standpoint.
Therefore, techniques are needed to continuously monitor temperature conditions throughout an entire data center.
The present invention provides techniques for monitoring and predicting environmental operating conditions in a data center. In one aspect of the invention, a method for real-time, three-dimensional analysis of environmental operating conditions in a data center is provided. The method comprises the following steps. High spatial resolution three-dimensional measurements of one or more environmental variables in the data center made at a time t1 are obtained. Real-time measurements of the environmental variables in the data center made at a time t2, wherein t2 is later in time than t1, are obtained. The high spatial resolution three-dimensional measurements are combined with the real-time measurements to derive a model for the environmental variables in the data center at the time t2. The model is used to predict three-dimensional distributions of the environmental variables in the data center at the time t2. The high spatial resolution three-dimensional measurements can be used to create a base model and the base model can be combined with the real-time measurements to derive the model for the environmental variables in the data center at the time t2.
In another aspect of the invention, a system for real-time, three-dimensional analysis of environmental operating conditions in a data center is provided. The system comprises a mobile off-line surveying system having a plurality of mobile off-line sensors configured to make high spatial resolution three-dimensional measurements of one or more environmental variables in the data center; a real-time sensing system having a plurality of real-time sensors placed throughout the data sensors configured to make real-time measurements of the environmental variables in the data center; a first functional module configured to take the high spatial resolution three-dimensional measurements from the mobile off-line surveying system and derive a base model at a time t1 for the environmental variables in data center; a second functional module configured to combine the base model with real-time measurements from the real-time sensing system at a given time tx to derive a data center model for the environmental variables in data center at the time tx, wherein tx is later in time than t1; a third functional module configured to monitor and compare data output from the data center model and the real-time sensing system; and a fourth functional module configured to derive a set of update parameters for the data center model.
A more complete understanding of the present invention, as well as further features and advantages of the present invention, will be obtained by reference to the following detailed description and drawings.
Presented herein are techniques for continuously monitoring environmental operating conditions, such as ambient temperature, thermal distributions, humidity, air flow, pressures and power distributions/current consumption, throughout an entire data center, i.e., with high spatial resolution. Thus, the present teachings provide a benefit over conventional data center monitoring techniques. As highlighted above, the conventional techniques can provide only either a spatially dense one-time snap-shot measurement (e.g., performed using a mobile off-line surveying system (MOSS), such as mobile measurement technology (MMT) V1.0, as described in U.S. Pat. No. 7,366,632, issued to Hamann et al., entitled “Method and Apparatus for Three-Dimensional Measurements” (hereinafter “U.S. Pat. No. 7,366,632”), the contents of which are incorporated by reference herein, or one-time computational fluid dynamics (CFD) modeling results) or spatially sparse continuous real-time measurements (e.g., by means of a real-time sensing system (RTSS), as offered by SynapSense of Folsom, Calif.).
When considering each one of these independent approaches, however, some opposing features and disadvantages become apparent. For example, with the MOSS methodology a snap-shot measurement of three-dimensional thermal conditions in a data center can be acquired within a few hours with high spatial resolution. Similarly, with a conventional data center CFD model it can easily take several hours to obtain a result with acceptable accuracy
In contrast, with a RTSS it is possible to monitor a data center environment continuously (high time resolution) but at significantly reduced spatial resolution, which could only be improved by deploying a much larger number of real-time sensor nodes, at a much elevated cost.
In
The ACUs typically receive chilled water from a refrigeration chiller plant (not shown). The ACUs, refrigeration chiller plant and various other related components and structures directed to cooling the data center make up a cooling infrastructure for the data center. Each ACU typically comprises a blower motor to circulate air through the ACU and to blow cooled air, e.g., into the sub-floor plenum. As such, in most data centers, the ACUs are simple heat exchangers consuming power needed to blow the cooled air into the sub-floor plenum.
Typically, one or more power distribution units (PDUs) (not shown) are present that distribute power to IT equipment racks 101. As will be described in detail below, power consumption by the PDUs can be an important consideration in the present techniques. In general, since the PDUs supply electrical power required by the IT equipment in a data center, a total electrical power intake of the PDUs represents an important parameter in determining the energy efficiency of a data center.
Data center 100 comprises one or more real-time sensors (not shown) which are part of an RTSS. The placement of the real-time sensors throughout the data center will be described in detail below. With the present techniques, for example, combined MOSS and RTSS technologies are employed to accurately model and analyze three-dimensional thermal and heat distributions in a data center in real-time.
The present techniques can be used to extend the corresponding capabilities of MMT V1.0. MMT V1.0 is a technology for optimizing data center infrastructures for improved energy and space efficiency which involves a combination of advanced metrology techniques for rapid measuring/surveying data centers (see, for example, U.S. Pat. No. 7,366,632) and physics-based modeling techniques for optimizing a data center facility within a given thermal envelope for optimum space and most-efficient energy utilization (see, for example, U.S. application Ser. No. 11/750,325, filed by Claassen et al., entitled “Techniques for Analyzing Data Center Energy Utilization Practices,” designated as Attorney Reference Number YOR920070242US1 (hereinafter “Claassen”), the contents of which are incorporated by reference herein).
The “static” MMT measurements (see, for example, U.S. Pat. No. 7,366,632) can provide essential data to build a base model of the respective data center (see step 204, described below). According to one exemplary embodiment, the static MMT measurements comprise three-dimensional thermal measurements, as well as air flow measurements at various levels of the IT equipment racks (e.g., IT equipment racks 101 of data center 100). Additionally, to determine total power consumption of a data center, power measurements of the PDUs are desirable, as well as a complete survey of operational parameters of the ACUs. The operational parameters of the ACUs can include, but are not limited to, inlet and outlet air flows of the ACUs and/or air temperature differentials between the inlets and outlets of the ACUs. For example, based on the three-dimensional thermal/air flow measurements and the operational parameters of the ACUs, an estimate of total thermal energy produced in a data center can be inferred and compared with direct power usage measurements (i.e., based on the fact that a total electrical energy consumed within the confines of a data center equates with the total thermal energy produced in that data center). In some cases, a combination of the PDU power measurements and name plate data (which may be available from data center asset databases) together with corresponding power distribution wiring schematics, can be used to determine power consumption of individual IT equipment components (i.e., power distribution). This approach allows for the power consumption of IT equipment components/IT equipment racks to be verified by comparing a spatial location of the components/racks with related MMT-based three-dimensional thermal maps.
MMT V1.0 allows for measurement of the environmental variables of the data center with a high spatial resolution. MMT V1.0, for example, is able to measure temperatures in a data center with a three-dimensional spatial resolution of less than one cubic foot. Exemplary MMT measurements are shown illustrated, for example, in
In step 204, the static three-dimensional measurements from MMT V1.0 are used to create a base model M for the one or more environmental variables in the data center, i.e., M({tilde over (x)}(t1)), wherein {tilde over (x)} is a vector of input parameters, such as power, flows, spatial coordinates and/or dimensions, for the model. The derivation of a base model (step 204) and manipulation thereof (steps 206/208, described below) are optional steps because, as is described in detail below, direct use of the static data (from time t1) and real-time data obtained at a time t2 (wherein t2 is later in time than t1, i.e., t2>t1) can be used directly to model the data center.
In general, the models described herein are used to predict, describe and/or represent the environmental variables in space, and possibly time, i.e.,
M({tilde over (x)})E(x,t)
wherein M is the particular model, E is an environmental variable, such as thermal conditions, humidity, air flow, pressures and power distributions/current consumption, {tilde over (x)} is a vector of input parameters for the model, such as power, flows, spatial coordinates and/or dimensions, x is a vector of dimensional variables (x, y, z), determining a location inside a data center, and t is time.
In the base model, there are a fixed set of input parameters and {tilde over (x)} is not a function of time, i.e., M({tilde over (x)}(t1))E(x). By contrast, in a subsequent model created at time t2 (see below) (wherein t2>t1), also referred to herein as a “dynamic model,” M({tilde over (x)}(t2))
E(x,t2), the input parameters {tilde over (x)} can change as a function of time and therefore are adjusted based on the latest sensor readings. x and {tilde over (x)} can have common elements but that is not required.
According to the present techniques, one or more of the following types of base models may be derived: a direct measurement model, a statistical model (i.e., fitting the MMT measurements to an analytical model), a physics-based model and a combination of at least one of the foregoing types of models. Suitable statistical models include, but are not limited to, Kriging techniques and interpolation schemes (e.g., linear, polynomial, Hermite and cosine models) and reduced order models, such as proper orthogonal decomposition (POD) models (see for example, Y. Joshi, et al., “Reduced Modeling Based Robust Thermal Design of Energy Efficient Data Centers,” The Eighteenth International Symposium on Transport Phenomena, Daejeon, KOREA (Aug. 27-30 2007), the contents of which are incorporated by reference herein). Suitable physics-based models include, but are not limited to, zoning/provisioning models and simplified heat convection/conduction models. See, for example, U.S. application Ser. No. 12/146,852, entitled “Techniques for Thermal Modeling of Data Centers to Improve Energy Efficiency,” designated as Attorney Reference No. YOR920080114US1, filed on Jun. 26, 2008, (hereinafter “Attorney Reference No. YOR920080114US1), the contents of which are incorporated by reference herein. As an example, the base model could also be constructed by the well-known Navier-Stokes-CFD modeling as well. Navier-Stokes-CFD modeling is described, for example, in Attorney Reference No. YOR920080114US1. It is also notable that all information about the data center that is relevant to assess energy consumption of the data center, such as the cooling system (refrigeration chiller plant technologies, refrigeration chiller plant coefficient of performance (COP) and outside temperatures), PDUs and perforated floor tiles, is included. Example 3 below provides an indication of how such infrastructure information can be used in the models.
The direct measurement model involves the direct use of the MMT measurements, for example, by arranging the MMT measurements in a data base, where each entry represents location coordinates (x,y,z) and associated measurement (value), e.g., (x, y, z; value). With the use of a statistical model of the data center, the thermal/air flow measurements are represented as a function of one or more variables, such as sensor location (where the sensors can be part of a MOSS network (base model) and/or an RTSS network (dynamic model, described below)), type of perforated floor tiles present and/or positioning of cooling air inlets/hot air outlets of an IT rack. Perforated floor tiles are typically characterized by their “opening” (e.g., 25 percent (%)) or flow impedance, or (as herein) by an amount of flow they provide. The “opening” is useful to calculate pressure under the data center floor. Compared to the zoning/provisioning model (see above), for example, the quality of the statistical model can benefit from a larger number of relevant input factors or data, such as the data provided by real-time sensors.
Kriging/interpolation models are described in detail below. The term “Kriging,” as described below, refers to a known interpolation method to predict unknown values from data measured at known locations. Known Kriging models include ordinary Kriging and universal Kriging. Ordinary Kriging includes, for example, spherical, circular exponential and linear methods, and assumes that a spatial variation in z-values is free of any structural component (drift). On the other hand, universal Kriging assumes that the spatial variation in z-values is the sum of three components, i.e., a structural component (drift), a random spatially correlated component and random noise representing residual errors. POD is a method to numerically predict velocity and temperature fields more efficiently than full-field simulations. The velocity and temperature fields are expanded into basis functions or POD modes. It can be shown that the basis functions can be represented as a matrix, where each column includes a complete velocity or temperature field data of an observation. These observations can come from measurements or CFD calculations.
For the physics-based models, such as the zoning/provisioning model illustrated immediately below, the MMT methodology yields modeling parameters for each of the IT equipment racks based on supplied air flow and temperature differentials, as well as the resulting temperature fields that surround each rack. For example, with the zoning/provisioning model, the data center is separated into different zones being fed by the individual ACUs. With this method, a provisioning zone is assigned to each ACU within a data center, i.e., this model assumes that cooling air supplied to each such zone is provided by the assigned ACU. The zone assignment can be done using air flow modeling with the base model. The MMT V1.0 air flow data is used to create zones, wherein each zone corresponds to an ACU. For example, if the discharge temperature of a particular ACU changes (e.g., read via RTSS with a sensor at an air outlet of the ACU), the corresponding zone temperatures change accordingly. Specifically, air flow sinks and sources can be identified in the data center using MMT V1.0 and an air flow model created, for example, as described in Attorney Reference No. YOR920080114US1 (another physics-based model). Other zoning concepts may use air flow from the respective ACU in conjunction with total data center area by geometrically partitioning up the total area in accordance with flow and position of the ACU. A model in the form of a spatial thermal footprint of each individual IT equipment rack can be evaluated by taking a ratio between an average temperature of (cooled) air provided to the air inlets of the IT equipment components in each rack and known or interpolated temperatures of air surrounding the IT equipment components.
In general, the category of physics-based models (also referred to herein as physics-driven models) includes, for example, the models described in Attorney Reference No. YOR920080114US1 (which also provides a description of simplified convection/conduction models that may be used herein) as well as the presently described zoning/provisioning models. Other suitable physics-based models may also be implemented, however Navier-Stokes-CFD models are generally excluded from use herein primarily because physics-based models are simpler (faster) to run than Navier-Stokes-CFD models (although standard CFD models may be used). With the zoning/provisioning models, what is first described is how to create zones and then how to use a provision concept.
In step 206, measurements from the real-time sensors made also at time t1, i.e., ERTSS(xs, t1), can be obtained to use in creating the base model. According to an exemplary embodiment, the real-time measurements are made using a RTSS that employs real-time sensors placed throughout the data center that continuously monitor the environmental operating conditions. See, for example, the description of
In step 208, based on the real-time sensor readings taken in step 206, described above, an altered base model (M({tilde over (x)}(t=0))) is created that includes both ERTSS(xs, t1) and EMOSS(xd), e.g., using sensor data fusion as described in detail below. Thus, strictly speaking, {tilde over (x)} in step 204 is different from {tilde over (x)} in step 208. For example, if sensor data fusion is used then there will be different spatial coordinates in step 208 as compared to step 204. Like step 204 (described above), steps 206 and 208 are also optional. For example, the base model can be created from the MMT V1.0 measurements (step 204), but it is not always necessary to obtain measurements from the real-time sensors at time t1 and/or alter the base model (steps 206 and/or 208, respectively).
In step 210, real-time measurements of the environmental variables in the data center made at a time t2 (wherein t2 is later in time than t1, i.e., t2>t1) are obtained. As in step 206, above, the real-time measurements can be made using a RTSS that employs real-time sensors placed throughout the data center that continuously monitor the environmental operating conditions.
The real-time sensors can comprise any type of sensor which monitors a relevant physical quantity, such as the environmental variables highlighted above, namely thermal conditions (i.e., ambient temperatures and/or thermal distributions), humidity, air flow, pressures and power distributions/current consumption, in real-time. The sensors can be wired or wireless. In many instances, these real-time sensors are already present as part of a building monitoring system. The placement of the real-time sensors is guided by which of the data center model(s), described above, is being used. By way of reference to an exemplary embodiment wherein a Kriging model is used, statistical sampling methods can also be used to determine both an optimum number and locations of real-time sensors for a given data center. Other exemplary methods for placing real-time sensors can involve use of (Latin) hypercube sampling techniques, described below.
Additional and/or existing real-time sensors placed in the data center can also be used to verify the base model prediction, if made, and to determine whether any additional surveys with the MOSS, i.e., MMT V1.0, need to be done. For example, mean-square error (MSE) can be determined between data obtained from the real-time sensors at time t2 and corresponding predictions from the base model. In this case, the MSE serves to indicate whether or not a new survey with the MOSS is required to readjust the base model for obtaining improved predictions. As highlighted above, the creation of a base model is optional, and embodiments are presented herein wherein a base model is not used in the data center analysis.
As compared to the MMT measurements, the real-time sensors provide (relatively) low spatial resolution, i.e., the real-time sensors are sparsely placed throughout the data center. This is due, at least in part, to the expense associated with the deployment, use and continual operation of real-time sensors. However, as highlighted above, the MMT measurements are temporally sparse, e.g., performed once or twice a year for a given data center, whereas the real-time sensors can provide measurements in real-time, or near real-time, e.g., with measurement intervals on an order of seconds or minutes.
The MMT surveys and the real-time sensors both provide measurements of one or more environmental variables E. The difference between the MMT surveys EMOSS(xd) and the real-time sensor data (ERTSS(xs, t1)) is given by the spatial coordinates, wherein xs is a vector with a sparse set of spatial coordinates while xd is a vector of a dense set of spatial coordinates. As highlighted above, xs can be a subset of xd but that is not required. Further, measurements are provided from the MMT surveys at t1 (not in real-time). In some embodiments presented herein, the real-time sensors are used to monitor one or more of the modeling input parameters, such as changing power levels {tilde over (x)}(t2).
The measurements obtained from the real-time sensors are used to adjust the data center model as a function of the differences in data, if any, from the time when the MMT measurements were taken to the present, i.e., from time t1 to time t2. This approach thus provides a real-time (or near real-time) analysis of the environmental operating conditions in the data center.
For example, in step 212, the base model is combined with the real-time measurements from the RTSS, i.e., combine M({tilde over (x)}(t1)) with ERTSS((xs,t2), to derive a “dynamic” model for the environmental operating conditions in the data center at time t2, i.e., M({tilde over (x)}(t2)). This can be achieved, for example, by applying known methods of sensor data fusion, i.e., by combining the real-time sensor measurements and base model data, such that the resulting model at time t2 produces an output of better quality (i.e., more accurate, more complete and/or more dependable modeling results) than the base model alone. In general, sources of data for the data fusion process can include measurements from one or more of a number of different types of sensors, such as wireless sensors, wired sensors or a combination thereof, that can provide real-time data. However, any data representing a measure of the existing environmental status of the data center would be useful in providing accurate up-to-date models. Further, any suitable data fusion techniques may be employed, such as direct fusion, indirect fusion and/or fusion of outputs of the direct and indirect fusion. Direct fusion generally refers to fusion of sensor data from a set of (real-time) heterogeneous (multiple-sensor) sensors and historic (i.e., MMT-based) sensor data, while indirect fusion uses information sources like a priori knowledge about the environment and/or human input. An example illustrating the derivation of a data center model wherein a base model is first created is provided in Example 2, described below.
However, as described above, the creation of the base model is optional. Thus, alternatively, in step 212, the measurements from the MMT survey, i.e., EMOSS(xd) (see description of step 202, above), can be combined with measurements from the real-time sensors, i.e., ERTSS(xs, t2 ), using the data fusion processes described immediately above, to derive a model for the data center at time t2, i.e., M({tilde over (x)}(t2)). An example illustrating the derivation of a data center model without first creating a base model is provided in Example 1, described below.
Like the base model described above, the “dynamic” model M({tilde over (x)}(t2)) can comprise one or more of a direct measurement model, a statistical model, a physics-based model and a combination of at least one of the foregoing types of models. Suitable statistical models include, but are not limited to, Kriging techniques and interpolation schemes (e.g., linear, polynomial, Hermite and cosine models) and reduced order models, such as POD models. Suitable physics-based models include, but are not limited to, zoning/provisioning models and simplified heat convection/conduction models. Each of these model types were described in detail above.
Since the RTSS is continually monitoring the environmental operating conditions in the data center, any changes in the environmental operating conditions measured by the RTSS and that are, for example, large enough to meet the above-mentioned MSE criterion, will change the dynamic model, accordingly. By way of example only, if external influences and/or disturbances to the data center infrastructure occur, such as the installation of new IT equipment, any changes to the environmental operating conditions in the data center brought about by these influences and/or disturbances will be detected by the real-time sensors, for example at time tx (wherein tx is later in time than t2), and a corresponding change(s) to the dynamic model will occur. The result will be an updated data center model at time tx, i.e., M({tilde over (x)}(tx)). This scenario is illustrated, for example, by arrow 218 in
Namely, in step 214, a set of update parameters is derived for the data center model, M({tilde over (x)}(tx)), provided that the chosen MSE criterion indicates that such an update will be necessary. Thus, whether or not a new set of update parameters needs to be derived depends on the error criterion (for example, besides the MSE, such an error criterion can be the absolute error between measured and modeled quantities). Error (absolute or MSE) values below some defined threshold value do not lead to any update action. Some analytic details on this procedure are provided below.
A basic high-level example is now provided that illustrates the update procedure of a temperature model using real-time sensing and dynamic models. As an example, consider the following simple method for combining real-time temperature data, T (tm; {right arrow over (r)}s), measured at an actual time tm at sensor locations {right arrow over (r)}s=(xs, ys, zs), with corresponding data generated by an MMT-based temperature model, {tilde over (T)}(t0;{right arrow over (r)}), earlier validated at time t0<tm for {right arrow over (r)}=(x, y, z)ε{right arrow over (R)}, wherein {right arrow over (R)} represents a validated location domain of the model. Based on the error functional,
ΔT(tm;{right arrow over (r)}s)=T(tm;{right arrow over (r)}s)−{tilde over (T)}(t0;{right arrow over (r)}={right arrow over (r)}s),
and the application of some suitable interpolation or fitting technique, such as a Kriging methodology described above, an estimator for the error functional, Δ{circumflex over (T)}(tm;{right arrow over (r)}), can be obtained for any required position vector {right arrow over (r)}=(x, y, z)ε{right arrow over (R)} not covered by the real-time sensor system network. The original MMT-based model {tilde over (T)}(t0;{right arrow over (r)}) can then be updated to reflect an improved model for the temperature distribution at time tm>t0, e.g., by applying a simple superposition approach:
{tilde over (T)}(tm;{right arrow over (r)})←{tilde over (T)}(t0;{right arrow over (r)})+Δ{circumflex over (T)}(tm;{right arrow over (r)}).
This and more sophisticated approaches to fusion of real-time data and corresponding historic models can be extended to environmental parameters other than temperature, e.g., parameters such as humidity, air flow, pressures and power distributions/current consumption.
In step 216, the updated model, in this example, M({tilde over (x)}(tx)), is then used to predict a three-dimensional distribution of the one or more environmental variables E in the data center at time tx, i.e., E(xd,tx). Steps 210, 212, 214 and 216 of methodology 200 can be repeated, as indicated by arrow 220 in
Mobile off-line sensors 304, shown in
Measurements made by MOSS 308 (output labeled “D”) are taken and used by functional module 312 to derive a base model (labeled “BM”) at time t1 for one or more environmental variables data center 302. Measurements from RTSS 310 combined with data from the base model (output labeled “F”) are used by functional module 314 to derive a “dynamic” data center model (labeled “DM”) for the one or more environmental variables in data center 302 at time tx, wherein tx is later in time than t1, i.e., tx>t1. As described in detail in conjunction with the description of
Due to an occurrence(s) 320 of one or more changes to data center 302, such as external influences and disturbances to the infrastructure of data center 302 (labeled “A”), e.g., the installation of new IT equipment, changes in the cooling infrastructure and/or rebalancing of an operational load of the IT equipment, a set of update parameters for the dynamic model are determined by functional module 318. As described in detail in conjunction with the description of
Like functional modules 312, 314, 316 and 318, described above, MOSS 308 and RTSS 310 may also be thought of as functional modules of system 300. Namely, MOSS 308 functions in making static measurements of the environmental operating conditions in data center 302, as described above. Similarly, RTSS 310 functions in making real-time measurements of the environmental operating conditions in data center 302, as described above.
According to an exemplary embodiment, the functional modules further comprise appropriate interfaces that enable them to communicate with each other and with other components (e.g., application software) by means of a messaging protocol based on a publish-subscribe principle. By way of example only, real-time sensors 306, which are part of RTSS 310, use such a protocol (for example message queue telemetry transport (MQTT) protocol or its version for wireless sensors (MQTT-S). MQTT-S is described, for example, in Stanford-Clark et al., “MQTT for Sensor Networks (MQTTs)” Protocol Specification, Version 1 (October 2007) (available at www.mqtt.org), the contents of which are incorporated by reference herein. This message-based method for communicating between system components has the advantage that application developers do not need to consider the intrinsic details of the various communication protocols and addressing schemes, since a message broker (e.g., a broker as shown in
Broker 414 comprises one or more MQTT/MQTT-S gateways 416 through which communication/exchange can take place based on a publish-subscribe protocol for wireless sensor networks. Namely, RTSS wired sensors 412 and/or wireless sensors 413 can communicate/exchange data through the MQTT/MQTT-S gateways 416. Although the MQTT-S protocol was designed for wireless sensors, it is equally applicable to wired sensors, thereby providing its advantages to both types of sensors. These advantages include modest memory requirements and low transmission overhead. Alternately, wired sensors can also use the basic but more complex MQTT protocol.
As shown in
The RTSS wired sensor functional module 412 can comprise application programming interfaces (labeled “API(s)”) 418 and an MQTT/MQTT-S adapter 420 which permits interface with MQTT/MQTT-S gateways 416. The MOSS functional module 410 can comprise MOSS application 422, MQTT client 424, communication protocols 426, i.e., transmission control protocol (TCP)/internet protocol (IP) (labeled “TCP/IP”) and MMT functional module 428. MOSS application 422 is an off-line software program, which receives data from MMT functional module 428 for post-processing either directly via an MQTT client 424 or a TCP/IP network or both. Typically, MOSS application 422 consolidates MMT-based measurement data and infrastructure information of a surveyed data center to produce suitable input for functional module BM 402.
According to the exemplary embodiment shown illustrated in
Based on measurement data from the MOSS, i.e., MMT V1.0 (see, for example, step 202 of
As highlighted above, as an alternate method for placing real-time sensors, Latin hypercube designs can be used to determine an optimum number of real-time sensors and their placement in a data center. According to the present techniques, space-filling Latin hypercube designs are used, i.e., designs that spread points evenly throughout an experimental region are often employed. One challenge, however, associated with using space-filling designs in the context of a data center is that a data center generally yields an irregular experimental region. By way of example only,
To accommodate an irregular experimental region, the present techniques employ probability-based Latin hypercube designs (see, for example,
Turning now to
Apparatus 900 comprises a computer system 910 and removable media 950. Computer system 910 comprises a processor 920, a network interface 925, a memory 930, a media interface 935 and an optional display 940. Network interface 925 allows computer system 910 to connect to a network, while media interface 935 allows computer system 910 to interact with media, such as a hard drive or removable media 950.
As is known in the art, the methods and apparatus discussed herein may be distributed as an article of manufacture that itself comprises a machine-readable medium containing one or more programs which when executed implement embodiments of the present invention. For instance, the machine-readable medium may contain a program configured to obtain high spatial resolution three-dimensional measurements of one or more environmental variables in the data center made at a time t1; obtain real-time measurements of the environmental variables in the data center made at a time t2, wherein t2 is later in time than t1; combine the high spatial resolution three-dimensional measurements with the real-time measurements to derive a model for the environmental variables in the data center at the time t2; and use the model to predict three-dimensional distributions of the environmental variables in the data center at the time t2.
The machine-readable medium may be a recordable medium (e.g., floppy disks, hard drive, optical disks such as removable media 950, or memory cards) or may be a transmission medium (e.g., a network comprising fiber-optics, the world-wide web, cables, or a wireless channel using time-division multiple access, code-division multiple access, or other radio-frequency channel). Any medium known or developed that can store information suitable for use with a computer system may be used.
Processor 920 can be configured to implement the methods, steps, and functions disclosed herein. The memory 930 could be distributed or local and the processor 920 could be distributed or singular. The memory 930 could be implemented as an electrical, magnetic or optical memory, or any combination of these or other types of storage devices. Moreover, the term “memory” should be construed broadly enough to encompass any information able to be read from, or written to, an address in the addressable space accessed by processor 920. With this definition, information on a network, accessible through network interface 925, is still within memory 930 because the processor 920 can retrieve the information from the network. It should be noted that each distributed processor that makes up processor 920 generally contains its own addressable memory space. It should also be noted that some or all of computer system 910 can be incorporated into an application-specific or general-use integrated circuit.
Optional video display 940 is any type of video display suitable for interacting with a human user of apparatus 900. Generally, video display 940 is a computer monitor or other similar video display.
The present techniques are further described by way of the following non-limiting examples.
Thermal data (EMOSS(xd)) was obtained via MMT V1.0 at time t1. Because MMT V1.0 provided such a dense data grid, a base model was not constructed. The real-time sensors were placed on the MMT V1.0 data grid and thus a sensor calibration was not required. New data was then gathered from the real-time sensors at a time, i.e., time t2, i.e., ERTSS(xs,t2) where no EMODS(xd) is available (see line 3 of example code below where value and, x, y, and z coordinates are read from the sensors). The ERTSS(xs,t2) was then compared with the initial EMOSS(xd) (at time t1) results at xs (see line 12 of example code below, where the initial values (room.GTemp) are compared with the latest readings (at time t2) (value)). The difference between ERTSS(xs,t2) and EMOSS(xd) was then modeled using a griding function (see line 15 of example code below). The model was then used to predict a complete temperature field at time t2 (see line 19 of example code below, where a new three-dimensional temperature distribution is calculated). Whenever there was new real-time data, for example, at time tx, i.e., ERTSS(xs,tx), available the above-described steps were repeated (i.e., called function mmt_fillNewThermalData).
EMOSS(xd) was obtained using temperature results from CFD calculations. Because these calculations are too slow to be repeated in real-time, the data was represented and modeled by a simple model, such as the Laplacian model approach as described in Attorney Reference No. YOR920080114US1 or a simple provisioning model (create M({tilde over (x)}(t1))). In the example using a Laplacian model approach, the complete data center was described by the physical dimensions and locations of its infrastructure and equipment, as well as by the power levels and the different air flow boundaries (and/or temperature boundaries) (see Attorney Reference No. YOR920080114US1 for a detailed discussion). Because the real-time sensors were not on the same grid as the CFD model results (compare with Example 1, described above), the real-time sensor data and the CFD model results had to be referenced to each other. In this example, the real-time data at time t1 ERTSS(xs,t1) was compared with the MOSS data EMOSS(xd) to obtain an updated base model M({tilde over (x)}(t1)), which took the differences between RTSS and MOSS sensors into account. Updated real-time data was then obtained at time t2, i.e., ERTSS(xs, t2). Some additional real-time data was obtained, such as PDU power consumption. The PDU power consumption can be related to the power consumption of each IT equipment component (as highlighted above). An updated set of input parameters for the model was obtained, i.e., ({tilde over (x)}(t2)). Real-time sensor readings at time t3, i.e., ERTSS(xs,t3), were then compared with the initial MOSS data (from time t1). A new model was then constructed with the updated parameters. For example, the updated model contained the real-time measurements of some of the input parameters at time t3. The updated model was then used to predict the dense distribution of the environmental variable.
To demonstrate the effectiveness of the present real-time, three-dimensional analysis techniques, both MMT measurements (
To predict a future temperature distribution, the MMT measurements, for reasons of simplicity called yMMT instead of EMOSS(xd) and the real-time sensor observations yS are related to each other according to the following steps:
y
S
=f(X, t)+ε(X, t), tε[0,T], Step 1:
y
t
MMT={circumflex over (β)}0(X)+{circumflex over (β)}lŷt
wherein f(X,t) is some function depending on factors X and time t, tε[0,T] and t0ε[0,T] represent time instances t and t0, respectively, with T being a time interval over which real-time sensor data are available, ε(X, t) is a Gaussian process with mean zero and covariance function σ2ψ, with a Gaussian correlation function applied to ψ. The quantity ŷt
ŷ
t
MMT={circumflex over (β)}0(X)+{circumflex over (β)}1ŷt
wherein {circumflex over (β)}0 and {circumflex over (β)}1 are estimates of initially unknown coefficients β0 and β1, respectively.
As highlighted above, other factors X can be incorporated in the analysis. By way of example only, the following factors X were used:
y
S=78.05+7.81x3l−3.99x3lx5q+f(t)+ε(X,t), tε[0,1000],
y
500
MMT=55.41−5.5x1l−2.2x4l+0.28ŷ500S,
An estimate (prediction) of the temperature at a future time tF=1050 is given by the following expression:
ŷ
1050
MMT=55.41−5.5x1l−2.2x4l+0.28ŷ1050S.
As indicated above, the following description of a “Kriging model” is considered part of an exemplary embodiment of a modeling (prediction) scheme for three-dimensional thermal distributions, according to the present techniques. Well-known extrapolation/interpolation methodologies, such as linear, cubic spline or Kriging, may be applied to data center areas not accessible during an MMT survey. While MMT measurements provide high spatial resolution (see above), they are typically temporally sparse, e.g., performed once or twice a year for a given data center. Thus, data obtained from real-time sensors are compared to corresponding data generated by the three-dimensional thermal distribution model that was originally derived from MMT measurements. Based on the differences found (i.e., difference between what the base model predicts and the “dynamic” model predicts) that may have occurred over time, the models derived from high resolution MMT measurements can be adjusted if necessary, as dictated by the chosen MSE criterion. One approach to accomplish this is to use Kriging techniques as described below. Here the term “Kriging” stands for a kind of interpolation method to predict unknown values from data points measured at known locations (see above).
A universal Kriging model can be stated as follows. Assume that the true function y(x), xεRP, is a realization from a stochastic process:
Y(x)=μ(x)+Z(x),
wherein μ(x)=Σi=0mμivi(x) and Z(x) is a weak stationary stochastic process with mean zero and covariance function σ2ψ, having vi represent known functions and μi represent unknown parameters. The covariance function is defined as cov{Y(x+h),Y(x)}=σ2ψ(h), where the correlation function ψ((h) is a positive semidefinite function with ψ(0)=1 and ψ(−h)=ψ(h). But, in practice, an ordinary Kriging model is commonly used. Ordinary Kriging includes, for example, spherical, circular exponential and linear methods, and assumes that the variation in z-values is free of any structural component (drift), while universal Kriging assumes that the spatial variation in z-values is the sum of three components: a structural component (drift), a random spatially correlated component and random noise representing residual errors. Ordinary Kriging is analytically defined by the following expression:
Y(x)=μ0+Z(x),
wherein μ0 is an unknown parameter.
There are, however, problems associated with these universal/ordinary Kriging models. Namely, the predication can be poor if there are some strong trends. See, for example, J. D. Martin, et al., “Use of Kriging Models to Approximate Deterministic Computer Models,” AIAA Journal, vol. 43, no. 4, pp. 853-863 (April 2005), the contents of which are incorporated by reference herein. Also, it is not easy to understand the effects of parameters and factors of these Kriging models just by looking at the predictor. Further, the predictor is not robust to possible misspecifications in the correlation parameters. Therefore, according to the present techniques, a blind Kriging model is used as the preferred method to circumvent these problems. See, for example, V. R. Joseph et al., “Blind Kriging: A New Method for Developing Metamodels,” ASME Journal of Mechanical Design, vol. 130, 031102 (March 2008) (hereinafter “Joseph 2008”), the contents of which are incorporated by reference herein.
The blind Kriging model is analytically defined by the following expression:
Y(x)=v(x)′μ+Z(x),
wherein v(x)′=(1,v1(x), . . . ,vm(x)), μ=(μ0,μ1, . . . ,μm) and m are unknown. The vi(x) terms of the functions are not assumed to be known. Instead, the vi(x) terms are identified through some data-analytic procedure. Because the vi(x) terms are unknown in this model, it is called “blind Kriging.”
The blind Kriging predictor is analytically defined by the following expression:
ŷ(x)=v(x)′{circumflex over (μ)}m+ψ(x)′Ψ−1(y−Vm{circumflex over (μ)}m),
wherein Vm=(v(x1), . . . ,v(xn))′ and {circumflex over (μ)}m=(V′mΨ−1Vm)−1V′mΨ−1y. Note that the blind Kriging predictor has the same form as that of the universal Kriging predictor. An important step in blind Kriging is to identify the unknown vi(x) terms of the functions, which can be chosen from a set of candidate functions (or variables) using variable selection techniques. A variable selection technique related to the Kriging methodology is described, for example, in V. R. Joseph, “A Bayesian Approach to the Design and Analysis of Fractionated Experiments,” Technometrics, vol. 48, no. 2, pp. 219-229 (2006) and in V. R. Joseph, et al., “Functionally Induced Priors for the Analysis of Experiments,” Technometrics, vol. 49, no. 1, pp. 1-11 (2007), the contents of each of which are incorporated by reference herein, and can be considered a Bayesian version of the forward selection strategy. The simplest data-driven model building approach is called “forward selection.” In this approach, one adds variables to the model one at a time. At each step, each variable that is not already in the model is tested for inclusion in the model and the most significant of these variables is added to the model. The variable selection technique is described below. Additional details of this technique and examples can be found in Joseph 2008.
With the variable selection technique, denote the candidate variables by u1, . . . , ut. Consider approximating y(x) by Σi=0mμivi+Σi=0tβiui, where μ0=1. With prior distributions β˜N(0,τm2R), a posterior can be calculated by,
A variable can be declared important if an absolute coefficient {circumflex over (β)}i of that variable is large. Thus, the variable to enter at each step m=1,2, . . . can be selected as the variable with the largest |{circumflex over (β)}i|. To help determine what is a best value of m, a cross validation prediction error (CVPE) is first defined as:
wherein cvi=yi−ŷ(i)(xi) is leave-out-one cross validation error. One can choose the value of m that minimizes CVPE(m).
In the estimation procedure, the following Gaussian product correlation function is used:
with θ=(θ1, . . . ,θp)′. The parameters θ, μm, and σm2 can be estimated by maximizing the likelihood. Thus,
Although illustrative embodiments of the present invention have been described herein, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be made by one skilled in the art without departing from the scope of the invention.
This application is related to the commonly owned U.S. application Ser. No. 12/146,852, entitled “Techniques for Thermal Modeling of Data Centers to Improve Energy Efficiency,” designated as Attorney Reference No. YOR920080114US1, filed herewith on the same day of Jun. 26, 2008, the contents of which are incorporated by reference herein.
