A data center may be defined as a location, for instance, a room that houses computer systems arranged in a number of racks. A standard rack, for instance, an electronics cabinet, is defined as an Electronics Industry Association (EIA) enclosure, 78 in. (2 meters) wide, 24 in. (0.61 meter) wide and 30 in. (0.76 meter) deep. These racks are configured to house a number of computer systems, about forty (40) systems, with future configurations of racks being designed to accommodate 200 or more systems. The computer systems typically dissipate relatively significant amounts of heat during the operation of the respective components. For example, a typical computer system comprising multiple microprocessors may dissipate approximately 250 W of power. Thus, a rack containing forty (40) computer systems of this type may dissipate approximately 10 KW of power.
Data centers are typically equipped with a raised floor with vent tiles configured to provide cool air to the computer systems from a pressurized plenum in the space below the raised floor. In certain instances, these vent tiles contain manually adjustable dampers for varying the flow rate of cool air therethrough. However, because these vent tiles cannot be remotely controlled, they are typically unable to vary the airflow to dynamically provision the data center with cooling resources. In addition, these vent tiles are typically manually actuated without knowledge of how each vent tile affects computer systems in its proximity. These actuations frequently have unintended consequences, such as, inadequate airflow delivery to the racks, adverse re-circulation of heated and cooled airflows, and wasted energy consumption. This may lead to inefficiencies in both cooling of the computer systems as well as in the operations of air conditioning units.
In other instances, automated vent tiles have been used in data centers to generally enable remote actuation of the vent tiles via feedback control algorithms. Conventional automated vent tiles are typically operated, however, without substantially accurate knowledge of how actuations of these vent tiles affect airflow in the data center. A process for associating vent tiles with racks would therefore be desirable.
According to an embodiment, the present invention pertains to a method for correlating vent tiles with racks. In the method, the vent tiles are opened to a first setting and the mass flow rates of air received by the racks and supplied through the vent tiles are determined. In addition, one of the vent tiles is closed to obtain a second setting and the mass flow rates of air received by the racks and supplied through the vent tiles are determined at the second setting. The vent tiles and the racks are correlated based upon the determined mass flow rates of air received by the racks and the mass flow rates of air supplied through the vent tiles at the first and second settings.
Features of the present invention will become apparent to those skilled in the art from the following description with reference to the figures, in which:
For simplicity and illustrative purposes, the present invention is described by referring mainly to an exemplary embodiment thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent however, to one of ordinary skill in the art, that the present invention may be practiced without limitation to these specific details. In other instances, well known methods and structures have not been described in detail so as not to unnecessarily obscure the present invention.
According to various examples, vent tiles in a data center are commissioned to enable determinations of their relationships with various racks of equipment housed in the data center. More particularly, procedures and algorithms are described herein for evaluating the relationships between the vent tiles and the racks. These relationships may be described by a coefficient which relates the mass flow rate of cool air delivered to the racks to the air flow rates through individual vent tiles. In certain instances, the coefficient may also relate to the temperatures of the airflow delivered to the racks through the vent tiles in addition to the mass flow rates of airflow. In other instances, the effects of airflow re-circulation may also be included in the determination of the coefficient. This coefficient has been termed the “Vent Tile Influence coefficient” or VTI, for simplicity of description purposes.
The VTI may be employed for the development of vent tile control algorithms configured to, for instance, enable relatively efficient and dynamic provisioning of cooling resources in the data center. In addition, establishment of the relationships between the vent tiles and the racks may enable the creation of the vent tile control algorithms. In order to maintain certain levels of efficiency, the vent tile control algorithms, in general, are required to compensate for the variations of the relationships between vent tiles and racks. The relationships may vary due to, for instance, movement, changing, or manipulation of the equipment, changes in airflow patterns, pressure distributions, etc. In addition, the relationships may vary as computer room air condition (CRAC) unit flow rates change relative to each other in a multi-CRAC unit data center, obstructions are added or removed from within the floor plenum, the pressure distribution within the floor plenum changes, etc. In this regard, the VTI disclosed herein may compensate for the varying relationships such that the vent tile control algorithms may also adapt as the data center environment changes.
Through a determination of VTI, the relationships between various racks and vent tiles may be established. Thus, for instance, the correlation between particular racks and vent tiles may be used to determine how airflow through one or more vent tiles should be varied to obtain desired airflow characteristics through the particular racks. In this regard, vent tile control algorithms may use these relationships in controlling vent tiles to achieve desired cooling results in the data center.
Although particular reference is made throughout the present disclosure to air conditioning units and vent tiles in data centers for cooling racks, it should be understood that certain principles presented herein may be applied to cooling systems in other types of buildings. For instance, correlations between ceiling mounted air supply vent tiles and various areas of a room containing a sensor network may be made using VTI. In this example, VTI may be used to develop control algorithms that operate the air supply vent tiles to ensure that the various areas of the room receive desired levels of airflow. Thus, the descriptions presented herein with respect to VTI should not be construed as being limited solely to data centers, but that the data center environment is an example of a suitable application of the principles presented herein.
With reference first to
It should be readily apparent that the data center 100 depicted in
The data center 100 is depicted as having a plurality of racks 102-108, for instance, electronics cabinets, aligned in parallel rows. Each of the rows of racks 102-108 is shown as containing four racks (a-d) positioned on a raised floor 110. A plurality of wires and communication lines (not shown) may be located in a space 112 beneath the raised floor 110. The space 112 may also function as a plenum for delivery of cooled air from one or more computer room air conditioning (CRAC) units 114 to the racks 102-108. The cooled air may be delivered from the space 112 to the racks 102-108 through vent tiles 118 located between some or all of the racks 102-108. The vent tiles 118 are shown as being located between racks 102 and 104 and 106 and 108.
As previously described, the CRAC units 114 generally operate to supply cooled air into the space 112. The cooled air contained in the space 112 may include cooled air supplied by one or more CRAC units 114. Thus, characteristics of the cooled air, such as, temperature, pressure, flow rate, etc., may substantially be affected by one or more of the CRAC units 114. By way of example, the cooled air supplied by one CRAC unit 114 may mix with cooled air supplied by another CRAC unit 114. In this regard, characteristics of the cooled air at various areas in the space 112 and the cooled air supplied to the racks 102-108 may vary, for instance, if the temperatures or the volume flow rates of the cooled air supplied by these CRAC units 114 differ due to mixing of the cooled air. In certain instances, the level of influence of a CRAC unit 114 over the racks 102-108 may be higher for those racks 102-108 that are in closer proximity to the CRAC unit 114. In addition, the level of influence of a CRAC unit 114 over the racks 102-108 may be lower for those racks 102-108 that are located farther away from the CRAC unit 114.
Moreover, a particular vent tile 118 may have greater levels of influence over particular racks 102-108 and have lesser levels of influence over other racks 102-108. The level of influence the vent tiles 118 have over various racks 102-108 is considered herein as a vent tile influence coefficient (VTI) and is described in greater detail herein below. In one respect, the VTI may be employed in vent tile control algorithms to relatively accurately control the level of cooled airflow delivered to the various racks 102-108.
The vent tiles 118 may comprise manually or remotely adjustable vent tiles. In this regard, the vent tiles 118 may be manipulated to vary, for instance, the mass flow rates of cooled air supplied to the racks 102-108. In addition, the vent tiles 118 may comprise the dynamically controllable vent tiles disclosed and described in commonly assigned U.S. Pat. No. 6,574,104, the disclosure of which is hereby incorporated by reference in its entirety. As described in the U.S. Pat. No. 6,574,104 patent, the vent tiles 118 are termed “dynamically controllable” because they generally operate to control at least one of velocity, volume flow rate and direction of the cooled airflow therethrough. In addition, specific examples of dynamically controllable vent tiles 118 may be found in U.S. Pat. No. 6,694,759, filed on Jan. 27, 2003, which is assigned to the assignee of the present invention and is incorporated by reference herein in its entirety.
The racks 102-108 are generally configured to house a plurality of components 116 capable of generating/dissipating heat (not shown), for instance, processors, micro-controllers, high-speed video cards, memories, semi-conductor devices, and the like. The components 116 may be elements of a plurality of subsystems (not shown), for instance, computers, servers, bladed servers, etc. The subsystems and the components may be operated to perform various electronic functions, for instance, computing, switching, routing, displaying, and the like. In the performance of these electronic functions, the components, and therefore the subsystems, may generally dissipate relatively large amounts of heat. Because the racks 102-108 have generally been known to include upwards of forty (40) or more subsystems, they may transfer substantially large amounts of heat to the cooled air flowing therethrough to maintain the subsystems and the components generally within predetermined operating temperature ranges.
The areas between the racks 102 and 104 and between the racks 106 and 108 may comprise cool aisles 120. These aisles are considered “cool aisles” because they are configured to receive cooled airflow from the vent tiles 118, as generally indicated by the arrows 122. In addition, the racks 102-108 generally receive cooled air from the cool aisles 120. The aisles between the racks 104 and 106, and on the rear sides of racks 102 and 108, are considered hot aisles 124. These aisles are considered “hot aisles” because they are positioned to receive air that has been heated by the components 116 in the racks 102-108, as indicated by the arrows 126. By substantially separating the cool aisles 120 and the hot aisles 124, for instance, with the racks 102-108, the heated air may substantially be prevented from re-circulating with the cooled air prior to delivery into the racks 102-108. In addition, the cooled air may also substantially be prevented from re-circulating with the heated air prior to returning to the CRAC units 114. However, there may be areas in the data center 100 where re-circulation of the cooled air and the heated air occurs. By way of example, cooled air may mix with heated air around the sides or over the tops of one or more of the racks 102-108.
The sides of the racks 102-108 that face the cool aisles 120 may be considered as the fronts of the racks and the sides of the racks 102-108 that face away from the cool aisles 120 may be considered as the rears of the racks 102-108. For purposes of simplicity and not of limitation, this nomenclature will be relied upon throughout the present disclosure to describe the various sides of the racks 102-108.
According to another example, the racks 102-108 may be positioned with their rear sides adjacent to one another (not shown). In this embodiment, the vent tiles 118 may be provided in each aisle 120 and 124. In addition, the racks 102-108 may comprise outlets on top panels thereof to enable heated air to flow out of the racks 102-108.
As described herein above, the CRAC units 114 generally operate to cool received heated air as indicated by the arrows 126. In addition, the CRAC units 114 may supply the racks 102-108 with airflow that has been cooled, through any reasonably suitable known manners and may thus comprise widely available, conventional CRAC units 114. For instance, the CRAC units 114 may comprise vapor-compression type air conditioning units, chiller type air conditioning units, etc. Examples of suitable CRAC units 114 may be found in co-pending and commonly assigned U.S. patent application Ser. No. 10/853,529, filed on May 26, 2004, and entitled “Energy Efficient CRAC Unit Operation,” the disclosure of which is hereby incorporated by reference in its entirety.
Also shown in
The data center 100 is illustrated in
Various manners in which the cooled airflow is supplied by the vent tiles 118 to the racks 102-108 will be described in greater detail with respect to
The vent tiles 118 are illustrated in
As described herein above with respect to
As the cooled air flows through the racks 102a-102d and 104a-104d and therefore the components 116, the cooled air may become heated by absorbing heat dissipated from the components 116. The heated air may exit the racks 102a-102d and 104a-104d through one or more outlets located on the rear sides of the racks 102a-102d and 104a-104d, as indicated by the arrows 126.
The vent tile 118a is illustrated as being in a fully closed position; whereas, the vent tiles 118b-118l are illustrated as being in fully open positions. However, the rack 102a may still draw cooled airflow from the area above the vent tiles 118a-118l as indicated by the arrow 122. The airflow drawn into the rack 102a may comprise airflow supplied into the area by one or more of the vent tiles 118b-118l. In addition, the airflow drawn into the rack 102a, as well as the other racks 102b-102d and 104a-104d, may also comprise airflow that has been heated, for instance, in one or more of the racks 102a-102d and 104a-104d. This airflow may be considered as re-circulated airflow since the heated airflow may have re-circulated into the cooled airflow.
The vent tiles 118a-118l may each include sensors 140 configured to detect one or more conditions of the cooled airflow supplied through the vent tiles 118a-118l. For instance, the sensors 140 may be equipped to detect the temperature of the airflow supplied through respective vent tiles 118a-118l. In this regard, the sensors 140 may include thermistors, thermocouples, or the like. As another example, the sensors 140 may be equipped to detect the mass flow rates of the airflow supplied through respective vent tiles 118a-118l. Thus, for instance, the sensors 140 may comprise anemometers or the like. Alternatively, the mass flow rates of airflow through the vent tiles 118a-118l may be estimated through a determination of, for instance, the pressure in the space 112 and the percentages that the vent tiles 118a-118l are open.
Some or all of the vent tiles 118a-118l may also comprise fans, as shown in
The racks 102a-102d and 104a-104d, may each also include sensors 142 configured to detect one or more conditions of the airflow drawn through the racks 102a-102d and 104a-104d. The sensors 142 may, for instance, be equipped to detect the respective temperatures of the air flowing into each of the racks 102a-102d and 104a-104d and may thus, include thermistors, thermocouples, or the like. In addition, the sensors 142 may be equipped to detect the mass flow rates of air flowing through the respective racks 102a-102d and 104a-104d and may thus include anemometers or the like. Alternatively, the mass flow rates of airflow through the racks 102a-102d and 104a-104d may be estimated through a determination of, for instance, the temperature increases from the inlets of the racks 102a-102d and 104a-104d to the outlets of the racks 102a-102d and 104a-104d along with the power drawn by the components 116 contained in the respective racks 102a-102d and 104a-104d, the speeds of various fans contained in the respective racks 102a-102d and 104a-104d, etc.
The system 202 includes a controller 204 configured to control the operations of the system 202. The controller 204 may, for instance, comprise the computing device 128 shown in
The controller 204 includes an input/output module 206 configured to receive data pertaining to measured or estimated conditions detected at a variety of locations in the data center 100. The input/output module 206 may also be configured to output various commands and other data by the controller 204 as described below. As shown in
The controller 204 may receive data from the sensors 140 and 142 through any reasonably suitable means. For instance, communications between the controller 204 and the sensors 140 and 142 may be effectuated through wired connections or through wireless protocols, such as IEEE 801.11b, 801.11g, wireless serial connection, Bluetooth, etc., or combinations thereof. In one regard, the input/output module 206 may thus also function as an adapter to enable the transfer of data from the sensors 140 and 142 to the controller 204.
Although the vent tiles 118a-118n and the racks 102a-102n are illustrated as including sensors 140 and 142, respectively, alternative means for detecting the temperatures and/or the mass flow rates at these locations may be employed without departing from a scope of the system 202. For instance, the temperatures and/or the mass flow rates may be detected by hand with a handheld device and inputted into the controller 204. As another example, the temperatures and/or mass flow rates may be detected with an adequately equipped semi-autonomous mobile sensor device (not shown). More particularly, the semi-autonomous mobile sensor device may be configured to travel around the vent tiles 118a-118n and the racks 102a-102n to detect the temperatures and/or mass flow rates of air at these locations and to communicate this information to the controller 204. In this regard, the semi-autonomous mobile sensor device may function to gather environmental condition information while requiring substantially fewer sensors in the data center 100. A more detailed description of the semi-autonomous mobile sensor device and its operability may be found in co-pending and commonly assigned U.S. application Ser. No. 10/157,892, filed on May 31, 2002, the disclosure of which is hereby incorporated by reference in its entirety.
In any regard, the data received by the controller 204 via the input/output module 206 may be stored in a memory 212. The memory 212 may also generally be configured to provide storage of software that provides the functionality of the controller 204. In one regard, the memory 212 may be implemented as a combination of volatile and non-volatile memory, such as DRAM, EEPROM, flash memory, and the like.
The data stored in the memory 212 may be accessed by a vent tile influence coefficient (VTI) calculation module 214. In addition, the memory 212 may comprise software or algorithms that the VTI calculation module 214 may implement in calculating the VTI. Although the VTI calculation module 214 has been shown in
In general, the VTI calculation module 214 operates to calculate the VTI for one or more racks 102a-102n. In other words, the VTI calculation module 214 is configured to determine how changes in flow rates of airflow through various vent tiles 118a-118n affect characteristics of airflow through various racks 102a-102n. Thus, for instance, the VTI may be used to determine how one or more vent tiles 118a-118n should be manipulated to vary the temperature of airflow delivered through a rack 102a-102n. Through this knowledge, the mass flow rate of airflow through a particular rack 102a-102n may substantially be controlled through a controlled variance in the airflows through one or more of the vent tiles 118a-118n.
The racks 102a-102n may comprise some or all of the racks 102-108 shown and described with respect to
Various manners in which the VTI may be calculated by the VTI calculation module 214 will now be described in greater detail. As stated herein above, the system 202 may be employed to commission the vent tiles 118a-118n in a data center 100. With respect to the equations set forth below, prior knowledge of rack 102a-102n and vent tile 118a-118n locations may be beneficial since this may reduce the number of VTIs calculated and the unknowns in the system of equations. However, in order to obtain the most accurate correlations between the racks 102a-102n and the vent tiles 118a-118n, the VTIs may be calculated for all possible rack 102a-102n and vent tile 118a-118n combinations.
According to an example, the VTIs for some of the vent tiles 118a-118n may be determined through approximation. In this example, a model may be created a priori to determine which of the vent tiles 118a-118n most affect a particular rack 102a-102n and to determine the VTIs for vent tiles 118a-118n that have relatively less effect on the particular rack 102a-102n by approximation. The approximated VTIs for the vent tiles 118a-118n may be based, for instance, upon their distances from the particular rack 102a-102n. Thus, those vent tiles 118a-118n that are closer to the particular rack 102a-102n may have higher approximated VTIs than those vent tiles 118a-118n that are farther from the particular rack 102a-102n. In this regard, the VTIs for all of the possible rack 102a-102n and vent tile 118a-118n combinations may not need to be determined, thus reducing the amount of time required to determine all of the VTIs.
The relationship between any particular vent tile 118a-118n and any particular rack 102a-102n may be evaluated to determine how the particular vent tiles 118a-118n influence the cooled air supplied to particular racks 102a-102n. This relationship is described herein as the VTI, which, in one respect, relates the mass flow rate of air delivered to racks 102a-102n to the air flow rates through individual vent tiles 118a-118n. This relationship may be written in matrix form as follows:
[VTI]=[MR]●[MVT]−1, Equation (1):
where MR is the vector of mass flow rates of air delivered to each rack 102a-102n and MVT is the vector of mass flow rates of air through each vent tile 118a-118n, of a particular group of racks 102a-102n and vent tiles 118a-118n. In addition, units of MR and MVT may be in kg/s, and VTI is dimensionless.
As may be seen from Equation (1), the matrix VTI may be determined through multiplication of the matrix MR by the inverse of the matrix MVT. In this regard, the VTI may be determined through variations in the mass flow rates of air delivered to the racks 102a-102n and the mass flow rates of air supplied through the vent tiles 118a-118n. The mass flow rates of air supplied through the vent tiles 118a-118n may be varied by varying the operating levels of the vent tiles 118a-118n. The operating levels of the vent tiles 118a-118n may include the percentages at which the vent tiles 118a-118n are open. In this regard, the vent tiles 118a-118n may include means for varying the airflow volumes through the vent tiles 118a-118n. The means for varying the airflow volumes may be manipulated by hand or they may be remotely actuated. In certain instances, for example, the vent tiles 118a-118n may include actuators 220 for controlling the means for varying the airflow volumes. In these instances, in determining the matrix VTI, the actuators 220 may be configured to vary the volume flow rates of air through particular vent tiles 118a-118n during different runs. A more detailed description of various manners in which the VTI may be determined according to various vent tile 118a-118n settings is set forth herein below.
The vent tile actuators 220 may comprise actuators configured to vary the airflows through the vent tiles 118a-118n. Examples of suitable vent tile actuators 220 and vent tiles 118a-118n configured to vary the cooling fluid flow therethrough may be found in commonly assigned U.S. Pat. No. 6,694,759, entitled “Pressure Control of Cooling Fluid Within a Plenum Using Automatically Adjustable Vents”, filed on Jan. 27, 2003, the disclosure of which is hereby incorporated by reference in its entirety. A discussion of various operational modes for these types of vents is disclosed in U.S. Pat. No. 6,574,104, which is also commonly assigned and hereby incorporated by reference in its entirety. In addition, the vent tile actuators 220 may comprise the motors 156 of the fan assemblies 152 depicted in
In addition, the mass flow rate MR through each rack 102a-102n may be determined by rearranging Equation (1) as follows:
[MR]=[VTI]●[MVT]. Equation (2):
As may be seen in Equation (2), if the VTI is known and a particular mass flow rate of air is desired through a particular rack 102a-102n, the VTI may be used to correlate how the settings of one or more vent tiles 118a-118n may be varied to achieve the desired mass flow rate of air through the particular rack 102a-102n.
In certain instances, the airflow supplied into the racks 102a-102n includes airflow that has not been directly supplied through one or more of the vent tiles 118a-118n. Instead, some of the airflow may include airflow that has been re-circulated into the supply airflow. This airflow may include, for instance, airflow that has been heated in one of more of the racks 102a-102n and exhausted into the data center 100. As this re-circulated airflow may affect the temperature of the airflow supplied to the racks 102a-102n, this airflow may be considered in determining VTI. In addition, the re-circulated airflow may be considered in determining the mass flow rates of airflow through the racks 102a-102n. More particularly, Equation (2) may be re-written to include the effects of re-circulation in terms of a re-circulation matrix Γ as follows:
[MR]=[VTI]●[MVT]+[Γ], Equation (3):
where Γ is the matrix of the re-circulation mass flow rate that infiltrates the inlets of the racks 102a-102n and its units may be in kg/s. The values that populate the matrix Γ may be determined through a calculation of the re-circulated airflow infiltration levels into the airflow supplied by the vent tiles 118a-118n into the racks 102a-102n. These re-circulation airflow infiltration levels may be characterized by an index of re-circulation. The index of re-circulation may be termed a supply heat index (SHI), which is defined according to the following equation:
where Q represents the total heat dissipation from all the components in the racks 102a-102n of the data center 100 and δQ represents the rise in enthalpy of the airflow before entering the racks 102a-102n. The SHI and manners in which it may be determined are described in greater detail in co-pending and commonly assigned U.S. patent application Ser. No. 10/446,854, filed on May 29, 2003, and entitled “Air Recirculation Index”, the disclosure of which is hereby incorporated by reference in its entirety.
The relationship between the matrix of the re-circulation mass flow rate Γ and SHI may be represented as follows:
where Γ* is the dimensionless form of Γ. Thus, by determining SHI, the matrix of re-circulation mass flow rate matrix Γ may also be estimated for the racks 102a-102n. In this regard, the controller 204 may optionally comprise an SHI calculation module 230 as shown in
The VTI calculation module 214 may also be configured to calculate VTI based upon the temperature and mass flow rates of airflow through the racks 102a-102n and the vent tiles 118a-118n. In addition, the mass flow rates and temperatures of airflow supplied to the racks 102a-102n may be calculated based upon the VTI and the mass flow rates and temperatures of airflow supplied by the vent tiles 118a-118n and a re-circulation matrix RM, according to the following equation:
[CpMR][TR]=[[VTI]●[CpMVT][TVT]]+[RM], Equation (6):
where TR is the inlet temperature matrix of the racks 102a-102n, TVT is the supply temperature matrix of the vent tiles 118a-118n, and Cp is the specific heat capacity of air. In Equation (6), the multiplication [X][Y] denotes elemental multiplication where each element in [X] is multiplied by its corresponding element in [Y]. In addition, RM=(Γ)*Trec* Cp and the units of RM may be Watts. Thus, for instance, if the mass flow rates of air supplied through the vent tiles 118a-118n and the temperatures at the racks 102a-102n are known, these values may be used to estimate F through, for instance, Equation (4). However, as described below, the change in RM may approach zero or may otherwise constitute a relatively small number and may, in most instances, be disregarded.
Equation (6) may be further refined through detection of the mass flow rates and temperatures at various times (t). For instance, if at time t=i+1, the mass flow rate through a single vent tile 118 changes while the supply temperature remains constant, Equation (6) may be evaluated at time t=i and again at time t=i+1. If the difference is taken for these times, the following equation is obtained:
[CpMR][TR]i−[CpMR][TR]i+1=[[VTI]●[CpMVT][TVT]]i−[[VTI]●[CpMVT][TVT]]i+1+[RM]i−[RM]i+1 Equation (7):
When a single rack (for instance, rack A) is considered with N number of vent tiles 118a-118n, Equation (7) may be re-written as:
[CpMRΔTR]A=[VTI]A●[CpΔMVTTVT]+[ΔRM]A, Equation (8):
where [MRΔTR]A is a 1×N matrix and represents the product of the mass flow rate and inlet temperature change for a single rack (A) in the data center, [VTI]A is a 1×N matrix and is the vent tile index coefficient for rack A, [ΔMVTTVT]A is an N×M matrix and is the product of change in tile flow rate and temperature of the airflow supplied by each vent tile 118a-118n in the data center 100 as the vent tile 118a-118n openings are varied sequentially N times, and [ΔRM]A is the 1×N re-circulation matrix for rack A.
In certain instances, it may be assumed that the change in re-circulation at a rack (A) due to the adjustment of a single vent tile 118 is insignificant. In these instances, it may be assumed then that [ΔRM]A→[10] and Equation (8) may be solved for [VTI]A, which may represent a single row in [VTI] of Equation (6).
In other instances, the re-circulation at the racks 102a-102n may be considered in determining VTI. In these instances, the following equation may be employed to determine VTI while factoring the effects of re-circulation:
where M denotes the mass flow rates of air, r denotes the racks 102a-102n, vt denotes the vent tiles 118a-118n, j denotes the jth rack 102a-102n, k denotes the kth vent tile 118a-118n and is an index for the vent tiles 118a-118n, and m denotes the number of vent tiles k, where k→{1,m}. One such equation will exist for each rack 102a-102n. Thus, each rack 102a-102n will have to solve m equations for the unknown coefficients VTI. In other words, a total of m runs have to be carried out while measuring data at the n racks 102a-102n to calculate the coefficients VTI for all the rack 102a-102n and vent tile 118a-118n combinations. Dividing both sides of Equation (9) by the rack mass flow M, the following equation is obtained:
Equation (10) may be re-written with dimensionless parameters, which are denoted by * superscripts, as follows:
The dimensionless re-circulation term Γ may also be re-written as follows:
where Tout and Tin are the temperatures at the outlet and the inlet of a rack 102a-102n, respectively. The numerator shows the re-circulation heat load dissipated by the re-circulation stream (or external path) and assumes minimal heat loss to the environment. In addition, the numerator shows that the re-circulation stream is fully mixed with the inlet stream prior to being pulled through the rack 102a-102n. The denominator shows the heat load gained by the airflow through the rack 102a-102n. The ratio of re-circulation flow over the mass flow rate of air supplied to the rack may thus be approximated to the ratio of re-circulation load to the actual heat load.
Through substitution, Equation (11) may be re-written as follows:
In determining VTI, a total of m runs may be performed while measuring data at the racks 102a-102n to calculate the coefficients for all of the rack 102a-102n and vent tile 118a-118n combinations. Therefore, for each rack 102a-102n, m number of equations is solved to determine all of the VTI values. Various manners in which VTI may be calculated according to performance of a number of runs will now be described in greater detail herein below with reference first to
The operational mode 300 may be initiated in response to a variety of stimuli at step 302. For example, the operational mode 300 may be initiated in response to a predetermined lapse of time, in response to receipt of a transmitted signal, manually initiated, etc. At step 304, all of the vent tiles 118a-118n of a particular group of vent tiles 118a-118n may be opened (or fan assemblies 152 may be activated). The particular group of vent tiles 118a-118n may include a particular rack 102a-102n, a plurality of racks in a group of racks 102a-102n, all of the racks 102a-102n in a data center 100, etc. In addition, the particular group of vent tiles 118a-118n may comprise all of the vent tiles 118a-118n in a particular row, all of the vent tiles 118a-118n that are supplied with cooled airflow by a particular CRAC unit 114, all of the vent tiles 118a-118n that are supplied with cooled airflow by a plurality of CRAC units 114, etc. Opening of the vent tiles 118a-118n may include activating the motors 156 of vent tiles 118a-118n equipped with fan assemblies 152 to cause the fans 154 to rotate at a predetermined speed. The fan 154 speed and the opening of the vent tiles 118a-118n are to be taken as being synonymous throughout the present disclosure. Thus, if the vent tiles 118a-118n are described as being 100% open, this is equivalent to a vent tile 118a-118n equipped with a fan assembly 152 operating at 100%.
In one example, the vent tiles 118a-118n may be opened to 100% open. In other examples, the vent tiles 118a-118n may be opened to various levels and the operational mode 300 may be repeated at the various levels. For instance, during a first iteration of the operational mode 300, the vent tiles 118a-118n may be opened to 100% open, to 90% open during a second iteration, to 80% open during a third iteration, and so forth. Determination of VTI at the various vent tile 118a-118n settings may be beneficial since VTI may change during operation of the vent tiles 118a-118n. More particularly, as the global set of vent tiles 118a-118n change, so too will the VTI. Thus, if the global set of vent tile 118a-118n configurations changes, the correct VTIi, where i=global vent tile 118a-118n configuration, may be employed to determine the relationships between the racks 102a-102n and the vent tiles 118a-118n.
In any regard, at step 306, the mass flow rates of airflow delivered to each of the racks 102a-102n may be determined. The mass flow rates of airflow delivered to each of the racks 102a-102n may be detected through use of mass flow rate sensors 210. Alternatively, the mass flow rates of airflow may be estimated through various other means. For instance, the mass flow rates may be estimated through use of temperature drop detection along with power draw detection, pressure differences, fan speeds, etc. The mass flow rates of airflow may be transmitted or otherwise sent to the input/output module 206 and may also be stored in the memory 212.
In addition, the mass flow rate of airflow supplied by each vent tile 118a-118n may be determined at step 308. Again, the mass flow rates of airflow supplied by each vent tile 118a-118n may be detected by mass flow rate sensors or they may be estimated through other means as described herein above. In addition, the mass flow rates of airflow supplied by each vent tile 118a-118n may be transmitted or otherwise sent to the input/output module 206 and may also be stored in the memory 212.
At step 310, one of the vent tiles 118a-118n, for instance, vent tile 118a, may be closed to substantially prevent the flow of air therethrough. The selection of which one of the vent tiles 118a-118n to close may be predetermined or it may be random. In any respect, the mass flow rates of airflow through the racks 102a-102n may be determined at step 312 and the mass flow rates of airflow through each open vent tile 118b-118n may be determined at step 314. The mass flow rates of airflow through the racks 102a-102n and the vent tiles 118b-118n may be determined as described herein above with respect to steps 306 and 308, respectively. In addition, the mass flow rates of airflow through the racks 102a-102n and the vent tiles 118b-118n may be stored in the memory 212.
The vent tile 118a that was closed at step 310 may be opened at step 316. This vent tile 118a may be opened to the percentage it was open at step 304.
At step 318, it may be determined whether another vent tile 118a-118n is to be closed. In one example, the “yes” condition may be reached at step 318 until conditions for each of the vent tiles 118a-118n being in closed positions have been determined. In another example, the “yes” condition at step 318 may be reached for a predetermined number of vent tiles 118a-118n equaling less than all of the vent tiles 118a-118n. In any respect, the determination of whether to close another vent tile 118a-118n may be made by the controller 204.
If it is determined that another vent tile 118a-118n is to be closed, another vent tile 118a-118n, for instance, vent tile 118b, may be closed at step 320. Again, the selection of which one of the vent tiles 118a-118n to close may be predetermined or it may be random. In any respect, the mass flow rates of airflow through the racks 102a-102n may be determined again at step 322 and the mass flow rates of airflow through each open vent tile 118b-118n may be determined at step 324. The mass flow rates of airflow through the racks 102a-102n and the vent tiles 118a, 118c-118n may be determined as described herein above with respect to steps 306 and 308, respectively. In addition, the mass flow rates of airflow through the racks 102a-102n and the vent tiles 118a, 118c-118n may be stored in the memory 212.
The vent tile 118b that was closed at step 320 may be opened at step 326. This vent tile 118b may also be opened to the percentage it was open at step 304.
Following step 326, it may be determined if another vent tile 118a-118n is to be closed at step 318. Steps 318-326 may be repeated for a predetermined number of times or until the conditions for each of the vent tiles 118a-118n being closed have been determined. In this case, which equates to a “no” condition at step 318, the controller 204 may calculate the VTI, as indicated at step 328. More particularly, the VTI calculation module 214 of the controller 204 may calculate the VTI based upon the information regarding the determined mass flow rates through the racks 102a-102n and the vent tiles 118a-118n with various vent tiles 118a-118n closed stored in the memory 212. As described in greater detail herein above, the VTI calculation module 214 may be configured to calculate the VTI in matrix form according to Equation (1).
Although the operational mode 300 may end following calculation of the VTI at step 328, optional procedures may be implemented in various circumstances. The optional procedures may be implemented, for instance, in cases where the blowers of CRAC units 114 are over-provisioned, which may lead to an excessive amount of airflow being supplied from the vent tiles 118a-118n. In some instances, this excessive flow may cause the airflow to dramatically exceed the flow necessary for the racks 102a-102n. In these instances, evaluation of VTI may not be possible. To overcome this possibility, the optional procedures depicted in
In other instances, the mass flow rates of air supplied through the vent tiles 118a-118n may fall below or exceed the required mass flow rates of air required to safely operate the equipment housed in the racks 102a-102n. In these instances, an optional pre-commissioning procedure depicted in
With reference first to
If it is determined that the VTI values are not near zero at step 352, it may be determined whether additional vent tiles 118a-118n are to be closed at step 356. The determination made at step 356 is similar to the determination made at step 318. Thus, for instance, if the VTI was calculated for a smaller number of vent tiles 118a-118n than the entire set of vent tiles 118a-118n to be considered, then it may be determined that additional vent tiles 118a-118n are to be considered and steps 320-328 may be repeated. In addition, steps 352-358 may also be performed depending upon the outcome of the VTI calculation performed at step 328.
If, however, it is determined that no further vent tiles 118a-118n are to be considered at step 356, the operational modes 300 and 350 may end as indicated at step 358. The end condition at step 358 may comprise a standby mode since the operational modes 300 and 350 may be re-initiated in response to a receipt of a signal to re-initiate, after a predetermined lapse of time, manually re-initiated, etc.
With reference now to
The outputs of the one or more CRAC units 114 may be reduced, for instance, by reducing the speed of the blowers configured to supply cooled airflow into the space 112. In addition, steps 306-328 may be repeated with the outputs of the one or more CRAC units 114 reduced by the predetermined amount. Moreover, steps 362-368 may also be performed following steps 306-328.
As with the operational mode 350, if it is determined that all of the VTI values are not near zero at step 362, it may be determined whether additional vent tiles 118a-118n are to be closed at step 366. The determination made at step 366 is similar to the determination made at step 318. Thus, for instance, if the VTI was calculated for a smaller number of vent tiles 118a-118n than the entire set of vent tiles 118a-118n to be considered, then it may be determined that additional vent tiles 118a-118n are to be considered and steps 320-328 may be repeated. In addition, steps 362-368 may also be performed depending upon the outcome of the VTI calculation performed at step 328.
If, however, it is determined that no further vent tiles 118a-118n are to be considered at step 366, the operational modes 300 and 360 may end as indicated at step 368. The end condition at step 368 may comprise a standby mode since the operational modes 300 and 360 may be re-initiated in response to a receipt of a signal to re-initiate, after a predetermined lapse of time, manually re-initiated, etc.
Referring to
As shown in
At step 376, the vent tiles 118a-118n may all be opened to uniform levels, such that the mass flow rates of airflow supplied through the vent tiles 118a-118n are approximately equivalent for each of the vent tiles 118a-118n. The vent tiles 118a-118n may be set between, for instance, around 25-100% open. At step 378, the mass flow rates of air supplied through each of the vent tiles 118a-118n may be determined. The mass flow rates may be determined in any of the manners described herein above with respect to step 308. In addition, the mass flow rates of air supplied through each of the vent tiles 118a-118n may be summed to obtain a total mass flow rate amount at step 380. At step 382, the total mass flow rate (TMF) of air supplied through each of the vent tiles 118a-118n (TMFVT) is compared with the total mass flow rate (TMF) of air required by the equipment housed in the racks 102a-102n (TMFR) to determine whether the TMFVT is within a predetermined percentage (X %) of the TMFR. Although the comparison between the flow rates at step 382 has been described with percentages, the comparison may be based upon any other reasonably suitable method of determining whether values are within a predetermined range from other values. For instance, the comparison may be based upon preset difference in flow rate value, etc.
In addition, the predetermined percentage (X %) may comprise any reasonably suitable percentage value and may be user defined. According to an example, however, the predetermined percentage (X %) may be approximately 10-20%.
In any regard, if the TMFVT is within the predetermined percentage (X %) of the TMFR, the operational mode 370 may end and the operational mode 300 may be performed beginning at step 304. In addition, the vent tiles 118a-118n may be at the level of opening indicated at step 376. However, if the TMFVT is not within the predetermined percentage (X %) of the TMFR, it may be determined as to whether the TMFVT falls below or equals the predetermined percentage (X %) of the TMFR at step 384.
If the TMFVT falls below or equals the predetermined percentage (X %) of the TMFR, the vent tiles 118a-118n may be opened by a predefined percentage (W %), as indicated at step 386. The predefined percentage (W) may be determined, for instance, through a trial and error process. By way of example, the openings of the vent tiles 118a-118n may be increased by 10% and may be further increased by 10% increments for subsequent iterations of step 386. If it is found that the 10% increase does not change VTI, then the predefined percentage (W) may be increased. In addition, steps 378-388 may be repeated until it is determined that the TMFVT is within the predetermined percentage (X %) of the TMFR.
If the TMFVT exceeds the predetermined percentage (X %) of the TMFR, the vent tiles 118a-118n may be closed by the predefined percentage (W %), as indicated at step 388. The predefined percentage (W) may be determined, for instance, through a trial and error process. By way of example, the openings of the vent tiles 118a-118n may be decreased by 10% and may be further decreased by 10% increments for subsequent iterations of step 388. If it is found that the 10% decrease does not change VTI, then the predefined percentage (W) may be decreased. The maximum levels to which the openings in the vent tiles 118a-118n are decreased may be limited to minimum levels of required airflow for the components 116 as set forth by, for instance, the component 116 manufacturers. Again, steps 378-388 may be repeated until it is determined that the TMFVT is within the predetermined percentage (X %) of the TMFR.
Through implementation of the operational mode 300 and one or more of the optional operational modes 350, 360, and 370, the VTI may be determined, which may be used to determine the relationships between the racks 102a-102n and the vent tiles 118a-118n.
In addition, the VTI may be employed to identify vent tiles 118a-118n that may have relatively little or no influence over any of the racks 102a-102n.
With reference first to
At step 392, it may be determined whether the number of vent tiles 118a-118n identified at step 391 exceed a predetermined threshold (T %). Again, the predetermined threshold (T %) may be user-defined and may be based upon a desired percentage. Thus, for instance; if a certain percentage of all of the vent tiles 118a-118n, such as, 10-20% or more, is identified at step 391, then the steps outlined in one or more of the operational modes 350, 360, and 370 may be performed as indicated at step 393. In other words, the pre-commissioning steps outlined in any of those operational modes 350, 360, 370 may be performed to reduce the number of vent tiles 118a-118n that have little or no influence over any of the racks 102a-102n.
If it is determined at step 392 that the number of vent tiles 118a-118n identified at step 391 falls below or equals the predetermined threshold (T %), those identified vent tiles 118a-118n may be deactivated or replaced as indicated at step 394. At step 394, those identified vent tiles 118a-118n may be deactivated by closing them completely, removing power supply to those vent tiles 118a-118n, etc. Alternatively, those identified vent tiles 118a-118n may be replaced with standard vent tiles 118a-118n or tiles that do not have vents. In addition, those identified vent tiles 118a-118n may be installed in locations where vent tiles are known to have greater levels of influence over the racks 102a-102n. In this regard, the relatively more expensive vent tiles 118a-118n may be used more efficiently at locations where they may be of greater utility and the relatively less expensive tiles may be used in locations where vent tiles 118a-118n would not be of great use.
With reference now to
The operational mode 400 may be initiated in response to a variety of stimuli at step 402. For example, the operational mode 400 may be initiated in response to a predetermined lapse of time, in response to receipt of a transmitted signal; manually initiated, etc. At step 404, all of the vent tiles 118a-118n of a particular group of vent tiles 118a-118n may be opened in manners similar to those described herein above with respect to step 304 in
At step 406, the temperatures at the inlets and the outlets of the racks 102a-102n may be detected. In certain instances, the temperatures at the inlets and the outlets of the racks 102a-102n may be detected by the temperature sensors 208. In other instances, the temperatures may be detected by hand with a handheld device, with an adequately equipped semi-autonomous mobile sensor device, etc. In any event, the detected temperatures may be inputted into or transmitted to the controller 204 and may be stored in the memory 212.
In similar fashion to step 306 in
At step 410, the temperatures of the airflow supplied by the vent tiles 118a-118n may be detected. These temperatures may be detected by the temperature sensors 208, by hand with a handheld device, with an adequately equipped semi-autonomous mobile sensor device, etc. In addition, the detected temperatures may be inputted or transmitted to the controller 204 and may be stored in the memory 212.
At step 412, the mass flow rate of airflow through the vent tiles 118a-118n may be determined in manners similar to those described herein above with respect to step 308. In addition, the mass flow rates of airflow supplied through the vent tiles 118a-118n may be transmitted or otherwise sent to the input/output module 206 of the controller 204 and may also be stored in the memory 212.
The temperatures detected at the racks 102a-102n and the vent tiles 118a-118n may be employed to calculate SHI, as indicated at step 414. In addition, the calculated SHI may be stored in the memory 212. As described in co-pending U.S. patent application Ser. No. 10/446,854, SHI may be determined through the following equation:
where (Γrin) and (Γrout)i,j are the respective inlet and outlet temperatures from the ith rack in the jth row of racks. In addition, Tref denotes the temperature of the cooled air supplied by one or more of the CRAC units 114 and may denote the average temperature of the airflow supplied through the vent tiles 118a-118n.
At step 416, one of the vent tiles 118a-118n, for instance, vent tile 118a, may be closed to substantially prevent the flow of air therethrough. The selection of which of the vent tiles 118a-118n may be based upon the manners described herein above with respect to step 310. In any respect, the temperatures at the inlets and the outlets of the racks 102a-102n may be detected at step 418, the mass flow rates of airflow delivered to each of the racks 102a-102n may be determined at step 420, the temperatures of the airflow supplied by the vent tiles 118a-118n may be detected at step 422, and the mass flow rate of airflow through the vent tiles 118a-118n may be determined at step 424. The temperatures detected at steps 418 and 422 may be employed to calculate the SHI at step 426. In addition, the determinations made at steps 418-426 may be stored in the memory 212.
The vent tile 118a that was closed at step 416 may be opened at step 428. This vent tile 118a may be opened to the percentage it was open at step 404.
At step 430, it may be determined whether another vent tile 118a-118n is to be closed. In one example, the “yes” condition may be reached at step 430 until conditions for each of the vent tiles 118a-118n being in closed positions have been determined. In another example, the “yes” condition at step 430 may be reached for a predetermined number of vent tiles 118a-118n equaling less than all of the vent tiles 118a-118n. In any respect, the determination of whether to close another vent tile 118a-118n may be made by the controller 204.
If it is determined that another vent tile 118a-118n is to be closed, another vent tile 118a-118n, for instance, vent tile 118b, may be closed at step 432. Again, the selection of which one of the vent tiles 118a-118n to close may be predetermined or it may be random. In any respect, the temperatures at the inlets and the outlets of the racks 102a-102n may be detected at step 434, the mass flow rates of airflow delivered to each of the racks 102a-102n may be determined at step 436, the temperatures of the airflow supplied by the vent tiles 118a, 118ac-118n may be detected at step 438, and the mass flow rate of airflow through the vent tiles 118a, 118c-118n may be determined at step 440. The temperatures detected at steps 434 and 438 may be employed to calculate the SHI at step 442. In addition, the determinations made at steps 434-442 may be stored in the memory 212.
The vent tile 118b that was closed at step 432 may be opened at step 444. This vent tile 118b may also be opened to the percentage it was open at step 404.
Following step 444, it may be determined if another vent tile 118a-118n is to be closed at step 430. Steps 432-444 may be repeated for a predetermined number of times or until the conditions for each of the vent tiles 118a-118n being closed have been determined. In this case, which equates to a “no” condition at step 430, the controller 204 may calculate the VTI, as indicated at step 446. More particularly, the VTI calculation module 214 of the controller 204 may calculate the VTI based upon the information stored in the memory 212 regarding the detected temperatures at the inlets of the racks 118a-118n, the determined mass flow rates through the racks 102a-102n and the vent tiles 118a-118n, and the SHI's with various vent tiles 118a-118n closed. As described in greater detail herein above, the VTI calculation module 214 may be configured to calculate the VTI in matrix form according to Equations (3), (8), (9), or (13).
Although the operational mode 400 may end following calculation of the VTI at step 446, optional procedures may be implemented in various circumstances. The optional procedures may be implemented, for instance, in cases where the blowers of CRAC units 114 are over-provisioned, which may lead to an excessive amount of airflow being ejected from the vent tiles 118a-118n. In some instances, this excessive flow may cause the airflow to dramatically exceed the flow necessary for the racks 102a-102n. In these instances, evaluation of VTI may not be possible. As described in detail herein above, to overcome this possibility, the optional procedures depicted in
In other instances, the mass flow rates of air supplied through the vent tiles 118a-118n may fall below or exceed the required mass flow rates of air required to safely operate the equipment housed in the racks 102a-102n. In these instances and as also described herein above, an optional pre-commissioning procedure depicted in
In addition, the operational modes 390 and 395 may be performed following operational mode 400 to identify and deactivate, replace, or group, vent tiles 118a-118n that have little or no influence over any of the racks 102a-102n.
In certain examples, calculation of SHI at steps 414, 426, and 442 may be omitted. For instance, these steps may be omitted if it may be assumed that changes in re-circulation due to the closure of a single vent tile 118a-118n are substantially minute. In this instances, the ΔΓ in Equations (3), (8), (9), or (13) may be considered as equaling zero. In this regard, the ΔΓ may be omitted from Equations (3), (8), (9), or (13) and Equations (3), (8), (9), or (13) may be solve determine VTI without considering ΔΓ.
The operations illustrated in the operational modes 300, 350,360,370,390,395, and 400 may be contained as a utility, program, or a subprogram, in any desired computer accessible medium. In addition, the operational modes and 300, 350, 360, 370, 390, 395, and 400 may be embodied by a computer program, which can exist in a variety of forms both active and inactive. For example, they can exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats. Any of the above can be embodied on a computer readable medium, which include storage devices and signals, in compressed or uncompressed form.
Exemplary computer readable storage devices include conventional computer system RAM, ROM, EPROM, EEPROM, and magnetic or optical disks or tapes. Exemplary computer readable signals, whether modulated using a carrier or not, are signals that a computer system hosting or running the computer program can be configured to access, including signals downloaded through the Internet or other networks. Concrete examples of the foregoing include distribution of the programs on a CD ROM or via Internet download. In a sense, the Internet itself, as an abstract entity, is a computer readable medium. The same is true of computer networks in general. It is therefore to be understood that any electronic device capable of executing the above-described functions may perform those functions enumerated above.
The computer system 500 includes one or more controllers, such as a processor 502. The processor 502 may be used to execute some or all of the steps described in the operational modes 300, 350, 360, 370, and 400. Commands and data from the processor 502 are communicated over a communication bus 504. The computer system 500 also includes a main memory 506, such as a random access memory (RAM), where the program code for, for instance, the computing device 128 or the controller 204, may be executed during runtime, and a secondary memory 508. The secondary memory 508 includes, for example, one or more hard disk drives 510 and/or a removable storage drive 512, representing a floppy diskette drive, a magnetic tape drive, a compact disk drive, etc., where a copy of the program code for the vent tile evaluation system may be stored.
The removable storage drive 510 reads from and/or writes to a removable storage unit 514 in a well-known manner. User input and output devices may include a keyboard 516, a mouse 518, and a display 520. A display adaptor 522 may interface with the communication bus 504 and the display 520 and may receive display data from the processor 502 and convert the display data into display commands for the display 520. In addition, the processor 502 may communicate over a network, for instance, the Internet, LAN, etc., through a network adaptor 524.
It will be apparent to one of ordinary skill in the art that other known electronic components may be added or substituted in the computer system 500. In addition, the computer system 500 may include a system board or blade used in a rack in a data center, a conventional “white box” server or computing device, etc. Also, one or more of the components in
What has been described and illustrated herein is a preferred embodiment of the invention along with some of its variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention, which is intended to be defined by the following claims—and their equivalents—in which all terms are meant in their broadest reasonable sense unless otherwise indicated.
This application is related to commonly assigned and co-pending U.S. patent application Ser. ______, (Attorney Docket No. 200407254-1) entitled “Correlation of Vent Tile Settings and Rack Temperatures”, filed on even date herewith, the disclosure of which is hereby incorporated by reference in its entirety.