Slide1: Data Center Power & Cooling Keeping up with Change 2003 IT Infrastructure Trends and Challenges Power Infrastructure Basics Planning and Deployment Cooling Best Practices Ken Baker MrRack@hp.com ISS Rack & Power May 9, 2003 Notice: The information contained herein is copyrighted property and may not be reproduced without the written permission of the Hewlett Packard Company Lone Star Chapter


why are we talking today?: why are we talking today? due to server and storage densification, yesterday’s data center infrastructure and design practices cannot keep pace with the growing demands for power and cooling the high density architectures and products we talked about for the last two years are here today


3 Key Metrics to Remember: 3 Key Metrics to Remember Supply Load Distribution


on the customer front: on the customer front what are the right questions to be asking? have I engaged facility engineering in planning for growth and hardware deployments? have I planned for growth in power and heat density? is my data center configured to provide sufficient airflow to the new servers & storage products? you own and manage the environment !


Slide5: power trends


power supplies trends: power supplies trends power supply capacities have not grown substantially and still range from 200 -1200 watts the physical size is decreasing high voltage operation (200+VAC) as a requirement is becoming more common power supply efficiency is starting to improve


power trends: power trends two years ago average servers per rack = 4 - 6 average watts per rack = 1500 - 3000 today average servers per rack = 8 - 12 average watts per rack = 5000 - 6000 not unusual to see 7- 12 - 18KW implementations


Slide8: processor trends Xeon MP 400Mhz 667 Mhz >90W 91W per CPU >100W 62-90W Xeon DP 533Mhz 667Mhz 800Mhz 95W 38W per CPU >100W 400Mhz 75W 88W 133Mhz 125W


What Key Metric is Changing Here?: What Key Metric is Changing Here? Load


historical server comparison: historical server comparison the original SystemPro first ProLiant PC Server 33MHz 386 processor 8 Mb RAM, 210 Mb HDD $36,000 400W BL 10e blade first ProLiant blade server 900 MHz processor 512 Mb RAM, 40 Gb HDD $1,800 20-25W(x24)


server trends: server trends generation over generation density growth rates 2p, 4gb, 2hdd, 1pci 2p, 4gb, 6hdd, 2pci 4p, 8gb, 4hdd, 3pci 1U 2U 4U


combining all of the trends: combining all of the trends Watts Per U Overall power density per Rack Unit “U” is becoming the key metric, capturing all of the key contributors to the densification. platform, form factor power in watts coolest hottest


Server Power Density Per Rack Unit: Server Power Density Per Rack Unit Storage, PL and ML Class Watts per U


Server Power Density Per Rack Unit: Server Power Density Per Rack Unit BL and DL Class Watts per U


Slide15: delivering power for hyper dense data centers:


Delivering More Power: Delivering More Power overcoming the “desktop mentality” standard 120V, 10A PC power cord is all you need open power receptacles indicates available unlimited power powering the infrastructure. Must Break the 15A and 20A branch barrier Options include multiple 30A and 3 Phase branches more power requires larger power distribution hardware


beyond the desktop: beyond the desktop larger copper cabling SJ/SO 3 x 10Awg SJ/SO 4 x 10Awg NEMA L6-30P, 208V, 30A, 3w NEMA L15-30P 208V, 30A 3ph, 4w


beyond the desktop: beyond the desktop IEC 309 Pin/Sleeve Plugs 16A 1Ø 32A 1Ø 32A 3Ø larger copper cabling HO5V - 3x4.0mm HO5V – 5x4.0mm


power density issues: power density issues on the electrical side, why is deploying full racks of servers a problem? total rack load not the problem issue lies with how power is distributed line cord/distribution outlet relationship and restrictions


commercial branch circuits in North America: commercial branch circuits in North America today’s power densities dictate leaving the 120V infrastructure, using multiple 208V 30A feeds and looking forward to using 3 phase power to meet the density demands. power limitations associated with each common branch circuit type


What Key Metric is Changing Here?: What Key Metric is Changing Here? Distribution


influence of the panel: influence of the panel 84 pole panel, 42 2-pole locations 208VAC breakers requires 2 poles 30 amp breaker is limited by NEC to 24A continuous duty 24A x 42 2-pole breakers 150kVA available! Appears to be plenty of overall power! typical panel with 84 pole positions


PDU/panel infrastructure: PDU/panel infrastructure x8 NEMA 5-15P (for Low Volt use ONLY) – 15A each/12A total x8 IEC 320 C13 outlets 10A each/12A total x4 IEC320 C19 Outlets 16A each/12A total Each IEC C19 Outlet is limited to 12A 24A continuous duty L6-30R limited to 24A continuous 2 pole 30A breaker typical panel with 84 pole positions


power requirements: power requirements


commercial branch circuits in North America: commercial branch circuits in North America


example of distribution limitations: example of distribution limitations 24A PDU 208V single/bi-phase PDU limited to 4992 VA subject cabinet of 21 DL380G2 would require 8560 VA total power or two 24A PDUs redundancy is required, that doubles the PDU count to 4 4 breakers (8 poles) per cabinet (4 PDUs) = 80 poles 10 cabinets per 84 pole panel 210 servers provided with power


increasing power capacity: increasing power capacity why not install larger PDU? any larger 1/2 phase PDU eliminates cost effective pluggable solutions (hardwiring) no cost-effective standard connectors > 30A only potential solution move to 3 phase power distribution cost effective pluggable solution > NEMA L15-30P


30A 3 phase PDU power : 30A 3 phase PDU power most efficient way to distribute power overall available power in a 30A circuit rises from 4995 VA to 8650VA total available current rises from 24 A to 42 A fewer overall panel positions used per rack of high density loads


panel utilization for racks of DL380G2: panel utilization for racks of DL380G2


Slide30: single-phase to 3-phase density comparison


What have we changed?: What have we changed? copper cabling SJ/SO 3 x 10Awg SJ/SO 4 x 10Awg NEMA L6-30P, 208V, 30A, 3w NEMA L15-30P 208V, 30A 3ph, 4w Before After


summary/power issues: summary/power issues server densification is toping out conventional power infrastructure and methods we have exceeded the ability of a plug and play solution to be deployed in a high density manner due to power distribution limitations without developing new approaches moving to 3 phase power to the rack enclosure is the future


Slide33: power planning


power planning methods: power planning methods use “name-plate” ratings worked in yesterdays environment costly method resulting in wasted infrastructure $ use actual power measurements most accurate approach numbers are difficult to generate and collect use “ProLiant Power Calculators” best practice for advanced planning numbers are more realistic Factoring for future growth Rate of change in today’s market continues at 25%-30%


ISS power sizing calculators: ISS power sizing calculators


calculator public website: calculator public website http://h18001.www1.hp.com/partners/microsoft/utilities/power.html


about power calculators: about power calculators based on actual system measurements taken on systems running NT and exercise utilities all major system components (CPU, memory and drives) are exercised at 100% duty cycle power results may be higher than your actual configuration leaving you extra headroom calculators can be found on each servers Active Answers page under configuration tools new calculator public website


Slide38: Issues in cooling hyper dense data centers


cooling trends: cooling trends as power consumption grows, so does the thermal demands cooling needs must be expressed in tons of AC today largest issue is not finding the cooling media, but finding a way to get it where it belongs driving new technologies to deliver cooled media


cooling requirements: cooling requirements


cooling trends: cooling trends 2 years ago average BTU load was 10-15K BTU’s average U size was 5-7U today average BTU load is approaching 28K BTU’s average U size is moving down to 3U


air-conditioning: air-conditioning issues we never thought we would have to tackle how many servers can a ton of A/C cool? how do we maximize efficiency? how much airflow does a densely populated cabinet require? how does the warmed air get back to the CRAC?


The mechanics of server airflow: The mechanics of server airflow Cool air in, warm air out All servers today are designed for front to back airflow


airflow data for servers: airflow data for servers


hot isle – cold isle data center layout: hot isle – cold isle data center layout


Datacenter air-conditioning: Datacenter air-conditioning one ton of air-conditioning is 12,000BTU’s how big is a ton of air-conditioning? one ton DX air conditioner takes roughly the cubic space of a 42U cabinet problem is delivery and return, not production


issues of poor CRAC placement: issues of poor CRAC placement If CRAC is placed to close to cabinets negative air pressure is created, robbing system of cooling combined with end of row recirculation, this can create some big hot spots


The negative affects of mixed air: The negative affects of mixed air Contributors: low ceiling - low return air volume blanking panels not used internal rack recirculation end of row recirculation low volume perf tiles low supply plenum pressure


blanking panels, not for looks any more: blanking panels, not for looks any more any ISS product any ISS product any ISS product any ISS product


dual supply plenum configuration for supporting high-density solutions : dual supply plenum configuration for supporting high-density solutions


data center research: data center research hp “cool labs” research on analysis of data center with CFD theoretical model constructed physical model of to prove theory physical model analysis proved theoretical model assumptions analyzed accuracy proven to be within 7%


hp data center validation services”Smart Cooling”: hp data center validation services ”Smart Cooling” Current setup, determine need for detailed validation services engage the data center services team front end information from the customer is compared against criteria document if the data center exceeds the key criteria the services team suggests a detailed 3D CFD analysis. rule of thumb, data centers exceeding 100 W per sq ft are primary candidates for detailed analysis. (gross load over gross area)


data center before “Static Smart Cooling”: data center before “Static Smart Cooling” Vertical Recirculation Highest temperature in room = 57 C (Goal is 45 C)


computational fluid dynamics: computational fluid dynamics detailed CFD analysis first models the customer data center’s physical layout air conditioning resources enterprise infrastructure equipment then provisions air conditioning resources CRAC unit settings, perforated floor tile layout return air vents / supply vents (if applicable) heat load distribution to provide the best possible support for the customer’s enterprise


data center after “Static Smart Cooling”: data center after “Static Smart Cooling” Highest temperature in room = 47 C (HPL goal was 45 C) Vertical Recirculation Removed Before After


What Key Metric is Changing Here?: What Key Metric is Changing Here? Distribution


Recommendations: Recommendations When arranging cabinets in a data center, arrange front to front, back to back Use “Blanking Panels” to fill all empty space in racks. This prevents the short circuiting of cold air to the hot aisle. Calculate air conditioning based on “Sensible Capacity” not rated tonnage. Map out a maximum load on the facility and keep to it May involve empty slots in racks


Recommendations (cont): Recommendations (cont) Avoid creating hot spots Work to balance load in your data center to maximize HVAC capabilities Biggest mistake in laying out equipment in data centers is arranging by component type Racks and rows full of servers (high heat loads) Racks and rows full of Storage


Slide59: MrRack@hp.com – Ken Baker MrPower@hp.com – Bob Pereira Questions?


References: References 1. Patel, C.D., Bash, C.E., Belady, C., Stahl, L., Sullivan, D., July 2001, “Computational fluid dynamics modeling of high compute density data centers to assure system inlet air specifications”, Proceedings of IPACK’01 – The PacificRim/ASME International Electronics Packaging Technical Conference and Exhibition, Kauai, Hawaii. [Modeling Validation] 2. Stahl, L., Belady, C. L., Oct 2001, “Designing an Alternative to Conventional Room Cooling”, Proceedings of the INTELEC’01 International Telecommunications Energy Conference, Edinburgh, Scotland ……………………………[Infrastructure] 3. Friedrich, R., Patel, C.D., Jan 2002, “Towards planetary scale computing - technical challenges for next generation Internet computing”, THERMES 2002, Santa Fe, New Mexico……………………………………………..[Pervasive Computing and Data Centers] 4. Patel, C.D., Sharma, R.K, Bash, C.E., Beitelmal, A, May 2002, “Thermal Considerations in Cooling Large Scale High Compute Density Data Centers”, ITherm 2002 - Eighth Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems, San Diego, California……………………. [Static Provisioning] 5. Sharma, R.K, Bash, C.E., Patel, C.D., Dimensionless Parameters for Evaluation of Thermal Design and Performance in Large Scale Data Centers, 8th AIAA/ASME Joint Thermophysics and Heat Transfer Conference, St. Louis, June 2002. [Data Center Figure of Merit, non-dimensional numbers for data centers] 6. DeLorenzo, D., Thermal Trends and Inflection, 7X24 Exchange 2002, Orlando, Florida [Trends]