ASHRAE Journal - December 2020 - 36

TECHNICAL FEATURE

commitment and adopts a " plan-do-check-act " continuous improvement cycle-based framework. Because of
its top energy consumer status and expected growth,
NERSC is the lab's " signiﬁcant energy and water use " in
the EWMS and is subject to more rigorous requirements
on operational control, workforce competency, and
procurement by the ISO standard. These requirements
in turn challenge the team to continuously explore additional opportunities while ensuring harvested savings
persist by monitoring any signiﬁcant deviation in energy
or water performance.
Some of the new challenges that the team is tackling
today include working with the HPC vendor on improved
control of the system's integral blower fans. The NERSC
team has developed a control script for the operating system, which interties the HPC system's cooling coil valve
controls with cooling plant water temperature controls,
enabling the capability to shift the load ratio between
the blower fans and AHU cooling coil under different
environmental conditions. The cooling towers consume
large volumes of fresh water every day, and the team is
exploring alternative heat rejection solutions that require
no water or much less water. In addition, the upcoming Perlmutter on-chip liquid cooling system opens new
options and energy efﬁciency strategies to cool the next
generation of high power density machines.

Key Lessons Learned and Outlook

The NERSC team at LBNL has demonstrated that,
through interdisciplinary collaboration and an open
mind to viewing risks and beneﬁts, it is possible for an
HPC facility to achieve extraordinarily low PUE and continuously improve and innovate leveraging data analytics. We hope the valuable lessons learned beneﬁt other
HPC facilities and other types of data centers.
* Backup power equipment such as UPSs and generators are capital-intensive investments, so consider limiting coverage to absolutely essential loads.
* " Compressor-free cooling " using cool outdoor air or
cooling tower water (or both) is an effective way to lower
a data center's PUE.
* A great deal of energy can be saved with control
changes and low-cost measures (or " ongoing commissioning " ) before capital investment is necessary. Leveraging a
team approach and data analytics will yield best results.
* Data analytics tools accessible by multidisciplinary
teams is a powerful process for breaking down silos

ASHRAE JOURNAL

ashrae.org

D ECEM BER 2020

among business units. A normal data center infrastructure management (DCIM) system with a prioritized metering conﬁguration could help data center teams make
better operational and business decisions.
* Manage energy and water using a holistic approach
with management's support rather than " project-by-project. " A matrixed team representing all key functions should
meet regularly and participate in making decisions.

Acknowledgment

The authors would like to acknowledge Rachel
Shepherd in the Federal Energy Management Program
(FEMP) Ofﬁce at the U.S. Department of Energy for
funding this case study. We would like to thank NERSC
energy efﬁciency team members Steve Greenberg, John
Eliot, Deirdre Carter and Jeff Broughton, who participated in interviews that provided rich information for
this case study. Their help with reviewing earlier drafts
is appreciated. We want to thank Natalie Bates at the
Energy Efﬁcient HPC Working Group (EE HPC WG) for
leading these interviews and others who participated.
We also appreciate Dale Sartor at LBNL for providing
materials and encouragement for this case study.

References

1. NERSC. 2016. " NERSC Mission. " National Energy Research
Scientiﬁc Computing Center. https://tinyurl.com/y4nqstse/
2. The Green Grid. 2012. " White Paper #49, PUE: A
Comprehensive Examination of the Metric. " The Green Grid.
3. Bautista, E., et al. 2019. " Collecting, monitoring, and analyzing
facility and systems data at the National Energy Research Scientiﬁc
Computing Center. " 48th International Conference on Parallel
Processing: Workshops (ICPP 2019).
4. Bourassa, N., et al. 2019. " Operational data analytics:
optimizing the National Energy Research Scientiﬁc Computing
Center Cooling Systems. " 48th International Conference on Parallel
Processing: Workshops (ICPP 2019).
5. LBNL. 2020. " Center of Expertise for Energy Efﬁciency in
Data Centers (CoE). " Lawrence Berkeley National Laboratory.
https://datacenters.lbl.gov/
6. NERSC. 2020. " Systems. " National Energy Research Scientiﬁc
Computing Center. https://www.nersc.gov/systems/
7. NERSC. 2020. " Perlmutter. " National Energy Research
Scientiﬁc Computing Center. www.nersc.gov/systems/perlmutter
8. ASHRAE. 2015. Thermal Guidelines for Data Processing
Environments, 4th Edition. Atlanta: ASHRAE.
9. ASHRAE. 2011. Thermal Guidelines for Data Processing Environments,
3rd Edition. Atlanta: ASHRAE.
10. ISO. 2018. " ISO 50001:2018, Energy
Management Systems-Requirements with
Guidance For Use. " International Organization
for Standardization.
https://bit.ly/3dTa21p
11. LBNL. 2020. " Sustainable Berkeley Lab:
Rate this Article
Energy and Water Management System Manual. "
Lawrence Berkeley National Laboratory. https://iso50001.lbl.gov

https://www.tinyurl.com/y4nqstse/ https://datacenters.lbl.gov/ https://www.nersc.gov/systems/ http://www.nersc.gov/systems/perlmutter https://www.bit.ly/3dTa21p https://iso50001.lbl.gov https://www.ashrae.org/

ASHRAE Journal - December 2020

Table of Contents for the Digital Edition of ASHRAE Journal - December 2020