Experiences in Building GPU Enabled HPC Clusters

Building a Successful Deep Learning Platform: Experiences in Building GPU Enabled HPC Clusters

Brian Michael Belgodere Software Engineer Cognitive Computing Cluster 1 Agenda

• Who am I • Why should you listen to IBM Research • What did we build • Why did we do this • How do we keep our users productive • How do we keep it running • Future Work

2 IBM 5/10/17 About Me

• Software Engineer • IBM Research • HPC and ML/Deep Learning • DevOps • Background in Computational Finance and Economics • Law Degree • Coffee and Beer Enthusiast • I Love Solving Problems 3 IBM 5/10/17 Why should you listen to IBM Research?

4 IBM 5/10/17 5 6 7 Bluemix Services

8 In 2 short years IBM Research...

• 20+ high value patents submitted • 50+ Papers Accepted at Top Conferences • NIPS, COLING, INTERSPEECH, AISTATS, EACL, ICASSP, EMNLP, etc. • Several Commercialized Services and Client Engagements • 5 Moonshots

9 IBM 5/10/17 What did we build?

10 IBM 5/10/17 Cognitive Computing Cluster

11 IBM 5/10/17 What did we build? "' ! !!!!!!!!!!!!!!! ! !!!!!!!!!!!!!!! "& ! !!!!!!!!!!!!!!! " ( ! # !!!!!!!!!!!!!!! ! !!!!!!!!!!!!!!! !% !!!!!!!!!!!!!!! !!!!!!!!!!!!!! !!!!!!!!!!!!!! !!!!!!!!!!!!!! ! $$$$$$$$$$ ! !!!!!!!!!! 12 !!!!!!!!!! What did we build?

• 263 x86 Compute Nodes • 130 P8 Compute Nodes • 946 GPU Devices • 3.45 Petabytes of Storage • 10GbE + Infiniband • 750+ Active Users

13 • 1.2IBM Million5/10/17 Jobs a Month Off the Shelf Parts IBM Software Defined Open Source Frameworks Infrastructure

14 Why did we do this?

• Users of BluegeneQ, Speech, and several other cluster looking for next-gen compute platform • Colocation of Data and Workload across Disciplines • Centralization of Management, Cost, and Workload Visibility • Advanced Scheduling to Maximize Cluster Utilization

15 • DemandIBM for5/10/17 Deep Learning & GPUs Why did we do this?

• The Cognitive Computing Cluster was built to explore the enormous possibilities of Deep Learning in order to accelerate the velocity of our researchers' experiments. This enables IBM Research to bring innovation to our customers faster and help extract the maximum value from their ever-growing

16 proprietaryIBM 5/10/17 data sets. IBM Research: A global research capability

Ireland Zurich Almaden Watson Tokyo China Austin Haifa

India

Africa

Brazil

Australia

Healthcare Cognitive IoT Big Data & Cognitive Data Centric Computing Cloud Core Technologies Ireland Quantum Computing Cognitive Zurich Healthcare Green Horizon Energy IoT & Mobile OpenPOWER Cloud Security Cognitive Health Almaden Watson Analytics Security Tokyo Big Data Nanotechnology Haifa China Cognitive Robotics Nanomaterials Austin Exascale Financial Services Neurosynaptics POWER Accessibility Mobile India Aging Blockchain Cognitive Fashion Africa Education & Skilling Cognitive Financial Services Cognitive Oil & Gas Healthcare Insurance Analytics Industry Cloud Industry Cloud IoT Blockchain Brazil Healthcare Government Financial Services Australia

19 IBM 5/10/17 Cognitive Systems Lifecycle

20 IBM 5/10/17 Providing a Full Software Stack

User files & project Cognitive Applications directories cross mounted across cluster Platform Enablers for GPU Tensorflow, Torch, Caffe, Application Libraries Theano Programming Platforms Licensed Tools: Intel MKL, CPU: C/C++, Python, Java Matlab Compilers & Runtime GPU: CUDA, OpenCL

Distributed Software: Distributed Runtime Support • SpectrumMPI, OpenMPI, & OpenMP OS: Redhat • Spark Filesystems: NFS, GPFS • TensorflowOnSpark OS Services & Data Access • CaffeOnSpark • SystemML Cluster and Job Manager: Ethernet & Infiniband Hardware Platform, • Spectrum LSF Software 21 GPU, Storage Parallelizing out Deep Learning

• SpectrumMPI & OpenMPI • Parameter Servers • Spark

22 23 IBM 5/10/17 24 IBM 5/10/17 How do we keep it running?

25 IBM 5/10/17 There are no silver bullets for this, only lead bullets. - Ben Horowitz

26 IBM 5/10/17 xCAT

27 Nagios

28 ELK Stack

29 Future Work

30 IBM 5/10/17 Thanks

31 IBM 5/10/17

For more information: http://research.ibm.com