The Platform Inside and Out Release 0.8 Joshua Patterson – GM, Data Science RAPIDS End-to-End Accelerated GPU Data Science Data Preparation Model Training Visualization Dask cuDF cuIO cuML cuGraph PyTorch Chainer MxNet cuXfilter <> pyViz Analytics Machine Learning Graph Analytics Deep Learning Visualization GPU Memory 2 Data Processing Evolution Faster data access, less data movement Hadoop Processing, Reading from disk HDFS HDFS HDFS HDFS HDFS Read Query Write Read ETL Write Read ML Train Spark In-Memory Processing 25-100x Improvement Less code HDFS Language flexible Read Query ETL ML Train Primarily In-Memory Traditional GPU Processing 5-10x Improvement More code HDFS GPU CPU GPU CPU GPU ML Language rigid Query ETL Read Read Write Read Write Read Train Substantially on GPU 3 Data Movement and Transformation The bane of productivity and performance APP B Read Data APP B GPU APP B Copy & Convert Data CPU GPU Copy & Convert Copy & Convert APP A GPU Data APP A Load Data APP A 4 Data Movement and Transformation What if we could keep data on the GPU? APP B Read Data APP B GPU APP B Copy & Convert Data CPU GPU Copy & Convert Copy & Convert APP A GPU Data APP A Load Data APP A 5 Learning from Apache Arrow ● Each system has its own internal memory format ● All systems utilize the same memory format ● 70-80% computation wasted on serialization and deserialization ● No overhead for cross-system communication ● Similar functionality implemented in multiple projects ● Projects can share functionality (eg, Parquet-to-Arrow reader) From Apache Arrow Home Page - https://arrow.apache.org/ 6 Data Processing Evolution Faster data access, less data movement Hadoop Processing, Reading from disk HDFS HDFS HDFS HDFS HDFS Read Query Write Read ETL Write Read ML Train Spark In-Memory Processing 25-100x Improvement Less code HDFS Language flexible Read Query ETL ML Train Primarily In-Memory Traditional GPU Processing 5-10x Improvement More code HDFS GPU CPU GPU CPU GPU ML Language rigid Query ETL Read Read Write Read Write Read Train Substantially on GPU RAPIDS 50-100x Improvement Same code Arrow ML Language flexible Query ETL Read Train Primarily on GPU 7 Faster Speeds, Real-World Benefits cuIO/cuDF – Load and Data Preparation cuML - XGBoost End-to-End 8762 6148 3925 3221 322 213 Time in seconds (shorter is better) cuIO/cuDF (Load and Data Prep) Data Conversion XGBoost Benchmark CPU Cluster Configuration DGX Cluster Configuration 200GB CSV dataset; Data prep includes CPU nodes (61 GiB memory, 8 vCPUs, 5x DGX-1 on InfiniBand joins, variable transformations 64-bit platform), Apache Spark network 8 Speed, UX, and Iteration The Way to Win at Data Science 9 RAPIDS Core 10 Open Source Data Science Ecosystem Familiar Python APIs Data Preparation Model Training Visualization Dask Pandas Scikit-Learn NetworkX PyTorch Chainer MxNet Matplotlib/Seaborn Analytics Machine Learning Graph Analytics Deep Learning Visualization CPU Memory 11 RAPIDS End-to-End Accelerated GPU Data Science Data Preparation Model Training Visualization Dask cuDF cuIO cuML cuGraph PyTorch Chainer MxNet cuXfilter <> pyViz Analytics Machine Learning Graph Analytics Deep Learning Visualization GPU Memory 12 Dask 13 RAPIDS Scaling RAPIDS with Dask Data Preparation Model Training Visualization Dask cuDF cuIO cuML cuGraph PyTorch Chainer MxNet cuXfilter <> pyViz Analytics Machine Learning Graph Analytics Deep Learning Visualization GPU Memory 14 Why Dask? • PyData Native • Built on top of NumPy, Pandas Scikit-Learn, etc. (easy to migrate) • With the same APIs (easy to train) • With the same developer community (well trusted) • Scales • Easy to install and use on a laptop • Scales out to thousand-node clusters • Popular • Most common parallelism framework today at PyData and SciPy conferences • Deployable • HPC: SLURM, PBS, LSF, SGE • Cloud: Kubernetes • Hadoop/Spark: Yarn 15 Why OpenUCX? Bringing hardware accelerated communications to Dask • TCP sockets are slow! • UCX provides uniform access to transports (TCP, InfiniBand, shared memory, NVLink) • Python bindings for UCX (ucx-py) in the works https://github.com/rapidsai/ucx-py • Will provide best communication performance, to Dask based on available hardware on nodes/cluster 16 Scale up with RAPIDS RAPIDS and Others Accelerated on single GPU NumPy -> CuPy/PyTorch/.. Pandas -> cuDF Scikit-Learn -> cuML Numba -> Numba PyData NumPy, Pandas, Scikit-Learn, Numba and many more Single CPU core In-memory data Scale Up / Accelerate 17 Scale out with RAPIDS + Dask with OpenUCX RAPIDS and Others RAPIDS + Dask with Accelerated on single GPU OpenUCX Multi-GPU NumPy -> CuPy/PyTorch/.. On single Node (DGX) Pandas -> cuDF Or across a cluster Scikit-Learn -> cuML Numba -> Numba PyData Dask NumPy, Pandas, Scikit-Learn, Multi-core and Distributed PyData Numba and many more NumPy -> Dask Array Single CPU core Pandas -> Dask DataFrame In-memory data Scikit-Learn -> Dask-ML … -> Dask Futures Scale Up / Accelerate Scale out / Parallelize 18 cuDF 19 RAPIDS GPU Accelerated data wrangling and feature engineering Data Preparation Model Training Visualization Dask cuDF cuIO cuML cuGraph PyTorch Chainer MxNet cuXfilter <> pyViz Analytics Machine Learning Graph Analytics Deep Learning Visualization GPU Memory 20 GPU-Accelerated ETL The average data scientist spends 90+% of their time in ETL as opposed to training models 21 ETL - the Backbone of Data Science libcuDF is… CUDA C++ Library ● Low level library containing function implementations and C/C++ API ● Importing/exporting Apache Arrow in GPU memory using CUDA IPC ● CUDA kernels to perform element-wise math operations on GPU DataFrame columns ● CUDA sort, join, groupby, reduction, etc. operations on GPU DataFrames 22 ETL - the Backbone of Data Science cuDF is… Python Library ● A Python library for manipulating GPU DataFrames following the Pandas API ● Python interface to CUDA C++ library with additional functionality ● Creating GPU DataFrames from Numpy arrays, Pandas DataFrames, and PyArrow Tables ● JIT compilation of User-Defined Functions (UDFs) using Numba 23 ETL - the Backbone of Data Science cuDF is not the end of the story Data Preparation Model Training Visualization Dask cuDF cuIO cuML cuGraph PyTorch Chainer MxNet cuXfilter <> pyViz Analytics Machine Learning Graph Analytics Deep Learning Visualization GPU Memory 24 ETL - the Backbone of Data Science String Support Now v0.8 String Support: • Regular Expressions • Element-wise operations • Split, Find, Extract, Cat, Typecasting, etc… • String GroupBys, Joins Future v0.9+ String Support: • Combining cuStrings into libcudf • Extensive performance optimization • More Pandas String API compatibility • Improved Categorical column support 25 Extraction is the Cornerstone of ETL cuIO is born • Follows the APIs of Pandas and provide >10x speedup • CSV Reader - v0.2, CSV Writer v0.8 • Parquet Reader – v0.7 • ORC Reader – v0.7 • JSON Reader - v0.8 • Avro Reader - v0.9 • HDF5 Reader - v0.10 • Key is GPU-accelerating both parsing and decompression wherever possible Source: Apache Crail blog: SQL Performance: Part 1 - Input File Formats 26 ETL is not just DataFrames! 27 RAPIDS Building bridges into the array ecosystem Data Preparation Model Training Visualization Dask cuDF cuIO cuML cuGraph PyTorch Chainer MxNet cuXfilter <> pyViz Analytics Machine Learning Graph Analytics Deep Learning Visualization GPU Memory 28 Interoperability for the Win DLPack and __cuda_array_interface__ mpi4py 29 Interoperability for the Win DLPack and __cuda_array_interface__ mpi4py 30 ETL – Arrays and DataFrames Dask and CUDA Python arrays • Scales NumPy to distributed clusters • Used in climate science, imaging, HPC analysis up to 100TB size • Now seamlessly accelerated with GPUs 31 Benchmark: single-GPU CuPy vs NumPy More details: https://blog.dask.org/2019/06/27/single-gpu-cupy-benchmarks 32 SVD Benchmark Dask and CuPy Doing Complex Workflows 33 Also…Achievement Unlocked: Petabyte Scale Data Analytics with Dask and CuPy Architecture Time Single CPU Core 2hr 39min Forty CPU Cores 11min 30s One GPU 1min 37s Cluster configuration: 20x GCP instances, each instance has: CPU: 1 VM socket (Intel Xeon CPU @ 2.30GHz), Eight GPUs 19s 2-core, 2 threads/core, 132GB mem, GbE ethernet, 950 GB disk GPU: 4x NVIDIA Tesla P100-16GB-PCIe (total GPU https://blog.dask.org/2019/01/03/dask-array-gpus-first-steps DRAM across nodes 1.22 TB) Software: Ubuntu 18.04, RAPIDS 0.5.1, Dask=1.1.1, Dask-Distributed=1.1.1, CuPY=5.2.0, CUDA 10.0.130 34 ETL – Arrays and DataFrames More Dask Awesomeness from RAPIDS https://youtu.be/gV0cykgsTPM https://youtu.be/R5CiXti_MWo 35 cuML 36 Machine Learning More models more problems Data Preparation Model Training Visualization Dask cuDF cuIO cuML cuGraph PyTorch Chainer MxNet cuXfilter <> pyViz Analytics Machine Learning Graph Analytics Deep Learning Visualization GPU Memory 37 Problem Data sizes continue to grow Massive Dataset Histograms / Distributions Better to start with as much data as possible and explore / preprocess to scale to performance needs. Dimension Reduction Feature Selection Time Increases Remove Outliers Iterate. Cross Validate & Grid Search. Iterate some more. Hours? Days? Sampling Meet reasonable speed vs accuracy tradeoff 38 ML Technology Stack Python Dask cuML Dask cuDF cuDF Cython Numpy cuML Algorithms Thrust Cub cuML Prims cuSolver nvGraph CUTLASS CUDA Libraries cuSparse cuRand CUDA cuBlas 39 Algorithms GPU-accelerated Scikit-Learn Decision Trees / Random Forests Linear Regression Classification / Regression Logistic Regression K-Nearest Neighbors Kalman