Exploiting HPC Technologies to Accelerate Big Data Processing
Exploi ng HPC Technologies to Accelerate Big Data Processing
Talk at Open Fabrics Workshop (April 2016) by
Dhabaleswar K. (DK) Panda The Ohio State University E-mail: [email protected] h p://www.cse.ohio-state.edu/~panda Introduc on to Big Data Applica ons and Analy cs • Big Data has become the one of the most important elements of business analy cs • Provides groundbreaking opportuni es for enterprise informa on management and decision making • The amount of data is exploding; companies are capturing and digi zing more informa on than ever • The rate of informa on growth appears to be exceeding Moore’s Law
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 2 Data Genera on in Internet Services and Applica ons • Webpages (content, graph) • Clicks (ad, page, social) • Users (OpenID, FB Connect, etc.) • e-mails (Hotmail, Y!Mail, Gmail, etc.) • Photos, Movies (Flickr, YouTube, Video, etc.) • Cookies / tracking info (see Ghostery) • Installed apps (Android market, App Store, etc.) • Loca on (La tude, Loopt, Foursquared, Google Now, etc.) • User generated content (Wikipedia & co, etc.) • Ads (display, text, DoubleClick, Yahoo, etc.) • Comments (Discuss, Facebook, etc.) • Reviews (Yelp, Y!Local, etc.) • Social connec ons (LinkedIn, Facebook, etc.) • Purchase decisions (Netflix, Amazon, etc.) • Instant Messages (YIM, Skype, Gtalk, etc.) • Search terms (Google, Bing, etc.) • News ar cles (BBC, NYTimes, Y!News, etc.) Number of Apps in the Apple App Store, Android Market, Blackberry, • Blog posts (Tumblr, Wordpress, etc.) and Windows Phone (2013) • Android Market: <1200K
• Microblogs (Twi er, Jaiku, Meme, etc.) • Apple App Store: ~1000K • Link sharing (Facebook, Delicious, Buzz, etc.) Courtesy: h p://dazeinfo.com/2014/07/10/apple-inc-aapl-ios-google-inc-goog- android-growth-mobile-ecosystem-2014/ Network Based Compu ng Laboratory OFA-BigData (April ‘16) 3 Not Only in Internet Services - Big Data in Scien fic Domains • Scien fic Data Management, Analysis, and Visualiza on • Applica ons examples – Climate modeling – Combus on – Fusion – Astrophysics – Bioinforma cs • Data Intensive Tasks – Runs large-scale simula ons on supercomputers – Dump data on parallel storage systems – Collect experimental / observa onal data – Move experimental / observa onal data to analysis sites – Visual analy cs – help understand data visually
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 4 Typical Solu ons or Architectures for Big Data Analy cs • Hadoop: h p://hadoop.apache.org – The most popular framework for Big Data Analy cs – HDFS, MapReduce, HBase, RPC, Hive, Pig, ZooKeeper, Mahout, etc. • Spark: h p://spark-project.org – Provides primi ves for in-memory cluster compu ng; Jobs can load data into memory and query it repeatedly • Storm: h p://storm-project.net – A distributed real- me computa on system for real- me analy cs, online machine learning, con nuous computa on, etc. • S4: h p://incubator.apache.org/s4 – A distributed system for processing con nuous unbounded streams of data • GraphLab: h p://graphlab.org – Consists of a core C++ GraphLab API and a collec on of high-performance machine learning and data mining toolkits built on top of the GraphLab API. • Web 2.0: RDBMS + Memcached (h p://memcached.org) – Memcached: A high-performance, distributed memory object caching systems
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 5 Big Data Processing with Hadoop Components • Major components included in this tutorial: User Applica ons – MapReduce (Batch) – HBase (Query) – HDFS (Storage) MapReduce HBase – RPC (Inter-process communica on) • Underlying Hadoop Distributed File System (HDFS) used by both MapReduce and HBase HDFS • Model scales but high amount of Hadoop Common (RPC) communica on during intermediate phases can be further op mized Hadoop Framework
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 6 Spark Architecture Overview • An in-memory data-processing framework Worker – Itera ve machine learning jobs Zookeeper
– Interac ve data analy cs Worker – Scala based Implementa on – Standalone, YARN, Mesos Worker • Scalable and communica on intensive HDFS Driver – Wide dependencies between Resilient SparkContext
Distributed Datasets (RDDs) Worker Worker – MapReduce-like shuffle opera ons to Master repar on RDDs – Sockets based communica on h p://spark.apache.org
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 7 Memcached Architecture
Main Main CPUs CPUs memory memory ... SSD HDD SSD HDD
Main Main CPUs CPUs memory memory High Performance Networks SSD HDD SSD HDD
... Main ... CPUs memory
SSD HDD • Distributed Caching Layer – Allows to aggregate spare memory from mul ple nodes – General purpose • Typically used to cache database queries, results of API calls • Scalable model, but typical usage very network intensive Network Based Compu ng Laboratory OFA-BigData (April ‘16) 8 Data Management and Processing on Modern Clusters
• Substan al impact on designing and u lizing data management and processing systems in mul ple ers – Front-end data accessing and serving (Online) • Memcached + DB (e.g. MySQL), HBase – Back-end data analy cs (Offline) • HDFS, MapReduce, Spark
Front-end Tier Back-end Tier
Memcached Data Analytics Apps/Jobs + DBMemcached (MySQL) + DBMemcached (MySQL) Web + DB (MySQL) Internet ServerWeb MapReduce Spark ServerWeb Server NoSQL DB HDFS (HBase)NoSQL DB Data Accessing (HBase)NoSQL DB and Serving (HBase)
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 9 Trends in HPC Technologies • Advanced Interconnects and RDMA protocols – InfiniBand – 10-40 Gigabit Ethernet/iWARP – RDMA over Converged Enhanced Ethernet (RoCE) • Delivering excellent performance (Latency, Bandwidth and CPU U liza on) • Has influenced re-designs of enhanced HPC middleware – Message Passing Interface (MPI) and PGAS – Parallel File Systems (Lustre, GPFS, ..) • SSDs (SATA and NVMe) • NVRAM and Burst Buffer
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 10 How Can HPC Clusters with High-Performance Interconnect and Storage Architectures Benefit Big Data Applica ons?
Can RDMA-enabled Can HPC Clusters with How much Can the bo lenecks be high-performance alleviated with new high-performance performance benefits storage systems (e.g. designs by taking interconnects can be achieved advantage of HPC SSD, parallel file through enhanced technologies? benefit Big Data systems) benefit Big designs? processing? Data applica ons? How to design What are the major benchmarks for bo lenecks in current Big evalua ng the Data processing performance of Big middleware (e.g. Hadoop, Data middleware on Spark, and Memcached)? HPC clusters?
Bring HPC and Big Data processing into a “convergent trajectory”!
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 11 Designing Communica on and I/O Libraries for Big Data Systems: Challenges
Applica ons Benchmarks Big Data Middleware Upper level (HDFS, MapReduce, HBase, Spark and Memcached) Changes? Programming Models (Sockets) Other Protocols? Communica on and I/O Library Point-to-Point Threaded Models Communica on and Synchroniza on Virtualiza on
I/O and File Systems QoS Fault-Tolerance
Commodity Compu ng System Networking Technologies Architectures Storage Technologies (InfiniBand, 1/10/40/100 GigE (Mul - and Many-core (HDD, SSD, and NVMe-SSD) and Intelligent NICs) architectures and accelerators)
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 12 The High-Performance Big Data (HiBD) Project • RDMA for Apache Spark • RDMA for Apache Hadoop 2.x (RDMA-Hadoop-2.x) – Plugins for Apache, Hortonworks (HDP) and Cloudera (CDH) Hadoop distribu ons • RDMA for Apache Hadoop 1.x (RDMA-Hadoop) Available for InfiniBand and RoCE • RDMA for Memcached (RDMA-Memcached) • OSU HiBD-Benchmarks (OHB) – HDFS and Memcached Micro-benchmarks • h p://hibd.cse.ohio-state.edu • Users Base: 166 organiza ons from 22 countries • More than 15,900 downloads from the project site • RDMA for Apache HBase and Impala (upcoming)
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 13 Different Modes of RDMA for Apache Hadoop 2.x
• HHH: Heterogeneous storage devices with hybrid replica on schemes are supported in this mode of opera on to have be er fault-tolerance as well as performance. This mode is enabled by default in the package. • HHH-M: A high-performance in-memory based setup has been introduced in this package that can be u lized to perform all I/O opera ons in- memory and obtain as much performance benefit as possible. • HHH-L: With parallel file systems integrated, HHH-L mode can take advantage of the Lustre available in the cluster. • MapReduce over Lustre, with/without local disks: Besides, HDFS based solu ons, this package also provides support to run MapReduce jobs on top of Lustre alone. Here, two different modes are introduced: with local disks and without local disks. • Running with Slurm and PBS: Supports deploying RDMA for Apache Hadoop 2.x with Slurm and PBS in different running modes (HHH, HHH-M, HHH- L, and MapReduce over Lustre).
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 14 Accelera on Case Studies and Performance Evalua on
• RDMA-based Designs and Performance Evalua on – HDFS – MapReduce – RPC – HBase – Spark – Memcached (Basic and Hybrid) – HDFS + Memcached-based Burst Buffer
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 15 Design Overview of HDFS with RDMA
Applica ons • Design Features – RDMA-based HDFS write HDFS – RDMA-based HDFS Others Write replica on Java Socket Interface Java Na ve Interface (JNI) – Parallel replica on OSU Design support
Verbs – On-demand connec on setup 1/10/40/100 GigE, IPoIB RDMA Capable Networks Network (IB, iWARP, RoCE ..) – InfiniBand/RoCE support • Enables high performance RDMA communica on, while suppor ng tradi onal socket interface • JNI Layer bridges Java based HDFS with communica on library wri en in na ve code
N. S. Islam, M. W. Rahman, J. Jose, R. Rajachandrasekar, H. Wang, H. Subramoni, C. Murthy and D. K. Panda , High Performance RDMA-Based Design of HDFS over InfiniBand , Supercompu ng (SC), Nov 2012 N. Islam, X. Lu, W. Rahman, and D. K. Panda, SOR-HDFS: A SEDA-based Approach to Maximize Overlapping in RDMA-Enhanced HDFS, HPDC '14, June 2014 Network Based Compu ng Laboratory OFA-BigData (April ‘16) 16 Enhanced HDFS with In-Memory and Heterogeneous Storage
Applica ons • Design Features – Three modes Triple-H • Default (HHH) Data Placement Policies • In-Memory (HHH-M) • Lustre-Integrated (HHH-L) Hybrid Replica on Evic on/Promo on – Policies to efficiently u lize the heterogeneous storage devices • RAM, SSD, HDD, Lustre Heterogeneous Storage
– Evic on/Promo on based on data usage RAM Disk SSD HDD pa ern – Hybrid Replica on Lustre – Lustre-Integrated mode: • Lustre-based fault-tolerance
N. Islam, X. Lu, M. W. Rahman, D. Shankar, and D. K. Panda, Triple-H: A Hybrid Approach to Accelerate HDFS on HPC Clusters with Heterogeneous Storage Architecture, CCGrid ’15, May 2015 Network Based Compu ng Laboratory OFA-BigData (April ‘16) 17 Design Overview of MapReduce with RDMA Applica ons • Design Features – RDMA-based shuffle MapReduce – Prefetching and caching map output Job Task Map – Efficient Shuffle Algorithms Tracker Tracker Reduce – In-memory merge
Java Socket Interface Java Na ve Interface (JNI) – On-demand Shuffle Adjustment – Advanced overlapping OSU Design • map, shuffle, and merge Verbs • shuffle, merge, and reduce 1/10/40/100 GigE, IPoIB RDMA Capable Networks – On-demand connec on setup Network (IB, iWARP, RoCE ..) – InfiniBand/RoCE support • Enables high performance RDMA communica on, while suppor ng tradi onal socket interface • JNI Layer bridges Java based MapReduce with communica on library wri en in na ve code M. W. Rahman, X. Lu, N. S. Islam, and D. K. Panda, HOMR: A Hybrid Approach to Exploit Maximum Overlapping in MapReduce over High Performance Interconnects, ICS, June 2014 Network Based Compu ng Laboratory OFA-BigData (April ‘16) 18 Performance Benefits – RandomWriter & TeraGen in TACC-Stampede
250 Reduced by 3x 250 IPoIB (FDR) IPoIB (FDR) Reduced by 4x 200 200
150 150
100 100
50 50 Execu on Time (s) Execu on Time (s) 0 0 80 100 120 80 100 120 Data Size (GB) Data Size (GB)
RandomWriter TeraGen Cluster with 32 Nodes with a total of 128 maps
• RandomWriter • TeraGen – 3-4x improvement over IPoIB – 4-5x improvement over IPoIB for 80-120 GB file size for 80-120 GB file size
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 19 Performance Benefits – Sort & TeraSort in TACC-Stampede
900 IPoIB (FDR) OSU-IB (FDR) 600 IPoIB (FDR) OSU-IB (FDR) 800 500 700 Reduced by 52% Reduced by 44% 600 400 500 300 400 300 200 200 Execu on Time (s) Execu on Time (s) 100 100 0 0 80 100 120 80 100 120 Data Size (GB) Data Size (GB)
Cluster with 32 Nodes with a total of Cluster with 32 Nodes with a total of 128 maps and 57 reduces 128 maps and 64 reduces
• Sort with single HDD per node • TeraSort with single HDD per node – 40-52% improvement over IPoIB – 42-44% improvement over IPoIB for 80-120 GB data for 80-120 GB data Network Based Compu ng Laboratory OFA-BigData (April ‘16) 20 Evalua on of HHH and HHH-L with Applica ons HDFS Lustre HHH-L Reduced by 79% 1000 HDFS (FDR) HHH (FDR) 800
600
400 60.24 s 48.3 s 200 Execu on Time (s)
0 4 6 8 Concurrent maps per host MR-MSPolyGraph CloudBurst • MR-MSPolygraph on OSU RI with • CloudBurst on TACC Stampede 1,000 maps – With HHH: 19% improvement over – HHH-L reduces the execu on me HDFS by 79% over Lustre, 30% over HDFS Network Based Compu ng Laboratory OFA-BigData (April ‘16) 21 Evalua on with Spark on SDSC Gordon (HHH vs. Tachyon/Alluxio) 700 180 IPoIB (QDR) Tachyon OSU-IB (QDR) Reduced by 25.2% 160 Reduced 600 140 by 2.4x 500 120 100 400 80 300 60 200 40 Execu on Time (s) 20 Execu on Time (s) 100 0 0 8:50 16:100 32:200 8:50 16:100 32:200 Cluster Size : Data Size (GB) Cluster Size : Data Size (GB) TeraGen TeraSort • For 200GB TeraGen on 32 nodes – Spark-TeraGen: HHH has 2.4x improvement over Tachyon; 2.3x over HDFS-IPoIB (QDR) – Spark-TeraSort: HHH has 25.2% improvement over Tachyon; 17% over HDFS-IPoIB (QDR)
N. Islam, M. W. Rahman, X. Lu, D. Shankar, and D. K. Panda, Performance Characteriza on and Accelera on of In-Memory File Systems for Hadoop and Spark Applica ons on HPC Clusters, IEEE BigData ’15, October 2015
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 22 Design Overview of Shuffle Strategies for MapReduce over Lustre • Design Features Map 1 Map 2 Map 3 – Two shuffle approaches • Lustre read based shuffle • RDMA based shuffle Intermediate Data Directory – Hybrid shuffle algorithm to take benefit from both shuffle approaches – Dynamically adapts to the be er Lustre shuffle approach for each shuffle request based on profiling values for Lustre Read / RDMA each Lustre read opera on Reduce 1 Reduce 2 – In-memory merge and overlapping of In-memory In-memory different phases are kept similar to reduce reduce merge/sort merge/sort RDMA-enhanced MapReduce design
M. W. Rahman, X. Lu, N. S. Islam, R. Rajachandrasekar, and D. K. Panda, High Performance Design of YARN MapReduce on Modern HPC Clusters with Lustre and RDMA, IPDPS, May 2015 Network Based Compu ng Laboratory OFA-BigData (April ‘16) 23 Performance Improvement of MapReduce over Lustre on TACC- Stampede • Local disk is used as the intermediate data directory 500 1200 Reduced by 44% IPoIB (FDR) OSU-IB (FDR) Reduced by 48% IPoIB (FDR) 450 1000 OSU-IB (FDR) 400 350 800 300 600 250 200 400 150 200 100 Job Execu on Time (sec) Job Execu on Time (sec) 50 0 0 300 400 500 20 GB 40 GB 80 GB 160 GB 320 GB 640 GB Data Size (GB) Cluster: 4 Cluster: 8 Cluster: 16 Cluster: 32 Cluster: 64 Cluster: 128
• For 500GB Sort in 64 nodes • For 640GB Sort in 128 nodes – 44% improvement over IPoIB (FDR) – 48% improvement over IPoIB (FDR) M. W. Rahman, X. Lu, N. S. Islam, R. Rajachandrasekar, and D. K. Panda, MapReduce over Lustre: Can RDMA-based Approach Benefit?, Euro-Par, August 2014. Network Based Compu ng Laboratory OFA-BigData (April ‘16) 24 Case Study - Performance Improvement of MapReduce over Lustre on SDSC-Gordon • Lustre is used as the intermediate data directory 900 900 IPoIB (QDR) Reduced by 34% Reduced by 25% 800 OSU-Lustre-Read (QDR) 800 700 700 OSU-RDMA-IB (QDR) 600 600 500 OSU-Hybrid-IB (QDR) 500 400 400 300 300 200 200 Job Execu on Time (sec) 100 Job Execu on Time (sec) 100 0 0 40 60 80 40 80 120 Data Size (GB) Data Size (GB) • For 80GB Sort in 8 nodes • For 120GB TeraSort in 16 nodes – 34% improvement over IPoIB (QDR) – 25% improvement over IPoIB (QDR)
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 25 Accelera on Case Studies and Performance Evalua on
• RDMA-based Designs and Performance Evalua on – HDFS – MapReduce – RPC – HBase – Spark – Memcached (Basic and Hybrid) – HDFS + Memcached-based Burst Buffer
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 26 Design Overview of Spark with RDMA
Spark Applications (Scala/Java/Python) • Design Features
Task Task Task Task – RDMA based shuffle Spark (Scala/Java) – SEDA-based plugins BlockManager BlockFetcherIterator – Dynamic connec on Java NIO Netty RDMA Java NIO Netty RDMA Shuffle Shuffle Shuffle Shuffle Shuffle Shuffle management and sharing Server Server Server Fetcher Fetcher Fetcher (default) (optional) (plug-in) (default) (optional) (plug-in) – Non-blocking data transfer
RDMA-based Shuffle Engine – Off-JVM-heap buffer Java Socket (Java/JNI) management
1/10 Gig Ethernet/IPoIB (QDR/FDR) Native InfiniBand – InfiniBand/RoCE support Network (QDR/FDR)
• Enables high performance RDMA communica on, while suppor ng tradi onal socket interface • JNI Layer bridges Scala based Spark with communica on library wri en in na ve code X. Lu, M. W. Rahman, N. Islam, D. Shankar, and D. K. Panda, Accelera ng Spark with RDMA for Big Data Processing: Early Experiences, Int'l Symposium on High Performance Interconnects (HotI'14), August 2014
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 27 Performance Evalua on on SDSC Comet – SortBy/GroupBy 250 300 IPoIB 80% IPoIB 57% 200 250 RDMA RDMA 200 150 150 Time (sec) 100 Time (sec) 100 50 50
0 0 64 128 256 64 128 256 Data Size (GB) Data Size (GB) 64 Worker Nodes, 1536 cores, SortByTest Total Time 64 Worker Nodes, 1536 cores, GroupByTest Total Time
• InfiniBand FDR, SSD, 64 Worker Nodes, 1536 Cores, (1536M 1536R) • RDMA-based design for Spark 1.5.1 • RDMA vs. IPoIB with 1536 concurrent tasks, single SSD per node. – SortBy: Total me reduced by up to 80% over IPoIB (56Gbps) – GroupBy: Total me reduced by up to 57% over IPoIB (56Gbps) Network Based Compu ng Laboratory OFA-BigData (April ‘16) 28 Performance Evalua on on SDSC Comet – HiBench Sort/TeraSort
600 450 38% IPoIB 400 IPoIB 15% 500 RDMA 350 RDMA 400 300 250 300 200 Time (sec) Time (sec) 200 150 100 100 50 0 0 64 128 256 64 128 256 Data Size (GB) Data Size (GB) 64 Worker Nodes, 1536 cores, Sort Total Time 64 Worker Nodes, 1536 cores, TeraSort Total Time
• InfiniBand FDR, SSD, 64 Worker Nodes, 1536 Cores, (1536M 1536R) • RDMA-based design for Spark 1.5.1 • RDMA vs. IPoIB with 1536 concurrent tasks, single SSD per node. – Sort: Total me reduced by 38% over IPoIB (56Gbps) – TeraSort: Total me reduced by 15% over IPoIB (56Gbps) Network Based Compu ng Laboratory OFA-BigData (April ‘16) 29 Performance Evalua on on SDSC Comet – HiBench PageRank
800 450 37% 43% 700 IPoIB 400 IPoIB 600 RDMA 350 RDMA 500 300 250 400 200 Time (sec) 300 Time (sec) 150 200 100 100 50 0 0 Huge BigData Gigan c Huge BigData Gigan c Data Size (GB) Data Size (GB) 32 Worker Nodes, 768 cores, PageRank Total Time 64 Worker Nodes, 1536 cores, PageRank Total Time
• InfiniBand FDR, SSD, 32/64 Worker Nodes, 768/1536 Cores, (768/1536M 768/1536R) • RDMA-based design for Spark 1.5.1 • RDMA vs. IPoIB with 768/1536 concurrent tasks, single SSD per node. – 32 nodes/768 cores: Total me reduced by 37% over IPoIB (56Gbps) – 64 nodes/1536 cores: Total me reduced by 43% over IPoIB (56Gbps) Network Based Compu ng Laboratory OFA-BigData (April ‘16) 30 Accelera on Case Studies and Performance Evalua on
• RDMA-based Designs and Performance Evalua on – HDFS – MapReduce – RPC – HBase – Spark – Memcached (Basic and Hybrid) – HDFS + Memcached-based Burst Buffer
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 31 RDMA-Memcached Performance (FDR Interconnect) Memcached GET Latency Memcached Throughput 1000 700 OSU-IB (FDR) IPoIB (FDR) 600 100 500 400 2X Latency Reduced 300
Time (us) 10 by nearly 20X 200 100 1 per Second (TPS) 0 1 2 4 8 16 32 64 128 256 512 1K 2K 4K
Thousands of Transac ons 16 32 64 128 256 512 1024 2048 4080 Message Size No. of Clients Experiments on TACC Stampede (Intel SandyBridge Cluster, IB: FDR) • Memcached Get latency – 4 bytes OSU-IB: 2.84 us; IPoIB: 75.53 us – 2K bytes OSU-IB: 4.49 us; IPoIB: 123.42 us • Memcached Throughput (4bytes) – 4080 clients OSU-IB: 556 Kops/sec, IPoIB: 233 Kops/s – Nearly 2X improvement in throughput
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 32 Micro-benchmark Evalua on for OLDP workloads Memcached-IPoIB (32Gbps) Memcached-IPoIB (32Gbps) Reduced by 66% 4000 8 Memcached-RDMA (32Gbps) Memcached-RDMA (32Gbps) 7 6 3000 5 4 2000 3
Latency (sec) 2 1000
1 Throughput (Kq/s) 0 0 64 96 128 160 320 400 64 96 128 160 320 400 No. of Clients No. of Clients • Illustra on with Read-Cache-Read access pa ern using modified mysqlslap load tes ng tool • Memcached-RDMA can - improve query latency by up to 66% over IPoIB (32Gbps) - throughput by up to 69% over IPoIB (32Gbps) D. Shankar, X. Lu, J. Jose, M. W. Rahman, N. Islam, and D. K. Panda, Can RDMA Benefit On-Line Data Processing Workloads with Memcached and MySQL, ISPASS’15 Network Based Compu ng Laboratory OFA-BigData (April ‘16) 33 Performance Benefits of Hybrid Memcached (Memory + SSD) on SDSC-Gordon IPoIB (32Gbps) 500 10 2X RDMA-Mem (32Gbps) 400 8 RDMA-Hybrid (32Gbps) 300 6 200 4 100 2 Average latency (us) 0 0 Throughput (million trans/sec) 64 128 256 512 1024 No. of Clients Message Size (Bytes) • ohb_memlat & ohb_memthr latency & throughput micro-benchmarks • Memcached-RDMA can - improve query latency by up to 70% over IPoIB (32Gbps) - improve throughput by up to 2X over IPoIB (32Gbps) - No overhead in using hybrid mode when all data can fit in memory Network Based Compu ng Laboratory OFA-BigData (April ‘16) 34 Performance Evalua on on IB FDR + SATA/NVMe SSDs
slab alloca on (SSD write) cache check+load (SSD read) cache update server response client wait miss-penalty
2500
2000
1500
1000 Latency (us) 500
0 Set Get Set Get Set Get Set Get Set Get Set Get Set Get Set Get IPoIB-Mem RDMA-Mem RDMA-Hybrid-SATA RDMA-Hybrid- IPoIB-Mem RDMA-Mem RDMA-Hybrid-SATA RDMA-Hybrid- NVMe NVMe Data Fits In Memory Data Does Not Fit In Memory
– Memcached latency test with Zipf distribu on, server with 1 GB memory, 32 KB key-value pair size, total size of data accessed is 1 GB (when data fits in memory) and 1.5 GB (when data does not fit in memory) – When data fits in memory: RDMA-Mem/Hybrid gives 5x improvement over IPoIB-Mem – When data does not fit in memory: RDMA-Hybrid gives 2x-2.5x over IPoIB/RDMA-Mem
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 35 On-going and Future Plans of OSU High Performance Big Data (HiBD) Project • Upcoming Releases of RDMA-enhanced Packages will support – HBase – Impala • Upcoming Releases of OSU HiBD Micro-Benchmarks (OHB) will support – MapReduce – RPC • Advanced designs with upper-level changes and op miza ons – Memcached with Non-blocking API – HDFS + Memcached-based Burst Buffer
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 36 Concluding Remarks
• Discussed challenges in accelera ng Hadoop, Spark and Memcached • Presented ini al designs to take advantage of HPC technologies to accelerate HDFS, MapReduce, RPC, Spark, and Memcached • Results are promising • Many other open issues need to be solved • Will enable Big Data community to take advantage of modern HPC technologies to carry out their analy cs in a fast and scalable manner
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 37 Second Interna onal Workshop on High-Performance Big Data Compu ng (HPBDC) HPBDC 2016 will be held with IEEE Interna onal Parallel and Distributed Processing Symposium (IPDPS 2016), Chicago, Illinois USA, May 27th, 2016 Keynote Talk: Dr. Chaitanya Baru, Senior Advisor for Data Science, Na onal Science Founda on (NSF); Dis nguished Scien st, San Diego Supercomputer Center (SDSC)
Panel Moderator: Jianfeng Zhan (ICT/CAS) Panel Topic: Merge or Split: Mutual Influence between Big Data and HPC Techniques
Six Regular Research Papers and Two Short Research Papers h p://web.cse.ohio-state.edu/~luxi/hpbdc2016 HPBDC 2015 was held in conjunc on with ICDCS’15 h p://web.cse.ohio-state.edu/~luxi/hpbdc2015
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 38 Thank You! [email protected]
Network-Based Compu ng Laboratory h p://nowlab.cse.ohio-state.edu/
The High-Performance Big Data Project h p://hibd.cse.ohio-state.edu/
Network Based Compu ng Laboratory OFA-BigData (April ‘16) 39