Full text available at: http://dx.doi.org/10.1561/1900000055 Algorithmic Aspects of Parallel Data Processing Paraschos Koutris University of Wisconsin-Madison [email protected] Semih Salihoglu University of Waterloo [email protected] Dan Suciu University of Washington [email protected] Boston — Delft Full text available at: http://dx.doi.org/10.1561/1900000055 R Foundations and Trends in Databases Published, sold and distributed by: now Publishers Inc. PO Box 1024 Hanover, MA 02339 United States Tel. +1-781-985-4510 www.nowpublishers.com [email protected] Outside North America: now Publishers Inc. PO Box 179 2600 AD Delft The Netherlands Tel. +31-6-51115274 The preferred citation for this publication is P. Koutris, S. Salihoglu and D. Suciu. Algorithmic Aspects of Parallel Data R Processing. Foundations and Trends• in Databases, vol. 8, no. 4, pp. 239–370, 2016. R This Foundations and Trends• issue was typeset in LATEXusingaclassfiledesigned by Neal Parikh. Printed on acid-free paper. ISBN: 978-1-68083-406-2 c 2018 P. Koutris, S. Salihoglu and D. Suciu • All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, mechanical, photocopying, recording or otherwise, without prior written permission of the publishers. Photocopying. In the USA: This journal is registered at the Copyright Clearance Cen- ter, Inc., 222 Rosewood Drive, Danvers, MA 01923. Authorization to photocopy items for internal or personal use, or the internal or personal use of specific clients, is granted by now Publishers Inc for users registered with the Copyright Clearance Center (CCC). The ‘services’ for users can be found on the internet at: www.copyright.com For those organizations that have been granted a photocopy license, a separate system of payment has been arranged. Authorization does not extend to other kinds of copy- ing, such as that for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. In the rest of the world: Permission to pho- tocopy must be obtained from the copyright owner. Please apply to now Publishers Inc., PO Box 1024, Hanover, MA 02339, USA; Tel. +1 781 871 0245; www.nowpublishers.com; [email protected] now Publishers Inc. has an exclusive license to publish this material worldwide. Permission to use this content must be obtained from the copyright license holder. Please apply to now Publishers, PO Box 179, 2600 AD Delft, The Netherlands, www.nowpublishers.com; e-mail: [email protected] Full text available at: http://dx.doi.org/10.1561/1900000055 R Foundations and Trends in Databases Volume 8, Issue 4, 2016 Editorial Board Editor-in-Chief Joseph M. Hellerstein University of California, Berkeley United States Editors Anastasia Ailamaki Ihab Ilyas EPFL University of Waterloo Peter Bailis Christopher Olston University of California, Berkeley Yahoo! Research Mike Cafarella Jignesh Patel University of Michigan University of Michigan Michael Carey Chris Re UC Irvine Stanford University Surajit Chaudhuri Gerhard Weikum Microsoft Research Max Planck Institute Saarbrücken Minos Garofalakis Yahoo! Research Full text available at: http://dx.doi.org/10.1561/1900000055 Editorial Scope Topics R Foundations and Trends• in Databases covers a breadth of topics re- lating to the management of large volumes of data. The journal targets the full scope of issues in data management, from theoretical founda- tions, to languages and modeling, to algorithms, system architecture, and applications. The list of topics below illustrates some of the in- tended coverage, though it is by no means exhaustive: Data models and query languages Data warehousing • • Query processing and Adaptive query processing • optimization • Data stream management Storage, access methods, and • • Search and query integration indexing • XML and semi-structured data Transaction management, • • concurrency control, and Web services and middleware • recovery Data integration and exchange • Deductive databases • Private and secure data • Parallel and distributed database management • systems Peer-to-peer, sensornet, and Database design and tuning • • mobile data management Metadata management Scientific and spatial data • • Object management management • Trigger processing and active Data brokering and • • databases publish/subscribe Data mining and OLAP Data cleaning and information • • extraction Approximate and interactive • query processing Probabilistic data management • Information for Librarians R Foundations and Trends• in Databases, 2016, Volume 8, 4 issues. ISSN pa- per version 1931-7883. ISSN online version 1931-7891. Also available as a combined paper and online subscription. Full text available at: http://dx.doi.org/10.1561/1900000055 R Foundations and Trends• in Databases Vol. 8, No. 4 (2016) 239–370 c 2018 P. Koutris, S. Salihoglu and D. Suciu • DOI: 10.1561/1900000055 Algorithmic Aspects of Parallel Data Processing Paraschos Koutris Semih Salihoglu University of Wisconsin-Madison University of Waterloo [email protected] [email protected] Dan Suciu University of Washington [email protected] Full text available at: http://dx.doi.org/10.1561/1900000055 Contents 1 Introduction 2 2 Models of Parallel Computation 6 2.1 The Massively Parallel Computation Model . 6 2.2 Other Models of Parallel Computation . 12 2.3 Comparing Sequential and Parallel Algorithms . 22 3 Two-way Join 28 3.1 Hash-Based Algorithms . 29 3.2 Sort-Based Algorithms . 39 3.3 Binary Join Algorithms in Existing Systems . 42 4 Multiway Joins 44 4.1 Single Round . 47 4.2 Multiple Rounds . 66 4.3 Multiway Join Algorithms In Existing Systems . 86 5 Sorting 89 5.1 Lower Bounds for Parallel Sorting . 90 5.2 Parallel Sorting Algorithms . 94 5.3 Discussion . 102 ii Full text available at: http://dx.doi.org/10.1561/1900000055 iii 6 Matrix Multiplication 104 6.1 Lower-bounds for Conventional Matrix Multiplication . 106 6.2 Algorithms . 112 6.3 Discussion . 118 6.4 Other Linear-Algebra Computations . 119 7 Conclusion 122 References 126 Full text available at: http://dx.doi.org/10.1561/1900000055 Abstract In the last decade or so we have witnessed a growing interest in process- ing large data sets on large distributed clusters. The idea was pioneered by the MapReduce framework, and has been widely adopted by sev- eral other systems, including PigLatin, Hive, Scope, U-SQL, Dremmel, Spark and Myria. A large part of the complex data analysis performed by these systems consists of a sequence of relatively simple query op- erations, such as joining two or more tables. This survey discusses re- cent algorithmic developments for distributed data processing. It uses a theoretical model of parallel processing called the Massively Paral- lel Computation (MPC) model, which is a simplification of the BSP model where the only cost is given by the amount of communication and the number of communication rounds. The survey studies several algorithms for multi-join queries, for sorting, and for matrix multiplica- tion, and discusses their relationships and common techniques applied across the different data processing tasks. P. Koutris, S. Salihoglu and D. Suciu. Algorithmic Aspects of Parallel Data R Processing. Foundations and Trends• in Databases, vol. 8, no. 4, pp. 239–370, 2016. DOI: 10.1561/1900000055. Full text available at: http://dx.doi.org/10.1561/1900000055 1 Introduction In the last decade we have witnessed a huge and growing interest in processing large data sets on large distributed clusters. This trend be- gan with the MapReduce framework [31], and has been widely adopted by several other systems, including PigLatin [69], Hive [83], Scope [24], Dremmel [65], Spark [91] and Myria [88] to name a few. While the applications of such systems are diverse (e.g., machine learning, data analytics), most involve relatively standard data processing tasks, such as identifying relevant data, cleaning, filtering, joining, grouping, trans- forming, extracting features, and evaluating results [25, 35]. This has generated great interest in the study of algorithms for data processing on large distributed clusters. This survey reviews some of the recent theoretical results on efficient data processing on large distributed architectures, as well as some of the relevant classical results on parallel sorting and parallel matrix multiplication. The survey begins in Chapter 2 with a review of parallel models used to analyze algorithms on large distributed clusters. Modern data analytics run on large, shared-nothing clusters, where the cost of com- munication during data reshuffling can dominate the running time. For example, individual jobs in Cosmos, Microsoft’s distributed file sys- 2 Full text available at: http://dx.doi.org/10.1561/1900000055 3 tem, often execute on over 10k nodes [72]. We introduce a very simple model of parallel computation, called the Massively Parallel Compu- tation model (MPC) where the cost of a distributed algorithm is mea- sured in the amount of communication per processor and the number of communication rounds. This model is a simplification of Valiant’s Bulk Synchronous Parallel (BSP) model [84], and allows us to separate the computation cost from the communication cost, and to focus solely on the latter. In this chapter we introduce the MPC model, then review several important classical models of parallel computation, and discuss their connection to the MPC model. In Chapter 3 we present and analyze two different approaches for computing in parallel the join of two large relations. Join operations are the bread and butter of most database processing tasks, and the support of efficient join algorithms is a top priority for all major big data systems. We discuss Parallel Hash join, and Parallel Sort Join. The preferred algorithm in practice is the Parallel Hash join, because on most datasets this algorithm is very effective and scales up linearly with the number of processors. However, the Parallel Hash join per- forms poorly on skewed data, when a large number of records have the same value of the join attribute and, thus, are hashed to the same processor.