Full text available at: http://dx.doi.org/10.1561/1900000052 Query Processing on Probabilistic Data: A Survey Guy Van den Broeck University of California Los Angeles Dan Suciu University of Washington Seattle Boston — Delft Full text available at: http://dx.doi.org/10.1561/1900000052 Foundations and Trends R in Databases Published, sold and distributed by: now Publishers Inc. PO Box 1024 Hanover, MA 02339 United States Tel. +1-781-985-4510 www.nowpublishers.com [email protected] Outside North America: now Publishers Inc. PO Box 179 2600 AD Delft The Netherlands Tel. +31-6-51115274 The preferred citation for this publication is G. Van den Broeck and D. Suciu. Query Processing on Probabilistic Data: A Survey. Foundations and Trends R in Databases, vol. 7, no. 3-4, pp. 197–341, 2015. R This Foundations and Trends issue was typeset in LATEX using a class file designed by Neal Parikh. Printed on acid-free paper. ISBN: 978-1-68083-314-0 c 2017 G. Van den Broeck and D. Suciu All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, mechanical, photocopying, recording or otherwise, without prior written permission of the publishers. Photocopying. In the USA: This journal is registered at the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923. Authorization to photocopy items for internal or personal use, or the internal or personal use of specific clients, is granted by now Publishers Inc for users registered with the Copyright Clearance Center (CCC). The ‘services’ for users can be found on the internet at: www.copyright.com For those organizations that have been granted a photocopy license, a separate system of pay- ment has been arranged. Authorization does not extend to other kinds of copying, such as that for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. In the rest of the world: Permission to photocopy must be obtained from the copyright owner. Please apply to now Publishers Inc., PO Box 1024, Hanover, MA 02339, USA; Tel. +1 781 871 0245; www.nowpublishers.com; [email protected] now Publishers Inc. has an exclusive license to publish this material worldwide. Permission to use this content must be obtained from the copyright license holder. Please apply to now Publishers, PO Box 179, 2600 AD Delft, The Netherlands, www.nowpublishers.com; e-mail: [email protected] Full text available at: http://dx.doi.org/10.1561/1900000052 Foundations and Trends R in Databases Volume 7, Issue 3-4, 2015 Editorial Board Editor-in-Chief Joseph M. Hellerstein University of California, Berkeley United States Editors Anastasia Ailamaki Ihab Ilyas EPFL University of Waterloo Peter Bailis Christopher Olston University of California, Berkeley Yahoo! Research Mike Cafarella Jignesh Patel University of Michigan University of Michigan Michael Carey Chris Re UC Irvine Stanford University Surajit Chaudhuri Gerhard Weikum Microsoft Research Max Planck Institute Saarbrücken Minos Garofalakis Yahoo! Research Full text available at: http://dx.doi.org/10.1561/1900000052 Editorial Scope Topics Foundations and Trends R in Databases covers a breadth of topics re- lating to the management of large volumes of data. The journal targets the full scope of issues in data management, from theoretical founda- tions, to languages and modeling, to algorithms, system architecture, and applications. The list of topics below illustrates some of the in- tended coverage, though it is by no means exhaustive: • Data models and query languages • Data warehousing • Query processing and • Adaptive query processing optimization • Data stream management • Storage, access methods, and • Search and query integration indexing • XML and semi-structured data • Transaction management, concurrency control, and recovery • Web services and middleware • Deductive databases • Data integration and exchange • • Parallel and distributed database Private and secure data systems management • • Database design and tuning Peer-to-peer, sensornet, and mobile data management • Metadata management • Scientific and spatial data • Object management management • Trigger processing and active • Data brokering and databases publish/subscribe • Data mining and OLAP • Data cleaning and information extraction • Approximate and interactive query processing • Probabilistic data management Information for Librarians Foundations and Trends R in Databases, 2015, Volume 7, 4 issues. ISSN paper version 1931-7883. ISSN online version 1931-7891. Also available as a com- bined paper and online subscription. Full text available at: http://dx.doi.org/10.1561/1900000052 Foundations and Trends R in Databases Vol. 7, No. 3-4 (2015) 197–341 c 2017 G. Van den Broeck and D. Suciu DOI: 10.1561/1900000052 Query Processing on Probabilistic Data: A Survey Guy Van den Broeck Dan Suciu University of California University of Washington Los Angeles Seattle Full text available at: http://dx.doi.org/10.1561/1900000052 Contents 1 Introduction 2 2 Probabilistic Data Model 7 2.1 Possible Worlds Semantics . 8 2.2 Independence Assumptions . 9 2.3 Query Semantics . 16 2.4 Beyond Independence: Hard Constraints . 19 2.5 Beyond Independence: Soft Constraints . 22 2.6 From Soft Constraints to Independence . 29 2.7 Related Data Models . 33 3 Weighted Model Counting 40 3.1 Three Variants of Model Counting . 40 3.2 Relationships Between the Three Problems . 45 3.3 First-Order Model Counting . 47 3.4 Algorithms and Complexity for Exact Model Counting . 53 3.5 Algorithms for Approximate Model Counting . 56 4 Lifted Query Processing 62 4.1 Extensional Operators and Safe Plans . 64 4.2 Lifted Inference Rules . 69 4.3 Hierarchical Queries . 73 ii Full text available at: http://dx.doi.org/10.1561/1900000052 iii 4.4 The Dichotomy Theorem . 75 4.5 Negation . 82 4.6 Symmetric Databases . 85 4.7 Extensions . 88 5 Query Compilation 96 5.1 Compilation Targets . 97 5.2 Compiling UCQ . 104 5.3 Compilation Beyond UCQ . 115 6 Data, Systems, and Applications 117 6.1 Probabilistic Data . 117 6.2 Probabilistic Database Systems . 119 6.3 Applications . 122 7 Conclusions and Open Problems 125 Acknowledgements 127 References 128 Full text available at: http://dx.doi.org/10.1561/1900000052 Abstract Probabilistic data is motivated by the need to model uncertainty in large databases. Over the last twenty years or so, both the Database community and the AI community have studied various aspects of probabilistic relational data. This survey presents the main ap- proaches developed in the literature, reconciling concepts developed in parallel by the two research communities. The survey starts with an extensive discussion of the main probabilistic data models and their relationships, followed by a brief overview of model counting and its relationship to probabilistic data. After that, the survey discusses lifted probabilistic inference, which are a suite of techniques devel- oped in parallel by the Database and AI communities for probabilis- tic query evaluation. Then, it gives a short summary of query compi- lation, presenting some theoretical results highlighting limitations of various query evaluation techniques on probabilistic data. The survey ends with a very brief discussion of some popular probabilistic data sets, systems, and applications that build on this technology. G. Van den Broeck and D. Suciu. Query Processing on Probabilistic Data: A Survey. Foundations and Trends R in Databases, vol. 7, no. 3-4, pp. 197–341, 2015. DOI: 10.1561/1900000052. Full text available at: http://dx.doi.org/10.1561/1900000052 1 Introduction The goal of probabilistic databases is to manage large volumes of data that is uncertain, and where the uncertainty is defined by a probabil- ity space. The idea of adding probabilities to relational databases goes back to the early days of relational databases [Gelenbe and Hébrail, 1986, Cavallo and Pittarelli, 1987, Barbará et al., 1992], motivated by the need to represent NULL or unknown values for certain data items, data entry mistakes, measurement errors in data, “don’t care” values, or summary information [Gelenbe and Hébrail, 1986]. Today, the need to manage uncertainty in large databases is even more pressing, as structured data is often acquired automatically by extraction, integra- tion, or inference from other large data sources. The best known exam- ples of large-scale probabilistic datasets are probabilistic knowledge bases such as Yago [Hoffart et al., 2013], Nell [Carlson et al., 2010], DeepDive [Shin et al., 2015], Reverb [Fader et al., 2011], Microsoft’s Probase [Wu et al., 2012] or Google’s Knowledge Vault [Dong et al., 2014], which have millions to billions of uncertain tuples. Query processing in databases systems is a mature field. Tech- niques, tradeoffs, and complexities for query evaluation on all pos- sible hardware architectures have been studied intensively [Graefe, 2 Full text available at: http://dx.doi.org/10.1561/1900000052 3 1993, Chaudhuri, 1998, Kossmann, 2000, Abadi et al., 2013, Ngo et al., 2013], and many commercial or open-source query engines exists to- day that implement these algorithms. However, query processing on probabilistic data is quite a different problem, since now, in addition to traditional data processing we also need to do probabilistic inference. A typical query consists of joins, projections with duplicate removal, grouping and aggregation, and/or negation. When the input data is probabilistic, each tuple in the query