DOCSLIB.ORG

Uva-DARE (Digital Academic Repository)

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Uva-DARE (Digital Academic Repository) UvA-DARE (Digital Academic Repository) On entity resolution in probabilistic data Ayat, S.N. Publication date 2014 Document Version Final published version Link to publication Citation for published version (APA): Ayat, S. N. (2014). On entity resolution in probabilistic data. General rights It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons). Disclaimer/Complaints regulations If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible. UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl) Download date:01 Oct 2021 On Entity Resolution in Probabilistic Data Naser Ayat On Entity Resolution in Probabilistic Data Naser Ayat On Entity Resolution in Probabilistic Data Naser Ayat On Entity Resolution in Probabilistic Data Academisch Proefschrift ter verkrijging van de graad van doctor aan de Universiteit van Amsterdam op gezag van de Rector Magnificus prof.dr. D.C. van den Boom ten overstaan van een door het college voor promoties ingestelde commissie, in het openbaar te verdedigen in de Agnietenkapel op dinsdag 30 september 2014, te 12.00 uur door Seyed Naser Ayat geboren te Esfahan, Iran Promotors: prof. dr. H. Afsarmanesh prof. dr. P. Valduriez Copromotor: dr. R. Akbarinia Overige leden: prof. dr. P.W. Adriaans prof. dr. P.M.A. Sloot prof. dr. A.P.J.M. Siebes prof. dr. M.T. Bubak prof. dr. ir. F.C.A Groen Faculteit der Natuurwetenschappen, Wiskunde en Informatica SIKS Dissertation Series No. 2014-32 The research reported in this thesis has been carried out under the auspices of SIKS, the Dutch Research School for Information and Knowledge Systems. Copyright c 2014 by Naser Ayat Printed by: GVO printers & designers. ISBN: 978 90 6464 812 0 to Aylar, and my parents iii Contents 1 Introduction 1 1.1 Uncertain Data . .1 1.2 Entity Resolution . .2 1.3 Entity Resolution for Probabilistic Data . .4 1.4 Research Questions . .5 1.5 Roadmap . .7 2 Background and Related Work 9 2.1 Uncertain Data Models . .9 2.1.1 Possible Worlds Semantics . .9 2.1.2 X-relation Data Model . 11 2.2 Entity Resolution . 12 2.2.1 Matching Methods . 14 2.2.2 Scalability Issues . 24 2.2.3 Entity Resolution for Probabilistic Data . 26 2.2.4 Conclusion . 29 3 Entity Resolution for Probabilistic Data 31 3.1 Introduction . 31 3.2 Problem Definition . 33 3.2.1 Data Model . 33 3.2.2 Context-Free and Context-Sensitive Similarity Functions . 33 3.2.3 Problem Statement . 34 3.3 Context-Free Entity Resolution . 36 3.3.1 CFA Algorithm . 36 3.3.2 Improving CFA Algorithm . 38 3.4 Context-sensitive Entity Resolution . 41 3.4.1 Monte Carlo Algorithm . 42 3.4.2 Parallel MC . 42 v 3.4.3 CB Similarity Function . 43 3.5 Performance Evaluation . 48 3.5.1 Experimental setup . 48 3.5.2 Results . 51 3.6 Analysis against related work . 54 3.7 Conclusion . 55 4 Entity Resolution for Distributed Probabilistic Data 57 4.1 Introduction . 57 4.2 Problem definition . 59 4.3 Distributed Computation of Most-probable Matching-pair . 59 4.3.1 Algorithm Overview . 60 4.3.2 Extract the Essential-set . 61 4.3.3 Merge-and-Backward Essential-Sets . 66 4.3.4 MPMP Computation and Data Retrieval . 67 4.4 FD Example . 68 4.5 Analysis of Communication Cost . 69 4.5.1 Distributed System Model . 69 4.5.2 Forward Messages . 71 4.5.3 Backward Messages . 72 4.5.4 Retrieve Messages . 73 4.6 Performance Evaluation . 73 4.6.1 Experimental and Simulation Setup . 74 4.6.2 Response Time . 77 4.6.3 Communication Cost . 79 4.6.4 Case Study on Real Data . 81 4.7 Analysis against related Work . 82 4.8 Conclusion . 83 5 Entity Resolution for Probabilistic Data Using Entropy Reduc- tion 85 5.1 Introduction . 85 5.2 Problem Definition . 87 5.2.1 Data Model . 87 5.2.2 Entropy in Probabilistic Databases . 87 5.2.3 Entity Resolution Over Probabilistic Database . 88 5.2.4 Problem Statement . 89 5.3 ERPD Using Entropy . 90 5.3.1 Computing Entropy in X-relations . 90 5.3.2 Merge Function . 92 5.3.3 ME Algorithm . 96 5.3.4 Time Complexity . 97 5.3.5 Multi-Alternative X-relation Case . 98 vi 5.4 Performance Evaluation . 99 5.4.1 Experimental Setup . 99 5.4.2 Performance Results . 102 5.5 Analysis against related Work . 107 5.6 Conclusion . 108 6 Pay-As-You-Go Data Integration Using Functional Dependen- cies 109 6.1 Introduction . 109 6.2 Problem Definition . 111 6.3 System Architecture . 113 6.4 Schema Matching . 114 6.4.1 Distance Function . 114 6.4.2 Schema Matching Algorithm . 120 6.4.3 Adding Data Sources Incrementally . 121 6.4.4 Execution Cost Analysis . 122 6.5 Performance Evaluation . 123 6.5.1 Experimental Setup . 123 6.5.2 Results . 124 6.6 Analysis against related Work . 127 6.7 Conclusion . 127 7 Conclusion 129 7.1 Answers to Research Questions . 129 7.2 Future Work . 132 A Uncertain Data Models 133 A.1 Incomplete Relational Models . 133 A.1.1 Codd tables . 133 A.1.2 C-tables . 134 A.2 Fuzzy Relational Models . 135 A.2.1 Fuzzy Set . 136 A.2.2 Fuzzy Relations . 136 A.3 Probabilistic Relational Models . 138 A.4 Probabilistic Graphical Models . 139 B The Courses Dataset 143 C Author's publications 147 Bibliography 149 Abstract 161 vii Samenvatting 165 Acknowledgments 169 SIKS Dissertation Series 171 viii Chapter 1 Introduction 1.1 Uncertain Data There are many applications that produce uncertain data. Untrusted sources, imprecise measuring instruments, and uncertain methods, are some reasons that cause uncertainty in data. Let us provide some examples of uncertain data which arise in different real-life applications. • Uncertainty inherent in data. There are many data sources that are inherently uncertain, such as scientific measurements, sensor readings, lo- cation data from GPS, and RFID data. • Data integration. An important step in automatically integrating a num- ber of structured data sources is to decide which schema elements have the same semantics, i.e. schema matching. However, the result of schema matching is also itself often uncertain. For instance, a schema matching approach outputs \phone" and \phone-no" as matching, while mentioning that they have a 90% chance to have the same semantics. • Information extraction. The process of extracting information from un- structured data sources, e.g. web, is an uncertain process, meaning that extracted information are typically associated with probability values indi- cating the likelihood that the extracted information is correct. Consider, for instance, an extractor which decides with 0.95 confidence that Bob lives in Amsterdam. • Optical Character Recognition (OCR). The aim of OCR is to recog- nize the text within handwritten or typewritten scanned documents. The output of OCR is often uncertain. For instance, an OCR method, which cannot recognize, with certainty, the 'o' character in the word \Ford", may produce the set f(\Ford" : 0.8), (\F0rd" : 0.2)g as output, meaning that 1 2 Chapter 1. Introduction each of the two output words may represent the original scanned word with the associated probability. Broadly speaking, there are two approaches in dealing with uncertain data: 1) ignoring it, and 2) managing it. While the former approach results in loss of information which is contained in uncertain data, in recent years, we have been witnessing much interest in the latter approach. As shown in the above examples, most real-life applications tend to quantify data uncertainty with probability values, referred to as probabilistic data. We also deal with probabilistic data in this thesis. 1.2 Entity Resolution Entity Resolution1 (ER) is the process of identifying duplicate tuples, refereing to the tuples that represent the same real-world entity. The ER problem arises in many real-life applications, such as data integration. Consider, for instance, that the large company A acquires another large company B. It is quite likely that A and B share many customers. On the other hand, it is not reasonable to expect a person, who is the customer of both companies, to be represented by identical tuples in the databases of the two companies. For example, consider tuple tA “ p\Thomas Michaelis"; \45, Main street"q in A's database and tuple tB “ p\T. Michaelis"; \45, Main st., 1951"q in B's database, both representing the same customer. In the above example, it is clear for humans that tA and tB refer to the same person. However, it is not reasonable to expect the humans to solve the ER problem on large datasets, since every tuple in the dataset should be compared against all other tuples in the same dataset, giving the ER process an Opn2q complexity. On the other hand, computational power exists on computers to deal easily with the ER problem on large datasets, although they lack the human intelligence, which greatly simplifies the identification of duplicate tuples.
Load more

Recommended publications
  • Probabilistic Databases
    Series ISSN: 2153-5418 SUCIU • OLTEANU •RÉ •KOCH M SYNTHESIS LECTURES ON DATA MANAGEMENT &C Morgan & Claypool Publishers Series Editor: M. Tamer Özsu, University of Waterloo Probabilistic Databases Probabilistic Databases Dan Suciu, University of Washington, Dan Olteanu, University of Oxford Christopher Ré,University of Wisconsin-Madison and Christoph Koch, EPFL Probabilistic databases are databases where the value of some attributes or the presence of some records are uncertain and known only with some probability. Applications in many areas such as information extraction, RFID and scientific data management, data cleaning, data integration, and financial risk DATABASES PROBABILISTIC assessment produce large volumes of uncertain data, which are best modeled and processed by a probabilistic database. This book presents the state of the art in representation formalisms and query processing techniques for probabilistic data. It starts by discussing the basic principles for representing large probabilistic databases, by decomposing them into tuple-independent tables, block-independent-disjoint tables, or U-databases. Then it discusses two classes of techniques for query evaluation on probabilistic databases. In extensional query evaluation, the entire probabilistic inference can be pushed into the database engine and, therefore, processed as effectively as the evaluation of standard SQL queries. The relational queries that can be evaluated this way are called safe queries. In intensional query evaluation, the probabilistic Dan Suciu inference is performed over a propositional formula called lineage expression: every relational query can be evaluated this way, but the data complexity dramatically depends on the query being evaluated, and Dan Olteanu can be #P-hard. The book also discusses some advanced topics in probabilistic data management such as top-kquery processing, sequential probabilistic databases, indexing and materialized views, and Monte Carlo databases.
    [Show full text]
  • Adaptive Schema Databases ∗
    Adaptive Schema Databases ∗ William Spothb, Bahareh Sadat Arabi, Eric S. Chano, Dieter Gawlicko, Adel Ghoneimyo, Boris Glavici, Beda Hammerschmidto, Oliver Kennedyb, Seokki Leei, Zhen Hua Liuo, Xing Niui, Ying Yangb b: University at Buffalo i: Illinois Inst. Tech. o: Oracle {wmspoth|okennedy|yyang25}@buffalo.edu {barab|slee195|xniu7}@hawk.iit.edu [email protected] {eric.s.chan|dieter.gawlick|adel.ghoneimy|beda.hammerschmidt|zhen.liu}@oracle.com ABSTRACT in unstructured or semi-structured form, then an ETL (i.e., The rigid schemas of classical relational databases help users Extract, Transform, and Load) process needs to be designed in specifying queries and inform the storage organization of to translate the input data into relational form. Thus, clas- data. However, the advantages of schemas come at a high sical relational systems require a lot of upfront investment. upfront cost through schema and ETL process design. In This makes them unattractive when upfront costs cannot this work, we propose a new paradigm where the database be amortized, such as in workloads with rapidly evolving system takes a more active role in schema development and data or where individual elements of a schema are queried data integration. We refer to this approach as adaptive infrequently. Furthermore, in settings like data exploration, schema databases (ASDs). An ASD ingests semi-structured schema design simply takes too long to be practical. or unstructured data directly using a pluggable combina- Schema-on-query is an alternative approach popularized tion of extraction and data integration techniques. Over by NoSQL and Big Data systems that avoids the upfront time it discovers and adapts schemas for the ingested data investment in schema design by performing data extraction using information provided by data integration and infor- and integration at query-time.
    [Show full text]
  • Finding Interesting Itemsets Using a Probabilistic Model for Binary Databases
    Finding interesting itemsets using a probabilistic model for binary databases Tijl De Bie University of Bristol, Department of Engineering Mathematics Queen’s Building, University Walk, Bristol, BS8 1TR, UK [email protected] ABSTRACT elegance and e±ciency of the algorithms to search for fre- A good formalization of interestingness of a pattern should quent patterns. Unfortunately, the frequency of a pattern satisfy two criteria: it should conform well to intuition, and is only loosely related to interestingness. The output of fre- it should be computationally tractable to use. The focus quent pattern mining methods is usually an immense bag has long been on the latter, with the development of frequent of patterns that are not necessarily interesting, and often pattern mining methods. However, it is now recognized that highly redundant with each other. This has hampered the more appropriate measures than frequency are required. uptake of FPM and FIM in data mining practice. In this paper we report results in this direction for item- Recent research has shifted focus to the search for more set mining in binary databases. In particular, we introduce useful formalizations of interestingness that match practical a probabilistic model that can be ¯tted e±ciently to any needs more closely, while still being amenable to e±cient binary database, and that has a compact and explicit repre- algorithms. As FIM is arguably the simplest special case of sentation. We then show how this model enables the formal- frequent pattern mining, it is not surprising that most of the ization of an intuitive and tractable interestingness measure recent work has focussed on itemset patterns, see e.g.
    [Show full text]
  • Open-World Probabilistic Databases
    Open-World Probabilistic Databases Ismail˙ Ilkan˙ Ceylan and Adnan Darwiche and Guy Van den Broeck Computer Science Department University of California, Los Angeles [email protected],fdarwiche,[email protected] Abstract techniques, from part-of-speech tagging until relation ex- traction (Mintz et al. 2009). For knowledge-base comple- Large-scale probabilistic knowledge bases are becoming in- tion – the task of deriving new facts from existing knowl- creasingly important in academia and industry alike. They edge – statistical relational learning algorithms make use are constantly extended with new data, powered by modern information extraction tools that associate probabilities with of embeddings (Bordes et al. 2011; Socher et al. 2013) database tuples. In this paper, we revisit the semantics under- or probabilistic rules (Wang, Mazaitis, and Cohen 2013; lying such systems. In particular, the closed-world assump- De Raedt et al. 2015). In both settings, the output is a pre- tion of probabilistic databases, that facts not in the database dicted fact with its probability. It is therefore required to de- have probability zero, clearly conflicts with their everyday fine probabilistic semantics for knowledge bases. The classi- use. To address this discrepancy, we propose an open-world cal and most-basic model is that of tuple-independent prob- probabilistic database semantics, which relaxes the probabili- abilistic databases (PDBs) (Suciu et al. 2011), which in- ties of open facts to intervals. While still assuming a finite do- deed underlies many of these systems (Dong et al. 2014; main, this semantics can provide meaningful answers when Shin et al. 2015).
    [Show full text]
  • A Compositional Query Algebra for Second-Order Logic and Uncertain Databases
    A Compositional Query Algebra for Second-Order Logic and Uncertain Databases Christoph Koch Department of Computer Science Cornell University, Ithaca, NY, USA [email protected] ABSTRACT guages for querying possible worlds. Indeed, second-order World-set algebra is a variable-free query language for uncer- quantifiers are the essence of what-if reasoning in databases. tain databases. It constitutes the core of the query language World-set algebra seems to be a strong candidate for a core implemented in MayBMS, an uncertain database system. algebra for forming query plans and optimizing and execut- This paper shows that world-set algebra captures exactly ing them in uncertain database management systems. second-order logic over finite structures, or equivalently, the It was left open in previous work whether world-set alge- polynomial hierarchy. The proofs also imply that world- bra is closed under composition, or whether definitions are set algebra is closed under composition, a previously open adding to the expressive power of the language. Compo- problem. sitionality is a desirable and rather commonplace property of query algebras, but in the case of WSA it seems rather unlikely to hold. The reason for this is that the algebra con- 1. INTRODUCTION tains an uncertainty-introduction operation that on the level Developing suitable query languages for uncertain data- of possible worlds is nondeterministic. First materializing a bases is a substantial research challenge that is only cur- view and subsequently using it multiple times in the query is rently starting to get addressed. Conceptually, an uncertain semantically quite different from composing the query with database is a finite set of possible worlds, each one a re- the view and thus obtaining several copies of the view def- lational database.
    [Show full text]
  • Answering Queries from Statistics and Probabilistic Views∗
    Answering Queries from Statistics and Probabilistic Views∗ Nilesh Dalvi Dan Suciu University of Washington, Seattle, WA, USA 1 Introduction paper we will assume the Local As View (LAV) data integration paradigm [17, 16], which consists of defin- Systems integrating dozens of databases, in the scien- ing a global mediated schema R¯, then expressing each tific domain or in a large corporation, need to cope local source i as a view vi(R¯) over the global schema. with a wide variety of imprecisions, such as: different Users are allowed to ask queries over the global, me- representations of the same object in different sources; diated schema, q(R¯), however the data is given as in- imperfect and noisy schema alignments; contradictory stances J1; : : : ; Jm of the local data sources. In our information across sources; constraint violations; or in- model all instances are probabilistic, both the local sufficient evidence to answer a given query. If standard instances and the global instance. Statistics are given query semantics were applied to such data, all but the explicitly over the global schema R¯, and the probabil- most trivial queries will return an empty answer. ities are given explicitly over the local sources, hence We believe that probabilistic databases are the right over the views. We make the Open World Assumption paradigm to model all types of imprecisions in a uni- throughout the paper. form and principled way. A probabilistic database is a probability distribution on all instances [5, 4, 15, 1.1 Example: Using Statistics 12, 11, 8]. Their early motivation was to model im- precisions at the tuple level: tuples are not known Suppose we integrate two sources, one show- with certainty to belong to the database, or represent ing which employee works for what depart- noisy measurements, etc.
    [Show full text]
  • BAYESSTORE: Managing Large, Uncertain Data Repositories with Probabilistic Graphical Models
    BAYESSTORE: Managing Large, Uncertain Data Repositories with Probabilistic Graphical Models Daisy Zhe Wang∗ , Eirinaios Michelakis∗ , Minos Garofalakisy∗ , and Joseph M. Hellerstein∗ ∗ Univ. of California, Berkeley EECS and y Yahoo! Research ABSTRACT Of course, the fundamental mathematical tools for managing un- certainty come from probability and statistics. In recent years, these Several real-world applications need to effectively manage and reason about tools have been aggressively imported into the computational do- large amounts of data that are inherently uncertain. For instance, perva- main under the rubric of Statistical Machine Learning (SML). Of sive computing applications must constantly reason about volumes of noisy special note here is the widespread use of Graphical Modeling tech- sensory readings for a variety of reasons, including motion prediction and niques, including the many variants of Bayesian Networks (BNs) human behavior modeling. Such probabilistic data analyses require so- and Markov Random Fields (MRFs) [13]. These techniques can phisticated machine-learning tools that can effectively model the complex provide robust statistical models that capture complex correlation spatio/temporal correlation patterns present in uncertain sensory data. Un- patterns among variables, while, at the same time, addressing some fortunately, to date, most existing approaches to probabilistic database sys- computational efficiency and scalability issues as well. Graphical tems have relied on somewhat simplistic models of uncertainty that can be models have been applied with great success in applications as di- easily mapped onto existing relational architectures: Probabilistic informa- verse as signal processing, information retrieval, sensornets and tion is typically associated with individual data tuples, with only limited pervasive computing, robotics, natural language processing, and or no support for effectively capturing and reasoning about complex data computer vision.
    [Show full text]
  • Brief Tutorial on Probabilistic Databases
    Brief Tutorial on Probabilistic Databases Dan Suciu University of Washington Simons 2016 1 About This Talk • Probabilistic databases – Tuple-independent – Query evaluation • Statistical relational models – Representation, learning, inference in FO – Reasoning/learning = lifted inference • Sources: – Book 2011 [S.,Olteanu,Re,Koch] – Upcoming F&T survey [van Den Broek,S] Simons 2016 2 Background: Relational databases Smoker Friend X Y X Z Database D = Alice 2009 Alice Bob Alice 2010 Alice Carol Bob 2009 Bob Carol Carol 2010 Carol Bob Query: Q(z) = ∃x (Smoker(x,’2009’)∧Friend(x,z)) Z Constraint: Q(D) = Bob Q = ∀x (Smoker(x,’2010’)⇒Friend(x,’Bob’)) Carol Q(D) = true Probabilistic Database Probabilistic database D: X y P a1 b1 p1 a1 b2 p2 a2 b2 p3 Possible worlds semantics: Σworld PD(world) = 1 X y P (Q) = Σ P (world) X Xy y D world |= Q D a1 b1 X y a1 a1b2 b1 X y a1 b2 a1 b1 X y p1p2p3 a2 a2b2 b2 a2 b2 X y a2 b2 a1 b2 a1 b1 X y (1-p1)p2p3 a1 b2 (1-p1)(1-p2)(1-p3) Outline • Model Counting • Small Dichotomy Theorem • Dichotomy Theorem • Query Compilation • Conclusions, Open Problems Simons 2016 5 Model Counting • Given propositional Boolean formula F, compute the number of models #F Example: X1 X2 X3 F F = (X1∨X2) ∧ (X2∨X3) ∧ (X3∨X1) 0 0 0 0 0 0 1 0 #F = 4 0 1 0 0 0 1 1 1 1 0 0 0 1 0 1 1 1 1 0 1 [Valiant’79] #P-hard, even for 2CNF 1 1 1 1 Probability of a Formula • Each variable X has a probability p(X); • P(F) = probability that F=true, when each X is set to true independently Example: X1 X2 X3 F F = (X1∨X2) ∧ (X2∨X3) ∧ (X3∨X1) 0 0 0 0 0 0 1 0 P(F) = (1-p1)*p2*p3 + 0 1 0 0 p1*(1-p2)*p3 + p1*p2*(1-p3) + 0 1 1 1 p1*p2*p3 1 0 0 0 1 0 1 1 n If p(X) = ½ for all X, then P(F) = #F / 2 1 1 0 1 1 1 1 1 Algorithms for Model Counting [Gomes, Sabharwal, Selman’2009] Based on full search DPLL: • Shannon expansion.
    [Show full text]
  • Efficient In-Database Analytics with Graphical Models
    Efficient In-Database Analytics with Graphical Models Daisy Zhe Wang, Yang Chen, Christan Grant and Kun Li fdaisyw,yang,cgrant,[email protected]fl.edu University of Florida, Department of Computer and Information Science and Engineering Abstract Due to recent application push, there is high demand in industry to extend database systems to perform efficient and scalable in-database analytics based on probabilistic graphical models (PGMs). We discuss issues in supporting in-database PGM methods and present techniques to achieve a deep integration of the PGM methods into the relational data model as well as the query processing and optimization engine. This is an active research area and the techniques discussed are being further developed and evaluated. 1 Introduction Graphical models, also known as probabilistic graphical models (PGMs), are a class of expressive and widely adopted machine learning models that compactly represent the joint probability distribution over a large number of interdependent random variables. They are used extensively in large-scale data analytics, including infor- mation extraction (IE), knowledge base population (KBP), speech recognition and computer vision. In the past decade, many of these applications have witnessed dramatic increase in data volume. As a result, PGM-based data analytics need to scale to terabytes of data or billions of data items. While new data-parallel and graph- parallel processing platforms such as Hadoop, Spark and GraphLab [1, 36, 3] are widely used to support scalable PGM methods, such solutions are not effective for applications with data residing in databases or with need to support online PGM-based data analytics queries. More specifically, large volumes of valuable data are likely to pour into database systems for many years to come due to the maturity of the commercial database systems with transaction support and ACID properties and due to the legal auditing and data privacy requirements in many organizations.
    [Show full text]
  • Indexing Correlated Probabilistic Databases
    Indexing Correlated Probabilistic Databases Bhargav Kanagal Amol Deshpande [email protected] [email protected] Dept. of Computer Science, University of Maryland, College Park MD 20742 ABSTRACT 1. INTRODUCTION With large amounts of correlated probabilistic data being Large amounts of correlated probabilistic data are be- generated in a wide range of application domains includ- ing generated in a variety of application domains includ- ing sensor networks, information extraction, event detection ing sensor networks [14, 21], information extraction [20, 17], etc., effectively managing and querying them has become data integration [1, 15], activity recognition [28], and RFID an important research direction. While there is an exhaus- stream analysis [29, 26]. The correlations exhibited by these tive body of literature on querying independent probabilistic data sources can range from simple mutual exclusion depen- data, supporting efficient queries over large-scale, correlated dencies, to complex dependencies typically captured using databases remains a challenge. In this paper, we develop Bayesian or Markov networks [30] or through constraints on efficient data structures and indexes for supporting infer- the possible values that the tuple attributes can take [19, 12]. ence and decision support queries over such databases. Our Although several approaches have been proposed for rep- proposed hierarchical data structure is suitable both for in- resenting complex correlations and for querying over corre- memory and disk-resident databases. We represent the cor- lated databases, the rapidly increasing scale of such databases relations in the probabilistic database using a junction tree has raised unique challenges that have not been addressed over the tuple-existence or attribute-value random variables, before.
    [Show full text]
  • Analyzing Uncertain Tabular Data
    Analyzing Uncertain Tabular Data Oliver Kennedy and Boris Glavic Abstract It is common practice to spend considerable time refining source data to address issues of data quality before beginning any data analysis. For ex- ample, an analyst might impute missing values or detect and fuse duplicate records representing the same real-world entity. However, there are many sit- uations where there are multiple possible candidate resolutions for a data quality issue, but there is not sufficient evidence for determining which of the resolutions is the most appropriate. In this case, the only way forward is to make assumptions to restrict the space of solutions and/or to heuristically choose a resolution based on characteristics that are deemed predictive of \good" resolutions. Although it is important for the analyst to understand the impact of these assumptions and heuristic choices on her results, evalu- ating this impact can be highly non-trivial and time consuming. For several decades now, the fields of probabilistic, incomplete, and fuzzy databases have developed strategies for analyzing the impact of uncertainty on the outcome of analyses. This general family of uncertainty-aware databases aims to model ambiguity in the results of analyses expressed in standard languages like SQL, SparQL, R, or Spark. An uncertainty-aware database uses descriptions of potential errors and ambiguities in source data to derive a correspond- ing description of potential errors or ambiguities in the result of an analysis accessing this source data. Depending on technique, these descriptions of uncertainty may be either quantitative (bounds, probabilities), or qualita- tive (certain outcomes, unknown values, explanations of uncertainty).
    [Show full text]
  • Entity Linkage for Heterogeneous, Uncertain, and Volatile Data Dr
    Entity Linkage for Heterogeneous, Uncertain, and Volatile Data Von der Fakultat¨ fur¨ Elektrotechnik und Informatik der Gottfried Wilhelm Leibniz Universitat¨ Hannover zur Erlangung des Grades Doktor der Naturwissenschaften Dr. rer. nat. genehmigte Dissertation von M.Sc. Ekaterini Ioannou geboren am 25. Juni 1980 in Lemesos, Zypern I II Referent Prof. Dr. Wolfgang Nejdl Institut fur¨ Verteilte Systeme Wissensbasierte Systeme (KBS) Universitat¨ Hannover Deutschland Ko-Referent Prof. Dr. Yannis Velegrakis Department of Information Engineering and Computer Science (DISI) University of Trento Italy Vorsitz Prof. Dr. Kurt Schneider Institut fur¨ Praktische Informatik Universitat¨ Hannover Deutschland Tag der Promotion 15. April 2011 Forschungszentrum L3S Hannover, Deutschland III IV Acknowledgments First and foremost, I would like to thank my advisor Prof. Wolfgang Nejdl for giving me the opportunity to join L3S and the freedom to pursue my research ideas. L3S is an exciting working environment, and for this I would like to also thank my friends and colleagues. I would also like to express my gratitude to Prof. Yannis Velegrakis for the collaboration and discussions. His always provided me with several interesting comments and ideas for improving and extending the work. I would like to thank my co-authors Dr. Peter Fankhauser, Dr. Claudia Niederee,´ George Papadakis, Odysseas Papapetrou, and Dr. Dimitrios Skoutas. Our join re- search work and discussions were always a pleasure. I thank Claudia for the sup- port, guidance, and of course for the fun times we had while working on projects and papers. I am especially grateful to Odysseas for his dual role as a co-author and as a husband, which made everything easier and more pleasant.
    [Show full text]