DOCSLIB.ORG

Distributed Complex Event Processing with Query Rewriting

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Distributed Complex Event Processing with Query Rewriting Distributed Complex Event Processing with Query Rewriting Nicholas Poul Matteo Migliavacca Peter Pietzuch Schultz-Møller Department of Computing Department of Computing Department of Computing Imperial College London Imperial College London Imperial College London United Kingdom United Kingdom United Kingdom [email protected] [email protected] [email protected] ABSTRACT is first stored and indexed and only queried afterwards. The nature of data in enterprises and on the Internet is In contrast, complex event processing (CEP) systems re- changing. Data used to be stored in a database first and gard new data (or updates to existing data) as a stream of queried later. Today timely processing of new data, repre- events [19]. Users can specify queries as high-level patterns sented as events, is increasingly valuable. In many domains, of events. The CEP system detects complex events matching complex event processing (CEP) systems detect patterns of these queries and notifies clients of matches in soft real-time. events for decision making. Examples include processing of For example, a credit card processing company may monitor environmental sensor data, trades in financial markets and fraudulent credit card use by specifying patterns that detect RSS web feeds. Unlike conventional database systems, most concurrent transactions caused by the same credit card at current CEP systems pay little attention to query optimi- geographically distant locations. sation. They do not rewrite queries to more efficient rep- Detecting complex patterns in high-rate event streams re- resentations or make decisions about operator distribution, quires substantial CPU resources. To make a CEP system limiting their overall scalability. scale in the number of concurrently detectable patterns, it needs additional computational resources by distributing de- This paper describes the Next CEP system that was espe- cially designed for query rewriting and distribution. Event tection across a set of machines or use the existing resources patterns are specified in a high-level query language and, be- more efficiently through query rewriting. However, the ma- fore being translated into event automata, are rewritten in jority of current CEP systems only support execution on a a more efficient form. Automata are then distributed across single machine and the semantics of their query languages a cluster of machines for detection scalability. We present make automated query rewriting challenging. algorithms for query rewriting and distributed placement. In this paper, we describe the design, implementation and Our experiments on the Emulab test-bed show a significant evaluation of the Next CEP system that focusses on dis- improvement in system scalability due to rewriting and dis- tributed event detection with query rewriting. The system tribution. uses an expressive automata-based approach for complex event detection. Users specify event patterns in a high-level, SQL-like language with six operators that are translated to 1. INTRODUCTION low-level event automata. The language facilitates rewriting Many application domains need timely processing of new of expressions into equivalent ones and the automata model data [8]. For example, a financial application may moni- eases deployment of detection operators across multiple ma- tor a continuous stream of credit card transactions for fraud chines. The system makes optimisation and placement deci- patterns. In environmental sciences, sensor networks output sions according to cost functions derived from the resource infinite sequences of measurements that may be processed consumption of event automata. Guided by the cost model, to respond quickly to physical events, such as an increase in our system rewrites queries containing two commonly used monitored pollution levels. On the Internet, RSS feeds from operators, \next" and \union", to reduce their CPU con- weblogs generate vast amounts of real-time information that sumption. The system also greedily selects distributed de- may be analysed to infer opinions of many people. In gen- ployment plans to perform event processing on a cluster of eral, the requirement of instant processing of new data is at machines while reusing already-existing operators. odds with the conventional database model, as implemented Our evaluation on the Emulab networks test-bed shows by database management systems (DBMSs), in which data that query rewriting increases the supported number of queries with the \next" operator by up to 100% and with the \union" operator of up to 20%. We also demonstrate how distribu- tion enables the system to scale linearly. Permission to make digital or hard copies of all or part of this work for In summary, the main contributions of this paper are: personal or classroom use is granted without fee provided that copies are 1.A high-level event pattern language based on a new ex- not made or distributed for profit or commercial advantage and that copies pressive event automata model that is suitable for query bear this notice and the full citation on the first page. To copy otherwise, to rewriting and distribution (x3); republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 2.A cost model for queries that allows automated optimi- DEBS’09, July 6-9, Nashville, TN, USA. sation (x4.2); Copyright 2009 ACM X-XXXXX-XX-X/XX/XX ...$5.00. 3. Algorithms for query rewriting into more efficient rep- Case for distributed detection. As estimated above, the resentations (x4.3) and for distributed query deployment data rate from a single source is relatively small. However, with operator reuse (x4.4); if a query correlates transaction events from several differ- 4. The design and implementation of a prototype CEP sys- ent sources, then hundreds of thousands of events have to tem in Erlang illustrating the above techniques (x5) and be processed per second. In this case, the CPU resources of an evaluation on the Emulab test-bed (x6). a single machine may become a bottleneck and limit system scalability. When the system runs out of CPU resources, it 1.1 Detection of Credit Card Fraud has to buffer events or discard them. Buffering events only We describe an application scenario | the detection of provides temporary relief and is only effective under tran- credit card fraud | that illustrates the scalability chal- sient overload conditions. A distributed CEP system with lenges in CEP and shows how efficient distribution can pro- query optimisation can distribute detection queries across vide relief. Credit card fraud has been committed for many machines and reduce the cost of individual queries by rewrit- years and remains a major problem. Just in the US alone, ing them into more efficient forms. 9:91 million people were victims of credit card fraud in 2007 with a total loss of US$ 52:6 billion [18]. Fraud typically 2. RELATED WORK happens when the credit card owner uses the card in a shop. During the payment, an employee can easily run the card A range of systems have been used to detect events us- through a card reader without the owner's knowledge. Later ing different detection approaches. This section provides an the copied credit card information is sold to other criminals overview of previous work, particularly focusing on propos- who manufacture a new card and commit the actual fraud. als for query optimisation in event processing. There are several patterns indicating such behaviour in Complex event processing systems. The goal of CEP credit card transaction logs, which can be detected by a systems is the fast detection of event patterns in streams. CEP system. A fast detection of suspicious patterns en- Specification languages for event patterns are frequently in- ables banks to contact customers early in the fraud or even spired by regular languages and therefore have automata- automatically alert users about suspicious credit card trans- based semantics [21]. In addition to commercial CEP sys- actions by sending text messages to their mobile phones. A tems, such as ruleCore CEP Server, Coral8 Engine and Es- typical fraud pattern is that the criminal starts by testing per, several open research prototypes exist. the credit card with a few small purchases followed by larger Cayuga [14, 13] is a high performance, single server CEP purchases. In our high-level query language described in x3, system developed at Cornell University. Event streams are this can be expressed as follows: infinite sequences of relational tuples with interval-based timestamps. Its event algebra has six operators: projec- SELECT * FROM ( t S1 ; t S2 ; t L ) tion, selection, renaming, union, conditional sequence and WHERE FILTER(S1.acc = S2.acc), FILTER(S2.acc = L.acc), iteration. Event algebra expressions are detected by non- S1.amount < 100 AND S2.amount < 100 AND deterministic finite automata, which can detect unbounded L.amount > 250 AND (S1, L) OCCURS WITHIN 12 hours sequences. To achieve high performance, Cayuga uses cus- Properties of data sources. We make the following as- tom heap management, indexing of operator predicates and sumptions about the workload of a CEP system for credit reuse of shared automata instances. However, Cayuga does card fraud detection. The system processes transactions not support automated query rewriting and distributed de- from multiple sources. The sources are credit card process- tection. The distribution of Cayuga automata is compli- ing companies responsible for a given credit card brand in a cated by the fact that it merges event algebra expressions given region, such as Northeastern US. into a single automaton, which would have to be partitioned We take credit card transactions made with VisaTM in the across nodes for distribution. US as an example. According to the Visa website [26], they The DistCED system [24] is a Java-based CEP frame- processed 27:612 billion transactions in 2007. This means work that uses extended, non-deterministic finite state au- an average of 875 credit transactions per second based on tomata.
Load more

Recommended publications
  • SAQE: Practical Privacy-Preserving Approximate Query Processing for Data Federations
    SAQE: Practical Privacy-Preserving Approximate Query Processing for Data Federations Johes Bater Yongjoo Park Xi He Northwestern University University of Illinois (UIUC) University of Waterloo [email protected] [email protected] [email protected] Xiao Wang Jennie Rogers Northwestern University Northwestern University [email protected] [email protected] ABSTRACT 1. INTRODUCTION A private data federation enables clients to query the union of data Querying the union of multiple private data stores is challeng- from multiple data providers without revealing any extra private ing due to the need to compute over the combined datasets without information to the client or any other data providers. Unfortu- data providers disclosing their secret query inputs to anyone. Here, nately, this strong end-to-end privacy guarantee requires crypto- a client issues a query against the union of these private records graphic protocols that incur a significant performance overhead as and he or she receives the output of their query over this shared high as 1,000× compared to executing the same query in the clear. data. Presently, systems of this kind use a trusted third party to se- As a result, private data federations are impractical for common curely query the union of multiple private datastores. For example, database workloads. This gap reveals the following key challenge some large hospitals in the Chicago area offer services for a cer- in a private data federation: offering significantly fast and accurate tain percentage of the residents; if we can query the union of these query answers without compromising strong end-to-end privacy. databases, it may serve as invaluable resources for accurate diag- To address this challenge, we propose SAQE, the Secure Ap- nosis, informed immunization, timely epidemic control, and so on.
    [Show full text]
  • WIN: an Efficient Data Placement Strategy for Parallel XML Databases
    WIN : An Efficient Data Placement Strategy for Parallel XML Databases Nan Tang† Guoren Wang‡ Jeffrey Xu Yu† Kam-Fai Wong† Ge Yu‡ † Department of SE & EM, The Chinese University of Hong Kong, Hong Kong, China ‡ College of Information Science and Engineering, Northeastern University, Shenyang, China {ntang, yu, kfwong}se.cuhk.edu.hk {wanggr, yuge}@mail.neu.edu.cn Abstract graph-based. Distinct from them, XML employs a tree struc- The basic idea behind parallel database systems is to ture. Conventional data placement strategies in perform operations in parallel to reduce the response RDBMSs and OODBMSs cannot serve XML data well. time and improve the system throughput. Data place- Semistructured data involve nested structures; there ment is a key factor on the overall performance of are much intricacy and redundancy when represent- parallel systems. XML is semistructured data, tradi- ing XML as relations. Data placement strategies in tional data placement strategies cannot serve it well. RDBMSs cannot, therefore, well solve the data place- In this paper, we present the concept of intermediary ment problem of XML data. Although graph partition node INode, and propose a novel workload-aware data algorithms in OODBMSs can be applied to tree parti- placement WIN to effectively decluster XML data, to tion problems, the key relationships between tree nodes obtain high intra query parallelism. The extensive ex- such as ancestor-descendant or sibling cannot be clearly periments show that the speedup and scaleup perfor- denoted on graph. Hence data placement strategies mance of WIN outperforms the previous strategies. for OODBMSs cannot well adapt to XML data either.
    [Show full text]
  • Tepzz¥ 67¥¥Za T
    (19) TZZ¥ ¥¥Z_T (11) EP 3 267 330 A1 (12) EUROPEAN PATENT APPLICATION (43) Date of publication: (51) Int Cl.: 10.01.2018 Bulletin 2018/02 G06F 17/30 (2006.01) (21) Application number: 16187202.3 (22) Date of filing: 05.09.2016 (84) Designated Contracting States: (72) Inventors: AL AT BE BG CH CY CZ DE DK EE ES FI FR GB • Abolhassani, Neda GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO Athens, GA Georgia 30605 (US) PL PT RO RS SE SI SK SM TR • Sheausan Tung, Teresa Designated Extension States: San Jose, CA California 95120 (US) BA ME • Gomadam, Karthik Designated Validation States: San Jose, CA California 95112 (US) MA MD (74) Representative: Müller-Boré & Partner (30) Priority: 07.07.2016 US 201662359547 P Patentanwälte PartG mbB 03.08.2016 US 201615227605 Friedenheimer Brücke 21 80639 München (DE) (71) Applicant: Accenture Global Solutions Limited Dublin 4 (IE) (54) QUERY REWRITING IN A RELATIONAL DATA HARMONIZATION FRAMEWORK (57) A query rewriting processor (processor) analyz- an SQL query). The processor may then pass the rela- es database semantic models (e.g., RDF knowledge tional database query to another system or process (e.g., graphs) that capture the interconnections (e.g., foreign a data virtualization layer) for execution against the indi- and primary key links to other tables) present in a rela- vidual relational databases. In this manner, the processor tional database. The processor generates an enriched automatically translates queries for information about the model query given an initial model query (e.g., a SPARQL relational database structure to a corresponding or query) against the semantic model.
    [Show full text]
  • Conjunctive Query Answering in the Description Logic EL Using a Relational Database System
    Conjunctive Query Answering in the Description Logic EL Using a Relational Database System Carsten Lutz - Universität Bremen, Germany David Toman - University of Waterloo, Canada FrankWolter - University of Liverpool, UK Presented By : Jasmeet Jagdev Background • One of the main application of ontologies is data access • Ontologies formalize conceptual information about the data • Calvanese et al. have argued, true scalability of CQ answering over DL ontologies can only be achieved by making use of RDBMSs • Not straight forward ! Background • RDBMSs are unaware of Tboxes (the DL mechanism for storing conceptual information) and adopt the closed-world semantics • In contrast, ABoxes (the DL mechanism for storing data) and the associated ontologies employ the open-world semantics Introduction • Approach for using RDBMSs for CQ answering over DL ontologies • We apply it to an extension of EL family of DLs • Widely used as ontology languages for large scale bio-medical ontologies such as SNOMED CT and NCI Introduction • EL allows – Concept Intersection – Existential restrictions dr • ELH⊥ = EL + bottom concept, role inclusions, and domain and range restrictions Main Idea • To incorporate the consequences of the TBox T into the relational instance corresponding to the given ABox A • To introduce the idea of combined first-order (FO) rewritability • It is possibly only if: – A and T can be written into an FO structure – q and T into FO query q* Main Idea • Properties of this approach: – It applies to DLs for which data complexity of CQ dr answering
    [Show full text]
  • Query Rewriting Using Monolingual Statistical Machine Translation
    Query Rewriting using Monolingual Statistical Machine Translation Stefan Riezler∗ Yi Liu∗∗ Google Google Long queries often suffer from low recall in web search due to conjunctive term matching. The chances of matching words in relevant documents can be increased by rewriting query terms into new terms with similar statistical properties. We present a comparison of approaches that deploy user query logs to learn rewrites of query terms into terms from the document space. We show that the best results are achieved by adopting the perspective of bridging the “lexical chasm” between queries and documents by translating from a source language of user queries into a target language of web documents. We train a state-of-the-art statistical machine translation (SMT) model on query-snippet pairs from user query logs, and extract expansion terms from the query rewrites produced by the monolingual translation system. We show in an extrinsic evaluation in a real-world web search task that the combination of a query-to-snippet translation model with a query language model achieves improved contextual query expansion compared to a state-of-the-art query expansion model that is trained on the same query log data. 1. Introduction Information Retrieval (IR) applications have been notoriously resistant to improvement attempts by Natural Language Processing (NLP). With a few exceptions for specialized tasks1, the contribution of part-of-speech taggers, syntactic parsers, or ontologies of nouns or verbs, has been inconclusive. In this paper, instead of deploying NLP tools or ontologies, we apply NLP ideas to IR problems. In particular, we take a viewpoint that looks at the problem of the word mismatch between queries and documents in web search as a problem of translating from a source language of user queries into a target language of web documents.
    [Show full text]
  • On Using an Automatic, Autonomous and Non-Intrusive Approach for Rewriting SQL Queries
    Simpósio Brasileiro de Banco de Dados - SBBD 2013 Short Papers On Using an Automatic, Autonomous and Non-Intrusive Approach for Rewriting SQL Queries Arlino H. M. de Araújo1;2, José Maria Monteiro2, José Antônio F. de Macêdo2, Júlio A. Tavares3, Angelo Brayner3, Sérgio Lifschitz4 1 Federal University of Piauí, Brazil 2 Federal University of Ceará, Brazil {arlino,monteiro,jose.macedo}@lia.ufc.br 3 University of Fortaleza, Brazil [email protected] 4 Pontifical Catholic University of Rio de Janeiro [email protected] Abstract. Database applications have become very complex, dealing with a huge volume of data and database objects. Concurrently, low query response time and high transaction throughput have emerged as mandatory requirements to be ensured by database management systems (DBMSs). Among other possible interventions regarding database performance, SQL query rewriting has been shown quite efficient. The idea is to rewrite a new SQL statement equivalent to the statement initially formulated, where the new SQL statement provides performance gains w.r.t. query response time. In this paper, we propose an automatic and non-intrusive approach for rewriting SQL queries. The proposed approach has been implemented and its behavior was evaluated in three different DBMSs using TPC-H benchmark. The results show that the proposed approach is quite efficient and can be effectively considered by database professionals. Categories and Subject Descriptors: H.2 [Database Management]: Miscellaneous Keywords: Query processing, Database tuning, Query rewriting 1. INTRODUCTION Database queries are expressed by means of high-level declarative languages, such as SQL (Structured Query Language), for instance. Such queries are submitted to the query engine, which is responsible for processing queries in database mangement systems (DBMSs).
    [Show full text]
  • Towards Practical Privacy-Preserving Data Analytics by Noah Michael
    Towards Practical Privacy-Preserving Data Analytics by Noah Michael Johnson A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Engineering - Electrical Engineering and Computer Sciences in the Graduate Division of the University of California, Berkeley Committee in charge: Professor Dawn Song, Chair Professor David Wagner Professor Deirdre Mulligan Fall 2018 Towards Practical Privacy-Preserving Data Analytics Copyright 2018 by Noah Michael Johnson 1 Abstract Towards Practical Privacy-Preserving Data Analytics by Noah Michael Johnson Doctor of Philosophy in Engineering - Electrical Engineering and Computer Sciences University of California, Berkeley Professor Dawn Song, Chair Organizations are increasingly collecting sensitive information about individuals. Extracting value from this data requires providing analysts with flexible access, typically in the form of data- bases that support SQL queries. Unfortunately, allowing access to data has been a major cause of privacy breaches. Traditional approaches for data security cannot protect privacy of individuals while providing flexible access for analytics. This presents a difficult trade-off. Overly restrictive policies result in underutilization and data siloing, while insufficient restrictions can lead to privacy breaches and data leaks. Differential privacy is widely recognized by experts as the most rigorous theoretical solution to this problem. Differential privacy provides a formal guarantee of privacy for individuals while allowing general statistical analysis of the data. Despite extensive academic research, differential privacy has not been widely adopted in practice. Additional work is needed to address performance and usability issues that arise when applying differential privacy to real-world environments. In this dissertation we develop empirical and theoretical advances towards practical differential privacy.
    [Show full text]
  • Data Management 09 Transaction Processing
    1 SCIENCE PASSION TECHNOLOGY Data Management 09 Transaction Processing Matthias Boehm, Arnab Phani Graz University of Technology, Austria Computer Science and Biomedical Engineering Institute of Interactive Systems and Data Science BMK endowed chair for Data Management Last update: Dec 06, 2020 2 Announcements/Org . #1 Video Recording . Link in TeachCenter & TUbe (lectures will be public) . Due to second lockdown: webex recording . #2 Exercise Submissions . Exercise 1: grading done . Exercise 2: in process of being grading . Exercise 3: published on Dec 01, submit by Dec 22 11:59pm . Issues dataset (Nov 19) / expected results (Nov 16, Nov 26) . Two approaches of handling actors . Incorrect expected results 5 extra points INF.01017UF Data Management / 706.010 Databases – 09 Transaction Processing Matthias Boehm, Graz University of Technology, WS 2020/21 3 Transaction (TX) Processing User 2 User 1 User 3 read/write TXs #1 Multiple users Correctness? DBS DBMS #2 Various failures Deadlocks (TX, system, media) Constraint Reliability? violations DBs Network Crash/power failure Disk failure failure . Goal: Basic Understanding of Transaction Processing . Transaction processing from user perspective . Locking and concurrency control to ensure #1 correctness . Logging and recovery to ensure #2 reliability INF.01017UF Data Management / 706.010 Databases – 09 Transaction Processing Matthias Boehm, Graz University of Technology, WS 2020/21 4 Agenda . Overview Transaction Processing . Locking and Concurrency Control . Logging and Recovery . Exercise
    [Show full text]
  • Probabilistic Query Rewriting for Efficient and Effective Keyword
    Probabilistic Query Rewriting for Efficient and Effective Keyword Search on Graph Data Lei Zhang Thanh Tran Achim Rettinger Karlsruhe Institute of Karlsruhe Institute of Karlsruhe Institute of Technology Technology Technology 76128 Karlsruhe, Germany 76128 Karlsruhe, Germany 76128 Karlsruhe, Germany [email protected] [email protected] [email protected] ABSTRACT connecting them, which represent final answers covering all The problem of rewriting keyword search queries on graph query keywords. The space of possible subgraphs is gener- data has been studied recently, where the main goal is to ally exponential in the number of query keywords. Through clean user queries by rewriting keywords as valid tokens ap- grouping keywords into larger meaningful units (called seg- pearing in the data and grouping them into meaningful seg- ments), the number of keywords to be processed and the ments. The main solution to this problem employs heuris- corresponding search space is reduced. tics for ranking query rewrites and a dynamic programming The two main tasks involved in query cleaning (henceforth algorithm for computing them. Based on a broader set of also called query rewriting) are token rewriting, where query queries defined by an existing benchmark, we show that the keywords are rewritten as tokens appearing in the data, and use of these heuristics does not yield good results. We pro- query segmentation, where tokens are grouped together as pose a novel probabilistic framework, which enables the op- segments representing compound keywords. Query rewrit- timality of a query rewrite to be estimated in a more princi- ing helps to improve not only the result quality but also the pled way.
    [Show full text]
  • First-Order Query Rewriting for Inconsistent Databases
    First-Order Query Rewriting for Inconsistent Databases Ariel D. Fuxman and Ren´ee J. Miller Department of Computer Science University of Toronto {afuxman, miller}@cs.toronto.edu Abstract. We consider the problem of retrieving consistent answers over databases that might be inconsistent with respect to some given integrity constraints. In particular, we concentrate on sets of constraints that consist of key dependencies. Most of the existing work has focused on identifying intractable cases of this problem. In contrast, in this paper we give an algorithm that computes the consistent answers for a large and practical class of conjunctive queries. Given a query q, the algorithm returns a first-order query Q (called a query rewriting) such that for ev- ery (potentially inconsistent) database I, the consistent answers for q can be obtained by evaluating Q directly on I. 1 Introduction Consistent query answering is the problem of retrieving “consistent” answers over databases that might be inconsistent with respect to some given integrity constraints. Applications that have motivated the study of this problem include data integration and data exchange. In data integration, the goal is to provide “a uniform interface to multiple autonomous data sources” [Hal01]. In data ex- change, “data structured under one (source) schema must be restructured and translated into an instance of a different (target) schema” [FKMP03]. In both contexts, it is often the case that the source data does not satisfy the integrity constraints of the global or target schema. The traditional approach to deal with this situation involves “cleaning” the source instance in order to remove data that violates the target constraints.
    [Show full text]
  • A Comparison of Query Rewriting Techniques for DL-Lite
    A Comparison of Query Rewriting Techniques for DL-Lite H´ector P´erez-Urbina, Boris Motik, and Ian Horrocks Oxford University Computing Laboratory Oxford, England {hector.perez-urbina,boris.motik,ian.horrocks}@comlab.ox.ac.uk 1 Introduction An incomplete database is defined by a set of constraints and a partial database instance [1]. Answering conjunctive queries over incomplete databases is an im- portant computational task that lies at the core of many problems, such as in- formation integration [12], data exchange [9], and data warehousing [17]. Given a query and an incomplete database, the task is to compute the so-called certain answers—the tuples that satisfy the query in every database instance that con- forms to the partial instance and satisfies the constraints [16]. It is well known that answering queries under general constraints is undecidable; therefore, the expressivity of the constraint languages considered is typically restricted in order to achieve decidability. Description Logics (DLs) [3] can be viewed as very expressive but decidable first-order fragments, which makes them natural candidates for constraint lan- guages. DLs are a family of knowledge representation formalisms that represent a given domain in terms of concepts (unary predicates), roles (binary predicates), and individuals (constants). A DL Knowledge Base (KB) K = hT , Ai consists of a terminological component T called the TBox, and an assertional compo- nent A called the ABox. In analogy to incomplete databases, the TBox can be seen as a conceptual schema containing the set of constraints, and the ABox as some partial instance of the schema. The use of DLs as constraint languages has already proven to be useful in a variety of scenarios such as ontology-based information integration [7] and the Semantic Web [10].
    [Show full text]
  • Polystore Query Rewriting: the Challenges of Variety
    Polystore Query Rewriting: The Challenges of Variety Yannis Papakonstantinou∗[email protected] Numerous databases marketed as SQL-on-Hadoop, FORWARD Middleware is a mediator. Figure 1 shows NewSQL [16] and NoSQL have emerged to catalyze Big an example of how the FORWARD middleware evaluates Data applications. These databases generally support the queries over different databases with varying capabilities. 3Vs [7]. (i) Volume: amount of data (ii) Velocity: speed An organization owns a PostgreSQL database and a Mon- of data in and out (iii) Variety: semi-structured and het- goDB database, where the PostgreSQL database contains a erogeneous data. As a result of differing use cases and de- sensors table and MongoDB contains a measurements ar- sign considerations around the Variety requirement, these ray of JSON objects. Conceptually, the FORWARD Middle- new databases have adopted semi-structured data models ware presents to its clients the virtual SQL++ views V1 and that vary among each other. Their query languages have V2 of these databases. (We describe the SQL-backwards- even more variations. Some variations are due to superficial compatible SQL++ below.) The sensors table of the Post- syntactic differences. Some variations arise from the data gres database is presented in the view V1 as a bag (bags are model differences. Other variations are genuine differences denoted by {{...}}) of JSON objects, since both tuples and in query capabilities. Yet another kind of variations involves JSON objects are sets of attribute/value pairs. Notice, the subtly different semantics for seemingly similar query func- virtual view V2 of MongoDB is identical to its native data tionalities.
    [Show full text]