Database Benchmarking for Supporting Real-Time Interactive
Total Page:16
File Type:pdf, Size:1020Kb
Database Benchmarking for Supporting Real-Time Interactive Querying of Large Data Leilani Battle, Philipp Eichmann, Marco Angelini, Tiziana Catarci, Giuseppe Santucci, Yukun Zheng, Carsten Binnig, Jean-Daniel Fekete, Dominik Moritz To cite this version: Leilani Battle, Philipp Eichmann, Marco Angelini, Tiziana Catarci, Giuseppe Santucci, et al.. Database Benchmarking for Supporting Real-Time Interactive Querying of Large Data. SIGMOD ’20 - International Conference on Management of Data, Jun 2020, Portland, OR, United States. pp.1571-1587, 10.1145/3318464.3389732. hal-02556400 HAL Id: hal-02556400 https://hal.inria.fr/hal-02556400 Submitted on 28 Apr 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Database Benchmarking for Supporting Real-Time Interactive Querying of Large Data Leilani Battle Philipp Eichmann Marco Angelini University of Maryland∗ Brown University University of Rome “La Sapienza”∗∗ College Park, Maryland Providence, Rhode Island Rome, Italy [email protected] [email protected] [email protected] Tiziana Catarci Giuseppe Santucci Yukun Zheng University of Rome “La Sapienza”∗∗ University of Rome “La Sapienza”∗∗ University of Maryland∗ [email protected] [email protected] [email protected] Carsten Binnig Jean-Daniel Fekete Dominik Moritz Technical University of Darmstadt Inria, Univ. Paris-Saclay, CNRS University of Washington Darmstadt, Germany Orsay, France Seattle, Washington [email protected]. [email protected] [email protected] de ABSTRACT for interactive and real-time performance, the database and In this paper, we present a new benchmark to validate the visualization communities have developed a variety of tech- suitability of database systems for interactive visualization niques, including approximate query processing [2, 14], on- workloads. While there exist proposals for evaluating data- line aggregation/progressive visualization [3, 19, 26], data base systems on interactive data exploration workloads, none cubes [8, 36, 38], spatial indexing [63], speculative query rely on real user traces for database benchmarking. To this execution [8, 30], and lineage tracking [53]. end, our long term goal is to collect user traces that repre- However, we still lack adequate benchmarks to empirically sent workloads with different exploration characteristics. In assess which of these resulting systems provide satisfactory this paper, we present an initial benchmark that focuses on performance, and which systems are actually better than oth- “crossfilter”-style applications, which are a popular interac- ers for interactive, real-time querying scenarios. This issue tion type for data exploration and a particularly demanding is exacerbated for the most demanding yet popular visualiza- scenario for testing database system performance. We make tion scenarios such as crossfilter [41, 58, 67], where one in- our benchmark materials, including input datasets, inter- teraction may generate hundreds of queries per second, with action sequences, corresponding SQL queries, and analysis an expectation of near-immediate results. Unfortunately, ex- code, freely available as a community resource, to foster fur- isting database benchmarks such as TPC-H [65], TPC-DS ther research in this area: https://osf.io/9xerb/?view_only= [64], or the Star Schema Benchmark (SSB) [47] are insuffi- 81de1a3f99d04529b6b173a3bd5b4d23. cient for making these comparisons. One main reason is that the workloads modeled in these benchmarks are not rep- CCS CONCEPTS resentative of how database queries are generated through user interactions, such as with tools like Tableau [59] or • Information systems → Data management systems; Spotfire [58]. In contrast, visualization benchmarks provide Data analytics; • Human-centered computing → Visu- realistic scenarios, but lack the configurability, precision and alization systems and tools. automation afforded by database benchmarks when test- 1 INTRODUCTION ing performance [7]. Furthermore, there exists no evaluation Motivation. The data science process often begins with platform to test database systems under a range of interactive users (i.e., analysts, data scientists) exploring possibly mas- analysis conditions, such as different dataset characteristics, sive amounts of data through interactions with a graphical exploration scenarios, interaction paradigms, or user profiles. user interface, oftentimes a data visualization tool [4, 9, 12, Therefore, our community is still woefully unequipped to 66]. However, each time a user interacts with a visualization answer the question “are database systems truly capable of interface, the underlying data must be processed (filtered, supporting real-time interactive data exploration at scale?” aggregated, etc.) such that the interface can quickly provide a visual response [39, 42, 56]. To meet this growing demand L. Battle et al. other interaction designs (e.g., mouse hovers or check-box se- lection), crossfilter-style applications are arguably the most demanding use case for database systems [41]. With one single interaction, a user can generate hundreds of database queries per second when moving a single crossfilter range-slider. For example, Figure 1 shows information about airline delays; changing the selection of the flight distance us- ing the crossfilter range-slider of the histogram in the upper row (center), causes all other histograms to update at each mouse drag. Crossfilter interfaces are instances of dynamic query interfaces [57], which require live updates from the underlying database system as the slider moves in real time. Figure 1: An example crossfilter application from our In the following, we discuss the contributions of this paper: user study, visualizing the Flights dataset, composed As a first contribution, we present the results ofa of six numerical attributes. Each attribute is visual- user study conducted by the different international re- ized as a histogram that can be filtered using a range- search groups who authored this article. In this user slider. When the user interacts with a range slider, the study, we asked 22 users with a broad range of data sci- entire dataset is filtered, the bins in each histogram ence experience to perform four representative interactive are recomputed, and the visualizations are updated. analysis tasks using a crossfilter visualization tool, across three different datasets, producing a total of 128 different Here, real-time means with a latency < 100 ms, or support- traces for further study. Using the log data and video record- ing at least ten frames per second [15, 41, 42]. Unfortunately, ings collected from the study, we analyze and summarize our study reveals more work is needed from the database our findings regarding the interaction patterns observed, and community to answer positively. their effects on the resulting query workload. This characteri- Contributions. While there exist proposals [7, 18, 28, 29, zation analysis provides useful insights into how interaction 61] for evaluating database systems on interactive data explo- patterns can be taken into account when evaluating and ration workloads, none of these approaches derive a bench- optimizing queries on a database system. mark from real user traces, introducing potential artifacts As a second contribution, we present the results of and possibly missing key behavioral patterns. To validate our analysis of the 128 collected traces and character- the suitability of database systems for interactive and visual ize the behavior of typical users on interactive tasks. data exploration workloads, this paper thus presents a bench- Although continuous interactions, such as dragging a slider, mark constructed from real traces of users interacting with require a latency < 100 ms to provide continuous feedback, an information visualization system. In this scenario, every other events such as mouse clicks only require a 1 s latency, query corresponds to an actual interaction from a real user. allowing for pre-computation strategies. Based on our analy- Given the innumerable analysis scenarios supported by vi- sis, we elaborate on design implications for database systems sualization tools [23, 33], it would be unrealistic to have one to support highly interactive use cases (e.g., crossfilter). benchmark represent all possible use cases. Instead, we be- As a third contribution, we present a new benchmark lieve that a database benchmark ecosystem is required based on our user traces. As mentioned before this bench- for evaluating interactive data exploration workloads, mark aims to measure the ability of a DBMS to sustain similar to the TPC benchmarks for classical database work- the challenging workloads of interactive and real-time data loads (OLTP, OLAP, etc.). Our long term goal is to collect user exploration applications. Contrary to traditional database traces, representing workloads with different exploration benchmarks focused on returning exact results without la- characteristics. As a first step,