On Improving User Response Times in Tableau Pawel Terlecki Fei Xu Marianne Shaw Tableau Software Tableau Software Tableau Software [email protected] [email protected] [email protected] Valeri Kim Richard Wesley Tableau Software Tableau Software [email protected] [email protected] ABSTRACT responsiveness of the interface. To achieve that across all the The rapid increase in data volumes and complexity of applied supported data sources, one often needs to apply dedicated analytical tasks poses a big challenge for visualization solutions. optimization techniques to alleviate efficiency and capability It is important to keep the experience highly interactive, so that shortcomings of various architectures. users stay engaged and can perform insightful data exploration. During data exploration, each user interaction with the application Query processing usually dominates the cost of visualization generates an adhoc query workload. This workload is difficult to generation. Therefore, in order to achieve acceptable response predict and requires powerful analytic databases, e.g. column times, one needs to utilize backend capabilities to the fullest and stores, to process requests with low latencies. Additionally, we apply techniques, such as caching or prefetching. In this paper we leverage caching and query optimization techniques to provide discuss key data processing components in Tableau: the query high levels of interactivity. processor, query caches, Tableau Data Engine [1, 2] and Data Sharing the results of data exploration via Tableau Server in the Server. Furthermore, we cover recent performance improvements intranet or Tableau Online, in the cloud, presents a different set of related to the number and quality of remote queries, broader reuse of workload challenges. Users can combine visualizations created cached data, and application of inter and intra query parallelism. during data exploration into dashboards and publish them to Server. A dashboard consists of multiple visualizations; they are 1. INTRODUCTION often linked by actions, such as filtering of one view based on Data visualization and big data processing have become key selection in another. elements of modern analytic platforms. Lower hardware prices The query workload generated by published dashboards is easier allow customers to purchase powerful computational clusters, to predict than the adhoc workload generated during exploratory collect high volumes of detailed data and perform sophisticated ad analysis. This is because queries are generated by interacting with hoc analysis. Moreover, there is a growing demand for schema components already defined in the dashboard. Moreover, standardized solutions in related areas, such as: data federation, caching is more efficient as it can be applied across multiple users exploration of semi-structured and graph data sets, and data accessing the same dashboards. preparation or knowledge discovery. The increasing popularity of Although with different emphasis, providing a highly interactive visual analytics is driving development in these fields and experience in both aforementioned scenarios depends upon the provides the research community with new data processing and efficient retrieval of new data from external sources and maximal usability challenges. utilization of previously obtained results. In this paper, we In the past years, Tableau has become a leader in providing easy- present recent performance improvements focused on increasing to-use data visualization tools. It currently connects to over 40 product interactivity, including: different data sources, from simple file-based sources, through SQL/MDX databases or web data sources, to Hadoop clusters. In Query batch preparation and concurrent query a common intuitive interface, it allows for rich analysis of execution to improve dashboard generation; different areas of data, including filtering, custom calculations – Persisted and distributed intelligent cache to improve potentially at different levels of detail, window and statistical response times across users and their interactions; functions and many others. Parallel plans and fast scanning of RLE encoded Tableau aims to provide a fast interactive environment for data columns in Tableau Data Engine; exploration. A key challenge, both today and as we continue to Managing data models and extracts in Data Server add more sophisticated data analysis, is maintaining high along with temporary table support. Permission to make digital or hard copies of all or part of this work for personal The text is organized accordingly. Section 2 covers the general or classroom use is granted without fee provided that copies are not made or background. In Sect. 3 we describe how dashboards are generated distributed for profit or commercial advantage and that copies bear this notice focusing on query processing and caching. Section 4 gives an and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is overview of the Tableau Data Engine and its recent performance permitted. To copy otherwise, or republish, to post on servers or to redistribute features. In Sect. 5 we look at connection and data model to lists, requires prior specific permission and/or a fee. Request permissions from management in the Data Server. Section 6 covers related work. [email protected]. Future plans are given in Sect. 7 and the paper is concluded in SIGMOD’15, May 31–June 4, 2015, Melbourne, Victoria, Australia. Sect. 8. Copyright © 2015 ACM 978-1-4503-2758-9/15/05…$15.00. http://dx.doi.org/10.1145/2723372.2742799 Figure 1: Sample dashboard for the FAA Flights On-Time dataset view it from different angles. Quality of experience depends on 2. BACKGROUND the degree of interactivity, a factor that distinguishes dashboards Tableau is an application for rapid interrogation of structured from old-fashioned static reports. data. The application has a simple, intuitive drag-and-drop A dashboard is a collection of zones organized according to a interface that generates queries against a wide spectrum of data certain layout. Different zone types include various charts, maps, sources. The application has a large number of built-in chart filters, calculated text, images etc. One defines the behavior of types. Users are offered a rich environment for analyzing their individual zones first and then specifies dependencies between data, including defining new calculations, forecasting, filtering, them. Complicated scenarios might involve tens or even hundreds and combining data from heterogeneous data sources [47]. of zones. User actions that add and remove items in a worksheet are Let us consider a simple example of a dashboard (Fig. 1) defined represented with VizQL [40], a set of specialized algebraic for the popular FAA Flights On-time dataset [43]. It covers all the operations. This representation is used to automatically generate flights in the US in the past decade. The two upper maps show the queries against a specified data source. Queries are issued and the number of flight origins and destination by state and, at the same results are interpreted to produce the requested visualization. As time, allow specifying origins and destinations for the slave charts the user interacts with elements of the visualization, additional at the bottom. Each chart is annotated with average delays and queries may be issued to the underlying database. flights per day. The bottom charts cover airlines operating the The Tableau Data Engine (TDE) is a proprietary column store flights, destination airports, breakdown of cancellations and database that can be used to allow users to extract data from an delays by weekdays, and distribution of arrival delays broken existing database or text file. When a TDE extract is incorporated down by hours of a day. The right-hand side has filtering, total into a user's workbook, a user can continue to interrogate their count of visible records and static legends. data when a network connection to the database is not available. Rendering a dashboard requires retrieving necessary data and Additionally, the extract reduces the query load on the backing post-processing it for visualization. Due to dependencies between database. Users can easily switch from an extract back to a live zones, the entire process might take several iterations in which connection, and extracts can be refreshed when appropriate. obtained responses are used to specify subsequent requests; the Tableau Server allows users to easily share their visualizations process is discussed in detail in Sect. 3.3. with others. Users publish data sources or visualizations to Server Most zones require data from one or more data sources in order to and specify sharing permissions for them. These visualizations be rendered. The requests are specified internally by queries with can be viewed and interacted with via another Tableau expressiveness matching the capabilities of the data sources. application, or simply through a Web browser. If visualizations These internal queries eventually get turned into textual queries in are published with accompanying TDE extracts, a schedule can be appropriate dialects and sent to the corresponding data sources. created to automatically refresh the extracts, ensuring the data is always current. In the next section we discuss efficient data retrieval and generation of high-quality remote queries as both vastly 3. QUERY PROCESSING FOR contribute to the latency