Fits and Starts: Enterprise Use of Automland the Role Of
Total Page:16
File Type:pdf, Size:1020Kb
Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop Anamaria Crisan Brittany Fiore-Gartland Tableau Research Tableau Software [email protected] [email protected] Figure 1: Levels of Automation in Data Science Work. From our interviews we illustrate the desired level of automation accord- ing to level of technical expertise in data science. We ground our findings in the levels of automation proposed by Parasuraman et al. [41] and Lee et al. [34] ABSTRACT KEYWORDS AutoML systems can speed up routine data science work and make Data Science, Automation, Machine Learning, Data Scientists machine learning available to those without expertise in statistics and computer science. These systems have gained traction in en- ACM Reference Format: terprise settings where pools of skilled data workers are limited. In Anamaria Crisan and Brittany Fiore-Gartland. 2021. Fits and Starts: En- CHI this study, we conduct interviews with 29 individuals from orga- terprise Use of AutoML and the Role of Humans in the Loop. In ’21: ACM Conference on Human Factors in Computing Systems, May 08– nizations of different sizes to characterize how they currently use, 13, 2021, Yokohama, Japan. ACM, New York, NY, USA, 15 pages. https: or intend to use, AutoML systems in their data science work. Our //doi.org/10.1145/1122445.1122456 investigation also captures how data visualization is used in con- junction with AutoML systems. Our findings identify three usage scenarios for AutoML that resulted in a framework summarizing 1 INTRODUCTION the level of automation desired by data workers with different levels Organizations are flush with data but bereft of individuals with the of expertise. We surfaced the tension between speed and human technical expertise required to transform these data into actionable oversight and found that data visualization can do a poor job bal- insights [45]. To bridge this gap, organizations are increasingly ancing the two. Our findings have implications for the design and turning toward automation in data science work beginning with arXiv:2101.04296v1 [cs.HC] 12 Jan 2021 implementation of human-in-the-loop visual analytics approaches. the adoption of techniques that automate the creation of machine learning models [13, 46]. However, the adoption of this technology CCS CONCEPTS into enterprise settings has not been seamless. Currently AutoML • Human-centered computing ! Empirical studies in HCI. offerings have limitations in what they can flexibly support. End-to- end systems encompassing the full spectrum of data science work, from data preparation to communication, are not yet fully real- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed ized [34, 54]. Consequently, AutoML systems still require human in- for profit or commercial advantage and that copies bear this notice and the full citation tervention to be practically applicable [42, 46]. This mode of human- on the first page. Copyrights for components of this work owned by others than ACM machine collaboration presents a number of challenges [2, 35], chief must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a among them being the importance of balancing the speed afforded fee. Request permissions from [email protected]. by AutoML with the agency of individuals to interpret, correct, CHI ’21, May 08–13, 2021, Yokohama, Japan and refine automatically generated models and results [20]. Data © 2021 Association for Computing Machinery. ACM ISBN 978-1-4503-XXXX-X/18/06...$15.00 visualization can play an important role in facilitating this human- https://doi.org/10.1145/1122445.1122456 machine collaborative process [20, 46], but there are few studies CHI ’21, May 08–13, 2021, Yokohama, Japan Crisan and Fiore-Gartland, et al. that examine if and how data visualization is used in real-world set- 2.1 AutoML in Data Science tings together with AutoML. To fill this gap, we conduct interviews Data science leverages techniques from machine learning to derive with 29 individuals from organizations of different sizes and that new and potentially actionable insights from real-world data [3, 6, extend across different domains to capture how they currently, or 12]. AutoML systems have been developed to automate the com- plan to, use AutoML to carry out data science work. We examine putational work involved in building a data analysis pipeline that specifically if and how participants use data visualization asaway enable individuals to derive these insights from data. Several com- to integrate the human in the automation loop. mercial systems already exists and are used within different types of Our investigation reveals that the practical use of AutoML tech- organizations, including AWS SageMaker AutoPilot [24], Google’s nology in real world settings requires considerable human effort. Cloud AutoML [26], Microsoft’s AutomatedML [29], IBM’s Au- This effort is complicated by the need to trade-off data workbe- toAI [28], H20 Driverless AI [27], and Data Robot [25]. There are tween individuals with different expertise, for example data sci- also implementations of AutoML that build upon widely used data entists and business analysts. This trade-off is exacerbated bya science packages, such as the scikit-learn [43] python library, auto- data knowledge gap that participants believe AutoML technology sklearn [15, 16] and TPOT [38, 39]. The focus of these AutoML is widening. While participants saw the value of data visualization systems are toward largely supervised tasks concerning feature en- as one way to facilitate human-in-the-loop interactions with Au- gineering, hyper-parameter tuning, and model selection [14, 50, 54]. toML tools, many still reported using visualization in a limited way. Recent innovations have proposed possible end-to-end solutions Participants found that creating quality visualizations for AutoML that also support data preparation [34, 50, 54] and it is likely that was often too difficult and time consuming and had the effect of AutoML technologies will continue to expand toward broader end- slowing down automation often with limited benefit. Moreover, to-end support. participants reported a lack of useful visualization tools to support The means and extent to which AutoML systems integrate with them in some of their more pressing needs, such as collaborating on a computational data science pipeline is variable. Some AutoML data work among their diverse teams and with AutoML technology. systems exist as a single component within a larger pipeline, such Altogether our study makes the following timely contributions to as automated feature selection step, that the analyst or data scien- the existing literature on AutoML and the design of human-in-the- tist creates. At other times, AutoML systems can also create these loop tools for data science: pipelines with minimal user input. In their comprehensive analysis • An interview study that presents real world uses of AutoML of existing AutoML tools, Zöller et al. [54] describe three common technologies in enterprise settings with a focus on the role configurations for including AutoML in data science work. Thetwo of the human-in-the-loop facilitated by data visualization configurations are "fixed structure pipelines”, where the AutoML • A summary of three use cases for AutoML according to system assumes a very specific configuration of computational different organizational needs pipeline. The authors differentiate between fixed pipelines that • A framework that illustrates the level of automation that is are optimized for specific AutoML methods (for example, neural desirable for individuals with different levels of technical networks or random forests) compared to those that are not. While expertise. these fixed systems are common, they have limitations when con- As AutoML systems continue to gain traction in enterprise set- fronted with different data types and tasks. For example, image tings, our contributions will be a resource to the research commu- data or text data demand more flexibility within the structure of the nities developing human-in-the-loop approaches that support an computational pipeline. The second category is a “variable struc- appropriate balance of automation and human agency. ture pipeline”, which refers to a fairly recent approach that aims to learn the appropriate steps within a data science pipeline [54]. 2 RELATED WORK TPOT [38, 39] is an example of one of the first variable pipelines. Unlike fixed models that execute a pre-determined set of processes, We review prior work that investigates the use of AutoML in data variable structure approaches learn a network of process in response science, the ways that humans act within these processes, and to different datasets and user objectives. current data visualization approaches that mediate these processes. While the stated goal of many of these AutoML systems is to As we reviewed this work, we were challenged by the varied effectively remove humans from many aspects of data science use of the term ‘AutoML’. The preliminary goals of automation work [50], a view that data scientists themselves express [46], to- in machine learning began with the objective of removing the hu- day these systems still rely on considerable human labor to be man specifically from hyper-parameter tuning and model selection of use [19]. These limitations stem from both the complexity of steps [50]. However, it quickly became clear that other steps, such data science work and the brittleness of fixed structure pipelines as data preparation or feature engineering, were also critical to the that are in common use [54]. Our study catalogs this human labor success of hyper-parameter tuning. The scope of the term AutoML, across data science work and examines how visualization is used and more recently “AutoAI” or “driverless AI”, began to encompass by individuals engaged in data work.