Digital Collaborator: Augmenting Task Abstraction in Visualization Design with Artificial Intelligence Aditeya Pandey* Andrea G. Parker Abstract Northeastern University Georgia Institute of Technology In the task abstraction phase of the visualization design pro- Boston, MA 02115, USA Atlanta, GA 30308, USA cess, including in “design studies”, a practitioner maps the [email protected] [email protected] observed domain goals to generalizable abstract tasks using visualization theory in order to better understand and address the user’s needs. We argue that this manual task abstraction Yixuan Zhang* Michelle A. Borkin process is prone to errors due to designer biases and a lack Northeastern University Northeastern University of domain background and knowledge. Under these circum- Boston, MA 02115, USA Boston, MA 02115, USA stances, a collaborator can help validate and provide sanity [email protected] [email protected] checks to visualization practitioners during this important task abstraction stage. However, having a human collaborator is not always feasible and may be subject to the same biases John A. Guerra-Gomez * Authors contributed equally and pitfalls. In this paper, we first describe the challenges as- arXiv:2003.01304v1 [cs.HC] 3 Mar 2020 Northeastern University sociated with task abstraction. We then propose a conceptual San Jose, CA 95138, USA Digital Collaborator—an artificial intelligence system that aims [email protected] to help visualization practitioners by augmenting their ability to validate and reason about the output of task abstraction. We also discuss several practical design challenges of designing and implementing such systems. Permission to make digital or Author Keywords hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and Information visualization; artificial intelligence; natural language that copies bear this notice and the full citation on the first page. Copyrights for third-party processing; task abstraction. components of this work must be honored. For all other uses, contact the owner/author(s). Copyright held by the owner/author(s). CHI’20,, April 25–30, 2020, Honolulu, HI, USA CCS Concepts ACM 978-1-4503-6819-3/20/04. •Human-centered computing ! Visualization; •Computing https://doi.org/10.1145/3334480.XXXXXXX methodologies ! Artificial intelligence; NLP; Introduction laborator (DC) that can serve as a feasible alternative to a Artificial intelligence (AI) has been used in the data information human collaborator. We envision that a DC can assist visual- Task-Abstraction Frame- community to help improve design of visualizations [5, 10]. A ization researchers by being up-to-date with task-abstraction works visualization practitioner can get help from a variety of tools frameworks to help identify the most appropriate framework for (e.g., Tableau, QlikView, SAS) [3] to select proper visual en- abstraction and can help validate the task analysis and abstrac- To highlight the variation in codings. However this step must be carefully considered in tion process to identify judgment errors or biases. With this task-abstraction method- the context of user goals and tasks. The visualization design paper we hope to open a discussion on the advantages and ologies, we present some process can be broadly divided into phases [6], including the challenges of building a DC for the visualization community. distinguishing characteristics design study process model [9], performing task analysis to of three common frameworks: understand domain problems, and task abstraction that aims to Challenges of Performing Task Abstraction A Multi-Level Typology of Ab- recast user goals from domain-specific languages to a gener- We first discuss the main challenges associated with the task stract Visualization Tasks [1]: A alized terminology for better understanding and readability [6]. abstraction process. generic task abstraction frame- Conducting task abstraction is an important but rigorous man- work that works well across ual process that requires in-depth understanding of domain A Wide Range of Task Abstraction Approaches: Visual- disciplines and data-set types. knowledge and familiarity with visualization literature [1,4, 11]. ization researchers have proposed various task abstraction For example, a biologist may be interested in results for tissue approaches (e.g., [1,4, 11]). On page2 (side-bar), we discuss Task Taxonomy for Graph samples treated with LL-37 matching up with the ones without three common visualization task abstraction frameworks and Visualization [4]: A descriptive the peptide. A visualization researcher may translate this task explain how they differ. Adopting an appropriate task abstrac- framework for tasks in the field to compare values between two groups [6]. However, to accu- tion approach is pivotal for visualization design as it impacts specific to graph visualizations. rately perform task abstraction, a visualization practitioner must the choice of visualization design and interaction idioms. How- This taxonomy provides more first choose the best abstraction framework and then the appro- ever, selecting a proper task abstraction framework requires an descriptive identification of priate abstraction. A practitioner has to keep up with the ever- extensive comparison of existing literature. visualization goals than a growing task abstraction literature [1,4, 11] and ensure that generalized framework [8]. Interpretation of a Task Abstraction Framework: Task ab- their personal biases that might come from previous work ex- straction is a subjective evaluation of the domain experts’ Hierarchical Task Abstrac- periences do not affect their ability to perform task abstraction. needs. Subjective assessments are prone to errors arising tion (HTA) [11]: HTA highlights from variability in the practitioner’s understanding of an abstrac- the importance of integrating As task abstraction is a manual and subjective phase of the tion framework or an innate bias such as recency bias where context and leverages existing visualization design process, we argue that it may be prone to the task-abstraction may be influenced by recent work. Such task abstraction frameworks in human-judgment errors. For example, domain experts often abstraction biases can lead to a “domino” effect of errors that combination with a systematic serve as project collaborators to help visualization researchers can only be objectively verified after prototyping [9]. Addition- analysis of user tasks, goals, and practitioners validate the task analysis and abstraction in ally, the analytic-task focused taxonomies require mastery of and processes. human-centered studies. However, it is challenging to have collaborators’ involvement in many situations. Furthermore, the terminology and definitions [4]. For example, in network human collaborators are still prone to pitfalls like keeping pace abstraction, it is common to use the term Topology for prop- with recent development of task-abstraction theories and prim- erties related to the structure of the network. Topology is a ing biases. Therefore, we propose an AI-enabled Digital Col- mathematical term, and practitioners coming from design back- to select the “right” framework. Therefore, the first challenge A Task Abstraction Process Example: grounds may be unfamiliar with its meaning. of building a DC will be to develop parameters to distinguish Diabetes Management between these abstraction frameworks. Automate Task Abstraction using AI We have discussed some challenges of performing task ab- Training Data: For automating the task-abstraction process, straction with human effort involved. Drawing inspiration from we need to train machine learning models with task data and the idiom of an Intelligent Personal Assistant (IPA) [2], we pro- their labeled outputs. One way to acquire training data is by pose a Digital Collaborator (DC)—a conceptual AI-enabled parsing domain goals and their abstractions from existing lit- system to support task abstraction for visualization research. erature. Smart data crawling tools may facilitate the process Figure1 shows an example of how AI can be used to facilitate of extracting tasks from research papers with little manual ef- task abstraction. fort. However, even after deploying web-crawlers, there might D a y 5 be problems with data quality. For instance, there might be Input and Output: Similar to the IPA system, our proposed conflicting abstractions where similar tasks have different ab- Visualization Designers’ Roles DC will adopt a question-and-answer-based interface. The Identify tasks through stractions. To counter the problem, we can think of human-in- observations, interviews, and questions (input) will be domain goals identified by visualiza- the-loop methodology where visualization researchers working field studies tion practitioners through interviews and observations with on the project can address quality issues. An Example Task/ Goal (Input): domain experts. The DC should generate a translation of a “How many days did the patient domain goal to a generalized task description by applying an Recommendation Validation: Task abstraction involves sub- have only high blood glucose levels jective evaluation and characterization of domain problems. within the 2 hours BEFORE lunch?” appropriate task-abstraction framework (output). To improve communication transparency, the DC should aim to