DOCSLIB.ORG

Increasing the Transparency of Research Papers with Explorable Multiverse Analyses Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, Fanny Chevalier

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Increasing the Transparency of Research Papers with Explorable Multiverse Analyses Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, Fanny Chevalier Increasing the Transparency of Research Papers with Explorable Multiverse Analyses Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, Fanny Chevalier To cite this version: Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, Fanny Chevalier. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. CHI 2019 - The ACM CHI Conference on Human Factors in Computing Systems, May 2019, Glasgow, United Kingdom. 10.1145/3290605.3300295. hal-01976951 HAL Id: hal-01976951 https://hal.inria.fr/hal-01976951 Submitted on 10 Jan 2019 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses Pierre Dragicevic Yvonne Jansen Abhraneel Sarma Inria CNRS – Sorbonne Université University of Michigan Orsay, France Paris, France Ann Arbor, MI, USA [email protected] [email protected] [email protected] Matthew Kay Fanny Chevalier University of Michigan University of Toronto Ann Arbor, MI, USA Toronto, Canada [email protected] [email protected] ABSTRACT ACM Reference Format: We present explorable multiverse analysis reports, a new ap- Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, proach to statistical reporting where readers of research and Fanny Chevalier. 2019. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. In CHI Conference on papers can explore alternative analysis options by interact- Human Factors in Computing Systems Proceedings (CHI 2019), May 4– ing with the paper itself. This approach draws from two 9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA, 15 pages. recent ideas: i) multiverse analysis, a philosophy of statistical https://doi.org/10.1145/3290605.3300295 reporting where paper authors report the outcomes of many different statistical analyses in order to show how fragile or robust their findings are; and ii) explorable explanations, 1 INTRODUCTION narratives that can be read as normal explanations but where The recent replication crisis in psychology and other disci- the reader can also become active by dynamically changing plines has dealt a blow to the credibility of human-subject re- some elements of the explanation. Based on five examples search and prompted a movement of methodological reform- and a design space analysis, we show how combining those [70]. Much of this movement calls for more transparency two ideas can complement existing reporting approaches and in the way statistics are reported, so that findings become constitute a step towards more transparent research papers. more trustworthy, more likely to be interpreted correctly, and easier to verify and replicate [29, 69, 72]. Concern for CCS CONCEPTS transparency in statistical reporting has spread to the HCI • Human-centered computing → Human computer in- community, which has published several articles [24, 31, 58] teraction (HCI). and hosted several workshops [56, 57, 96] on the topic. While much of the current discussions around transparent KEYWORDS statistics in HCI focus on how the community can improve Multiverse analysis, explorable explanation, statistics, trans- its practice, it has been suggested that HCI can do more than parent reporting, interactive documents. endorse and promote the transparent statistics movement— it can actively contribute to it by proposing novel user in- terfaces for better doing and better communicating statis- tics [31, 97]. In this article, we consider the research paper as Permission to make digital or hard copies of all or part of this work for a user interface, and seek to understand how we can enrich personal or classroom use is granted without fee provided that copies this user interface to better support and promote transparent are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights statistics reporting. for components of this work owned by others than the author(s) must While there are many ways a statistical report can lack be honored. Abstracting with credit is permitted. To copy otherwise, or transparency, a common and damaging form of opacity is republish, to post on servers or to redistribute to lists, requires prior specific undisclosed flexibility (see Figure 1a), i.e., not reporting the permission and/or a fee. Request permissions from [email protected]. different options that have been tried during the analysis [85, CHI 2019, May 4–9, 2019, Glasgow, Scotland UK 98], or the options that would have been chosen had the © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. data been different [40]. Undisclosed flexibility is damaging ACM ISBN 978-1-4503-5970-2/19/05...$15.00 because it substantially increases the chances of reporting https://doi.org/10.1145/3290605.3300295 erroneous findings, while being invisible to the reader. Analyzed Reported Analyzed Reported Analyzed Reported A second option consists of summarizing the multiverse in the research paper itself using tables and custom plots. This is the approach put forward by the two papers intro- ducing multiverse analysis [86, 87], which use as examples histograms of p-values [87], grids of p-values [87], and sorted a. traditional analysis b. planned analysis c. multiverse analysis dot-plots of point estimates where color encodes statistical significance [86]. Although such visualizations are very use- Figure 1: Three reporting strategies, from the least trans- parent to the most transparent: a) traditional analysis with ful as overviews, they force the paper author to collapse the undisclosed flexibility; b) planned analysis; c) multiverse results of each analysis into a single point estimate, p-value analysis. Each branching represents a choice between differ- or statistical significance verdict. In doing so, a lot ofthe ent analysis options [71]. richness and nuance of well-crafted statistical reports is lost. A third approach, which we propose here, consists of One response to the issue of undisclosed flexibility has making the results section of the research paper interac- been to encourage researchers to commit to a single sta- tive. More specifically, we introduce and explore the concept tistical analysis that has been planned [17, 31] and ideally of explorable multiverse analysis report (emar). An emar is registered [24] ahead of time (Figure 1b). Although plan- a multiverse analysis report that is presented in the form ning eliminates the problem of undisclosed flexibility, some of an explorable explanation in Bret Victor’s sense [94]. Ex- statisticians and methodologists are starting to argue that plorable explanations are explanations that “enable and en- more transparency can be achieved by letting researchers try courage truly active reading” and allow “the reader to play many analyses and report all of them in their paper [86, 87] with the author’s assumptions and analyses, and see the (Figure 1c). This is partly motivated by evidence that dif- consequences” [94]. At the same time, they can be read like ferent researchers who analyze the same data will make normal prose: “the reader is not forced to interact in order to different choices and will thus get slightly—and sometimes learn” [94]. Consistent with this design philosophy, an emar widely—different results [84]. looks like a regular results section, but readers are given In a multiverse analysis, researchers identify sets of de- the possibility to change some of the analysis options and fensible analysis choices (e.g., different ways of excluding immediately see the results within the research paper itself. outliers, different data transformations), implement them all, We posit that emars can be a useful and compelling com- and then report the outcomes of all analyses resulting from plement to existing multiverse reporting approaches. Our all possible choice combinations. This approach increases goal with the present paper is to provide a better under- transparency because readers can appreciate the “fragility standing of the design space of possible approaches. We will or robustness of a claimed effect” [86] by checking whether explore this design space through five examples of short the findings are dependent on arbitrary analysis choices. interactive papers1 we have written to demonstrate the con- Multiverse analysis promises an unprecedented level of cept of explorable multiverse analysis report. We will discuss transparency for research papers, but the idea is in its infancy. the trade-offs between different emar reporting strategies, Writing papers with a multiverse analysis are difficult and as well as the benefits and challenges raised by emars com- there is currently very little guidance. An important part of pared to alternative reporting strategies such as the use of the difficulty lies in reporting the outcomes of potentially supplemental material or static multiverse analysis reports. hundreds or thousands of analyses in a single research paper (past examples
Load more

Recommended publications
  • Increasing the Transparency of Research Papers with Explorable Multiverse Analyses
    Increasing the Transparency of Research Papers with Explorable Multiverse Analyses Pierre Dragicevic Yvonne Jansen Abhraneel Sarma Inria CNRS – Sorbonne Université University of Michigan Orsay, France Paris, France Ann Arbor, MI, USA [email protected] [email protected] [email protected] Matthew Kay Fanny Chevalier University of Michigan University of Toronto Ann Arbor, MI, USA Toronto, Canada [email protected] [email protected] ABSTRACT ACM Reference Format: We present explorable multiverse analysis reports, a new ap- Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, proach to statistical reporting where readers of research and Fanny Chevalier. 2019. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. In CHI Conference on papers can explore alternative analysis options by interact- Human Factors in Computing Systems Proceedings (CHI 2019), May 4– ing with the paper itself. This approach draws from two 9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA, 15 pages. recent ideas: i) multiverse analysis, a philosophy of statistical https://doi.org/10.1145/3290605.3300295 reporting where paper authors report the outcomes of many dierent statistical analyses in order to show how fragile or robust their ndings are; and ii) explorable explanations, 1 INTRODUCTION narratives that can be read as normal explanations but where The recent replication crisis in psychology and other disci- the reader can also become active by dynamically changing plines has dealt a blow to the credibility of human-subject re- some elements of the explanation. Based on ve examples search and prompted a movement of methodological reform- and a design space analysis, we show how combining those [70].
    [Show full text]
  • Increasing the Transparency of Research Papers with Explorable Multiverse Analyses
    CHI 2019 Paper CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK Increasing the Transparency of Research Papers with Explorable Multiverse Analyses Pierre Dragicevic Yvonne Jansen Abhraneel Sarma Inria CNRS – Sorbonne Université University of Michigan Orsay, France Paris, France Ann Arbor, MI, USA [email protected] [email protected] [email protected] Matthew Kay Fanny Chevalier University of Michigan University of Toronto Ann Arbor, MI, USA Toronto, Canada [email protected] [email protected] ABSTRACT ACM Reference Format: We present explorable multiverse analysis reports, a new ap- Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, proach to statistical reporting where readers of research and Fanny Chevalier. 2019. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. In CHI Conference on papers can explore alternative analysis options by interact- Human Factors in Computing Systems Proceedings (CHI 2019), May 4– ing with the paper itself. This approach draws from two 9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA, 15 pages. recent ideas: i) multiverse analysis, a philosophy of statistical https://doi.org/10.1145/3290605.3300295 reporting where paper authors report the outcomes of many different statistical analyses in order to show how fragile or robust their findings are; and ii) explorable explanations, 1 INTRODUCTION narratives that can be read as normal explanations but where The recent replication crisis in psychology and other disci- the reader can also become active by dynamically changing plines has dealt a blow to the credibility of human-subject re- some elements of the explanation. Based on five examples search and prompted a movement of methodological reform- and a design space analysis, we show how combining those [70].
    [Show full text]
  • 2018 Fellows
    2018 Undergraduate Fellows From Fall 2017 to Summer 2018, over 500 Stanford students engaged in immersive service opportunities around the world. See a map (http://bit.ly/cardinalquartermap2018) illustrating where all of our summer Cardinal Quarter participants served. The students listed below are supported by the Haas Center for PuBlic Service’s Undergraduate Fellowships Program. Advancing Gender Equity Fellows The Advancing Gender Equity Fellowship is a joint program with the Women’s Community Center and enaBles students to learn aBout gender, diversity, and social justice through a summer practicum with a nonprofit organization or government agency addressing social, political, or economic issues affecting women. • Mayahuel Ramirez, ‘20 (Psychology); Sunrise Children’s Foundation, Las Vegas, NV. Mayahuel was able to directly work with WIC staff in regards to providing low-income families’ access to nutritional and Breastfeeding information as needed. She spent the summer oBserving how WIC staff provide support, education, and empowerment to mothers of all Backgrounds. African Service Fellows The African Service Fellowship is a joint program with the Center for African Studies supporting students’ work on social and economic issues in Africa. • Asrat Alemu, ‘19 (Human Biology); NBA Academy Africa, Saly, Senegal. Asrat worked as a film coordinator and assisted the strength and conditioning coach and the player development coach while also doing several miscellaneous managerial tasks. • James Bicamumpaka, ‘21 (Undeclared); Sandra Lee Centre, Mbabane, Eswatini. James spent his Cardinal Quarter helping orphaned and aBandoned children attending under-resourced government schools overcome their academic Barriers, particularly with STEM subjects. He equally focused his efforts on advocating self-expression as Being therapeutic and an indication of personal strength, rather than the culturally Believed sign of weaknesses.
    [Show full text]
  • Prizes and Awards
    SAN DIEGO • JAN 10–13, 2018 January 2018 SAN DIEGO • JAN 10–13, 2018 Prizes and Awards 4:25 p.m., Thursday, January 11, 2018 66 PAGES | SPINE: 1/8" PROGRAM OPENING REMARKS Deanna Haunsperger, Mathematical Association of America GEORGE DAVID BIRKHOFF PRIZE IN APPLIED MATHEMATICS American Mathematical Society Society for Industrial and Applied Mathematics BERTRAND RUSSELL PRIZE OF THE AMS American Mathematical Society ULF GRENANDER PRIZE IN STOCHASTIC THEORY AND MODELING American Mathematical Society CHEVALLEY PRIZE IN LIE THEORY American Mathematical Society ALBERT LEON WHITEMAN MEMORIAL PRIZE American Mathematical Society FRANK NELSON COLE PRIZE IN ALGEBRA American Mathematical Society LEVI L. CONANT PRIZE American Mathematical Society AWARD FOR DISTINGUISHED PUBLIC SERVICE American Mathematical Society LEROY P. STEELE PRIZE FOR SEMINAL CONTRIBUTION TO RESEARCH American Mathematical Society LEROY P. STEELE PRIZE FOR MATHEMATICAL EXPOSITION American Mathematical Society LEROY P. STEELE PRIZE FOR LIFETIME ACHIEVEMENT American Mathematical Society SADOSKY RESEARCH PRIZE IN ANALYSIS Association for Women in Mathematics LOUISE HAY AWARD FOR CONTRIBUTION TO MATHEMATICS EDUCATION Association for Women in Mathematics M. GWENETH HUMPHREYS AWARD FOR MENTORSHIP OF UNDERGRADUATE WOMEN IN MATHEMATICS Association for Women in Mathematics MICROSOFT RESEARCH PRIZE IN ALGEBRA AND NUMBER THEORY Association for Women in Mathematics COMMUNICATIONS AWARD Joint Policy Board for Mathematics FRANK AND BRENNIE MORGAN PRIZE FOR OUTSTANDING RESEARCH IN MATHEMATICS BY AN UNDERGRADUATE STUDENT American Mathematical Society Mathematical Association of America Society for Industrial and Applied Mathematics BECKENBACH BOOK PRIZE Mathematical Association of America CHAUVENET PRIZE Mathematical Association of America EULER BOOK PRIZE Mathematical Association of America THE DEBORAH AND FRANKLIN TEPPER HAIMO AWARDS FOR DISTINGUISHED COLLEGE OR UNIVERSITY TEACHING OF MATHEMATICS Mathematical Association of America YUEH-GIN GUNG AND DR.CHARLES Y.
    [Show full text]
  • Many Tiny Things Explorable Explanations on Statistical Physics
    Many Tiny Things Explorable Explanations on Statistical Physics Master’s thesis in Complex Adaptive Systems PONTUS GRANSTRÖM Department of Physics CHALMERS UNIVERSITY OF TECHNOLOGY Gothenburg, Sweden 2016 Master’s thesis 2016 Many Tiny Things Explorable Explanations on Statistical Physics PONTUS GRANSTRÖM Department of Physics Chalmers University of Technology Gothenburg, Sweden 2016 Many Tiny Things Explorable Explanations on Statistical Physics Pontus Granström © Pontus Granström, 2016. Supervisor and Examiner: Göran Wahnström, Department of Physics Master’s Thesis 2016 Department of Physics Chalmers University of Technology SE-412 96 Gothenburg Telephone +46 31 772 1000 Cover: Screenshot of interactive simulations of a gas, a liquid and a solid. These simulations are featured in the explorable explanations described in the report. Typeset in LATEX Gothenburg, Sweden 2016 Many Tiny Things Explorable Explanations on Statistical Physics Pontus Granström Department of Physics Chalmers University of Technology Abstract This thesis report describes the development of Many Tiny Things, a series of ex- plorable explanations on basic statistical physics. The explanations guide the reader through a molecular dynamics model, which is simulated and visualised on the reader’s computer. The reader can interact with the simulated model, and explore concepts such as energy, temperature, friction, entropy and state transitions. The target audience is people with little physics or mathematics background, and the focus is on conveying an intuitive understanding, not a quantitative one. The goal is for the reader to play and experiment with the model, and hopefully learn something new about the world in the process. In the report I detail the design of the simulated model and the reader’s interaction with it, as well as the way the guiding explanations are presented.
    [Show full text]
  • Signature Redacted Jonathan Bobrow Program in Media Arts and Sciences May 6, 2016
    AutomaTiles Tangible Cellular Automata for Playful Engagement with Systems Thinking by Jonathan Bobrow B.A. University of California Los Angeles, 2008 Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial fulfillment of the requirements for the degree of Master of Science in Media Arts and Sciences at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2016 Massachusetts Institute of Technology 2016. All rights reserved. Signature of author: Signature redacted Jonathan Bobrow Program in Media Arts and Sciences May 6, 2016 /Y, Certified by: Signature redacted Kevin Slavin Assistant Professor of Media Arts and Sciences Thesis Supervisor Accepted by: Signature redacted Pattie Maes, Ph.D. MASSACHUSETTS INSTITUTE Academic Head OF TECHNOLOGY (0) LUJ Program in Media Arts and Sciences JUL 14 2016 LIBRARIES 2 AutomaTiles I Tangible Cellular Automata for Playful Engagement with Systems Thinking by Jonathan Bobrow Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, May 6, 2015 in Partial Fulfillment of the Requirements for the Degree of Master of Science in Media Arts and Sciences Abstract There is an increasingly vital awareness that our world is an aggregate of complex systems, emergent behavior, and system dynamics. The perceptual and analytical tools for exploring and studying these systems, however, have generally been relegated to scientists (whether mathematicians, physicists, biologists, economists, or computer scientists). Thus, as more and more people become aware of such systems, most people are still excluded from engaging with complex systems. By inventing a new tool and interface, consisting of playful objects called AutomaTiles, I propose a new approach for fostering a more aware society of systems thinkers.
    [Show full text]
  • University of California Santa Cruz
    UNIVERSITY OF CALIFORNIA SANTA CRUZ SEARCHING WITHIN AND ACROSS GAMES A dissertation submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in COMPUTATIONAL MEDIA by Barrett Anderson December 2020 The Dissertation of Barrett Anderson is approved: ________________________ Professor Noah Wardrip-Fruin, chair ________________________ Professor Adam M. Smith ________________________ Professor Sri Kurniawan ________________________ Henry Lowood, Ph.D. _____________________________ Quentin Williams Acting Vice Provost and Dean of Graduate Studies Copyright © by Barrett Anderson 2020 Table of Contents List of Figures xi List of Tables xvi Abstract xvii Chapter 1. Introduction 1 Contributions 6 Videogame Moments in STEAM Education 6 Videogame Moments and Information Retrieval 10 Research Questions 11 Research Paradigms 12 Chapter 2. Background 14 How Search Engines Work 14 Vector Space Model for Information Retrieval 18 Visualizing High-Dimensional Information 21 Quantitative Evaluation of Information Retrieval Systems 23 Videogame Search 25 The Game Metadata and Citation Project (GAMECIP) 26 Game Moments as Primary Sources 27 iii Chapter 3. Requirements Analysis for Videogame Moment Search 28 Project Summary 28 Research Questions 29 Related Work 31 HCI of IR 32 Indexing Moments in Games 33 Rich-media Search Engines 35 Interview Methods 36 Recruitment 36 Interview Protocol 39 Anchoring Demonstration 40 Analysis 41 Findings 43 Speedrunners 44 Speeedrunner Interview Subjects 44 What kinds of videogame moments
    [Show full text]
  • Increasing the Transparency of Research Paperswith Explorable
    Increasing the Transparency of Research Papers with Explorable Multiverse Analyses Pierre Dragicevic Yvonne Jansen Abhraneel Sarma Inria CNRS – Sorbonne Université University of Michigan Orsay, France Paris, France Ann Arbor, MI, USA [email protected] [email protected] [email protected] Matthew Kay Fanny Chevalier University of Michigan University of Toronto Ann Arbor, MI, USA Toronto, Canada [email protected] [email protected] ABSTRACT ACM Reference Format: We present explorable multiverse analysis reports, a new ap- Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, proach to statistical reporting where readers of research and Fanny Chevalier. 2019. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. In CHI Conference on papers can explore alternative analysis options by interact- Human Factors in Computing Systems Proceedings (CHI 2019), May 4– ing with the paper itself. This approach draws from two 9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA, 15 pages. recent ideas: i) multiverse analysis, a philosophy of statistical https://doi.org/10.1145/3290605.3300295 reporting where paper authors report the outcomes of many different statistical analyses in order to show how fragile or robust their findings are; and ii) explorable explanations, 1 INTRODUCTION narratives that can be read as normal explanations but where The recent replication crisis in psychology and other disci- the reader can also become active by dynamically changing plines has dealt a blow to the credibility of human-subject re- some elements of the explanation. Based on five examples search and prompted a movement of methodological reform- and a design space analysis, we show how combining those [70].
    [Show full text]
  • University of California Santa Cruz the Stabilization
    UNIVERSITY OF CALIFORNIA SANTA CRUZ THE STABILIZATION, EXPLORATION, AND EXPRESSION OF COMPUTER GAME HISTORY A dissertation submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in COMPUTER SCIENCE by Eric Kaltman September 2017 The Dissertation of Eric Kaltman is approved: Noah Wardrip-Fruin, Chair Michael Mateas Henry Lowood Tyrus Miller Vice Provost and Dean of Graduate Studies Copyright © by Eric Kaltman 2017 Table of Contents List of Figures vi List of Tables viii Abstract ix Dedication xi Acknowledgments xii 1 Introduction 1 1.1 On the history of technology . .4 1.2 On the history of software . 10 1.3 On the history of computer games in the history of software in the history of technology . 12 1.4 On preservation . 13 1.5 On knowledge accumulation, exploration and expression in the his- tory of technology . 15 1.6 On an intermediary perspective for the history of games as software 20 1.7 Stabilization . 25 1.8 Exploration . 27 1.9 Expression . 29 2 Appraisal 31 2.1 Compiling the Record . 31 2.2 Appraisal . 33 2.2.1 Related Work . 38 2.3 Prom Week .............................. 41 2.3.1 Choice of Prom Week ..................... 42 2.3.2 Process . 43 2.3.3 Context . 57 2.3.4 Documentary Enumeration . 62 iii 2.4 Conclusion . 79 3 Description 82 3.1 Introduction . 82 3.1.1 A Brief on Controlled Vocabularies . 84 3.1.2 A Course Through the Thicket . 86 3.2 Controlled Vocabularies . 90 3.2.1 A Brief Record Example . 92 3.2.2 Vocabulary and Ontology Best Practices .
    [Show full text]
  • Explorable Explanations
    Explorable Explanations What are they? What do they explain? How do we work with them? Let's find out. by Jesper Hyldahl Fogh May, 2018 Thesis-project – Interaction Design Master at K3 / Malmö University / Sweden Examiner: Maria Engberg Supervisor: Simon Niedenthal Time of examination: 28th of May, 13:00 Page 1 / 35 ​ TABLE OF CONTENTS TABLE OF CONTENTS 1 ABSTRACT 3 1 · INTRODUCTION 3 1.1 · Research Question 3 1.2 · Delimitation 3 1.3 · Structure 3 2 · THEORY 5 2.1 · Generic Design 5 Critiques 6 2.2 · Educational games theory 6 Educational games 7 Games research 7 Play, games, toys, and simulations 7 Mechanics, dynamics and aesthetics 8 Intro to relevant educational theory 9 The cognitive process dimension 9 The knowledge dimension 10 2.3 · What's next? 10 3 · ANALYSIS 10 3.1 · Explorable explanations 11 The categories of analysis 12 General examples 13 Simulating the World In Emojis 14 Introduction to A* 15 Notable examples 15 4D Toys 15 Pink trombone 16 Page 2 / 35 ​ Something Something Soup Something 16 Hooked 17 The Monty Hall Problem 17 Talking with God 17 Fake it to Make It 18 So what are explorable explanations then? 18 What's next? 19 4 · METHODOLOGY 19 4.1 · Sketching and prototyping 19 4.2 · Evaluation 19 5 · DESIGN PROCESS 20 5.1 · The design goals 20 5.2 · Neural networks 20 What are neural networks? 20 Why neural networks make sense as a subject 21 5.3 · The three iterations 21 Iteration 1 · The Visualized Network 21 Evaluating the iteration 22 Iteration 2 · The World's Dumbest Dog 22 Evaluating the iteration 23 Iteration 3 · A Tale of 70.000 Numbers 24 Evaluating the iteration 24 5.4 · A summary of the whole design process 25 6 · REFLECTION & FUTURE WORK 25 6.1 · Reflection 25 Method 25 6.2 · Future work 26 For designers 26 For researchers 27 7 · CONCLUSION 27 Page 3 / 35 ​ ACKNOWLEDGEMENTS 27 REFERENCES 28 APPENDICES 31 Page 4 / 35 ​ ABSTRACT In this paper, the author examines the concept of explorable explanations.
    [Show full text]
  • Data Theater: a Live Programming Environment for Prototyping Data-Driven Explorable Explanations Sam Lau Philip J
    Data Theater: A Live Programming Environment for Prototyping Data-Driven Explorable Explanations Sam Lau Philip J. Guo UC San Diego UC San Diego La Jolla, CA, USA La Jolla, CA, USA [email protected] [email protected] ABSTRACT In our experience with implementing explorables, we found Explorable explanations (a.k.a. ‘explorables’) enable readers that even simple explorables require writing hundreds of lines to learn concepts in domains such as math, physics, and the of JavaScript. However, the amount of code corresponding social sciences by interacting with live visualizations. Despite to their core logic is usually an order of magnitude smaller their popularity, there is currently a high barrier to creating than the total code. For instance, a representative statistics explorables since one must be adept at UI and visualization explorable within Seeing Theory [8] contains under 20 lines programming. To learn about these challenges, we interviewed of logic to simulate dice rolls intertwined within 200 lines to 6 educators who were interested in explorables but lacked the set up UI event listeners and visualize results using D3 [3]. skills to create them from scratch. These interviews gave us To surface opportunities to abstract away this boilerplate code, design insights to lower some of these implementation bar- we conducted formative interviews with six instructors of riers. We used these insights to create a live programming statistics and data science courses. We asked the instructors to system called Data Theater that enables programmers to pro- make pen-and-paper mockups of explorables based on their totype explorables by writing their simulation logic in Python lecture slides.
    [Show full text]
  • Ucsc-Soe-17-14
    GameSpace: An Explorable Visualization of the Videogame Medium JAMES RYAN, Expressive Intelligence Studio, University of California, Santa Cruz ERIC KALTMAN, Expressive Intelligence Studio, University of California, Santa Cruz ANDREW MAX FISHER, Games and Playable Media, University of California, Santa Cruz TAYLOR OWEN-MILNER, 2K Games, Novato, California MICHAEL MATEAS, Expressive Intelligence Studio, University of California, Santa Cruz NOAH WARDRIP-FRUIN, Expressive Intelligence Studio, University of California, Santa Cruz We present GameSpace, a visualization of the videogame medium as a 3D explorable space. GameSpace takes the form of a galaxy of 15,000 stars, where each star represents an existing game and stars are positioned such that related games are nearer to one another. Users may fly around the space, using conventional game controls, to explore the medium and serendipitously discover new games. This application is made possible by a combination of techniques from natural language processing and machine learning, namely latent semantic analysis and multidimensional scaling, and builds upon our earlier work in producing tools for videogame discovery. Beyond demonstrating an increase in user engagement relative to our primary existing tool, in this paper we outline our design goals, describe our development process, and discuss the project along various dimensions of intrigue. GameSpace is available online at http://gamespace.io. Additional Key Words and Phrases: visualization, game discovery, game genre, machine learning ACM Reference format: James Ryan, Eric Kaltman, Andrew Max Fisher, Taylor Owen-Milner, Michael Mateas, and Noah Wardrip-Fruin. 2017. GameSpace: An Explorable Visualization of the Videogame Medium. 1, 1, Article 1 (January 2017), 18 pages. DOI: 1 INTRODUCTION The Internet is replete with text about videogames.
    [Show full text]