BNL-113170-2016-JA Crowdsourcing quality control for Dark Energy Survey images P. Melchior, E. Sheldon, A. Drlica-Wagner, E. S. Ryko, T. M. C. Abbott, F. B. Abdalla, S. Allam, A. Benoit-L´evy, D. Brooks, E. Buckley-Geer, A. Carnero Rosell, M. Carrasco Kind, J. Carretero, M. Crocce, C. B. D’Andrea, L. N. da Costal, S. Desait, P. Doel, A. E. Evrard, D. A. Finley, B. Flaugher, J. Frieman, E. Gaztanaga, D. W. Gerdes, D. Gruen, R. A. Gruendl, K. Honscheid, D. J. James, M. Jarvisa, K. Kuehn, T. S. Lia, M. A. G. Maial, M. Marcha, J. L. Marshall, B. Nord, R. Ogando, A. A. Plazas, A. K. Romera, E. Sanchez, V. Scarpine, I. Sevilla-Noarbe, R. C. Smith, M. Soares-Santos, E. Suchyta, M. E. C. Swanson, G. Tarle, V. Vikram, A. R. Walker, W. Wester, Y. Zhang Submitted to Astronomy and Computing October 2016 Physics Department Brookhaven National Laboratory U.S. Department of Energy USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25) Notice: This manuscript has been authored by employees of Brookhaven Science Associates, LLC under Contract No. DE-SC0012704 with the U.S. Department of Energy. The publisher by accepting the manuscript for publication acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. DISCLAIMER This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, nor any of their contractors, subcontractors, or their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or any third party’s use or the results of such use of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof or its contractors or subcontractors. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. BNL-113170-2016-JA Crowdsourcing quality control for Dark Energy Survey images P. Melchiora,b, E. Sheldonc, A. Drlica-Wagnerd, E. S. Rykoffe,f, T. M. C. Abbottg, F. B. Abdallah,i, S. Allamd, A. Benoit-Levy´ j,h,k, D. Brooksh, E. Buckley-Geerd, A. Carnero Roselll,m, M. Carrasco Kindn,o, J. Carreterop,q, M. Croccep, C. B. D’Andrear,s, L. N. da Costal,m, S. Desait,u, P. Doelh, A. E. Evrardv,w, D. A. Finleyd, B. Flaugherd, J. Friemand,x, E. Gaztanagap, D. W. Gerdesw, D. Grueny,z, R. A. Gruendln,o, K. Honscheida,b, D. J. Jamesg, M. Jarvisaa, K. Kuehnab, T. S. Liac, M. A. G. Maial,m, M. Marchaa, J. L. Marshallac, B. Nordd, R. Ogandol,m, A. A. Plazasad, A. K. Romerae, E. Sanchezaf, V. Scarpined, I. Sevilla-Noarbeaf,n, R. C. Smithg, M. Soares-Santosd, E. Suchytaaa, M. E. C. Swansono, G. Tarlew, V. Vikramag, A. R. Walkerg, W. Westerd, Y. Zhangw aCenter for Cosmology and Astro-Particle Physics, The Ohio State University, Columbus, OH 43210, USA bDepartment of Physics, The Ohio State University, Columbus, OH 43210, USA cBrookhaven National Laboratory, Bldg 510, Upton, NY 11973, USA dFermi National Accelerator Laboratory, P. O. Box 500, Batavia, IL 60510, USA eKavli Institute for Particle Astrophysics & Cosmology, P. O. Box 2450, Stanford University, Stanford, CA 94305, USA fSLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA gCerro Tololo Inter-American Observatory, National Optical Astronomy Observatory, Casilla 603, La Serena, Chile hDepartment of Physics & Astronomy, University College London, Gower Street, London, WC1E 6BT, UK iDepartment of Physics and Electronics, Rhodes University, PO Box 94, Grahamstown, 6140, South Africa jCNRS, UMR 7095, Institut d’Astrophysique de Paris, F-75014, Paris, France kSorbonne Universit´es,UPMC Univ Paris 06, UMR 7095, Institut d’Astrophysique de Paris, F-75014, Paris, France lLaborat´orioInterinstitucional de e-Astronomia - LIneA, Rua Gal. Jos´eCristino 77, Rio de Janeiro, RJ - 20921-400, Brazil mObservat´orioNacional, Rua Gal. Jos´eCristino 77, Rio de Janeiro, RJ - 20921-400, Brazil nDepartment of Astronomy, University of Illinois, 1002 W. Green Street, Urbana, IL 61801, USA oNational Center for Supercomputing Applications, 1205 West Clark St., Urbana, IL 61801, USA pInstitut de Ci`enciesde l’Espai, IEEC-CSIC, Campus UAB, Carrer de Can Magrans, s/n, 08193 Bellaterra, Barcelona, Spain qInstitut de F´ısicad’Altes Energies (IFAE), The Barcelona Institute of Science and Technology, Campus UAB, 08193 Bellaterra (Barcelona) Spain rInstitute of Cosmology & Gravitation, University of Portsmouth, Portsmouth, PO1 3FX, UK sSchool of Physics and Astronomy, University of Southampton, Southampton, SO17 1BJ, UK tExcellence Cluster Universe, Boltzmannstr. 2, 85748 Garching, Germany uFaculty of Physics, Ludwig-Maximilians University, Scheinerstr. 1, 81679 Munich, Germany vDepartment of Astronomy, University of Michigan, Ann Arbor, MI 48109, USA wDepartment of Physics, University of Michigan, Ann Arbor, MI 48109, USA xKavli Institute for Cosmological Physics, University of Chicago, Chicago, IL 60637, USA yMax Planck Institute for Extraterrestrial Physics, Giessenbachstrasse, 85748 Garching, Germany zUniversit¨ats-Sternwarte, Fakult¨atf¨urPhysik, Ludwig-Maximilians Universit¨atM¨unchen, Scheinerstr. 1, 81679 M¨unchen, Germany aaDepartment of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA 19104, USA abAustralian Astronomical Observatory, North Ryde, NSW 2113, Australia acGeorge P. and Cynthia Woods Mitchell Institute for Fundamental Physics and Astronomy, and Department of Physics and Astronomy, Texas A&M University, College Station, TX 77843, USA adJet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Dr., Pasadena, CA 91109, USA aeDepartment of Physics and Astronomy, Pevensey Building, University of Sussex, Brighton, BN1 9QH, UK afCentro de Investigaciones Energ´eticas,Medioambientales y Tecnol´ogicas(CIEMAT), Madrid, Spain agArgonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USA Abstract We have developed a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the “DES exposure checker”, it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized oth- erwise. We conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a sufficiently large user base. We also release open-source code of the web application and host an online demo version at http://des-exp-checker.pmelchior.net. Keywords: surveys, Information systems: Crowdsourcing, Human-centered computing: Collaborative filtering 1. Introduction The application, dubbed the “DES exposure checker”, is geared for a professional user base of several hundred scien- Large astronomical surveys produce vast amounts of data tists and seeks to identify flaws in the DES images that will be for increasingly demanding science applications. At the same used for most science analyses. Problems discovered with this time, the complexity of the instruments, operations, and the application can then be fixed in hardware or in subsequent data subsequent data analyses renders glitches and flaws inevitable, processing runs. Our approach ties in with other quality control particularly during the early phases of experiments. Thus, mech- efforts, which automatically analyze the latest exposures and anisms that facilitate the discovery and reporting of problems in flag cases of e.g. bad observing conditions (Honscheid et al., the data, whether they originate from unexpected instrumen- 2012; Diehl et al., 2014), and allow the inspection of final coad- tal behavior or insufficient treatment in software, are impor- ded data products, both images and object catalogs (Balbinot tant components of a data quality program. Due to the often et al., 2012). Our concept profits from the experience of a prior unexpected nature of these features, algorithmic approaches to ad hoc crowdsourcing effort in DES. During the so-called Sci- identify artifacts are generally infeasible and human inspection ence Verification phase in 2012, flaws in DES imaging have remains necessary. Human-generated reports can then be fed been identified by a small “eyeball squad”, whose reports were back to algorithm developers and hardware experts to mitigate relayed to DES operations and data management experts on a and eliminate