PROGRAMMING BY VOICE: A HANDS-FREE APPROACH FOR MOTORICALLY CHALLENGED CHILDREN by AMBER WAGNER DR. JEFF GRAY, COMMITTEE CHAIR MARCUS BROWN NICHOLAS KRAFT RANDY SMITH SUSAN VRBSKY A DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science in the Graduate School of The University of Alabama TUSCALOOSA, ALABAMA 2015 Copyright Amber Wagner 2015 ALL RIGHTS RESERVED ABSTRACT Computer Science (CS) educators frequently develop new methodologies, languages, and programming environments to teach novice programmers the fundamental concepts of CS. A recent trend has focused on new environments that reduce the initial challenges associated with the heavy syntax focus of textual programming languages. There are numerous Initial Programming Environments (IPEs) available that have been created for student use that in some cases have fostered self-discovery and inquiry-based exploration. In this dissertation, three IPEs are discussed: Scratch (2015), Lego Mindstorms (2015), and Blockly (2015). Although the block-based nature of IPEs can be helpful for learning concepts in CS, a small group of students (approximately 5%) is being left out from learning experiences and engagement in CS due to block-based environments’ dependence on the Windows Icon Mouse Pointer (WIMP) metaphor. Block-based environments often require the use of both a mouse and keyboard, which motorically challenged users often are unable to operate. Based on research performed and presented in this dissertation, a Vocal User Interface (VUI) is a viable solution that offers a “Programming by Voice” (PBV) capability (i.e., a capability to describe a program without using a keyboard or mouse). However, adapting legacy applications can be time consuming, particularly, if multiple applications (such as the three IPEs previously mentioned) require specialized VUIs. Each environment has its own visual layout and its own commands; therefore, each application requires a different VUI. In order to create a more generic solution, a Domain-Specific Language ii (DSL) can be applied to create a semi-automated process allowing a level of abstraction that captures the specific needs of each IPE. From the specification of each IPE, a customized VUI can be generated that integrates with the legacy application in a non-invasive manner. The nine chapters included in this dissertation were motivated by the following four research questions: 1. How can we improve initial programming instruction? 2. Can all children participate in programming instruction? 3. How do we implement PBV to allow children to take advantage of creative, block- based programming environments? 4. What are some potential ideas that can assist in generalizing the process of voice enabling IPEs? iii DEDICATION To my loving husband, Ray, and my adorable daughter, Izzy, thank you for your support and faith. To my mentor, Isabel, thank you for your guidance. Philippians 4:13 iv LIST OF ABBREVIATIONS AND SYMBOLS ACM Association for Computing Machinery AP Advanced Placement CDC Cemter for Disease Control CMU Carnegie Mellon University CS Computer Science CSCP Computer Science Collaboration Project CSTA Computer Science Teachers’ Association DART Daily Automated Regression Tester DO-IT Disabilities, Opportunities, Internetworking and Technology DSL Domain-Specific Language DTT Dialog Traversal Testing EATA Educational and Assistive Technology Application EFG Event-Flow Graph EIC Event Interaction Coverage EIG Event-Interaction Graph GPS Global Positioning System GUI Graphical User Interface HCI Human Computer Interaction IDE Integrated Development Environment v IPA International Phonetic Alphabet IPE Initial Programming Environment IVR Interactive Voice Response JSAPI Java Speech Application Program Interface LOC Lines of Code LOCAL Location and Context Aware Learning MEEC Minimal Effective Event Context MIT Massachusetts Institute of Technology MS Microsoft MVC Model View Controller NIH National Institute of Health OCR Optical Character Recognition PBV Programming by Voice RAPTOR Rapid Algorithmic Prototyping Tool for Ordered Reasoning SDK Software Development Kit SPEED for SPEech EDitor STEM Science, Technology, Engineering, and Mathematics TACCESS ACM Transactions on Accessible Computing TAIT Train and Assess Information Technology TCI Translator-Computer Interaction UA University of Alabama UAT User Acceptance Testing UCP United Cerebral Palsy of Greater Birmingham vi USF University of San Francisco VASDE Voice-Activated Syntax-Directed Editor VRT VUI Review Testing VUI Vocal User Interface WIMP Windows Icon Mouse Pointer WOZ Wizard of Oz vii ACKNOWLEDGMENTS First, I would like to acknowledge my husband, Raymond Wagner, without whom I would never have had the opportunity to complete this degree. His support throughout this process was incredible, and I am forever grateful for his faith in me. Dinner conversations were always interesting and extremely helpful to my work, and I am so thankful that he never hides his infinite love for me. I would like to thank Isabel Rosenblitt who inspired me from my CS beginnings in undergrad. She advised me in ways no one else could and was always willing to serve as a sounding board for my work. There is not a day that goes by that I do not think of her and miss her. She was a pioneer in the CS field and paved the way for young women like me to study CS and work in the field. Without the help of my parents, especially during this past year when my daughter was born, I never would have had the ability to finish. Their never ending support is always appreciated, and I am thankful they raised me to be the person I am today. I would like to thank Dr. Jeff Gray for all of his help and guidance throughout this process. It is with his encouragement and support that I am able to produce this research. Dr. Gray has been an incredible mentor throughout this entire process from advising me on courses to guiding my research, and the opportunities he provided me with over these past five years will never be forgotten. viii I would also like to thank my Committee members: Dr. Susan Vrbsky, Dr. Marcus Brown, Dr. Randy Smith, and Dr. Nicholas Kraft, who provided me with assistance during my studies from providing research material to lending an open ear during stressful times. It was an absolute pleasure to work with Dr. Gary Edwards at United Cerebral Palsy of Birmingham. He and his staff express their passion for working with children and adults with disabilities daily. It is by their example that this same passion has blossomed within me, and I hope to keep persons with disabilities at the center of my future research endeavors. This project was supported by NSF grant IIS-1117940, with previous support from a Google Research Award. ix CONTENTS ABSTRACT ............................................................................................................ ii DEDICATION ....................................................................................................... iv LISTS OF ABBREVIATIONS ............................................................................... v ACKNOWLEDGMENTS .................................................................................... viii LIST OF TABLES ............................................................................................... xiv LIST OF FIGURES ............................................................................................... xv LIST OF LISTINGS ........................................................................................... xviii 1. INTRODUCTION ............................................................................................... 1 1.1. CS Education on the Rise ........................................................................... 1 1.2 Why Initial Programming Environments are Used Heavily ........................ 4 1.3 Addressing Accessibility Needs of IPEs .................................................. 12 1.4 Dissertation Roadmap ............................................................................... 16 2. TECHNOLOGY EDUCATION AND SPECIAL NEEDS ............................... 18 2.1 Computers and People with Disabilities: Then and Now .......................... 19 2.2 Technology in the Classroom .................................................................... 21 2.3 Technology in the Special Needs Classroom ............................................ 26 2.4 Getting Involved ........................................................................................ 27 2.5 Conclusion ................................................................................................. 31 x 3. BACKGROUND RESEARCH ........................................................................ 33 3.1 Ability-Based Design ................................................................................ 34 3.2 Programming by Voice .............................................................................. 38 3.3 Vocal Limitations ...................................................................................... 43 3.4 Additional Input Modalities ...................................................................... 44 3.5 Conclusion ................................................................................................. 46 4. MYNA: HANDS-FREE PROGRAMMING .................................................... 47 4.1 Overview of Early Myna ..........................................................................