EVALUATION OF EFFORTS TO EXPOSE MIDDLE SCHOOL STUDENTS TO COMPUTATIONAL THINKING: A REPORT ON THE COSMIC PROGRAM A Thesis by KARA ELISE BEASON Submitted to the School of Graduate Studies at Appalachian State University in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE December 2019 Department of Computer Science EVALUATION OF EFFORTS TO EXPOSE MIDDLE SCHOOL STUDENTS TO COMPUTATIONAL THINKING: A REPORT ON THE COSMIC PROGRAM A Thesis by KARA ELISE BEASON December 2019 APPROVED BY: __________________________________________ Dr. James B. Fenwick Jr. Chairperson, Thesis Committee ____________________________________________ Dr. Cindy Norris Member, Thesis Committee __________________________________________ Dr. Alice McRae Member, Thesis Committee __________________________________________ Dr. Rahman Tashakkori Chairperson, Department of Computer Science __________________________________________ Mike McKenzie, Ph.D. Dean, Cratis D. Williams School of Graduate Studies Copyright by Kara Elise Beason 2019 All Rights Reserved Abstract EVALUATION OF EFFORTS TO EXPOSE MIDDLE SCHOOL STUDENTS TO COMPUTATIONAL THINKING: A REPORT ON THE COSMIC PROGRAM Kara Elise Beason B.S., Florida State University M.S., Appalachian State University Chairperson: Dr. James B. Fenwick Jr. Computational thinking (CT) is a set of concepts and problem solving skills that are not only imperative for computer scientists, but important and applicable to nearly every discipline. In the past decade, many efforts have been made to develop and evaluate CT in primary and secondary students. This push for CT development in students seeks to prepare their problem-solving skills for a world where technology is ubiquitous, as well as to understand and mitigate the underrepresentation of women and minorities in STEM careers through exposure to computer science early on. COSMIC is one such effort that took place in Caldwell County middle schools from 2015 through 2017. The COSMIC program was created and supervised by researchers at Appalachian State University who supported teachers in hosting after school clubs and summer camps to teach students CT concepts through the use of the CS First curriculum and Scratch programming language. This thesis analyzes the impact of COSMIC using a mixed-mode approach of quantitative and qualitative data. The COSMIC effort was successful in its efforts to improve student awareness, knowledge, and skill of CT concepts, perspectives, and practices. iv Acknowledgements First, I would like to thank my advisor Dr. Jay Fenwick. His passion for education and inclusivity is inspirational, and the work he has done through the COSMIC program has touched the lives of so many. Without his constant support and encouragement, I probably would not have made it through my master’s degree. I would also like to thank the members of my thesis committee, Dr. Cindy Norris and Dr. Alice McRae. Thank you not only for believing in me, but for being two amazing role models for women like me in computer science. I would also like to thank the Caldwell county school teachers who are working every day to improve the lives of future generations. It was a pleasure to meet and work with and learn from so many passionate and caring individuals. I’d also like to thank all of the delightfully smart and kind middle school students I was fortunate enough to work with through COSMIC club. Without them this thesis would not have been possible. Next, I would like to thank Cassidy Baker, Ethan Fenwick, and Tyler Stein for the countless hours they spent doing data entry, as well as running Dr. Scratch on hundreds of projects. Their hard work and thorough analysis contributed greatly to the work in this thesis and made my life a whole lot easier. Finally, I would like to thank my friends and family, whose unending emotional support was vital throughout my master’s degree and the writing of this thesis. A master’s degree may technically be awarded to only one person, but it truly takes a village of support. I am infinitely grateful. v Table of Contents Abstract ................................................................................................................................................ iv Acknowledgements ............................................................................................................................... v List of Tables ..................................................................................................................................... viii List of Figures ...................................................................................................................................... ix Chapter 1- Introduction ......................................................................................................................... 1 Chapter 2 - Background ........................................................................................................................ 4 2.1 Computational Thinking ............................................................................................................. 4 2.2 Scratch ......................................................................................................................................... 7 2.3 CS First ..................................................................................................................................... 10 2.4 COSMIC Club Structure ........................................................................................................... 11 Chapter 3 - Related Work ................................................................................................................... 13 Chapter 4 - Data Collection Methodology and Tools ......................................................................... 16 4.1 CS First Data ............................................................................................................................. 16 4.2 Qualitative Data Collection ....................................................................................................... 19 4.2.1 Surveys ............................................................................................................................... 19 4.2.2 Interviews ........................................................................................................................... 20 4.3 Quantitative Scratch Project Analysis ....................................................................................... 21 vi 4.3.1 Scratch Project Structure .................................................................................................... 21 4.3.2 Hairball ............................................................................................................................... 22 4.3.3 Dr. Scratch.......................................................................................................................... 22 4.3.4 Other Tools......................................................................................................................... 27 4.4 Data Analysis Methodology ...................................................................................................... 28 4.4.1 Dr. Scratch Analysis ........................................................................................................... 29 4.4.2 Sentiment Analysis ............................................................................................................. 31 Chapter 5 - Results .............................................................................................................................. 33 5.1 Internal Evaluation Reports ....................................................................................................... 33 5.1.1 COSMIC Year 1 ................................................................................................................. 33 5.1.2 COSMIC Year 2 ................................................................................................................. 35 5.2 Quantitative Analysis ................................................................................................................ 36 5.3 Qualitative Analysis .................................................................................................................. 41 5.3.1 Survey Data ........................................................................................................................ 41 5.3.2 Interviews ........................................................................................................................... 44 Chapter 6 - Conclusions ...................................................................................................................... 48 Bibliography ........................................................................................................................................ 51 Vita ...................................................................................................................................................... 53 vii List of Tables Table 4.1 Dr. Scratch rubric. ............................................................................................................... 27 Table 4.2 A breakdown of the Dr. Scratch analysis for the “Game Design” theme’s activity example projects. ............................................................................................................................................... 29 Table 4.3 The Dr. Scratch breakdown of the provided