PROGRAM CONSTRUCTION EXAMPLES IN COMPUTER SCIENCE EDUCATION: FROM STATIC TEXT TO ADAPTIVE AND ENGAGING LEARNING TECHNOLOGY by Roya Hosseini B.S. in IT Engineering, Shiraz Univ. of Technology, 2009 M.S. in IT Engineering (E-Commerce), Amirkabir Univ. of Technology, 2012 M.S. in Intelligent Systems, Univ. of Pittsburgh, 2015 Submitted to the Graduate Faculty of the School of Computing and Information in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Pittsburgh 2018 UNIVERSITY OF PITTSBURGH SCHOOL OF COMPUTING AND INFORMATION This dissertation was presented by Roya Hosseini It was defended on July 24th 2018 and approved by Peter Brusilovsky, PhD, Professor Christian D. Schunn, PhD, Professor Diane Litman, PhD, Professor Vincent Aleven, PhD, Associate Professor Dissertation Director: Peter Brusilovsky, PhD, Professor ii PROGRAM CONSTRUCTION EXAMPLES IN COMPUTER SCIENCE EDUCATION: FROM STATIC TEXT TO ADAPTIVE AND ENGAGING LEARNING TECHNOLOGY Roya Hosseini, PhD University of Pittsburgh, 2018 My dissertation is situated in the field of computer science education research, specifi- cally, the learning and teaching of programming. This is a critical area to be studied, since, primarily, learning to program is difficult, but also, the need for programming knowledge and skills is growing, now more than ever. This research is particularly fo- cused on how to support a student’s acquisition of program construction skills through worked examples, one of the best practices for acquiring cognitive skills in STEM areas. While learning from examples is superior to problem-solving for novices, it is not recommended for intermediate learners with sufficient knowledge, who require more attention to problem-solving. Thus, it is critical for example-based learning environ- ments to adapt the amount and type of assistance given to the student’s needs. This important matter has only recently received attention in a few select STEM areas and is still unexplored in the programming domain. The learning technologies used in pro- gramming courses mostly focus on supporting student problem-solving activities and, with few exceptions, examples are mostly absent or presented in a static, non-engaging form. To fill existing gaps in the area of learning from programming examples, my disser- tation explores a new genre of worked examples that are both adaptive and engaging, to support students in the acquisition of program construction skills. My research ex- iii amines how to personalize the generation of examples and how to determine the best sequence of examples and problems, based on the student’s evolving level of knowledge. It also includes a series of studies created to assess the effectiveness of the proposed technologies and, more broadly, to investigate the role of worked examples in the process of acquiring programming skills. Results of our studies show the positive impact that examples have on student engagement, problem-solving, and learning. Adaptive technologies were also found to be beneficial: The adaptive generation of examples had a positive impact on learning and problem-solving performance. The adaptive sequencing of examples and problems engaged students more persistently in activities, resulting in some positive effects on learning. iv TABLE OF CONTENTS 1.0 INTRODUCTION ..............................1 1.1 MOTIVATION..............................1 1.2 OVERVIEW AND RESEARCH QUESTIONS.............3 1.2.1 Design and development of interactive program examples (PCEX)4 1.2.2 Adaptive fading in the PCEX examples.............5 1.2.3 Adaptive recommendation of PCEX examples and problems.5 1.3 DISSERTATION TASKS.........................6 1.4 CONTRIBUTIONS............................7 1.5 THESIS ORGANIZATION.......................8 2.0 BACKGROUND AND RELATED WORK .............. 10 2.1 WORKED EXAMPLES IN PROBLEM-SOLVING.......... 10 2.2 WORKED EXAMPLES IN PROGRAMMING............ 12 2.3 ADAPTIVE LEARNING TECHNOLOGIES.............. 15 2.4 PERSONALIZING WORK WITH PROGRAM EXAMPLES..... 17 2.4.1 Personalized access........................ 19 2.4.2 Problem-solving support..................... 22 2.4.3 Adaptive scaffolding....................... 22 2.5 PROGRAM CONSTRUCTION ASSESSMENT TOOLS....... 23 2.6 INTEGRATED SYSTEMS........................ 24 3.0 TOOL DESIGN ................................ 25 3.1 ITERATIVE DESIGN PROCESS.................... 25 v 3.2 TARGETED ASPECTS OF ENGAGEMENT............. 26 3.3 PCEX: CHARACTERISTICS AND DESIGN............. 27 4.0 AN OVERVIEW OF STUDIES ...................... 32 4.1 CLASSROOM AND USER STUDIES................. 32 4.1.1 Classroom Study 1........................ 33 4.1.2 Classroom Study 2........................ 33 4.1.3 User Study............................ 34 4.1.4 Classroom Study 3........................ 35 4.2 OVERVIEW OF COMMON METRICS AND INSTRUMENTS... 35 4.2.1 Engagement metrics....................... 35 4.2.2 Performance metrics....................... 36 4.2.3 Learning metrics......................... 36 4.2.4 Survey instruments........................ 36 4.2.4.1 Example evaluation survey............... 36 4.2.4.2 Recommendation evaluation survey........... 37 5.0 CLASSROOM STUDY 1: EXPLORATORY STUDY OF PCEX . 38 5.1 RESEARCH QUESTIONS........................ 38 5.2 STUDY DESIGN............................. 39 5.3 PRACTICE SYSTEM.......................... 41 5.4 COLLECTED DATA........................... 41 5.5 RESULTS................................. 44 5.5.1 Relationship between usage of PCEX and student’s learning. 44 5.5.1.1 Correlation between usage of PCEX and learning gain 44 5.5.1.2 Correlation between usage of PCEX and performance in coding exercises.................... 45 5.5.1.3 Correlation between usage of PCEX and course perfor- mance........................... 45 5.5.2 Correlation between usage of PCEX and student’s learning over time................................ 46 vi 5.5.2.1 Correlation analysis during the first and second half of the course........................ 46 5.5.2.2 Correlation analysis of the regular and exam prepara- tion usage......................... 47 5.6 SURVEY ANALYSIS........................... 49 5.7 SUMMARY AND DISCUSSION.................... 52 6.0 CLASSROOM STUDY 2: CONTROLLED STUDY OF PCEX .. 54 6.1 RESEARCH QUESTIONS........................ 54 6.2 CONTROL GROUP INTERFACE................... 55 6.3 THE STUDY............................... 56 6.3.1 Hypotheses............................ 57 6.3.2 Study design........................... 58 6.3.3 Study procedure......................... 58 6.3.4 Materials............................. 59 6.3.4.1 Practice Content..................... 59 6.3.4.2 Pre- and Post-Tests................... 59 6.3.5 Metrics.............................. 60 6.3.5.1 Engagement metrics................... 61 6.3.5.2 Performance metrics................... 61 6.3.5.3 Learning metrics..................... 63 6.4 RESULTS................................. 63 6.4.1 Students participation and collected data........... 63 6.4.2 Engagement analysis....................... 65 6.4.3 Performance analysis....................... 68 6.4.4 Learning analysis......................... 71 6.4.5 Survey analysis.......................... 74 6.5 SUMMARY AND DISCUSSION.................... 77 6.5.1 Summary............................. 77 6.5.1.1 Overall effects on engagement.............. 77 vii 6.5.1.2 Overall effects on problem-solving performance.... 77 6.5.1.3 Overall effects on learning outcomes.......... 78 6.5.1.4 Students’ feedback.................... 78 6.5.2 Discussion............................. 79 7.0 USER STUDY: CONTROLLED STUDY OF ADAPTIVE FADING 81 7.1 RESEARCH QUESTIONS........................ 81 7.2 ADAPTIVE FADING STRATEGY................... 82 7.3 BAYESIAN NETWORK STUDENT MODEL............. 83 7.4 THE STUDY............................... 84 7.4.1 Hypotheses............................ 84 7.4.2 Study design........................... 85 7.4.3 Participants and procedure.................... 85 7.4.4 Materials............................. 86 7.4.4.1 Practice content..................... 86 7.4.4.2 Pre- and post-tests.................... 87 7.4.5 Metrics.............................. 87 7.5 RESULTS................................. 89 7.5.1 Overall practice.......................... 90 7.5.2 Effects of adaptive fading on performance in practice problems 93 7.5.3 Effects of adaptive fading on performance in test problems.. 96 7.5.4 Effects of adaptive fading on learning.............. 100 7.6 SUMMARY AND DISCUSSION.................... 101 8.0 CLASSROOM STUDY 3: CONTROLLED STUDY OF ADAP- TIVE RECOMMENDATION ....................... 104 8.1 RESEARCH QUESTIONS........................ 104 8.2 ADAPTIVE RECOMMENDATION STRATEGY........... 105 8.2.1 Proactive recommendation.................... 105 8.2.2 Reactive recommendation.................... 111 8.3 THE STUDY............................... 113 viii 8.3.1 Hypotheses............................ 113 8.3.2 Study design........................... 113 8.3.3 Participants and procedure.................... 114 8.3.4 Materials............................. 115 8.3.4.1 Practice content..................... 115 8.3.4.2 Pre- and post-tests.................... 115 8.3.5 Metrics.............................. 115 8.4 RESULTS................................. 116 8.4.1 Overall engagement analysis................... 117 8.4.2 Persistence analysis........................ 120 8.4.3 Learning analysis......................... 123 8.4.3.1 Impact of the system on midterm score........ 123 8.4.3.2 Impact of the