Exploring the Role of Visualization and Engagement in Computer Science Education Report of the Working Group on ”Improving the Educational Impact of Algorithm Visualization” Thomas L. Naps (co-chair) Rudolf Fleischer Myles McNally U Wisconsin Oshkosh, USA Hong Kong U Sc. & Techn. Alma College, USA [email protected] [email protected] [email protected] Guido Roßling¨ (co-chair) Chris Hundhausen Susan Rodger Darmstadt U Techn., Germany U of Hawaii, USA Duke University, USA [email protected] [email protected] [email protected] Vicki Almstrum Ari Korhonen J. Ang´ el Velazquez-Iturbide´ U Texas Austin, USA Helsinki U Techn., Finland U Rey Juan Carlos, Spain [email protected] [email protected] [email protected] Wanda Dann Lauri Malmi Ithaca College, USA Helsinki U Techn., Finland [email protected] [email protected] Abstract meeting in person during the ITiCSE conference in Arhus,˚ Denmark. Throughout the remainder of this report, “we” Visualization technology can be used to graphically illus- refers to the Working Group, which was composed of the in- trate various concepts in computer science. We argue that dividuals listed at the beginning of the report. We also had such technology, no matter how well it is designed, is of three remote members, who are acknowledged in a section little educational value unless it engages learners in an ac- at the end of this report. tive learning activity. Drawing on a review of experimental studies of visualization effectiveness, we motivate this posi- The impetus for visualization in computing comes from the tion against the backdrop of current attitudes and best prac- inherent abstractness of the basic building blocks of the field. tices with respect to visualization use. We suggest a new Intuition suggests that, by making these building blocks taxonomy of learner engagement with visualization technol- more concrete, graphical representations would help one to ogy. Grounded in Bloom’s well-recognized taxonomy of un- better understand how they work. Visualization software derstanding, we suggest metrics for assessing the learning emerged in the late 1980’s for the purpose of creating and in- outcomes to which such engagement may lead. Based on teractively exploring graphical representations of computer these taxonomies of engagement and effectiveness metrics, science concepts [8, 54]. we present a framework for experimental studies of visu- Our recent surveys of computer science educators suggest alization effectiveness. Interested computer science educa- a widespread belief that visualization technology positively tors are invited to collaborate with us by carrying out studies impacts learning. However, experimental studies designed within this framework. to substantiate the educational effectiveness of such visu- alization technology simply do not bear this out [28]. On 1 Introduction top of this, a major deterrent to adopting visualization that emerged from our pre-conference survey is the time and ef- This report is the culmination of efforts by the Working fort required for instructors to integrate the technology into Group on Improving the Educational Impact of Algorithm their curricula. These findings point to two key obstacles to Visualization. The group, which was convened by Tom Naps visualization technology’s widespread adoption: and Guido Roßling,¨ began work during spring 2002. Using a groupware tool and a listserv mailing list, the group dis- cussed a number of issues, prepared and conducted an on- From the learner’s perspective, the visualization technol- line survey, and developed a rough draft of the report before ogy may not be educationally beneficial. From the instructor’s perspective, the visualization tech- Section 4 identifies metrics for assessing the learning out- nology may simply incur too much overhead to make it comes to which such engagement may lead. Based on our worthwhile. taxonomy of engagement and the effectiveness metrics we propose, we present a framework for empirical studies of al- Given the belief of computer science educators that visual- gorithm visualization effectiveness in Section 5. Section 6 ization technology, under the right conditions, can greatly concludes by discussing the future of this framework: stud- benefit learners and instructors alike, what can be done to ies we are planning, and opportunities for interested com- overcome these obstacles? With respect to educational ef- puter science educators to collaborate with us. fectiveness, it makes sense to look more closely at past ex- perimental studies of effectiveness. Indeed, closer inspection 2 Background reveals an important trend in those studies: that learners who are actively engaged with the visualization technology have Interactive visualization has been employed in computer sci- consistently outperformed learners who passively view visu- ence education since the 1980s (see, for example, [8, 10]). alizations [28]. For example, visualization technology has During that time, a set of “best practices” has evolved been successfully used to actively engage learners in such through instructors’ experiences with the technology. We activities as: summarize the most important of these in Section 2.1. To determine current practice and instructors’ attitudes toward the efficacy and impact of these practices, we recently con- Constructing their own input data sets [40, Chapter 9], ducted a survey of computing educators; we summarize the Making predictions regarding future visualization states key results in Section 2.2. In Section 2.3, we summarize the [11], widely mixed results of past experimental studies of visual- ization effectiveness and motivate the need for a new set of Programming the target algorithm [31], experiments. Answering strategic questions about the visualization [23, 2.1 Overview of Best Practices 44], Pedagogical visualization draws on many related disciplines, Constructing their own visualizations [26]. including typography, psychology, and algorithms. This makes it difficult to summarize the lessons learned, although Given this trend, it makes sense to design further experimen- some general recommendations about typography and lay- tal studies that more closely examine the educational bene- out apply. Khuri [35] summarizes recommendations on dis- fits of various forms of active engagement. In this report, we play layout, use of color and sound, and interactivity issues. present a framework for this type of empirical study. The While there is no agreed-upon standard for the “command- framework is flexible enough to allow such studies to take ments of algorithm animation” [18], the following eleven place both in controlled situations and in classrooms. Our points are commonly accepted suggestions drawn from ex- thesis is that visualization technology, no matter how well perience: it is designed, is of little educational value unless it engages learners in an active learning activity. If this is true, then the 1. Provide resources that help learners interpret the graph- key question to consider is what, if any, forms of active en- ical representation. As concrete representations, visual- gagement with visualization technology can have a positive izations may assist learners in understanding algorithms. impact on how much a learner learns. However, visualizations may also be difficult to interpret; The second major obstacle to visualization technology iden- learners may find it difficult to map a visualization to tified in our survey, instructor overhead, is not directly ad- the underlying algorithm it is designed to represent. The dressed in this report. However, if experimental results show meaning of the graphical representations and their rela- that some forms of active engagement with visualization lead tion to program elements can be clarified for learners in to very positive educational outcomes, we are hopeful that a one of two ways: explain the relationship by embedding variety of educators will begin the development of instruc- the representations in the system using text or narration, or tional materials that take advantage of those forms of en- reinforce the relationship by allocating instructional time gagement. to the topic during the course. Drawing on a review of experimental studies of visualization 2. Adapt to the knowledge level of the user. Novice learn- effectiveness and well-known best practices for algorithm vi- ers can become quickly overwhelmed by too many details sualization, Section 2 motivates our thesis against the back- or windows, and they usually prefer to test an animation drop of current attitudes and best practices with respect to with predefined input data. In contrast, advanced learn- algorithm visualization use. Section 3 presents a taxonomy ers may benefit from additional facilities for controlling of learner engagement with visualization technology, while complexity and for navigation, or from the capability to invent input data to more fully exercise algorithms. In ad- interface for visualization control mirrors a video player, dition, novices may more easily understand the structure with buttons for the following functions: stop, pause, one of animations that are based on well-known metaphors, step forward, continuous advance, advance to the end, one for example, comic strips [4], theater performances [21], step backwards and backtrack to the beginning [43]. electronic books [9] or slide presentations [43]. In con- trast, advanced learners may benefit from facilities for 7. Support learner-built visualizations. Stasko [55] advo- large data sets and