HackEd: A Pedagogical Analysis of Online Vulnerability Discovery Exercises Daniel Votipka Eric Zhang and Michelle L. Mazurek Tufts University University of Maryland
[email protected] [email protected],
[email protected] Abstract—Hacking exercises are a common tool for security exercises provide the most effective learning, and researchers education, but there is limited investigation of how they teach do not have a broad view of the landscape of current exercises. security concepts and whether they follow pedagogical best As a step toward expanding this analysis, we review online practices. This paper enumerates the pedagogical practices of 31 popular online hacking exercises. Specifically, we derive a set hacking exercises to address two main research questions: of pedagogical dimensions from the general learning sciences and • RQ1: Do currently available exercises apply general educational literature, tailored to hacking exercises, and review pedagogical principles suggested by the learning sciences whether and how each exercise implements each pedagogical di- literature? If so, how are these principles implemented? mension. In addition, we interview the organizers of 15 exercises to understand challenges and tradeoffs that may occur when • RQ2: How do exercise organizers consider which princi- choosing whether and how to implement each dimension. ples to implement? We found hacking exercises generally were tailored to students’ To answer these questions we performed an in-depth qual- prior security experience and support learning by limiting extra- itative review of 31 popular online hacking exercises (67% neous load and establishing helpful online communities. Con- versely, few exercises explicitly provide overarching conceptual of all online exercises we identified). As part of our analysis, structure or direct support for metacognition to help students we completed a sample of 313 unique challenges from these transfer learned knowledge to new contexts.