Learning Obstacles in the Capture The Flag Model Kevin Chung, NYU Polytechnic School of Engineering Julian Cohen, NYU Polytechnic School of Engineering Abstract game is an IRC channel in which competitors discuss Capture The Flag (CTF) competitions have been used in the competition and interact with the. the computer security community for education and evaluation objectives for over a decade. These competi- At its core, the CTF is a type of “open-book test” re- tions are often regarded as excellent approaches to learn volving around computer security. This lends itself to deeply technical concepts in a fun, non-traditional learn- testing competitors on very obscure types of knowledge. ing environment, but there are many difficulties associ- The technical knowledge that is communicated during a ated with developing and competing in a CTF event that CTF is often only mentioned in passing in documenta- are rarely discussed that counteract these benefits. CTF tion or a classroom. For example, the Python documen- competitions often have issues related to participation, tation briefly, and without much discussion, mentions quality assurance, and confusing challenges. These that the pickle module is “not intended to be secure problems affect the overall quality of a CTF competi- against erroneous or maliciously constructed data.” This tion and describe how effective they are at catalyzing ambiguity was leveraged in the 2012 Zombie Reminder learning and assessing skill. In this paper, we present (Hack.lu CTF) and Minesweeper (29c3 CTF) to create insights and lessons learned from organizing CSAW challenges that involved un-pickling untrusted data. The CTF, one of the largest and most successful CTFs. Zombie Reminder challenge from Hack.lu asked teams to break a simple hash to load arbitrary pickled data 1. Introduction to Capture The Flag while the Minesweeper challenge asked competitors to Capture The Flag (CTF) competitions have been used in break an XOR cipher to load arbitrary pickled data (See the security community as tests of offensive security Figure 1) [7]. These challenges serve as examples of skill since at least 1996. The first DEFCON CTF was in how CTF competitions can introduce and evaluate secu- 1999 and many other CTF competitions have spawned rity concepts that are not often covered in traditional since then, including CSAW CTF, PlaidCTF, PicoCTF, academic settings, or in documentation. UCSB iCTF, Ghost in the Shellcode, Codegate, Boston def load(self, data): Key Party, 30c3, and RuCTF amongst others. There are self.__dict__ = pickle.loads(data) now more than 70 annual CTF competitions tracked by def save(self): CTF ranking site ctftime.org. Each of these competi- return pickle.dumps(self.__dict__, 1) tions seeks to realize different goals but at its core, CTF Vulnerable source code from the 29c3 Minesweeper has always been about education and skill evaluation challenge [3][4][5][6]. CTF competitions aim to bring security community members from all skill levels together to msg = "cos\nsystem\n(S'/bin/sh'\ntR.'\ntR." msg = pickle.dumps(Exploit()) share information and to see who among them can solve h = hashlib.sha1() the most challenges. The community has a huge appre- h.update(msg) msg = "4n71cH3aT" + h.digest() + msg ciation for CTFs as a learning platform, but very few tmp = "" members have directly addressed the pitfalls and obsta- Figure 1: Snippet of exploit for the 29c3 Minesweep- cles associated with developing and participating in er challenge. CTF competitions. Due to the unstructured nature of CTFs, students typi- For reference, CTFs are commonly structured with a cally learn individually and must directly apply the group of judges/challenge developers led by a competi- knowledge they’ve gained to succeed in a CTF competi- tion organizer. Judges create challenges that are then tion. A common and important aspect of CTF is that it given a point value and placed online to be played by is often not possible to earn “partial credit” for a chal- the public. Challenges are made available on a game lenge: the challenge is either solved or not, resulting in board that lists the challenges, their point values, de- either all or none of the points. By presenting challeng- scriptions, and solve counts. Alongside the challenges, es in this way, competitors are forced to continually there is typically a scoreboard that lists teams’ points modify their approach until they succeed, providing and comparative ranking. Also typically included in a multiple opportunities to learn about a specific technol- ogy. This continuous development and learning not only lar to how CodeAcademy rates its problem sets. This allows challenge developers to use CTF as a platform to allows teams to strategize; self-selecting challenges that teach competitors about a particular area of security, but they believe they can solve immediately or with some also allows organizers to identify competitors with ex- effort. CCDC doesn’t share this property as problems tremely driven personalities. Even if a driven individual are handled as they come instead of giving the players is not technically competent enough to score well in a the choice of which problems to solve. We believe, that CTF, we believe that their tenacity would resolve that the CCDC approach is more inline with the “real problem given enough effort and study. It is our belief world,” but for learning purposes, allowing competitors that students who are willing to immerse themselves in to choose their learning is preferred. the studying and culture that’s associated with CTF will eventually succeed and build worthwhile habits for fur- 2. Problems in the Capture The Flag Model thering their own knowledge. Despite the claimed success of CTFs as both an educa- tional and skill evaluation platform, there are still prob- 1.1. Learning in Capture The Flag lems that hamper the success of the model. One of the CTF is not the only alternative means of competitive most important problems seen by us, as organizers, is security education currently in use. Other examples of the difficultly for a newcomer to start playing and be- “gamified” learning include the Collegiate Cyber De- come immediately immersed in the competition. In our fense Competition (CCDC) and CodeAcademy. experience, potential competitors are introduced to CTFs by meeting a security professional (or fellow stu- CCDC is one such example where a student team (blue dent who tells them about an upcoming event, and that team) must exclusively defend their network against a they should participate in to get more exposure to the team of professional attackers (red team). CCDC’s main security industry. However, it becomes difficult to con- focus is communicating the administrative aspects of tinue playing CTFs if the competitors are unable to defending a fictional corporation from attack, leaving solve problems or independently further themselves in aside many of the technical aspects of computer securi- some way. Competitors that cannot solve any challenges ty. Additionally, CCDC fails to properly model the situ- will quickly get discouraged and stop playing. However, ations that it aims to [11]. The inherent imbalance of we believe CTF competitions have the potential to pro- power between the red and blue teams result in a dispar- vide a scaffolding to help build player confidence, and a ity between the benefits enjoyed by each. For example, reward system to encourage player engagement. Here the notoriety associated with being a part of the red we discuss some of these deficiencies, as well as how team is far more desirable than being part of the blue CSAW has attempted to overcome them. team. 2.1. Difficulties in Playing In comparison, the CTF model focuses entirely on self- The qualities that make games easily playable (e.g. education by providing teams the technical problems those that are self-describing, have low skill pre- that they must solve. Instead of worrying about adminis- requisites, and are language neutral) are often not found trative tasks, the competitors focus entirely on the prob- in many CTF, meaning CTF is not easy to immediately lem. Similarly important, notoriety is gained based on pick up and play. Other games, like Checkers and your success in the CTF, not by being chosen to be on a GoFish, have very low initial barriers to entry and, as a team. result, are popular worldwide. A Checkers player does not need to research the game and understand fine tech- CTFs also provide a truer game-like experience, and in nical details in order to win a game. The same does not particular, provide incremental feedback. The process hold true for CTF where deep knowledge of technology of solving a challenge, submitting the answer, and assists greatly with participation and success. In this knowing whether the answer is correct or not provides way, CTF is more like Chess, where even if the basic competitors feedback on where they stand in the game rules are straightforward, they may take many years of and where they stand on their current problem. We be- experience to master. lieve these incremental rewards help motivate competi- tors and keep them engaged throughout the competi- Unfortunately, many CTFs are not designed with the tion’s duration. This is in contrast to CCDC, which only beginner in mind, relying on heavy technical require- provides feedback at the end of the competition. ments that lead to beginners quickly becoming stuck and, ultimately, giving up. To a newcomer, the first step In addition to feedback, CTFs often have a difficulty is often the hardest. The motions that become second estimation associate with a challenge giving some indi- nature to a CTF veteran are not always easily under- cation of how difficult a challenge will be. This is simi- stood or accessible. Specifically, CTF challenges often revolve around complex, highly technical skills, such as of the Imgception (GitS 2014) asked competitors to byte code de-compilation (DotNet CSAW CTF 2013), forensically analyze images in order to find a hidden machine code disassemblers (Impossible CSAW CTF key.