ISSN 1822-7732 INTERNATIONAL OLYMPIAD IN INFORMATICS INSTITUTE OF MATHEMATICS AND INFORMATICS INTERNATIONAL FEDERATION FOR INFORMATION PROCESSING OLYMPIADS IN INFORMATICS Volume 3 2009 Selected papers of the International Conference joint with the XXI International Olympiad in Informatics Plovdiv, Bulgaria, August 8–15, 2009 OLYMPIADS IN INFORMATICS ISSN 1822-7732 Editor-in-Chief Valentina Dagiene˙ Institute of Mathematics and Informatics, Lithuania, [email protected] Executive Editor Richard Forster British Informatics Olympiad, UK, [email protected] International Editorial Board Gerald Futschek, Vienna University of Technology, Austria, [email protected] Bruria Haberman, Holon Institute of Technology, Israel, [email protected] Marcin Kubica, Warsaw University, Poland, [email protected] Ville Leppänen, University of Turku, Finland, [email protected].fi Krassimir Manev, Sofia University, Bulgaria, [email protected]fia.bg Fredrik Niemelä, KTH University, Sweden, [email protected] Rein Prank, University of Tartu, Estonia, [email protected] Miguel A. Revila Ramos, University of Valladolid, Spain, [email protected] Peter Taylor, University of Canberra, Australia, [email protected] Troy Vasiga, University of Waterloo, Canada, [email protected] Peter Waker, International Qualification Alliance, Republic of South Africa, [email protected] http://www.mii.lt/olympiads_in_informatics c Institute of Mathematics and Informatics, 2009 Olympiads in Informatics, Volume 3, 2009 2009.06.30. 13 leidyb. apsk. l. Tirazˇas 200 egz. Uzˇsakymas Nr. 1748 Printed by Printing house “Mokslo aidai”, Gosˇtauto 12, 01108 Vilnius, Lithuania Olympiads in Informatics, 2009, Vol. 3 2009 Institute of Mathematics and Informatics, Vilnius Foreword OLYMPIADS IN INFORMATICS is an annual refereed journal that provides an inter- national forum for presenting research in teaching and learning informatics through competition. The journal is substantially connected with the conference held during the International Olympiad in Informatics (IOI). The papers in this volume are being presented during IOI’2009 in Plovdiv, Bulgaria. What a difference 20 years makes In May 1989, when the teams were gathering for the first IOI (also in Bulgaria), the 80486 microprocessor had just been released on the world and Tim Berners-Lee’s “Information Management: A Proposal” was being distributed to CERN’s management. That processor was available at 25 Mhz; the one this introduction is being written on is running at 2.5 Ghz. That proposal was the seed of the World Wide Web, which has become so ubiquitous that access has become a basic utility, along with water and power, in many countries. And what of the IOI? The foundation of the IOI, thanks to the efforts of Professor Blagovest Sendov and the support of UNESCO, saw the emergence of a second phase of international science olympiads. The 50s and 60s saw Mathematics, Physics and Chemisty flourish. It was to be another 20 years before another olympiad, the IOI, came into existence. Since then there has been an explosion of interest, from Astronomy to Earth Sciences, from Biology to Linguistics. The 13 countries who participated in 1989 have grown to around 80. The contest has run every year since its inauguration, has been held in 19 different countries across 5 dif- ferent continents. Contestants from earlier years are now lead delegations. Leaders who come for ‘just one year’ find themselves coming back year after year. The IOI commu- nity – the IOI family – continues to grow, and long may it continue. The contest itself has been polished over the years, but is still fundamentally faithful to the original vision. The number of questions may have changed and their difficulty tuned, but contestants today require the same skills as those 20 years ago. Machines may have hundreds or thousands of times the speed or memory, but it is a testimony to the types of problems set that, with perhaps a little tweak here and there, the challenges from the contest’s history remain a challenge. In a world that demands ‘bigger’ and ‘faster’, the IOI demands ‘smarter’. And what of the wider community? National olympiads have become an established part of the educational system in many countries. You need only read the papers in these proceedings to see how many students we reach and have reached. Some contests are run with governmental support and guid- ance, some independently. In all cases opportunities, which might not otherwise exist, are given to students. Not just to those who know they are interested in informatics, but often at a junior level where such an interest can be inspired. In this year, when the IOI returns to its birthplace, we should think back over the last 20 years. Think of the thousands of students who have been given the wonderful opportunity of attending an IOI, and of the hundreds of thousands (if not more) who have been touched across the world by national endeavours. Think of the teams of volunteers who have given their time and resources to making the contests the success that they are. Organisers who work tirelessly, often for no reward or even recognition, time and time again. Thanks are due to everyone who has contributed to this volume and the IOI con- ference. In particular, we would like to thank Prof. Krassimir Manev and the Bulgarian organisation of this year’s IOI for giving us the opportunity to host the conference. Editors Olympiads in Informatics, 2009, Vol. 3, 3–16 3 © 2009 Institute of Mathematics and Informatics, Vilnius Using Item Response Theory to Rate (not only) Programmers Michal FORIŠEK Department of Informatics, Faculty of Mathematics, Physics and Informatics Comenius University Mlynská dolina, 842 48 Bratislava, Slovakia e-mail: [email protected] Abstract. We show how Item Response Theory (IRT) can be used to define a new type of rating system, one that is especially suitable for programming competitions (and other types of compe- titions where difficulty of competitions varies between rounds). We show some useful theoretical properties of this rating system, including the ability to argue about hardness of past competition tasks, and about the precision of contestants’ skill estimates. We also define an objective method of comparing different rating systems. In the final section of the paper we apply our methods on real competition data. Key words: item response theory, ranking, rating, programming competitions. 1. Overview In this section we provide an overview of topics relevant to this article: – research of scoring and ranking in programming competitions, – research in the area of rating systems, – Item Response Theory. 1.1. Programming Competitions For a few years the International Olympiad in Informatics (IOI) community is con- cerned about the accuracy of the testing, scoring and ranking process. Several publica- tions that research various aspects of this problem include (Cormack, 2006; Cormack et al., 2006; Forišek, 2004; Forišek, 2006; Opmanis, 2006; Van Leeuwen, 2005; Verhoeff, 2006; Yakovenko, 2006). A publication particularly relevant to the topic of this paper is (Kemkes et al., 2006) where Kemkes et al. use Item Response Theory to analyze scoring used at IOI 2005, and in particular the impact of the proportion of “easy” and “hard” test cases on the relevancy of the competition results. Based on the results of the analysis, new scoring methods with better discrimination are suggested. (We would like to note that similar research has recently been conducted for other competitions as well, for example see (Gleason, 2008) for an analysis of two mathemati- cal competitions.) 4 M. Forišek However, we would like to note that while these publications use IRT only passively, as a tool for analysis of tasks only. In parts of this paper, we will use IRT as an active tool – not only to rate tasks and participants, but also to make estimates and predictions. 1.2. Rating Systems The idea of a rating system has been studied for several decades. Competitiveness is a part of our human nature, and when there is competition, there is the need to rate and/or rank the competitors. Also, the need for rating is often encountered in educational systems, and many other areas. In this context, rating means assigning a vector of properties to each subject (i.e., contestant), and ranking means arranging the subjects into a linear order according to some set of criteria. Usually, ranking is the goal, and rating represents possible means to achieve this goal. The first rating systems were reward-based: A good performance was rewarded by granting the subject rating points. The main advantage of these rating systems was their simplicity. Due to this reason, such rating systems are still used in many popular sports, such as tennis and Formula 1. These systems are usually designed so that their discrimination ability is highest among the top subjects, and rapidly decreases as the skill of the subjects decreases. More- over, the reward values are usually designed ad-hoc, and the rating systems usually have little to no scientific validity. One of the first areas to adopt a more scientific-based rating (and thus ranking) system was chess. The United States Chess Federation (USCF) was founded in 1939. Initially, USCF used the Harkness rating system (Harkness, 1967). This was a reward-based sys- tem determined by a table that listed the reward size as a simple function of the difference between the players’ current ratings. After discovering many inaccuracies caused by this system, a new system with a more solid statistical basis was designed by Arpád Elo,˝ and first implemented in 1960. For details of this rating system see (Elo, 1978). The Elo rating system was based on the following set of assumptions: • The performance of a player in a game is a normally distributed random variable. • For a fixed player, the mean value of his performance remains constant (or varies negligibly slowly).