An Explorative Study of Applying Prediction Markets in Course Evaluation
Total Page:16
File Type:pdf, Size:1020Kb
An Explorative Study of Applying Prediction Markets in Course Evaluation University of Oulu Department of Information Processing Science Master’s Thesis Kang Wang Date 20.5.2013 2 Abstract Prediction Markets is a new emerging efficient method for predicting the likelihood of uncertain future events, and also it has been used successfully in various fields. Compare to traditional forecasting methods, it can provide more accurate results and reflect continuous real-time information. In this research, we focus on its application in course evaluation. This research includes a Prediction Markets-based systematic literature review, in which there are 120 articles included. They are analysed based on four main categories: description, theoretical work, organization management, and applications. The results demonstrate that few of research take Prediction Markets into consideration as an evaluation system and it has not been used for course evaluation purpose. Course evaluation in universities is quite meaningful. Traditional course evaluation methods have been proved to be low response rate, time consuming, inaccurate data and lacking interactions between teachers and students. In this research, the Prediction Markets-based course evaluation system is designed as a new solution of evaluating the course quality. We present an explorative experiment which is conducted in a real course environment with 49 students involved. The students are divided into two groups: Prediction Markets-based group and Traditional group, with the purpose of comparing the difference between Prediction Markets-based course evaluation system and traditional in use course evaluation system. In conclusion, the study confirms the feasibility of extending Prediction Markets in course evaluation use. Additionally, a possible solution of designing and implementing Prediction Markets-based course evaluation system is presented. Further studies should focus on the issues of optimizing the Prediction Markets-based course evaluation system. Keywords Prediction Markets, Course Evaluation, Students Feedback 3 Abbreviations AMM Automated Market Maker CA Call auctions CDA Continuous double auctions DA Double auctions DPM Dynamic pari-mutuel market GQM Goal question metric method HSX Hollywood Stock Exchange IEM Iowa Electronic Markets IT Information technology LMSR Logarithmic market scoring rules PMCES Prediction Markets-based Course Evaluation System UBC-ESM UBC Election Stock Market 4 Foreword During the thesis studying, I have learnt many things that they are not only the knowledge relating to the research topic, but also including many aspects of how to do a better research; for instance, I may rich the experience of holding a small experiment, managing a questionnaire survey and making research interviews. In addition, understanding a little bit more of the life living wisdom can be recognised as extra obtains. I sincerely thank my supervisor Jouni Markkula and Muhammad Ovais Ahmad very much, and all the people who take part in my thesis work. Without their supports the thesis would not have been done. Especially, I have received much good guidance and significant helps from my supervisor. I owe many thanks to him. Last but not least, I cherish the opportunity of study abroad and always holding gratitude for University of Oulu. Of course, my parents have been encouraging me and supporting my study, many thanks for them as well. Kang Wang Oulu, May 20, 2013 5 Contents Abstract ............................................................................................................................. 2 Abbreviations .................................................................................................................... 3 Foreword ........................................................................................................................... 4 Contents ............................................................................................................................ 5 1. Introduction .................................................................................................................. 7 2. Research problem and method ................................................................................... 10 2.1 Research problem .............................................................................................. 10 2.2 Research strategy ............................................................................................... 10 2.2.1 Systematic literature review ...................................................................... 13 2.2.2 System development ................................................................................. 13 2.2.3 Experiment ................................................................................................ 14 2.2.4 Interviews .................................................................................................. 14 3. Systematic literature review ....................................................................................... 15 3.1 Background and related work ............................................................................ 15 3.2 Classification of Prediction Markets .................................................................. 15 3.3 Review questions ............................................................................................... 17 3.4 Methods ............................................................................................................. 17 3.4.1 Inclusion and exclusion ............................................................................ 18 3.4.2 Search strategy .......................................................................................... 18 3.5 Results ................................................................................................................ 21 3.5.1 Overview of results ................................................................................... 21 3.5.2 Analysis .................................................................................................... 21 3.6 Relationship of results between systematic review and thesis work ................. 25 4. Fundamentals of Prediction Markets .......................................................................... 26 4.1 How does it work? ............................................................................................. 26 4.2 Types of Prediction Markets .............................................................................. 27 4.3 Trading mechanisms .......................................................................................... 27 4.4 Does money matter? .......................................................................................... 29 4.5 Why do Prediction Markets work so well? ........................................................ 29 4.6 Previous usage of Prediction Markets ............................................................... 30 4.6.1 Iowa Electronic Markets ........................................................................... 30 4.6.2 Prediction Markets for sports game .......................................................... 30 4.6.3 Hollywood Stock Exchange ...................................................................... 31 4.6.4 Other applications ..................................................................................... 32 5 Course evaluation system in universities ................................................................... 33 5.1 Definition of course evaluation .......................................................................... 33 5.2 Paper-based VS online-based course evaluation method .................................. 34 5.3 Current course evaluation in University of Oulu ............................................... 34 6 An explorative experiment of Prediction Markets-based course evaluation system ......................................................................................................................... 36 6.1 Design the Prediction Markets-based course evaluation system ....................... 36 6.1.1 Contracts ................................................................................................... 36 6.1.2 Trading mechanism ................................................................................... 37 6.1.3 Incentives .................................................................................................. 37 6.1.4 Traders ...................................................................................................... 38 6.1.5 Trading platform ....................................................................................... 38 6 6.2 Establish the Prediction Markets-based course evaluation system .................... 38 6.3 Define the experiment ........................................................................................ 42 6.4 Results ................................................................................................................ 44 6.4.1 Participants ................................................................................................ 44 6.4.2 Number of trades ...................................................................................... 44 6.4.3 Answers of the questions .......................................................................... 45 7 Experiment analysis ................................................................................................... 50 7.1 Defining the