Statistical Tests of Real-Money Versus Play- Money Prediction Markets
Total Page:16
File Type:pdf, Size:1020Kb
RESEARCH AbstractStatistical Tests of Real-Money versus Play- Prediction markets are mechanisms that aggregate information such that an esti- Money Prediction Markets mate of the probability of some future event is produced. It has been established that both real-money and play-money prediction E. S. ROSENBLOOM AND WILLIAM NOTZ markets are reasonably accurate. An SPRT- like test is used to determine whether there are statistically significant differences in accuracy between the two markets. The results establish that real-money markets are significantly more accurate for non- sports events. We also examine the effect of volume and whether differences between forecasts are market specific. Keywords: prediction markets, sequential probability ratio test INTRODUCTION eventually use to bid on real prizes. If a participant loses his or her stake of play Prediction markets are mechanisms money, the participant can replenish or that aggregate information such that refill their account back to their original the estimate of the probability of stake. There is considerable evidence some future event is produced. The that such prediction markets, using real probability of some future event money or play money, generate reason- is evoked by contract payoffs; for ably accurate probability forecasts (Berg example, a contract might pay $100 et al. 2000, 2003; Forsythe et al. 1999; if the New England Patriots win the Surowiecki 2004). Super Bowl, or zero if they do not. To date, however, there has only The price at which this contract been one study that examined trades therefore represents the col- directly the question of whether lective consensus of its expected real-money or play-money made value, or the subjective probability any difference in market predictive that the New England Patriot will performance. Servan-Schreiber et al. win the Super Bowl. (2004) compared the predictions Downloaded By: [Schmelich, Volker] At: 11:26 24 March 2010 There are two types of prediction of NewsFutures with those of markets: real-money markets and TradeSports regarding the outcomes Authorsplay–money markets. Examples of of NFL football games during the real-money exchanges are the fall-winter 2003 season. They found E. S. Rosenbloom Iowa Electronic Markets (http:// by using a randomization test that ([email protected]) is Professor www.biz.uiowa.edu/iem) and there was no statistically significant of Management Science at the I. H. TradeSports.com (http:// difference in accuracy between the Asper School of Business at the University of Manitoba. His primary www.TradeSports.com). Examples of TradeSports’ predictions and the research interests are in decision theory, play-money exchanges are the NewsFutures’ predictions. In our analytic hierarchy process and Hollywood Stock Exchange (http:// research we directly tested the pre- 2006 Electronic Markets mathematics of gaming. www.hsx.com) and NewsFutures dictive accuracy of the NewsFutures ß William Notz World News Exchange (http:// play-money market with the ([email protected]) is Professor www.us.NewsFutures.com). In real- TradeSports real-money market by of Organizational Behaviour at the I. H. money prediction markets the par- using an SPRT-like statistical test. In Asper School of Business at the Volume 16 (1): 63–69. www.electronicmarkets.org DOI: 10.1080/10196780500491303 Copyright ticipants risk their own money. In addition, we examined the effects of University of Manitoba. His primary play-money prediction markets partici- differences in trading volume, since research interests are in the extension pantsbearnofinancialrisk.Asan the two markets differ substantially and application of theories of cognition, conflict and motivation to incentive to participate in play-money in that respect, as well as in their understanding the behaviour of markets, participants are initially given a currencies. The results of that organizational participants. stakeofplaymoneythattheycan experiment suggested that the 64 E. S. Rosenbloom and William Notz & Real-Money versus Play-Money results were market related. We then tested the forecasts makers, as well as being relatively few in number. Thus, of TradeSports and NewsFutures in specific markets. the performance of prediction markets would seem to depend less on large numbers of decision makers than it does on a core of active, informed and motivated traders. WHICH MARKET SHOULD PERFORM BETTER? While it is not obvious that either of our prediction markets would have a monopoly on such marginal The dominant theoretical position that bears on the traders, it is nevertheless clear that their mere presence in operation of markets is the proposition that market the TradeSports market would remove any comparative prices summarize trader information so as to produce disadvantage in predictive accuracy caused by the greater efficient outcomes. Prices, in other words, represent the volume in NewsFutures. aggregation of all information in the system. Moreover, since the market reacts almost instantaneously and correctly to new information, this theoretical position SPRT-LIKE TEST would predict no difference between the two markets (assuming the information would be equally available to In order to determine whether the real-money market or participants across markets). the play-money market is more accurate, an SPRT-like There are arguments to suggest that the real-money hypothesis test was performed. An advantage of an SPRT- market would be more accurate. It is not unreasonable like test is that it requires no assumption about which to assume that the prospect of making money as well as model is the null hypothesis. It can treat each model the risk of losing money will make participants more equally and allow the data to determine which model is motivated in seeking accurate information. Many of the more appropriate or alternatively conclude that there is no play-money participants may not be very serious in their statistically significant difference in accuracy between the choices. In addition, there is a Darwinian aspect to real- two models. The corresponding hypotheses are: money markets. Eventually weak players lose their money and no longer participate in the market. Real- H1: Probabilities generated by NewsFutures (play-money) are money markets become a competition between stronger accurate players. versus However, there are counter arguments to suggest that H2: Probabilities generated by TradeSports (real-money) are the play-money market could perform better. Kahneman accurate. and Tversky’s (1979) prospect theory posits that the way in which a decision maker frames a problem as either a SPRT or Sequential Probability Ratio Test, due to Wald loss or a gain will produce systematic deviation from (1947), is a sequential test for a simple hypothesis H1 what would be predicted from both expected value and against an alternative simple hypothesis H2. At the end expected-utility theory. More specifically, problems of each stage in the sampling the likelihood ratio L1/L2 framed in terms of potential losses will generally evoke is computed where the suffixes 1 and 2 refer to the H1 risk-taking behavior, while the same problem framed in and H2 hypotheses respectively and L is the likelihood terms of possible gains will usually produce risk aversion function of all sample members so far drawn. If B,L1/ Downloaded By: [Schmelich, Volker] At: 11:26 24 March 2010 behavior. In addition, prospect theory posits that we L2,A the sampling is continued to another stage. If L1/ ( tend to overweigh the probability of low probability L2 B, the H2 hypothesis is accepted. If L1/L2>A, the events, under-weigh the probability of moderate and H1 hypothesis is accepted. The two positive constants, A high probability events and that our response to loss is and B, are determined by reference to prescribed more extreme than our response to gain. However, this requirements concerning the two types of errors made theory is based on gains and losses of real money. These in testing hypotheses, the rejection of H1 when it is true behavioral biases may diminish when dealing with play- and the acceptance of H1 when it is false. money rather than real-money. Defining a and b by; a5Probability of accepting H2 A second factor that impacts predictive accuracy is given that H1 is true (the probability of a Type I error) differential volume. In general, higher volumes are and b5Probability of accepting H1 given that H2 is true associated with greater predictive accuracy, and this (the probability of a Type II error), Wald established the relationship extends to prediction markets (Berg et al. relationship between (a,b) and (A,B). He showed that an 1997). However, Forsythe et al. (1999) reported upper bound for A is (1-b)/a and that a lower bound for impressive findings from both election stock markets Bisb/(1-a). and laboratory markets that call into question just how An actual determination of A and B is a difficult much of a necessary condition volume is to accuracy. computational problem. Therefore, the constants A and Their data indicates that only a small core of traders was B are almost always approximated by (1-b)/a and b/(1- necessary to drive markets to efficient outcomes. These a) respectively. This will mean that when the SPRT so-called marginal traders tended to be more experi- terminates, the probability of a Type I error is at most a enced, more educated and more knowledgeable market and the probability of a Type II error is at most b. Electronic Markets Vol. 16 No 1 65 Wald proved that under certain regularity conditions For the test of NewsFutures versus TradeSports, a SPRT will terminate in a finite number of steps with significance a5.01 and the maximum sample size probability one provided that the data are independent M51,000 was chosen. The experiment began 2 June and identically distributed. The data in this experiment 2004. Whenever NewsFutures and TradeSports offered consisted of the results of events simultaneously fore- markets on the same event, the probability forecasts casted by NewsFutures and TradeSports.