Outcome bias and Football 1 Running head: OUTCOME BIAS AND FOOTBALL Outcome Bias in Subjective Ratings of Performance: Evidence from the (Football) Field Edgar E. Kausel School of Management Pontificia Universidad Católica de Chile Av.Vicuña Mackenna 4860 Macul, Santiago, Chile. Phone: (56-2) 235-44018 [email protected] Santiago Ventura School of Management Pontificia Universidad Católica de Chile Av.Vicuña Mackenna 4860 Macul, Santiago, Chile. Phone: (56-2) 235-47303 [email protected] Arturo Rodríguez Faculty of Economics and Business University of Chile Diagonal Paraguay 257 Santiago, Chile 6510015 Phone: (56-2) 978-3770 Fax: (56-2) 222-0639 [email protected] Outcome bias and Football 2 Abstract The outcome bias occurs when people assess others’ decision making process or performance and put an unwarranted weight to their outcomes. This bias has important implications for the judgment and choice as well as the performance appraisal literatures. However, virtually every extant study has been conducted in the lab, likely due to endogeneity concerns in field. Penalty shoot-outs in association football (‘soccer’) offer an interesting way of studying outcome bias, as recent research suggests that their outcome is unrelated to in-game performance. We use Goal (goal.com) to study subjective performance ratings by reporters given to 1,157 players in 43 games from important football competitions. Using both multilevel mixed-effects and fixed-effects (within- players design) modeling, we found that winning on penalties was linked to higher performance ratings. This result persisted even after we removed players who took part in the penalty shoot-outs; thus, supporting the idea of outcome bias. We discuss implications for applied settings. Keywords: Judgment and Decision Making; Behavioral Economics; Heuristics and Biases; Outcome Bias; Football (Soccer); Performance Appraisal. Outcome bias and Football 3 Outcome Bias in Subjective Ratings of Performance: Evidence from the (Football) Field This is the lottery of penalty shootouts (Franz Beckenbauer during the 1990 World Cup; Vecsey, 1990) For me, the national team is over (Lionel Messi’s reaction after losing in penalty shootouts the Copa América final against Chile; Das, 2016) Some people use a similar argument to amplify a behavior after a win and to condemn the same behavior after a defeat. For example, if Neymar recovers a ball and then the team scores, journalists say: ‘Ah, the coach tamed Neymar and made him play collectively.’ But if the team loses, they say: ‘Useless coach, how can he make Neymar chase the wing-back instead of making him play close to the penalty box!’ The media specializes in corrupting human beings depending on a win or a loss. (Coach Marcelo “El Loco” Bielsa; El Dinamo, 2017) 1. Introduction From a normative perspective, we should distinguish the decision making process from its consequences (Skitka & Tetlock, 1993; Hastie & Dawes, 2009). Given that real decisions are made in an uncertain world, an appropriate decision process may lead to poor results—or a poor decision process may lead to positive outcomes (Baron, 1985; Bazerman & Moore, 2013; Brockner, 2015). For example, even if a well-intended physician follows a procedure flawlessly and exercises sound judgment, the operation may lead to patient death. More generally, we should be able to differentiate behavior from outcomes. This is consistent with Kahneman’s (2011) observation that “because luck plays a large role, the quality of leadership and management practices cannot be inferred reliably from observations of success” (p. 207). Indeed, the behavior vs. outcome issue is important in performance assessment within the industrial-organizational and educational psychology literatures. The aim in Outcome bias and Football 4 performance appraisal is to judge an individual's behavior, not factors beyond his or her control (Cascio & Aguinis, 2010). This is one of the reasons many managers and school principals often tend to put emphasis on subjective ratings to assess employees or teachers, as opposed (or in addition) to so-called ‘objective measures’ (Murphy & Cleveland, 1995). Although objective measures can tap job performance, they are more often indicators of outcomes—likely to be affected by external factors. For example, an objective measure of scholar performance is number of papers published. However, this outcome is affected by external factors such as the journal editors’ attitude toward the scholar’s specific field of study. Thus, researchers agree that subjective ratings are more likely to tap job performance1 than objective measures are (Landy & Conte, 2004). In particular, subjective ratings are said to cover psychologically relevant aspects of job performance given the context and task at hand (Campbell, 2012; Landy & Farr, 1983). Also, subjective ratings can capture different dimensions of job performance, such as task performance, contextual performance, and counterproductive behavior (Rotundo & Sackett, 2002). A relevant issue regarding subjective ratings is, however, the potential influence of outcome bias (Savani & King, 2015; Gino, Moore, & Bazerman, 2009). Outcome bias occurs when outcome knowledge affects the evaluation of how well an individual made a decision or performed a task (Baron & Hershey, 1988; Johnston & Marshall, 2016). For example, Baron and Hershey showed two groups of participants the same description of a medical decision making process but different outcomes. People who were told that the 1 We should underscore that performance is defined as behavior—something done by an individual (Campbell, Mcenry, & Wise 1990). Outcome bias and Football 5 outcome was positive perceived the decision making process as more favorable than those told that the outcome was negative. Yet, although there are several published studies showing evidence of outcome bias using subjective ratings in the laboratory, where outcome information can be randomized, there are virtually no studies in field settings (cf. Lefren, Platt, & Price, 20142). This is unfortunate because, as the authors of a seminal paper on outcome bias recognized, the “difficulty with the method we used is that it is restricted to somewhat artificial situations” (Hershey & Baron, 1992, p. 91). The reason for a lack of field studies on outcome bias is that, in the field, an individual’s performance is correlated with his or her outcomes (Bommer, Johnson, Rich, Podsakoff, & Mackenzie, 1995; Ilgen & Favero, 1985). As such, it is difficult to distinguish whether correspondence between subjective ratings and outcomes is the result of a biased assessment by the rater or actual performance by the ratee. In other words, there is a problem of endogeneity, whereby an individual’s performance may have an effect on both subjective ratings and outcomes. 2. Penalty shoot-outs and performance in the field Association football (‘soccer’) gives us an interesting setting to examine outcome bias in subjective performance ratings. In some competitions, after the game ends in a draw, the teams proceed to penalty shoot-outs to determine the winner. Many coaches and 2 Lefren et al. is a notable exception. Using American Football games, they found that coaches were more likely to change their team strategy after a close loss than after a close win. Although impressive in terms of the data used, this result could be explained by action bias rather than outcome bias (Bar-Eli, Azar, Ritov, Keidar-Levin, & Schein, 2007). After losing, coaches tend to take action because this ‘looks better’ and thus is more justifiable (Connolly & Zeelenberg, 2002). Relatedly, the study by Leftren et al. does not directly include subjective performance ratings, which is the focus of our study. Outcome bias and Football 6 journalists believe that chance plays a crucial role in the outcome of penalty shoot-outs, as reflected by Franz Beckenbauer’s quote at the outset of the present paper.3 This belief seems to be supported by data. Nate Silver—from the popular Five Thirty-Eight website—examined 37 official games from 2005 to 2013 played in the knockout stages of different continental championships among men’s national teams (Silver, 2014). For each game, he classified teams as favorites and underdogs by using the teams’ ELO ranking, a measure of team skill that has relatively strong predicting power in forecasting football outcomes (Lasek, Szlávik, and Bhulai, 2013). Silver found that favorites and underdogs had roughly equal chance to win the game on the penalty shoot- outs. We used Silver’s data and also found no significant differences in ELO rating between teams that won (M = 1,803; SD = 172) and teams that lost (M = 1,800; SD = 153) on penalty shoot-outs (t [36] = .12, p = .91). A different study with much more data at the individual level is consistent with this finding. The Economist’s Data Team (2017) analyzed 2,788 penalty kicks awarded in three European leagues between 2007 and 2016. They found no statistically significant relationship between a player’s past conversion rate and his future success (Data Team, 2017). In sum, the outcome of a penalty shoot-out (i.e. who wins and who loses) seems to be unrelated to in-game skill and performance at both the team- and player- levels of analysis. We provide more evidence regarding this issue in our results section. Note that we are not claiming that, in games defined by penalty shoot-outs, the teams’ in-game performances are equal. Indeed, we report below some evidence 3 We included two other quotes to reflect (a) how people in football are intuitively aware of the outcome bias and its problems (Marcelo Bielsa’s quote) and (b) how dramatic can an outcome of a penalty shootout be. Messi’s quote reflects this, as right after Argentina lost that game on penalties, Messi announced his retirement from the national team. Outcome bias and Football 7 suggesting that the there are some systematic differences in performance between teams in these games (i.e., favorites tend to have better performance than underdogs). What we are claiming is that, whichever the differences between two teams in this type of games are, these are unrelated to the penalty shoot-out outcome.