MANAGEMENT SCIENCE Vol

The Loser’s Curse: Decision Making and Market Efﬁciency in the National Football League Draft

Cade Massey The Wharton School, University of Pennsylvania, Philadelphia, Pennsylvania 19104, [email protected] Richard H. Thaler Booth School of Business, University of Chicago, Chicago, Illinois 60637, [email protected]

question of increasing interest to researchers in a variety of fields is whether the biases found in judgment Aand decision-making research remain present in contexts in which experienced participants face strong economic incentives. To investigate this question, we analyze the decision making of National Football League teams during their annual player draft. This is a domain in which monetary stakes are exceedingly high and the opportunities for learning are rich. It is also a domain in which multiple psychological factors suggest that teams may overvalue the chance to pick early in the draft. Using archival data on draft-day trades, player performance, and compensation, we compare the market value of draft picks with the surplus value to teams provided by the drafted players. We find that top draft picks are significantly overvalued in a manner that is inconsistent with rational expectations and efficient markets, and consistent with psychological research. Key words: overconfidence; judgment under uncertainty; efficient market hypothesis; organizational studies; decision making History: Received August 9, 2011; accepted August 31, 2012, by Uri Gneezy, behavioral economics. Published online in Articles in Advance March 18, 2013.

1. Introduction economic analysis is that teams often trade picks. For Two of the building blocks of modern neoclassical example, a team might give up the 4th pick and get economics are rational expectations and market effi- the 12th pick and the 31st pick in return. In aggregate, ciency. Agents are assumed to make unbiased pre- such trades reveal the market value of draft picks. dictions about the future, and markets are assumed Although it is not immediately obvious what the rate to aggregate individual expectations into unbiased of exchange should be for such picks, a consensus has estimates of fundamental value. Tests of either of emerged over time that is highly regular. One reason these concepts are often hindered by the lack of for this regularity is that a price list—known in the data. Although there are countless laboratory demon- league circles as “the Chart”—has emerged and teams strations of biased judgment and decision making now routinely refer to the Chart when bargaining for (for recent compendiums, see Gilovich et al. 2002, picks. What our analysis shows is that although this Kahneman and Tversky 2000), there are far fewer chart is widely used, it has the “wrong” prices. That studies of predictions by market participants with is, the prices on the chart do not correspond to the substantial amounts of money at stake (for a recent correct relative value of the players. We are able to review, see DellaVigna 2009). Similarly, tests of finan- say this because player performance is observable. cial market efficiency are often plagued by the inabil- To determine whether the market values of picks ity to measure fundamental value. are “correct,” we compare them to the surplus value In this paper we investigate how rational expecta- (to the team) of the players chosen with the draft tions and market efficiency play out in an unusual but picks. We define surplus value as the player’s perfor- interesting labor market: the National Football League mance value—estimated from the labor market for NFL (NFL), specifically, its annual draft of young players. veterans—less his compensation. In the example just Every year the NFL holds a draft in which teams mentioned, if the market for draft picks is rational, take turns selecting players. A team that uses an early then the surplus value of the player taken with the 4th draft pick to select a player is implicitly forecasting pick should equal (on average) the combined surplus that this player will do well. Of special interest to an value of the players taken with picks 12 and 31. Thus,

1479 Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft 1480 Management Science 59(7), pp. 1479–1495, © 2013 INFORMS our null hypothesis is that the ratio of pick values will comparing the benefit of using a pick to the opportu- be equal to the ratio of surplus values. nity cost of foregone trades. We then perform a more The alternative hypothesis we investigate is detailed cost-benefit analysis of each player selected that a combination of well-documented behavioral in the draft, by calculating the surplus value that the phenomena, all working in the same direction, cre- player provides to the team, namely, the performance ates a systematic bias causing teams to overvalue value (estimated by the price of an equivalent veteran the highest picks in the draft. For example, this is player) minus the salary paid. These analyses allows the result we would expect if teams overestimate us to test for and reject market efficiency. their ability to determine the quality of young players. Market forces will not necessarily eliminate this mispricing because even if there are a few smart 2. Background Information teams, they cannot correct the mispricing of draft Although it is not necessary to know the difference picks through arbitrage. There is no way to sell the between an outside linebacker and a cheerleader to early picks short, and successful franchises typically follow the analysis in this paper, it is important to do not “earn” the rights to the very highest picks, and have some background regarding the nature of this so cannot offer to trade them away. labor market. There are three essential features. First, Our findings strongly reject the hypothesis of mar- new players to the league are allocated to teams via ket efficiency. Although the market prices of picks an annual draft. Teams take turns selecting players in decline sharply initially (the Chart prices the first pick an order determined by the previous year’s record. at three times the 16th pick), we find surplus value There are seven rounds of the draft, and in each of the picks during the first round actually increases round the worst team chooses first and the champion throughout most of the round: the player selected chooses last (with some minor exceptions). The play- with the final pick in the first round, on average, pro- ers selected are then signed to a contract, historically duces more surplus to his team than the first pick! four to six years. Players can only sign with the team The market seems to have converged on an inefficient that selected them. equilibrium. As we discuss in §4, both the Chart and Second, the league has adopted a rule setting a a robust rule of thumb regarding the trading of a pick maximum amount any team can pay its players in this year for a pick next year have emerged as norms a given year. This is called the salary cap. The cap in the league, norms that appear to be difficult to dis- has increased over time, from $34.6 million in 1994 lodge even though the values these norms imply are to $128 million in 2009. When players are signed to demonstrably wrong. multiple-year contracts, there is usually a guaranteed The setting for this study is unusual, but we sug- up-front bonus payment plus annual salaries. The gest that the implications are quite general. It is accounting for the salary cap rule allows the teams known in financial economics that limits to arbi- to allocate the bonus equally across the years of the trage can allow prices to diverge from intrinsic value, contract. Whenever we report player compensation but some version of market efficiency remains the in this paper we are using the official cap charge as 1 working hypotheses (sometimes implicitly) even in reported to the league. The existence of this salary markets where there are no arbitrage opportunities. cap makes it easier to draw robust conclusions about Are competition and high stakes enough to produce market efficiency because all owners face the same efficiency? We show that they are not. We study a upper limit on what they can spend, unlike in pro- domain in which it is arguably easier to predict per- fessional baseball or European soccer. In those sports, formance than, say, the market for most employees, rich owners can buy the rights to star players to suit even CEOs. Teams have been able to watch prospects their own preferences, and it would be impossible for several years play the same game they will play in to say they are paying “too much” without knowing the pros, and also have administered days of physical their utility function. In the NFL, hiring a star to a big and mental tests. Still, we find their ability to predict salary limits what can be offered to other players, so performance incommensurate with their confidence. owners are forced to choose which players they wish Similar judgments about the future undergird many to spend their budget on. important decisions. Whether deciding to hire a CEO, Third, there is also a special “rookie salary cap” invest in a new technology, or to use military force, that limits the amount of money a team can spend on it is critical that one’s confidence level is appropriate. first-year players, both drafted and undrafted (most For the initial step in our analysis, we use a data set teams typically sign several of these each year). This of 407 draft-day trades to estimate the market value rookie salary cap is a “cap within a cap,” meaning of draft picks. We then ask whether the highest picks are too expensive, as the relevant psychology predicts. 1 For an excellent summary of salary cap rules, see Hall and To do so, we first take a nonparametric approach, Lim (2002). Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft Management Science 59(7), pp. 1479–1495, © 2013 INFORMS 1481 that the money spent on rookies counts toward the in the draft. Specifically, we believe teams will overall cap, but is an extra constraint. A key feature systematically pay too much for the right to draft one of the rookie salary cap is that, unlike the overall cap, player over another. This will be reflected in the rel- it varies by team. Specifically, the team’s rookie salary ative price for draft picks as observed in draft-day cap depends on the portfolio of picks the team has trades. Specifically, we predict that (subsequent to all trades), and teams with high first- M E4S 5 round picks are given larger amounts to spend on i > i 3 (2) rookie salaries. As we shall show, these rookie salary Mi+k E4Si+k5 cap allocations largely determine the compensation of i.e., that the market value of draft picks will decline draft picks.2 more steeply than the surplus value of players drafted A few other features of the league are worth not- with those picks.4 ing. The teams earn most of their revenue from televi- The bases for our prediction that top picks will sion contracts, and these revenues are divided equally. be overvalued are rooted in numerous findings in Teams also share all revenues from sales of team para- the psychology of decision making. The NFL draft phernalia such as hats or jerseys. Finally, during the involves predicting the future, a task that has received period we study, the salary cap is a binding con- considerable attention from psychological researchers. straint, or nearly so, for most teams. This research suggests that behavior can deviate systematically from rational models. In this particular 3. Research Hypothesis domain, all the psychological biases point in the direc- Draft picks are valuable. Because of the rookie salary tion of our central prediction, so we will not bela- cap, and teams’ exclusive rights to the players they bor our discussion of these various findings. Briefly, draft, teams spend less on drafted players than they the following robust empirical findings support our would for veteran players of the same expected qual- prediction. ity. (We document this fact below.) The greater this Nonregressive Predictions. One of the earliest find- “surplus,” the more a team can spend to sign high- ings in this literature is that intuitive predictions quality players in the free-agent market. This suggests are insufficiently regressive (Kahneman and Tversky that if teams are profit maximizing in their decisions 1973). That is, intuitive predictions are more extreme to trade (or not trade) draft picks, the relative value and more varied than is justified by the evidence on of any two picks will equal the relative expected sur- which they are based. Normatively, one should com- plus of the picks.3 Specifically, for the ith and ith + k bine evidence (e.g., player’s running speed) with the picks in the draft, prior probabilities of future states. Teams should find these prior odds quite sobering. For example, over M E4S 5 i = i 1 (1) their first five years, first-round draft picks (that is, Mi+k E4Si+k5 the top 32 picks) have more seasons with zero starts (15.3%) than with selections to the Pro Bowl5 (12.8%). where Mi is the market value of the ith draft pick rel- To the extent that the evidence about an individ- ative to other picks, and E4Si5 is the expected surplus value (i.e., salary-cap savings) of players drafted with ual player is highly diagnostic of a player’s NFL the ith pick. We assume that the player’s value can future, prior probabilities such as these can be given be observed on the field and estimated from the labor less weight. However, if the evidence is imperfectly market, although we stress test those assumptions in related to future performance, then teams should the penultimate section of the paper. “regress” player forecasts toward the prior probabili- In contrast to this null hypothesis, we predict ties. However, to be regressive is to admit to a limited that teams will overvalue the right to choose early ability to differentiate the good from the great; this perceived ability to differentiate is the very thing that

2 The NFL signed a new labor agreement in 2011. The rules we describe are those that were in place during the period we stud- 4 Note that this expression, by itself, does not imply which side of ied. In brief, the pay to the top draft picks has been cut, a change the equation is “wrong.” Whereas our hypothesis is that the left- consistent with viewing our results as both accurate and problem- hand side is the problem, an alternative explanation is that the error atic for the league. However, the changes were small relative to is on the right-hand side. This is the claim Bronars (2004) makes, the effects we document, and primarily focused on the top three in which he assumes the draft-pick market is rational and points to five picks. The broad patterns we find under the previous labor out its discrepancy with subsequent player compensation. The key agreement still exist under the new one. difference in our approaches is that we appeal to a third, objec- 3 Alternative assumptions, that firms try to maximize profits, win- tive measure—player performance—to determine which of the two ning percentage, or chance of winning the Super Bowl, are concep- sides, or markets, is wrong. tually quite similar. Teams do make more money if they win, and 5 The Pro Bowl is held at the end of each year with the best players the salary cap means that they have to win without spending an selected to play. We use the selection to play in this game as one unlimited amount on players. measure of outstanding performance. Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft 1482 Management Science 59(7), pp. 1479–1495, © 2013 INFORMS has secured NFL scouts and general managers their on for long enough (since 1936) that teams have had jobs. Hence, we suspect NFL decision makers put ample time to learn. Indeed, sports provides one of more weight on scouting evidence than is justified.6 the few occupations (academia is perhaps another) Overconfidence. Another robust finding in psychol- where employers can easily monitor the performance ogy, similar in spirit to the aforementioned ten- of the candidates they do not hire as well as those dency to make excessively extreme forecasts, is that they do. This could facilitate learning. This same fea- people are overconfident in their judgments (Alpert ture, that performance is observable, is what makes and Raiffa 1982). Furthermore, overconfidence is this research project possible. exacerbated by information—the more information We expect the deviation between market prices and experts have, the more overconfident they become.7 surplus value to be most acute at the top of the NFL teams face a related challenge—making judg- draft, because the psychological mechanisms we have ments about players while accumulating increasing highlighted above are most acute there. Regression amounts of information about them as the draft to the mean is strongest for more extreme samples, approaches. so we expect the failure to regress predictions to be 9 Other Psychological Factors. There are additional fac- strongest there as well. Players at the top of the draft tors that could reinforce the tendency for teams to also receive a disproportionate amount of the teams’ overvalue top picks. The winner’s curse suggests that attention and analysis, so information-facilitated over- 10 teams will fail to adjust for the fact that the win- confidence should be most extreme there. ner among many bidders for an object of uncertain A finding supporting our hypothesis implies teams but common value is likely to overpay (for a review, should trade down when endowed with a very high see Thaler 1988).8 False consensus suggests that teams pick. There are of course limits to this strategy, because will overestimate the need to trade up to acquire a the roster size constrains the number of new play- player they value because they will believe, unduly, ers a team can acquire via trading down. These are not tight constraints though, because teams routinely that other teams value him similarly (Ross et al. 1977). invite 5 to 10 undrafted free agents to summer training Also, anticipated regret can lead teams to exercise camp. Sufficient for a test of our hypothesis is the pos- rights to high-profile players because to miss out on sibility of converting (at market rates) one first-round a superstar would be particularly painful (Lerner and draft pick into just two lower picks. Indeed, this is the Tetlock 1999). Together these biases all push teams modal type of trade we observe, and also matches well toward overvaluing picking early. our theoretical focus on very high picks. Of course there are strong incentives for teams to More generally, we are investigating whether well- overcome these biases, and the draft has been going established judgment and decision-making biases are robust to market forces. Gary Becker asserts that, 6 In unreported analyses we find that team scouts predict excep- “Division of labor strongly attenuates if not elimi- tional performance by college players in the NFL more frequently nates any effects caused by bounded rationality. 0 0 0 [It] than is warranted, and that among these players predicted to be doesn’t matter if 90% of people can’t do the com- superstars there is no relation between ratings and performance. plex analysis required to calculate probabilities. The 7 Research subjects have included clinical psychologists (Oskamp 1965) and horserace bettors (Russo and Schoemaker 2002, Slovic 10% of people who can will end up in the jobs where and Corrigan 1973). it’s required” (Stewart 2005, p. 41). Romer’s (2006) 8 Although values are not perfectly common—there is certainly insightful analysis of the decision about whether to some true heterogeneity in the value teams place on players— punt or “go for it” on 4th down suggests that NFL there are multiple reasons this characterization fits. First, we simply coaches are not members of Becker’s elite 10% (see assert that there is far more variation in value across players (i.e., also Carter and Machol 1978). Here we see whether uncertainty) than there is variation in value within players across market forces can help NFL owners and general man- team (i.e., heterogeneity). Second, any true heterogeneity is muted agers to do better. by the relatively liquid trading market in players. Finally, and most important, almost all the heterogeneity is determined by player position due to team needs. This is fine, because the winners curse 4. The Market for NFL Draft Picks should apply within position as well. Harrison and March (1984) In this section we estimate the market value of NFL suggest that a related phenomenon, “expectation inflation,” occurs draft picks as a function of draft order. We value the when a single party selects from multiple alternatives. If there is uncertainty about the true value of the alternatives, the decision 9 maker, on average, will be disappointed with the one she chooses. Similarly, De Bondt and Thaler (1985) found the strongest mean Harrison and Bazerman (1995) point out that nonregressive predic- reversion in stock prices for the most extreme performers over the tions, the winner’s curse, and expectation inflation have a common past three to five years. underlying cause—the role of uncertainty and individuals’ failure 10 The tendency to overweight small probabilities, well documented to account for it. Harrison and Bazerman (1995) emphasize that in the psychological literature, also suggests that overvaluation will these problems are exacerbated when uncertainty increases and be worse at the top of the draft. For example, consider how suspi- when the number of alternatives increases—precisely the condi- ciously often we hear a college prospect described as a “once-in-a- tions of the NFL draft. lifetime player.” Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft Management Science 59(7), pp. 1479–1495, © 2013 INFORMS 1483 draft picks in terms of other draft picks. We would Figure 1 Estimated Trade Value of Draft Picks like to know, for example, how much the first draft 1.0 pick is worth, relative to, say, the 10th, 16th, or 32nd. Top pick acquired (all possible) We infer these values from draft-day trades observed Picks exchanged for top pick (observed) 0.8 over 26 years.

4.1. Data 0.6 The NFL draft consists of multiple rounds, with each team owning the right to one pick per round.11 We 0.4 designate each pick by its overall order in the draft. During the period we observe, the NFL expanded 0.2 from 28 to 32 teams and reduced the number of Value relative to no. 1 pick rounds from 12 to 7. This means the number of 0 draft picks per year ranges from 222 (1994) to 336 1 33 65 97 129 161 (1990). The data we use are trades of these draft Overall draft pick 12 picks from 1983–2008. Over this period, we observe Notes. A comparison of estimated values for “both sides” of a trade: the 1,078 draft-pick trades. Of these, we exclude 663 (61%) top pick acquired and the net exchange of all other picks in the trade. These that involve NFL players in addition to draft picks equate to the left- and right-hand sides of expression (7), respectively, calcu- and 7 4< 1%5 with inconsistencies implying a report- lated with the estimated Weibull parameters. There are at least two interpre- ing error. We separate the remaining trades into two tations of this graph. First, it provides an evaluation of the fit of the estimated model. Second, it suggests the relative “bargain” of each trade; those below groups: 314 (29%) involving draft picks from the cur- the line represent trades that cost less (from the perspective of the party rent year only and 94 (9%) involving draft picks from trading up) than expected by the model, whereas those above the line rep- both the current and future years.13 Although we resent trades that cost more (from the perspective of the party trading up) observe trades in every round of the draft, the major- than expected. ity of the trades (n = 171, 54%) involve a pick in one of the first two rounds, precisely the domain in which A striking feature of these data is how steep the we are predicting the strongest deviations from mar- curve is. The drop in value from the 1st pick to the ket efficiency.14 We observe every team trade both up 10th is roughly 50%, and values fall another 50% from and down at least once. there to the end of the first round. As we report in the following section, compensation costs follow 4.2. Results and Discussion a very similar pattern. Although the curve is not We estimate the market value draft picks using a as steep as it used to be, this flattening has slowed two-parameter Weibull distribution (see the appendix over time. In an efficient market the curve’s steep- for details). Figure 1 plots this function, showing the ness would imply both that player performance falls value of the first 160 draft picks (the first five rounds) sharply at the top of the draft, and that teams are relative to the first draft pick. It does this by com- highly skilled in their ability to identify these perfor- paring the estimated values for “both sides” of a mance differences. trade: the value of the top pick acquired by the team Another notable feature is the remarkably high dis- moving up, and the value paid for that pick by the count rate, which we estimate to be 136% per year. team moving up, net of the value of additional picks Although this finding is not the focus of the paper, it acquired. The model fits the data exceedingly well, in is clear that teams who “borrow” picks on these terms part because of the reliance on the Chart, discussed are displaying highly impatient behavior. Although it in detail below. is not possible to say whether this behavior reflects the preferences of the team owners, of their employ- 11 The order that teams choose depends on the team’s won–lost ees who typically make the decisions (general manger, record in the previous season—the worst team chooses first, and head coach, etc.), or both, it provides a significant the winner of the Super Bowl chooses last. opportunity for teams with a longer-term perspective. 12 This data set was compiled and cross-checked from a variety of Norms. As noted above, one reason why our publicly available sources, including newspapers and ESPN.com. trading-price estimates fit the data so well is that 13 See the supplemental analysis for a summary table of draft- teams have come to rely on the Chart to help them pick trades. The supplement is available online at http://consafo negotiate the terms of trade. The Chart was origi- .com/losers_curse_supplemental_analysis.pdf. nally estimated in 1991 by Mike McCoy, then a part 14 The average number of picks acquired by the team trading down owner of the Dallas Cowboys (McCoy 2006). An engi- was 2.3 (SD = 0061), with a maximum of 6. The average number of picks acquired by the team trading up is 1.1 (SD = 0038), neer, McCoy estimated the values from a subset of the with a maximum of 3. The modal trade was 2-for-1, occurring trades that occurred from 1987 to 1990. His goal was 212 times (68%). merely to characterize past trading behavior rather Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft 1484 Management Science 59(7), pp. 1479–1495, © 2013 INFORMS than to determine what the picks should be worth. than a coin flip.17 This (overly) simple observation The Chart then made its way through the league as suggests a discrepancy between the teams’ perceived personnel moved from the Cowboys to other teams, and actual ability to discriminate between prospective taking the Chart with them. In 2003 ESPN.com posted players. We explore this potential discrepancy in the a graphical version of the Chart, reporting that it was sections that follow. representative of curves that teams use (ESPN 2004). McCoy’s original curve, as well as the ESPN curve, closely approximates the one we estimate for the 5. Opportunity Costs 1983–2008 period. In our first analysis, we evaluate the benefit of using a Teams were beginning to agree about the market draft pick relative to the opportunity cost of trading it value of picks by the time McCoy estimated his chart. for two lesser picks. We focus only on directly observ- As the Chart spread around the league, it became able performance—starts and pro bowls—taking a standard for teams to openly use it to negotiate the nonparametric approach free from any monetary cal- terms of trades.15 A norm emerged for trades involv- culations. We can avoid the step of estimating the rela- ing future picks as well: “gain a round by waiting tive value of these two performance measures because a year.” For example, this year’s third-round pick the results show that trading down is a dominant would bring a pick in next year’s second round. strategy; the players acquired by trading down make 18 McCoy mentioned this heuristic explicitly when dis- significantly more starts and just as many pro bowls. cussing his construction of the Chart, and it is clear in To conduct this analysis, we evaluate all possible the data. This rule of thumb leads to huge discount 2-for-1 trades. We focus exclusively on first-round rates because they must equate the value of picks in picks, i.e., each first-round pick and the 2-for-1 trades two adjacent rounds. It is also surprisingly arbitrary. down that are possible from that position. The possi- Consider that it depends on the number of teams in bility of a trade depends on the Chart—we consider the league (which in fact has changed over time). all two-pick combinations whose total Chart value is Predictably, the emergence of widely accepted 90%–100% of the value of the first-round pick.19 For prices made trading easier; between 1983 and 2008, example, a team could trade the top pick in the draft the deviation in prices from the Chart dropped by for the 2nd and 181st, for the 14th and 15th, or a 50% (and the year-to-year volatility of that deviation number of combinations in between. Every draft-pick shrunk considerably). Also, trading activity tripled position has, on average, 19 of these two-player com- to over 20 trades per year.16 Thus, the emergence of binations. We use the 1991–2004 drafts, stopping in consensus—a norm—seems to have lent the consider- 2004 so that we have five years of performance data. able power of precedent and conventional wisdom to We estimate means and standard errors separately for the overvaluation we suggest has psychological roots. each draft-pick position, clustering on draft year. For The very steep curve we document in this sec- each possible trade, we consider the number of starts tion suggests that teams believe they have the abil- and pro bowls generated by the players involved over ity to distinguish great players from the merely good. their first five seasons. Before moving to full cost-benefit analyses, let us con- We analyze 8,526 potential trades over the 14-year sider a simple question: What is the likelihood that period and find overwhelming evidence that a team a player is better than the next player chosen at his would do better in the draft by trading down. The position (e.g., linebacker) by some reasonable mea- average gain from trading down is 5.4 starts per sure of performance, such as games started in his first five seasons? After all, this is the question teams face 17 The median number of picks between players at the same posi- as they decide whether to trade up to acquire a spe- tion is 7. cific player. 18 Of course even this analysis is not assumption free. There could The answer is 52%. Across all rounds, all positions, be some measure of performance that is sacrificed by this strat- all years, the chance that a player proves to be better egy, at least partially offsetting the gains in starts. This is unlikely than the next-best alternative is only slightly better given our unsuccessful search for elite performance measures, documented in §7.1.1. Moreover, even if present, this kind of performance is by definition exceedingly rare and must be weighted 15 In a conversation with the authors, McCoy (2006) stated, “It gave accordingly. It would also require that such performance is tightly us more confidence. If you just had a sticker—bread is 49 cents— related to draft order, a common intuition with scant empirical everything would be easier.” It also provided cover. “A standard support. price list also protects you.” McCoy added, “Now nobody gets 19 Hence, the trades we consider always weakly favor the team skinned.” trading up according to the chart. Compensation costs are very 16 By 2008 the average absolute deviation from the Chart was equiv- similar, although slightly lower for a single pick. We also estimate alent in value to a mid-4th-round pick, 1/50th the value of the top compensation-constant portfolios (rather than these chart-value pick in draft. See the supplemental analysis for a figure showing constant) and find the same pattern of results. See the supplemental these trends. analysis for details. Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft Management Science 59(7), pp. 1479–1495, © 2013 INFORMS 1485 season. We estimate this gain to be reliably positive a more complicated analysis. For this we turn to a for 31 of the 32 draft-pick positions in the first two-stage cost-benefit analysis. In the first stage, we round. Indeed, the mean gain is greater than three establish the value that teams place on performance starts/trade for most of the round (25 of the 32 posi- by looking at the compensation of veteran players. In tions). Importantly, these gains are generated with- the second stage, we apply these values to all drafted out cost in terms of pro bowls; the net change in pro players. We estimate the “surplus value” of these play- bowls for most (20 of 32) draft-pick positions is not ers to their teams by subtracting their compensation different than zero, and there are more that are posi- from these performance values. Our interest is the tive (9) than negative (3). relation between surplus value and draft order. Of course not every possible trade will work out well; sometimes the team with the high pick will trade 6.1. Data away a star for two duds, but this strategy has a very Because we want to include players in every posi- high hit rate. For 74% of the trades, a team would tion in our analyses, we rely on three performance have acquired more starts by trading down than by statistics that we can use for all positions: whether using a pick. Also, it is not the case that these gains the player is on a roster (i.e., in the NFL), the num- come at the expense of giving up the chance at a big ber of games he starts, and whether he makes the hit. In fact, in terms of starts and pro bowls, trading Pro Bowl (a season-ending “All-Star” game). We have down is a stochastically dominant strategy; 61% of the these data for the 1991–2008 seasons.20 (Later we time the team trading down would have done better replicate these analyses with more detailed perfor- in terms of starts without doing worse in terms of pro mance data, using only wide receivers.) Using these bowls. This is 2.5 times the risk of trading down and statistics, we create five comprehensive and mutu- doing worse on pro bowls without doing any better ally exclusive performance categories for each player- on starts (24%). season: players elected to the Pro Bowl (“Pro Bowl”), We also conducted an alternative version of this those who start at least 14 of the 16 regular sea- analysis by comparing the maximum (as opposed to son games (“regular starter”), those who start fewer the sum) of the two players acquired by trading down than 14 games (“occasional starter”), those who do to the player taken with the original (higher) pick. not start any games (“backup”), and those not in This is an extremely conservative test of the value the league (“NIL”).21 For player i in his tth year in of trading down because, (1) the player costs almost the league, this gives the measure Cat_ni1 t = 801 19, 50% less than the original player and (2) it neglects indicating qualification for performance category n the possibly high value of the second player. How- according to the criteria described above. ever, even setting aside those benefits, trading down We rely on a sample of experienced players to esti- is beneficial. Teams would have gained an average mate the value that teams place on these performance of 0.83 starts per season by trading down and keep- categories. These are veteran players who have signed ing only the best of the two acquired players, with a at least one free-agent contract. We limit this sample 0.0 change in the number of pro bowls. In fact, this to players drafted in 1991–2001 who are in their sixth, strategy is almost stochastically dominant; 48% of the seventh or eighth year in the NFL, and restrict our time the best acquired player is better than the origi- analysis to the 1996–2008 seasons so we can observe nal player on either starts or pro bowls without being five years of performance for each player in previous worse on the other (versus 40% of the time being years. As shown in Table 1, panel A, this leaves 3,014 worse on one without being better on the other). This players-seasons. These veteran players averaged 16% means that even a team simply trying to fill a sin- of their previous five seasons as a backup, 41% as an gle spot on their roster would be better taking two occasional starter, 33% as a regular starter, and 10% draws later in the draft than one draw in the first elected to the Pro Bowl. They are paid an average round. This strategy is even more appealing when of $3.4 million per year (median = $207 million, SD = considering the reduced compensation cost, as well as $207 million). the option value from the second player. It is difficult to overstate the strength of these results. 20 Performance data are from Stats. Inc. The 1991 season is the earliest for which the “games started” measure is reliable. 6. Cost-Benefit Analysis 21 Most of these category boundaries are obvious. The exception is The opportunity-cost analysis above indicates that dividing the two “starter” categories at 14 games. We do this to teams make a mistake by holding onto a single first- avoid excluding a player from the top starter category because of round draft pick rather than trading it for two lower very small perturbations due to injury, chance, coaching, etc. Esti- mation results are robust to moving this cutoff higher or lower. picks. However, the analysis is silent on the magni- Players elected to the Pro Bowl are assigned to that category regard- tude of the mistake, as well as how it varies at differ- less of how many games they started, with the exception of special- ent points in the draft. These finer questions require teams players. Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft 1486 Management Science 59(7), pp. 1479–1495, © 2013 INFORMS

Table 1 Player Samples

Historical performance (%)

Years in Occasional Regular Pro Compensation league N NIL Backup starter starter Bowl ($ mm)

Panel A: Experienced-player sample 6 11169 0 18 40 31 10 30052 7 993 0 14 42 34 10 30439 8 852 0 16 41 33 11 30883 Total 31014 0 16 41 33 10 30414 Panel B: Drafted-player sample 1 31483 21 38 34 7 1 00747 2 31456 23 26 33 15 3 00881 3 31438 30 19 29 18 3 00952 4 31430 38 15 26 17 4 10186 5 31348 45 11 22 17 4 10472 Total 171155 31 22 29 15 3 10044

Notes. Panel A: Includes players drafted 1991–2001 who are in the league during their 6th–8th years. Means for each category over the previous five seasons. Observations are player-seasons. Compensation is the player’s salary-cap charge, in 2008 dollars. Panel B: Includes drafted players in their first five seasons, 1994–2008. Limited to players in the first seven rounds of the draft (this is binding only 1991–1993). Observations are player-seasons. Compensation is the player’s salary-cap charge, in 2008 dollars.

Ultimately, we are interested in the value of the of the player’s performance history, estimating the player to the drafting team. To assess this, we turn best-fitting “memory” parameter for these weights. to a second sample consisting of players in their first Specifically, for player i in year t, we estimate that five years after being drafted. We restrict our analy- = + ˜ + P + T + sis to the salary cap era, 1994–2008. We also limit our log4Compi1 t5 BCat_ni1 t çIi KIi1 t i1 t1 (3) analysis to the first seven rounds of the draft because where BCat_n˜ is a weighted average of the player’s the draft has included only seven rounds since 1994. i1 t nth performance category over the previous five As shown in Table 1, panel B, this yields 17,155 player- years, IP is a vector of indicator variables for the seasons. Thirty-one percent of the player-seasons are i player’s position (quarterback, running back, etc.), NIL, 22% are backup, 29% are occasional starter, 15% T and Ii is a vector of indicator variables for the are regular starter, and 3% are Pro Bowl. Note that player’s year in the league (6th–8th). Weights are we avoid survivorship bias by retaining players in our given by wt = exp4−4r − 155 for player performance analysis who are not in the league. Players in this r years in the past. This model lets “memory” in com- sample are paid an average of $1.04 million per year pensation decay at an exponential rate. The amount (median = $00606 million, SD = $10438 million). of decay is determined by , which we estimate. The As one would expect, the correlation between com- special case of full memory, in which all five years are pensation and player performance is much higher in equally weighted, is given when = 0. By construc- years 6–8 (0.73) than in the players’ first five years tion the weight is 1 for the most recent year.22 (0.55). Performance becomes easier to predict after a The model’s predicted values provide the estimated player has played several years in the league, and market value for each position-performance pair.23 the market (rather than draft order) is determining This general approach is similar to that of previ- compensation. This is the primary motivation for bas- ous research on NFL compensation (Ahlburg and ing the compensation model on the sample of experienced players. 22 We allow the memory parameter, , to vary by player year. This is because we expect the distant past to carry less weight for a his- 6.2. Analysis and Results tory covering years 1–5 (at the beginning of which a player has just entered the league and sometimes does not even play) than 6.2.1. Performance Value. We are interested in for a history covering years 3–7. Let = 4t−65t for a player’s tth the market value of different levels of player per- 1 2 year in the league. We estimate 1, which provides the exponen- formance: backup, Pro Bowl, etc. To do this, we tial memory parameter, and 2, which modifies that parameter by investigate the relation between a player’s compen- player year. See the supplemental analysis for a depiction of these sation (salary-cap value) in years 6–8 and his per- functions. 23 formance during the previous five seasons. Recent Of course this is an approximation, because there is variation in true value within a position-performance pair. For our purposes, years are likely to carry more weight because they these approximations will be adequate as long as they are unbi- are more closely related to future performance. ased relative to draft order. We relax this assumption to test its To allow this possibility we use a weighted average implications (see Footnote 26). Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft Management Science 59(7), pp. 1479–1495, © 2013 INFORMS 1487

Table 2 Compensation Models cally distinct.25 The timing of the performance matters, Coefficient (1) (2) as well—estimates for the memory parameter indicate that a player’s performance two years before has only 1 Year 7 001251 −000607 65% as much influence on salary as the most recent 4000195 4000315 year. Comparable values for three, four, and five years 2 Year 8 002449 −000943 4000245 4000555 past are 42%, 28%, and 18%, respectively. Hence, there

2 Starts = 0 000725 is considerable “decay” in memory, providing a more 4000315 predictive model for future performance. Finally, hav- 3 Starts ≤ 14 00342 4000295 ing controlled for performance and hence survivorship, we no longer ﬁnd compensation rising with 4 Starts > 14 00683 4000335 player experience. This model explains a considerable

5 Pro Bowl 00886 portion of the variance in player compensation, with 4000395 an adjusted R-squared of 0.59.26 1 Memory 00360 4000315 In Figure 2 we present the predicted values of

2 Memory (yearly modifier) 00896 this final model for each position and performance 4000405 category, transformed into dollars. As we saw in Constant 14022 13022 the model estimates, values increase with perfor- 4000705 4000815 mance. The mean values increase from $917,000 for Observations 3,014 3,014 Adjusted R-squared 0.07 0.59 the player-seasons without any starts, to $1.9 million, $4.8 million and $8.2 million for occasional starters, Notes. Nonlinear regression results for compensation in years 6–8. Com- pensation is the salary-cap charge. Sample is all players who were drafted full-time starters, and Pro Bowl players, respectively. 1991–2001 and on an NFL roster 6–8 years later (excluding kickers and pun- One striking feature of the results is the variation in ters). Position fixed-effects are included but suppressed for presentation. The compensation for various positions. Most notable is omitted player year is 6th. In model (2), the omitted performance category the incremental value of quarterbacks, who are paid is NIL. Standard errors are shown in parentheses. more than 50% above the next-highest paid position, defensive end. Dworkin 1991, Kahn 1992, Leeds and Kowalewski 6.2.2. Compensation Cost. 2001), though, aside from our analysis of wide NFL teams care about receivers below, we rely on performance categories salary costs for two reasons. First, and most obviously, salaries are outlays, and even behavioral economists rather than performance statistics. Our modeling com- believe that owners prefer more money to less. Sec- pensation as a function of past performance is con- ond, as we discussed above, the NFL teams operate sistent with these earlier approaches, as well as with under rules restricting how much they are allowed to industry practice, in football and other professional pay their players—the salary cap. sports.24 Indeed, player negotiations are often con- The compensation data we use are from a variety of sidered “boring” because they are largely a matter public sources and have been checked for accuracy by of finding comparable players based on historical an NFL team.27 Our sample includes the first 15 years performance (Burke 2012). of the free-agency era, 1994–2008. We focus on a We present the results from this estimation in player’s salary-cap charge each year, which includes Table 2. Model (1) provides a baseline, including only indicator variables for player position and player year. Results indicate that compensation increases 25 Estimates are in log terms and therefore difficult to interpret from year 6 to 7 and from year 7 to 8. This model directly; we transform their values in Figure 3 to see the results in explains only 6.7% of the variance in player com- real terms. 26 pensation. In model (2) we add player performance In unreported analyses we consider two further elaborations of this model. First, we interacted player position with performance categories, including the memory parameters. Most categories. This increased the R-squared 0.01 (to 0.60) and did not important, we find that values increase monotonically change the results of our subsequent tests. Second, we included the with performance category, with each category statisti- player’s original draft-pick value, using an exponential distribution estimated from the data. This parameter was significantly positive, indicating that model (2) does not completely capture the relation 24 We also experimented with models that used predicted perfor- between draft pick and player value as a free agent. However, the mance in years 6–8 as the explanatory variable rather than past effect was quite small, and incorporating it into our subsequent performance. Models of this type make more sense theoretically, tests did not change any of our results. because teams should be interested in what the players will do in 27 Player contracts have to be submitted in full to the league, and the future rather than how they performed in the past, but explain the details are made available to all the teams and registered player much less of the variance in salary. Apparently, teams pay based agents. In other words, compensation is common knowledge within on the past rather than a rational forecast of the future. the league. Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft 1488 Management Science 59(7), pp. 1479–1495, © 2013 INFORMS

Figure 2 Performance Valuation

Starts = 0 Starts > 0 Starts ≥ 14 All pro

6 Value per season (millions)

TE S RB OT CB QB TE S RB OT CB QB TE S RB OT CB QB TE S RB OT CB QB G/C LB DT WR DE G/C LB DT WR DE G/C LB DT WR DE G/C LB DT WR DE Position Notes. The labor market value of a player’s previous five years. These are the predicted values from model (2) in Table 2, estimated from compensation in player years 6–8 for draft classes 1991–2001. Note that these are values for player performance that falls into a category 100% of the five-year history. Reported in 2008 dollars. his salary and a prorated portion of his bonus.28 There contract, even if remaining with their initial teams.30 are also minimum salaries, which vary by year and The slope of this curve approximates the draft-pick with player experience. In our sample only 12% of value curve estimated in the previous section. Thus, players are paid the league minimum. players taken early in the draft are thus expensive on The data reveal a very steep relation between com- both counts: foregone picks and salary paid. pensation and draft order at the top of the draft.29 6.2.3. Surplus Value. The third and final step This general pattern holds through the players’ first in our analysis is to evaluate the costs and bene- five years, after which virtually all players have fits of drafting a player. To do this we apply the reached free agency and are therefore under a new performance-value estimates from the previous section to performances in the players’ first five years. This provides an estimate of the benefit teams derive 28 Our compensation data include only players who appear on a roster in a given season, meaning that our cap charges do not from drafting a player, having exclusive rights to that include any accelerated charges incurred when a player is cut player for three years and restricted rights for another before the end of his contract. This creates an upward bias in our two. Specifically, we calculate the surplus value for cap-based surplus estimates. We cannot say for sure whether the player i in year t, bias is related to draft order, although we strongly suspect it is ˆ ˆ negatively related to draft order—i.e., that there is less upward bias Si1 t = Pi1 t−Ci1 t1 (4) at the top of the draft—and therefore works against our research ˆ hypothesis. The reason for this is that high draft picks are much where Pi1 t is the performance value estimated from more likely to receive substantial signing bonuses. Recall that such the compensation model above for his position and bonuses are paid immediately but are amortized across years for cap purposes. Thus when a top pick is cut, we may miss some of 30 After four years, players are eligible for restricted free agency. what he was really paid, thus underestimating his costs. After five years, players are unrestricted free agents and can nego- 29 There is also a distinct discontinuity after pick 32, the last pick in the tiate with any team. This time frame can be superseded by an ini- first round. Compensation shifts down sharply at this point, creating tial contract that extends into the free-agency period, e.g., six years a first-round premium, although of course there is no such disconti- and longer. Such contracts were exceedingly rare in the period we nuity in performance. See the supplemental analysis for a figure. observe, although they are becoming more common. Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft Management Science 59(7), pp. 1479–1495, © 2013 INFORMS 1489

actual performance, and Ci1 t is the player’s actual Figure 3 Surplus Value by Draft Order compensation costs. Our interest is in the relationship Panel A: Performance, compensation, and surplus between surplus value and draft order. Across all rounds, the mean salary-cap charge is 5 Performance value $1,044,029, whereas the mean estimated performance Compensation value is $1,703,390, resulting in a mean surplus 4 Surplus value of $659,361. For an initial look at the relation between these values and draft order, we consider 3 how the values for first-round players compare to Millions those for second-round players. Surprisingly, we find 2 that the mean surplus value is higher in the second round ($1,171,834) than in the first round ($1,016,797). 1 Indeed, the median surplus value is more than 60% higher in the second round ($762,785) than in the first 0 round ($462,634). Recall that the draft-pick market 1 33 65 97 129 161 values the first pick approximately four times higher Draft pick than the first pick in the second round. Panel B: Surplus vs. trade value In Figure 3, panel A, we graph all three variables 1.25 as a function of draft order, fitting lowess curves Expected surplus Trade market to the underlying player-seasons. It is noteworthy 1.00 that performance value is everywhere higher than compensation costs, and so surplus is always pos- 0.75 itive. This implies that the rookie salary cap keeps initial contracts artificially low relative to the more 0.50 experienced players who form the basis of our compensation analysis. More central to the thrust of this 0.25 paper is the fact that although both performance and Value relative to no. 1 pick compensation decline with draft order, compensation 0 declines more steeply. Consequently, surplus value 1 33 65 97 129 161 increases at the top of the order, rising to its maximum Draft pick of approximately $1,200,000 near the beginning of the Notes. Panel A: Summary of lowess curves for player performance value, second round before declining through the rest of the compensation, and surplus (performance value less compensation) in the draft. That treasured first pick in the draft is, accord- player’s first five years. Underlying observations are player-seasons, 1994– 2008; n = 161502. Reported in 2008 dollars. Panel B: “Expected surplus” is ing to this analysis, actually the least valuable pick in the lowess curve for the relationship between estimated surplus value and the first round! To be clear, the player taken with the draft order (panel A). Observations are player-seasons. The sample is for the first pick does have the highest expected performance 1994–2008 seasons, including all drafted players in their first five years in the (that is, the performance value curve is monotonically NFL, excluding punters and kickers. “Trade market” is the Weibull estimated from draft-day trades (Figure 1). decreasing), but he also has the highest salary, and in terms of performance per dollar, is less valuable than this relation is less negative than the one between mar- most players taken in the second round.31 ket value and draft order. Certainly it appears to be Clearly we should be cautious in interpreting this less negative, as shown in Figure 3, panel B. Whereas surplus curve; it is meant to summarize the results the market value of draft picks drops immediately simply. Whereas the general shape is robust to a wide and precipitously, the surplus value expected from the range of modeling decisions, the precise values are draft pick actually increases. Having established in not. More important for our hypothesis is a formal test §4 that the market value relationship is strongly neg- of the relation between the estimated surplus value ative and measured quite precisely, we will take as and draft order. Specifically, we need to know whether a sufficient, and conservative, test of our hypothesis whether the relationship between surplus value and 31 We also find that the standard deviation of surplus value is draft order is positive over a substantial part of the strongly negatively related to draft order. That is, not only do the draft. This relationship varies with draft order, so the top picks have lower mean surplus value than those in the second formal tests should be specific to regions of the draft. round, they also have the highest variance in the draft. Of course We are most interested in the top of the draft, where teams might value variance if it means there is a fat right tail offer- ing the chance of a superstar, but the opportunity-cost analysis in the majority of trades—and the overwhelming major- §5 shows this is not the case. See also our discussion of superstars ity of value-weighted trades—occur. Also, the psy- in §7.1.1. chological findings on which we base our hypothesis Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft 1490 Management Science 59(7), pp. 1479–1495, © 2013 INFORMS suggest that the overvaluation will be most extreme at than the single pick they gave up.33 Mispricing this the top of the draft. pronounced raises red flags: Is there something we have left out of our analysis that can explain the dif- 6.2.4. Spline Regressions. We regress estimated ference between market value and expected surplus? surplus value on a linear spline of draft order. The We turn to this question next. spline is linear within round and knotted between rounds. Specifically, we estimate 7. Additional Empirical Evidence ˆ In this section we consider a variety of alternative Si1 t = + 1Rd1 + 2Rd2 + 3Rd3 + 4Rd4 + 5Rd5 explanations and provide additional empirical evi- + 6Rd6 + 7Rd7 + i1 t1 (5) dence relevant to the most common questions about these results. We also construct a new test of our where Rdj is the linear spline for round j. In this research hypothesis, based on a very different depen- model, j provides the estimated per-pick change in dent variable—wins. The objective throughout is to surplus value during round j. Estimation results are determine whether the main results are robust to significantly positive for round 1. This is true whether alternative empirical formulations. using ordinary least squares (M = 00025, SE = 00003) or quantile regressions for the 25th (M = 00048, SE = 7.1. Alternative Explanations = = = 00001), 50th (M 00017, SE 00000), and 75th (M 7.1.1. Superstars. One might worry that our = 00012, SE 00002) percentiles. In contrast, rounds 2–5 results could be produced by a failure to capture the are negative in all models, significantly so in all the true value of superstar players who can singlehand- 32 models for the second and fifth rounds. edly transform a team. We are skeptical of this explanation on three counts. First, a football team has so 6.3. Discussion many players (53 on the roster, of which 22 are regu- We have shown that the market value of draft picks lar starters, not including specialists such as kickers) declines steeply with draft order: the last pick in the that it is difficult for a single player to have such a first round is worth only 25% of the first pick even profound effect (unlike in basketball, for example). though the last pick will command a much smaller Second, not all great players come from the top of salary than the first pick. These simple facts are incon- the draft. The two best quarterbacks in recent years, trovertible. In a rational market, such high prices Peyton Manning and Tom Brady, are cases in point. would forecast high returns; in this context, stellar Manning was taken with the first pick in the draft, performance on the field. And, teams do show skill but Brady was taken 199th. Also, as we showed in the in selecting players: Using any performance measure, opportunity-cost analysis above, trading down to get the players taken at the top of the draft perform more players does not reduce the chance of getting top better than those taken later. In fact, performance players. declines steadily throughout the draft. Still, perfor- Third, we are already valuing the performance of mance does not decline steeply enough to be consis- top players quite highly, and the valuation function tent with the very high prices of top picks. Indeed, is extremely convex. We estimate the value of the we find that the expected surplus to the team declines top (99th) percentile of players at more than twice throughout the first round. The first pick, in fact, has that of players at the 94th percentile, and, in turn, an expected surplus lower than any pick in the sec- value those twice as highly as players at the 72nd per- 34 ond round and is riskier as well. Furthermore, the centile. This is one of the reasons we have found risk associated with very high picks is mostly on the that there is no need for an additional “elite” perfor- downside. Because top picks are paid so much, there is mance category beyond our all-pro designation. We little room for a player to greatly exceed expectations, have estimated a wide range of compensation models but when top picks turn out to be complete busts, tens of millions of dollars are wasted. 33 Theoretically, roster limits constrain the extent to which a team The magnitude of the market discrepancy we have could pursue this strategy. However, as a practical matter, this is uncovered is strikingly large. A team blessed with not binding. Teams can carry 80 players into summer training camp (versus 53 during the season), a six-week period that provides the first pick could, in principle, through a series of a much more thorough assessment of the player. Teams usually trades, swap that pick for four or more picks in the include 10–18 rookies on this roster, meaning they might have as top of the second round, each of which is worth more many undrafted rookies as drafted. Hence, the marginal player dis- placed by an extra draft pick is an undrafted rookie. A team could certainly trade down enough to double the number of picks it has 32 The four models produce patterns that are broadly similar; see from 7 to 14 without bumping up against roster constraints. the supplemental analysis for a complete table and graph. 34 See the supplemental analysis for a more complete summary. Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft Management Science 59(7), pp. 1479–1495, © 2013 INFORMS 1491 using an additional, sixth performance category for player will not last long if the player does not con- the players elected to some combination of the all-pro tribute to the team winning on the field. However, teams approved by the collective bargaining agree- to be certain, we replicated our analysis using only ment (CBA).35 No matter how exclusively or inclu- offensive linemen, the very large players who pro- sively we construct the super-elite category, the labor tect the quarterback and create holes for the running market does not appear to distinguish it from our backs to run through, but who are forbidden to carry existing top category. the ball. While the football cognoscenti may tell you Still, we test the plausibility of this hypothesis by they are the most important unit on the field, they arbitrarily increasing by 50% the performance value attract little fan attention (or jersey sales). However, of players who are consensus All-Pro, that is, elected we find an almost identical relation between surplus to all three all-star teams approved in the CBA. There value and draft order in this subsample.37 are, on average, about seven players a season (the top Finer Performance Measures. Our main analysis of 0.4%) in this elite group of superstars. Despite this player valuation includes all NFL players. This increase, which if fully compensated would almost restricts the performance measures we can use to certainly violate the salary cap of every team with those common across all positions, measures that are one of these players, our estimated surplus value still admittedly coarse. A question that naturally arises increases during the first round of the draft accord- is whether a more fine-grained evaluation of player ing to the spline regressions estimated as in the pre- performance might alter our results. To evaluate this vious section ( = 00022, t = 7022, p < 0001). Indeed, possibility, we estimate a separate valuation model even doubling the value of these elite players does for wide receivers (n = 304), the players whose main not alter this pattern ( = 00016, t = 4092, p < 0001). job is to catch the passes thrown by the quarter- 38 Thus, it does not appear that undervaluing superstars back. We use the same estimation strategy as in our is a valid explanation for our results. Although this main analysis, except that instead of using broad cate- exercise is clearly arbitrary, these results and others gories to measure performance (e.g., starter, Pro Bowl, from similar exercises demonstrate the robustness of etc.), we use a continuous measure of performance— the pattern we observe.36 receiving yards—and explicitly allow for nonlineari- ties. As in the general model, we find that surplus 7.1.2. Other Alternative Explanations. Off-Field values increases sharply through the first round, Utility. A more subtle argument is that the utility peaking somewhere in the second before gradually to the team of signing a high draft pick is derived declining.39 This relation is strikingly similar to that from something beyond on-field performance. The which we found in our general model. intuition is that a very exciting player might help Traded-For Players. A conclusion from our analysis sell tickets and team paraphernalia in a way his per- is that teams should trade down, not up. A possible formance statistics do not reflect. Setting aside the objection to this conclusion is that when teams trade fact that paraphernalia sales are shared equally across up, they might have a special need at the position teams (unlike in European soccer, where jersey sales and/or believe they have particularly good informa- can yield a team millions of Euros), such arguments tion about the acquired player. To assess this possibil- are dubious in American football. Very few football ity, we compare the performance of players “traded players are able to bring in fans without performing for”—the highest drafted player obtained by a team well on the field, the value of which we have cap- trading up (n = 221)—with the performance of all tured in our analysis. The fans’ interest in an exciting other players (n = 31409). We calculate the player’s performance over his first five years using four measures: probability of being in the NFL, games played, 35 Pro Football Writers of America, the Associated Press, and the Sporting News. games started, and the probability of making the Pro 36 We perform an additional robustness check by increasing the Bowl. Using Tobit regressions, we estimate a separate value of all players by 50%. This addresses the concern—setting model for each performance measure. In each model aside its validity—that the salary cap artificially reduces the wage that would be paid under a free market, and because this is done by 37 See the supplemental analysis for the complete analysis. a fixed percentage, the impact is greatest among the best-paid play- 38 ers. If these players are also systematically drafted early, our esti- We chose wide receivers over quarterbacks because a wide mate of the draft-pick–performance relation will be muted. We find receiver’s contribution is better captured by a single statistic than that after inflating the performance value of all players by 50%, the is a quarterback’s. We chose wide receivers over running backs spline regression of surplus value on draft-pick order does become because we know from separate analyses that running backs are the reliably negative for the first round. However, the decline (∼20% poorest value of any position in the draft and therefore might bias over the first round) is much less steep than the decline of draft- the results in favor of our hypothesis. Still, the results are similar pick values (∼75%). We also note that for smaller increases in perfor those positions. formance value (e.g., 20%), the surplus value still reliably increases 39 R-squared is 0.80. Full results can be found in the supplemental over the first round. analysis. Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft 1492 Management Science 59(7), pp. 1479–1495, © 2013 INFORMS we regress player performance on draft order (using Table 3 Wins Analysis both linear and quadratic terms) and a dummy vari- Win % able for whether the player was “traded for.” Eval- uating 14 draft classes (1991–2004) over 18 seasons Coefficient (1) (2) (3) (4) (1991–2008), we find that “traded-for” players do not Win %, lag 1 002191 002213 002161 002174 perform differently than other players. In each of 40004275 40004265 40004405 40004425 the four models, the dummy variable for traded-for Win %, lag 2 001203 001186 001261 001249 players is not statistically different than zero.40 This 40004885 40004795 40004835 40004735 Win %, lag 3 000629 000609 000736 000721 means that the players targeted in these trades per- 40005495 40005545 40005555 40005635 form no better than would be expected for their draft Win %, lag 4 000346 000341 000411 000407 position. 40005865 40005845 40005945 40005945 No. of picks (4-yr. accum.) 000015 000009 7.2. Alternative Test: Wins 40000265 40000265 In this section we construct a final test of our Net trade value (4-yr. accum.) 000057 000056 40000285 40000295 hypothesis. Specifically, we test the implication that Constant 002834 002355 002732 002452 those teams who make wise trades according to 40003665 40009145 40003535 40009225 these estimates—wittingly or unwittingly—will per- Observations 393 393 393 393 form better on the football field. R-squared 000932 000940 001011 001014 The discrepancy between the market prices implied Notes. Regression results evaluating the impact of draft-pick trades on team by the Chart and the surplus values we estimate winning percentage. Observations are team-seasons, 1997–2008. No. of suggest that teams who successfully exploit this picks is a team’s total number of draft picks. The net trade value is the net difference can substantially improve their on-field value exchanged by a team via draft-day trades, as estimated by the surplus performance. In this section we investigate whether curve in the previous section. Both no. of picks and net trade value are accumulated over a four-year window. Standard errors are clustered on team and the teams that make “smart” trades, by our mea- reported in parentheses. sure, end up winning more games. There are various aspects of the National Football League that make the four-year trade value accumulated on that year’s finding a statistically significant relationship unlikely. roster. We include four years of a team’s lagged win- With so many players on the roster, a single draft-pick ning percentages to control for previous performance. trade may not have much effect. Also, teams play only Finally, we cluster standard errors by team. 16 games a season, each with a substantial random Regression results are presented in Table 3. Model (1) component, so there is not much power to detect an is simply the time series of winning percentage; it increase in the chance of winning. Nevertheless, we shows that winning is reliably persistent for two years test this implication. before dropping off. Model (3) adds our trade-value Observations are team-years. For each draft year we variable. This variable is positive and significant calculate the net surplus value of the picks exchanged 4p < 00055. Additional analyses reveal that the strength in trades. If teams do not trade any picks in a par- of the effect has increased over time and is strongest ticular year, their net value is 0. We arbitrarily assign for the last four years of the sample. During this the first pick a value of 1.0 and compare the sur- final period, a one-standard-deviation improvement plus values of other picks to that.41 To reflect the net in draft-pick trading produced an estimated 1.5 wins value on a roster at a given time, we accumulate these per year, a huge number in a 16-game season. single-year values into a rolling four-year sum for These results should be interpreted cautiously. each team-year. We analyze the period following the We can say that teams that make the type of draft- introduction of free agency (1993). Given a four-year day trades that our model evaluates positively tend to lag, this means our sample is limited to performance subsequently win more games. Of course it is likely in years 1997–2008. As would be expected because that such teams also do other things right, so it is trades are zero sum, the median value for a team-year possible that draft-pick trading might be serving as a is 0 (M = −0003). There is wide range, though, with proxy for overall management intelligence.42 a minimum of −12006, a maximum of 11.23, and an interquartile range of −1080 to 1.86 (SD = 300). 8. Conclusion We then evaluate the relation between this measure Our modest claim in this paper is that the owners of draft-trade acuity and a team’s winning percentage. and managers of National Football League teams are To do so, we regress a team’s winning percentage on

42 In fact, two of the teams that do well in our evaluation of draft 40 See the supplemental analysis for full regression results. trading (New England and Philadelphia) also try more fourth- 41 For example, we estimate that surplus value peaks at approxi- down conversions than average, a smart strategy as judged by mately 1.2 near the end of the first round. Romer (2006). Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft Management Science 59(7), pp. 1479–1495, © 2013 INFORMS 1493 subject to the same biased judgments found in count- salary cap reduced the cost of the very top draft picks, less other domains. Furthermore, market forces have but not enough of them to alter our results. not been strong enough to overcome these human We think that although our results are surprising, failings. The task of picking players, as we have they are plausible. We suspect that some teams, even described in this paper, is an extremely difficult one, after 15 years, have not fully come to grips with the much more difficult than the tasks psychologists typ- implications of the salary cap. Buying expensive play- ically pose to their subjects. Teams must first make ers, even if they turn out to be great performers, predictions about the future performance of (fre- imposes opportunity costs elsewhere on the roster. quently) immature young men. Then they must make Some of the most successful franchises seem to under- judgments about their own abilities: How much con- stand these concepts, but others do not. However, fidence should the team have in its forecasting skills? notice that if a few teams do learn and have win- As we detailed in §3, human nature conspires to ning records, there is no market action they can take make it extremely difficult to avoid overconfidence in to make the implied value of draft picks rational. this task. The more information teams acquire about Indeed, the irony of our results is that the supposed players, the more overconfident they will feel about benefit bestowed on the worst team in the league, their ability to make fine distinctions. And, although the right to pick first in the draft, is only a benefit it would seem that there are good opportunities for if the team trades it away. The first pick in the draft teams to learn, true learning would require the type of is the loser’s curse. systematic data collection and analysis effort that we The loser’s curse can persist even in competitive have undertaken here. Organizations rarely have the markets for a reason similar to why the winner’s inclination to indulge in such time-intensive analysis. curse can persist: there are limits to arbitrage. If naïve In the absence of systematic data collection, learning oil companies bid too much for drilling rights, then will be inhibited by bad memories and hindsight bias. sophisticated competitors can only sit on the side- The Chart is an example of an especially interesting lines and hope their competitors go broke—or even- social phenomenon in which a bias or wrong belief tually learn. Because there is no way to sell the oil becomes conventional wisdom and then, eventually, leases short, the smart money cannot actively drive a norm. The NFL draft is a situation with both great the prices of those leases down. Similarly, because there is no way to sell the first draft pick short, there uncertainty and the need to coordinate, making a is no way for any team other than the one that owns norm—such as the Chart—especially valuable. How- the pick to exploit the teams that put too high a value ever, which norm is determined by the psychological on it. Finally, now that the Chart is widely used and biases at play. The early trades on which the original accepted in the NFL, a team that owns a top draft Chart was based were priced according to the intu- pick and would like to trade it may be reluctant to itions of team decision makers. As we have argued, make a trade at less than “full value.” So, even trad- we have reason to expect these intuitions are overcon- ing down will be hard unless there is a buyer willing fident. Once distilled as a norm, this overconfidence is to pay the inflated but conventional price. self-enforcing via the confirmation bias (Klayman and The implications of this study extend beyond the Ha 1987), status quo bias (Samuelson and Zeckhauser gridiron. At its most general, these findings stand 1988), and regret aversion (Bell 1983). It also changes as a reminder that decision makers often know less the incentives that decision makers face, because there than they think they know. This lesson has been may be sanctions from fans, media, and possibly even implicated in disaster after disaster, from financial ownership for deviating from such a widely accepted markets to international affairs. Closer to the topic practice. Because norms exert such power, biases once at hand, football players are surely not the only codified are particularly pernicious. Hence, the Chart employees whose future performance is difficult to appears both a symptom of biased judgment and predict. In fact, football teams almost certainly are also a self-perpetuating cause. This dynamic between in a better position to predict performance than most biases and norms deserves greater investigation. employers choosing workers, whether newly minted Our findings are strikingly strong. Rather than a MBAs or the next CEO.43 In our judgment, there is treasure, the right to pick first appears to be a curse. If picks are valued by the surplus they produce, then 43 Kaplan et al. (2012) find that observable traits reliably predict the first pick in the first round is the worst pick in the the performance of CEOs in buyout firms and, to a lesser extent, round, not the best. In paying a steep price to trade venture-capital backed firms. The explanatory power of these mod- up, teams are paying a lot to acquire a pick that is els is understandably modest (R-squared ranges from 0.14 to 0.42). worth less than the ones they are giving up. We have The present research suggests decision makers in these environ- ments will overestimate this predictive ability. This has conse- conducted a wide range of empirical tests, and every quences both before the hiring decision (e.g., settling on candidates analysis gives qualitatively similar results. The same is too soon, paying too much for one over another) and afterward true under the 2011 labor agreement. The new rookie (e.g., insufficient monitoring, misconstruing early performance). Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft 1494 Management Science 59(7), pp. 1479–1495, © 2013 INFORMS little reason to think that the market for CEOs is Table A.1 Draft-Pick Trade Value: Regression Results more efficient than the market for football players. Model: (1) (2) (3) (4) (5) The problem is not that future performance is difficult Years: 1983–2008 1983–2008 1983–1992 1993–2000 2001–2008 to predict, but that decision makers do not appreciate Future picks: No Yes No No No how difficult it is. Or, as Mark Twain supposedly put it, more colorfully: “It ain’t what you don’t know that Parameter estimates 00146 000996 00199 00184 000994 gets you into trouble. It’s what you know for sure that 4000275 4000165 4000865 4000685 4000215 just ain’t so.” 00698 00745 00642 00662 00764 4000305 4000265 4000685 4000605 4000355 Acknowledgments 10358 The authors thank Marianne Bertrand, Jim Baron, Rodrigo 4000845 Canales, Russ Fuller, Shane Frederick, Rob Gertner, Jon Implied values 4relative to the first pick5 (%) Guryan, Tom Hubbard, Rick Larrick, Michael Lewis, Toby 5th pick 68 76 62 63 75 10th pick 51 60 44 45 59 Moskowitz, Barry Nalebuff, Devin Pope, Olav Sorenson, 16th pick 38 47 32 33 46 David Robinson, Yuval Rottenstreich, Suzanne Shu, Jack 32nd pick 20 28 16 17 25 Soll, George Wu, and especially David Romer. They also 64th pick 7 11 6 6 9 thank workshop participants at the University of California N 313 407 70 108 135 at Berkeley, Carnegie Mellon University, Cornell University, R-squared 0.99 0.99 0.98 0.99 0.99 Duke University, the Massachusetts Institute of Technology, Notes. Results from using nonlinear regression to estimate parameter values the University of Pennsylvania, Stanford University, the for a Weibull-function model of draft-pick value. Data are draft-day trades, University of California at Los Angeles, the University of 1983–2008. Excludes trades involving players (n = 663). Standard errors California at San Diego, the University of Chicago, and are in parentheses. Yale University for valuable comments; and Chad Reuter, Al Mannes, and Wagish Bhartiya for very helpful research decay at either an increasing or decreasing rate, depending assistance. on whether its value is greater than or less than 1. If = 1, we have a standard exponential with a constant rate of Appendix. Estimating the Trade-Market decay. Also, note that for the first pick in the draft, v415 = Value of Draft Picks e−41−15 = 100. Substituting (7) into (6) and solving in terms of the high- A.1. Methodology est pick in the trade, we have We are interested in estimating the value of a draft pick in n m 1/ terms of other draft picks, as a function of its order. We let L H 1 X −4t −15 X −4tH −15 t = − log e j − e i + 11 (8) the first pick be the standard by which we measure other 1 picks. We assume that the value of a draft pick drops mono- j=1 i=2 tonically with the pick’s relative position, and that it can be which expresses the value of the top pick acquired by well described using a Weibull distribution.44 Our task is the team trading up in terms of the other picks involved then estimating the parameters of this distribution. in the trade. Recall that this value is relative to the first pick r Let ti denote the tth pick in the draft, either for the team in the draft. We can now estimate the value of the parame- with the relatively higher draft position (if r = H) and there- ters and in expression (8) using nonlinear regression.45 fore “trading down,” or the team with the relatively lower draft position (if r = L) and therefore “trading up.” The A.2. Results index i indicates the rank among multiple picks involved We estimate (8) using the 313 current-year trades only, find- in a trade, with i = 1 for the top pick involved. ing = 00146 (se = 00027) and = 00698 (se = 00030). These For each trade, we observe the exchange of a set of draft results are summarized in Table A.1, column (1). As shown picks that we assume are equal in value. Thus, for each in the bottom half of the table, these values imply a steep trade we have drop in the value of draft picks. In short, the 5th pick is m n X H X L valued approximately 2/3 as much as the first pick, the v4ti 5 = v4tj 51 (6) i=1 j=1 10th pick 1/2 as much, and the last pick in the first round about 1/5 as much.46 We also estimated an extended ver- where m picks are exchanged by the team trading down sion of (8) that includes a parameter for the discount rate.47 for n picks from the team trading up. Assuming the value of the picks follow a Weibull distribution, and taking the overall first pick as the numeraire, let the relative value of 45 We first take the log of both sides of expression (8) before esti- a pick be mation to adjust for lognormal errors. − r − r 4ti 15 46 v4ti 5 = e 1 (7) We drop one trade from our estimation because of its disproportionate influence. We identify this trade by repeatedly estimating where and are parameters to be estimated. Note that this model while dropping one observation at a time. Results are the presence of the parameter allows the draft value to robust to the exclusion of all trades except one, the inclusion of which changes values dramatically. Excluding this observation pro- 44 The Weibull distribution is a two-parameter function that nests vides a conservative test of our main hypothesis since the valuation the exponential, providing a more flexible estimation than a stan- curve is flatter without it. dard exponential would provide. 47 See the supplemental analysis for the full derivation. Massey and Thaler: Decision Making and Market Efficiency in the NFL Draft Management Science 59(7), pp. 1479–1495, © 2013 INFORMS 1495

This expression allows us to include trades involving future Hall BJ, Lim JP (2002) A note on incentives in the NFL. HBS Case picks, expanding our sample to 407 observations. Results 902–129, Harvard Business School, Boston. are presented in Table A.1, column (2). The estimated curve Harrison JR, Bazerman MH (1995) Regression to the mean, is close to the previous one, with = 000996 (SE = 00016) expectation inflation, and the winner’s curse in organizational and = 00745 (SE = 00026), although a bit flatter—e.g., the contexts. Kramer R, Messick DM, eds. The Social Context of Negotiation (Sage Publications, Thousand Oaks, CA), 69–94. 10th pick is valued at 60% (versus 51%) of the first. The Harrison JR, March JG (1984) Decision-making and post-decision estimated discount rate, , is a staggering 136% (se = 00084) surprises. Admin. Sci. Quart. 29(1):26–42. per year. Kahn LM (1992) The effects of race on professional football players Finally, we investigate how these draft-pick values have compensation. Indust. Labor Relations Rev. 45(2):295–310. changed over time, focusing on trades for current picks Kahneman D, Tversky A (1973) On the psychology of prediction. only. We find that the valuation curve has flattened some in Psych. Rev. 80(4):237–251. recent years, meaning pick values do not decline as rapidly. Kahneman D, Tversky A (2000) Choices, Values, and Frames This is consistent with teams learning, however slowly and (Russell Sage Foundation, Cambridge University Press, insufficiently, that the top picks are relatively overpriced. Cambridge, UK). Kaplan S, Klebanov M, Sorensen M (2012) Which CEO characteris- tics and abilities matter? J. Finance 67(3):973–1007. References Klayman J, Ha Y (1987) Confirmation, disconfirmation, and information in hypothesis testing. Psych. Rev. 94(2):211–228. Ahlburg DA, Dworkin JB (1991) Player compensation in the National Football League. Staudohar PD, Mangan JA, eds. Leeds MA, Kowalewski S (2001) Winner take all in the NFL: The The Business of Professional Sports (University of Illinois Press, effect of the salary cap and free agency on the compensation Urbana), 61–70. of skill position players. J. Sports Econom. 2(3):244–256. Alpert M, Raiffa H (1982) A progress report on the training of Lerner J, Tetlock P (1999) Accounting for the effects of accountabil- probability assessors. Kahneman D, Slovic P, Tversky A, eds. ity. Psych. Bull. 125(2):255–275. Judgment Under Uncertainty: Heuristics and Biases (Cambridge McCoy M (2006) Personal correspondence with the authors, University Press, Cambridge, UK), 294–305. September 12. Bell D (1983) Risk premiums for decision regret. Management Sci. Oskamp S (1965) Overconfidence in case-study judgments. J. Con- 29(10):1156–1166. sulting Psych. 29(3):261–265. Bronars SG (2004) Bargaining in professional football: Why NFL Romer D (2006) Do firms maximize? Evidence from professional superstars are underpaid. Working paper, Georgetown Univer- football. J. Political Econom. 114(2):340–365. sity, Washington, DC. Ross L, Greene D, House P (1977) The false consensus effect: An Burke B (2012) Art and analytics of negotiation. Panel discussion, egocentric bias in social perception and attribution processes. MIT Sloan Sports Analytics Conference, March 2, Boston. J. Experiment. Soc. Psych. 13(3):279–301. Carter V, Machol RE (1978) Optimal strategies on fourth down. Russo J, Schoemaker P (2002) Winning Decisions: Getting It Right the Management Sci. 24(16):1758–1762. First Time (Doubleday, New York). De Bondt WFM, Thaler R (1985) Does the stock-market overreact. Samuelson W, Zeckhauser R (1988) Status quo bias in decision mak- J. Finance 40(3):793–805. ing. J. Risk Uncertainty 1(1):7–59. DellaVigna S (2009) Psychology and economics: Evidence from the Slovic P, Corrigan B (1973) Behavioral problems of adhering to field. J. Econom. Literature 47(2):315–372. a decision policy. Seminar presentation, May 1, Institute for ESPN (2004) NFL draft-pick value chart. ESPN Insïder (March 1), Quantitative Research in Finance, Napa, CA. http://sports.espn.go.com/nfl/draft06/news/story?id=2410670. Stewart S (2005) Can behavioral economics save us from ourselves? Gilovich T, Griffin DW, Kahneman D (2002) Heuristics and Biases: Univ. Chicago Magazine 97(3):36–42. The Psychology of Intuitive Judgement (Cambridge University Thaler RH (1988) The winner’s curse. J. Econom. Perspect. 2(1): Press, Cambridge, UK). 191–202.