Flipping Coins in the War Room: Skill and Chance in the NFL Draft

Home , Draft (sports), Dwight Freeney, Edgerrin James, Reggie Wayne

Cade Massey Yale University

MIT Sports Analytics Conference March 4, 2011

Skill and Chance in the Draft

Clearly there is skill involved.

But are there differences in skill?

Are some teams better than others at picking players? Pro Bowl Seasons by Players Drafted '97-'07 IND PIT DAL SEA PHI CHI MIN BAL NE WAS GB ARI SD CAR STL ATL NYJ DEN CIN NYG SF KC OAK MIA TEN NO TB BUF HOU JAC CLE DET 0 5 10 15 20 25 30 35 40 “Indianapolis Colts…not just with Peyton Manning… Dwight Freeney, Edgerrin James and Reggie Wayne were genius picks, too.”

“The Cleveland Browns…screwed the Chihuahua. Their run of No. 1 picks from 1999 to 2002 is the single worst stretch of drafting since the Iraqi Republican Guard. Were they using an Ouija board?” Skeptical, due to research on NFL draft

Massey & Thaler (“Loser’s Curse”)

. What’s the chance that a player is better than the next player taken at his position?

. E.g., the 3rd cornerback vs. the 4th cornerback Probability 0 .1 .2 .3 .4 .5 .6 .7 1234567 Better Than The Next Guy? The Than Better by round draft Skeptical, due to research in psychology

. People under-appreciate the role of chance . E.g., belief in the “law of small numbers”

. An important implication: “Fictitious Variation”

. Matthew Rabin (2002)

. Suggests that people over-attribute differences in outcomes to differences in skill.

. Provides our main hypothesis: There is less difference in draft-picking skill than people believe. Methodology

Use games started . Results are robust to pro bowl, Massey-Thaler $s, etc. Adjusted for draft position . Compared against long-run average for that pick Use the player’s first 5 years Two tests: . Within year, across teams (intra-class correlation) . Within team, across years (autocorrelation) Performance evaluated over player's 1st 5 years. Starts/Season 0 2 4 6 8 10 12 14 16 Draft-pick Performance vs. Expectation vs. Performance Draft-pick 1 33 65 97 Draft Pick Draft 129 161 193 225 257 Draft-pick Performance vs. Expectation

16 Patriots, 2003

Koppen 14 Warren 12 Wilson Samuel 10 8 6 Starts/Season 4 Banta-Cain

2 Johnson Kelley Klecko

0 Kingsbury Nead 1 33 65 97 129 161 193 225 257 Draft Pick Performance evaluated over player's 1st 5 years. Games Started vs. Expected -10 -5 0 5 10 Pick Performance by Team (1995 Draft) ARI

ATL

BAL

BUF

CAR

CHI

CIN Team CLE

DAL

DEN

DET

HOU

IND

JAC

KC Intraclass Correlation 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 1991 1993 1995 1997 Draft Year Draft 1999 2001 2003 2005 Avg Games Started vs. Expected -2 -1 0 1 2 1991-2006 Drafts t 5 r 5 n 5 o 5 t 5 r 5 n 5 Top 25% 2nd 25% 3rd 25% 4th 25% Top 25% 2nd 25% 3rd 25% 4th 25% hsYa Next Year This Year r=-0.14 (n.s) Skill and Chance in the Draft

So it’s all chance on the margin!

Is this nihilistic?

Quite the contrary. Decision strategy depends on situation

Skill-based Chance-based

? Logic of Chance (LOC)

Distinct set of prescriptions for operating in an environment with high uncertainty. LOC: First, acknowledge role of chance

Roethlisberger on the number of times they’ve beaten the Ravens in close games: “We got lucky.” LOC: Maximize your # of chances LOC: Minimize the cost of choosing

vs. LOC: Value the process (and protect it) LOC: Seek broad, independent input LOC: Be slow to place credit, blame In sum: At the margin, picking players is chance 1

.9 Teams don’t vary at .8

.7 this… .6 .5 .4 .3 Intraclass Correlation .2 .1 0

1991 1993 1995 1997 1999 2001 2003 2005 Draft Year

1991-2006 Drafts 2 …and there are no 1 persistent differences

0 in drafting ability. -1 Avg Games Started Expected vs. -2

4th 25% 3rd 25% 2nd 25% Top 25% 4th 25% 3rd 25% 2nd 25% Top 25% This Year Next Year Managing chance is where the skill is.