Journal of Sports Analytics 1 (2015) 111–119 111 DOI 10.3233/JSA-150015 IOS Press You can beat the “market”: Estimating the return on investment for NHL team scouting Michael E. Schuckersa,∗ and Steve Argerisb aSt. Lawrence University, Romoda Drive, Canton, NY, USA bGeorge Mason University, Fairfax, VA, USA Abstract. Scouting is a major part of talent acquisition for any professional sports team. In the National Hockey League (NHL), the ‘market’ for scouting is set by the NHL’s Central Scouting Service which develops a ranking of draft eligible players. In addition to the Central Scouting rankings, NHL teams use their own internal scouting to augment their knowledge of eligible players and develop their own rankings. Using a novel statistical approach we show in this paper that the additional information possessed by teams provides better rankings than those of Central Scouting. Using data from the 1998 to 2002 NHL drafts, we estimate that the average yearly gain per team from their internal scouting is between $1.7MM and $5.2MM. These values are consistent across the three measures of player productivity that we consider: cumulative Games Played, cumulative Time On Ice and cumulative Goals Versus Threshold where we aggregate these metrics across the first seven years post draft. We used this time frame since teams generally retain rights to their draft picks for seven years. Further, we find that no individual team outperformed the others in terms of draft performance. Keywords: Ice hockey, NHL, player draft, return on investment 1. Introduction and background why to pick Eberle over Nemisz, as well as other cen- ters that were available with the 22nd pick, including The Edmonton Oilers selected Jordan Eberle, a cen- Daultan Leveille, who has yet to play in the NHL, and ter, with the 22nd pick in the 2008 National Hockey Derek Stepan, who has played more than 400 regular League (NHL) Entry Draft. Three picks later, at 25, the season and playoff games and represented the United Calgary Flames selected another center, Greg Nemisz. States at the 2014 Olympics. This was not the obvious order of things at the time. Sometimes the information teams possess misleads Going into the draft, the NHL’s Central Scouting Ser- them. Infamously, the New York Rangers used the vice (CSS) ranked Eberle as the 33rd best North 12th overall pick in 2003 to select Hugh Jessiman, a American skater while Nemisz ranked 22nd in that right wing from Dartmouth ranked the 20th-best North same category; in other words, the league’s own ama- American skater by CSS. The Rangers preferred Jessi- teur scouts ranked Eberle as a second-round pick at man to a virtual all-star team’s worth of talent (Dustin best, even if you ignored the dozens of North American Brown, ranked the No. 2 North American skater by goaltenders and Europeans available. Since the draft, CSS, along with Brent Seabrook, Zach Parise, Ryan Eberle has played 356 NHL games, while Nemisz has Getzlaf, Brent Burns, Ryan Kesler, Mike Richards and played in 15. The Oilers presumably had additional Corey Perry). Jessiman has since played two NHL information from their internal scouting staff about games, and played seven games in the 2014-15 season with the Capitals, albeit in Vienna, not Washington. ∗Corresponding author: Michael E. Schuckers, St. Lawrence University, 23 Romoda Drive, Canton, NY 13617, USA. Tel.: CSS was formed by former NHL general man- +1 315 229 5794; E-mail: [email protected]. ager Jack Button in 1975 as a service to NHL clubs ISSN 2215-020X/15/$35.00 © 2015 – IOS Press and the authors. All rights reserved This article is published online with Open Access and distributed under the terms of the Creative Commons Attribution Non-Commercial License. 112 M.E. Schuckers and S. Argeris / You can beat the “market”: Estimating the ROI on NHL scouting to scout amateur prospects and later administer the serve teams as “a na´ıve model against which their in- NHL’s annual combine, which invites the top 100 house gunslingers can measure their prowess” as Paul prospects to come to a centralized location for physical Samuelson (1974) wrote of a market index fund and examinations, light drills, and interviews with teams. money managers. The data that we analyze here is from (Malloy, 2011). Other leagues have had some form of the five NHL Entry Drafts from 1998 to 2002. For each centralized scouting as a cost-saving service to their player selected we have their selection number, their member clubs, such as the Major League Baseball position, time on ice (TOI), the games played (GP) Scouting Bureau, begun in 1974, or the National Foot- in the NHL, the ranking by player type by CSS and ball League’s dueling BLESTO and National Football their career goals versus threshold (GVT). GVT is an Scouting organizations (dating to the 1960 s, along attempt to quantity the contribution of a given player with a third, the now-defunct Quadra).1 A significant similar to baseball’s VORP (Value Over Replacement part of CSS’s appeal to the league is the media atten- Player). Teams have the rights to players for at least the tion its rankings generate, as no league has placed such first seven years after they are drafted.2 Consequently, a public emphasis on its scouting service’s rankings we focus our analysis on player performance during as the NHL, which publishes various CSS lists several this period after a player is drafted. Below we use the times per year to great fanfare. The CSS employs nearly total of each metric for the first seven seasons after they 30 scouts in the field, more than any team’s in-house were drafted. Since for all of the players in our sample department, including six European-based scouts via this includes the 2004-5 lockout season, we use eight Goran Stubb and his Finland-based European Scout- total years to capture seven seasons worth of data for ing Services. (Morreale, 2011; Shoalts, 2010). Each each player. These data were compiled from nhl.com, NHL team has available to them information about hockey-reference.com, and eliteprospects.com. We possible draftees from the CSS as well as informa- use the CSS final rankings released prior to the draft. tion compiled by their own staffs. The outcomes of These rankings were obtained via contemporaneous draft decisions like those of Edmonton and Calgary media accounts accessed via Internet searches and discussed above are full of variation—more than half the Lexis/Nexis database. A player’s position was of all players selected in the NHL Entry Draft never recorded as either center (C), defensemen (D), forward play a single game in the NHL. (F), goalie (G), left wing (L) or right wing (R). In the In this paper, we look at the quality of rankings by analyses that follow we categorize players as forwards the CSS and draft order by team so as to evaluate and (F) which includes C’s, L’s, and R’s, defensemen (D) quantify the value of the additional information that or goalies (G). The selection number for a player is the teams have. In doing so, we estimate the average annual place in the draft order that they were selected. That is, return that teams get from their internal scouting. This the 10th player selected will have a selection number is analogous to an approach used in evaluating the of 10, the 100th player selected will have a selection success of portfolio managers in finance—the search number of 100, etc. for “alpha”, or risk-adjusted returns in excess of those CSS ranks players by category, either North Amer- readily available in a market index. (Jensen, 1967). ican or European, and by whether they are a skater or Were the “index”, in this case the freely available a goalie. Thus, CSS produces four separate rankings CSS rankings, comparably successful at picking tal- without a correspondence between them for compar- ent after adjusting for the cost of teams running their ison purposes. To better utilize the CSS rankings, we own scouting departments, it would endorse a more employ Iain Fyffe’s “Central Scouting Integratinator” passive approach to talent evaluation, similar to that of (CESCIN) metric, Fyffe (2011). CESCIN takes the John Bogle when he launched the first index mutual rankings of players by CSS within their given category fund at Vanguard in 1975. If nothing else, the CSS can 2NHL free agency is slightly more complicated, but generally speaking players reach free agency at age 27 or after seven NHL 1We did not contemplate whether teams would benefit from seasons, whichever comes first. The age limit was gradually reduced smaller organizations with greater degrees of exclusivity similar to after the 2004-5 lockout from 31 to 27 for the players in our sample, the NFL, or no organizations at all. (Buchanan, 1965). Theoretically, but they did have collectively bargained options to have some access if NHL teams employ more than 400 scouts, reducing that salary to market forces while under their drafting team’s control such as expense to an enlarged CSS could result in significant economies of salary arbitration and restricted free agency, the latter of which allows scale, though teams would lose the ability to customize their scouting for players to sign with another team in exchange for that team’s picks priorities and reports. in the following NHL draft. M.E. Schuckers and S. Argeris / You can beat the “market”: Estimating the ROI on NHL scouting 113 Table 1 considered alternatives to the 20 minutes for goalies as Statistical Summaries of First Seven TOI, GP and GVT well as weighting minutes for defensemen differently Median Mean 75th percentile Max Std.