Playoff Shot Quality

Examining 2003-04 NHL playoffs shot quality using a regular season model

Ken Krzywicki – November 2005

Abstract

It is often said that playoff hockey is a different brand of hockey. Using the model1 built on the NHL 2003-04 regular season, we wish to see how, if at all, playoff hockey differed from the regular season with regards to shot quality.

Recall that a logistic regression shot quality model was constructed using the 2003-04 regular season play-by-play (PBP) and game summary (GS) files obtained from the National Hockey League’s web site (www.nhl.com). This model was validated against out-of-time data (2002-03 regular season) provided by Alan Ryder and proven stable.

The 2003-04 playoff data was run through this shot quality model and each shot given a probability of being a goal—one minus this being the probability of a save. The model Kolmogorov-Smirnov (KS) statistic2 was 35.05 for the 2003-04 regular season. When examining this statistic for the 2003-04 playoffs, a KS of 36.02 was observed. This small difference, 0.97 points (2.8%), indicates that the model’s ability to discriminate goals from saves is roughly the same for the 2003-04 regular season and the playoffs of that same year.

Model characteristics were analyzed and a population stability index (PSI)3 calculated in order to determine what, if any, shot quality differences between the regular season and playoffs could be found. The PSI was a very low 0.0114, indicating a stable population comparison between the two datasets. In general, the distribution of model variables did not change much from the regular season to playoffs, indicating similarities in shot quality. That is, there was no need to recalibrate or rebuild the original model.

Given these findings, it is reasonable to conclude that the 2003-04 playoffs did not differ from the regular season with respect to shot quality. While actual shooting percentage (and goals per game—5.14 vs. 4.42) was significantly down in the playoffs, the quality of shots taken was not, leading to the conclusion that teams could not capitalize at the same rate on the same shots as in the regular season and that goaltenders

1 See Shot Quality Model – a logistic regression approach to assessing NHL shots on goal [Krzywicki – January 2005]. http://www.hockeyanalytics.com/Research_files/Shot_Quality_Krzywicki.pdf 2 See Glossary for KS definition. 3 See Glossary for characteristics analysis and PSI definitions.

Playoff Shot Quality Ken Krzywicki – November 2005 1 outperformed predictions. Is this attributed to the series format of playing the same team multiple times as opposed to playing against a variety of teams? Are goalies and video coaches recognizing better their opponents’ shooting style? Do goaltenders basically step it up a notch in the playoffs, or is it simply harder to sustain a high save percentage over the course of a long regular season? Then, again, the playoffs come after a grueling regular season where starting goaltenders faced 1,000 shots or more and all primary goalies faced a meaningful number of shots in the playoffs. Combinations of all of these factors probably attributed to the higher playoff save percentage, given the relatively same shot quality as seen in the regular season.

Data

Data from the 89 playoff games was collected from the NHL PBP and GS files that come from the Real Time Scoring System (RTSS) employed by the league. Of the 4,876 shots on goal (392 goals) listed by the NHL, the data files contained 4,816 shots (389 goals). This difference can be accounted for: the PBP file for game number 0134 was empty—a 3-0 Philadelphia victory over New Jersey in the Eastern Conference Quarterfinals. This difference in our data versus the actual number of shots on goal and goals scored is insignificant and did not materially affect the outcome of our results.

Results

First we wish to see if the model built on the 2003-04 regular season data still does a good job of separating goals from saves and rank orders this result well. One way to test this is by comparing the KS statistic from the modeling data to that of the playoffs data. The regular season dataset was divided into approximately 10% (decile) breaks and compared to the playoffs data.

Playoff Shot Quality Ken Krzywicki – November 2005 2 KS Report for 2003-04 Regular Season Data Pred Probability (%) Totals Cuml % Saves Int Rate Cuml % Goals Int Rate Cuml % KS 17.05 - 100.00 7,092 10.28% 5,278 74.42% 8.42% 1,814 25.58% 28.72% 20.30 14.22 - 17.04 6,958 20.36% 5,852 84.10% 17.76% 1,106 15.90% 46.22% 28.47 12.07 - 14.21 6,327 29.53% 5,504 86.99% 26.54% 823 13.01% 59.25% 32.71 8.70 - 12.06 6,894 39.53% 6,129 88.90% 36.32% 765 11.10% 71.36% 35.05 6.13 - 8.69 7,393 50.24% 6,799 91.97% 47.17% 594 8.03% 80.77% 33.60 4.85 - 6.12 5,496 58.21% 5,199 94.60% 55.46% 297 5.40% 85.47% 30.01 3.76 - 4.84 8,534 70.58% 8,141 95.39% 68.45% 393 4.61% 91.69% 23.24 2.93 - 3.75 6,148 79.49% 5,997 97.54% 78.02% 151 2.46% 94.08% 16.06 2.35 - 2.92 7,065 89.73% 6,834 96.73% 88.92% 231 3.27% 97.74% 8.82 0.00 - 2.34 7,087 100.00% 6,944 97.98% 100.00% 143 2.02% 100.00% 0.00 Total 68,994 62,677 90.84% 6,317 9.16% 35.05

KS Report for 2003-04 Playoffs Data Pred Probability (%) Totals Cuml % Saves Int Rate Cuml % Goals Int Rate Cuml % KS 17.05 - 100.00 484 10.05% 370 76.45% 8.36% 114 23.55% 29.31% 20.95 14.22 - 17.04 465 19.71% 397 85.38% 17.33% 68 14.62% 46.79% 29.46 12.07 - 14.21 409 28.20% 357 87.29% 25.39% 52 12.71% 60.15% 34.76 8.70 - 12.06 448 37.50% 409 91.29% 34.63% 39 8.71% 70.18% 35.55 6.13 - 8.69 524 48.38% 480 91.60% 45.47% 44 8.40% 81.49% 36.02 4.85 - 6.12 394 56.56% 371 94.16% 53.85% 23 5.84% 87.40% 33.55 3.76 - 4.84 555 68.09% 540 97.30% 66.05% 15 2.70% 91.26% 25.21 2.93 - 3.75 493 78.32% 478 96.96% 76.85% 15 3.04% 95.12% 18.27 2.35 - 2.92 427 87.19% 418 97.89% 86.29% 9 2.11% 97.43% 11.14 0.00 - 2.34 617 100.00% 607 98.38% 100.00% 10 1.62% 100.00% 0.00 Total 4,816 4,427 91.92% 389 8.08% 36.02

Table 1: KS Comparison – Model Development Data vs. Playoffs Data (incl. Empty Net Goals)

The model development data, 2003-04 regular season, yielded a KS of 35.05 while the playoffs data saw a 36.02 KS. To say that the model still separates goals from saves in a similar manner, we wish the KS difference to be small. Here we see only a 0.97-point difference, or 2.8%. We may confidently say that the model built on 2003-04 regular season data holds muster when used to assess shot quality for the 2003-04 playoffs.

Examining the cumulative distribution of goals identified by score break (approximate deciles), we see that the model identifies approximately the same percentage of goals for both the regular season and the playoffs. This can be seen graphically in Illustration 1 below:

Playoff Shot Quality Ken Krzywicki – November 2005 3

100%

90%

80% d e i f

i 70% t n e d I

s l 60% a o G

f o

t 50% n e c r e P

e 40% v i t a l u

m 30% u C

20%

10%

0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Cumulative Percent of Shots on Goal

Regular Season Playoffs Random

Illustration 1: Lift Curve For Regular Season and Playoffs

The curved lines represent the cumulative goals identified by the model from Table 1. The solid, diagonal line represents a random assignation of the goals, i.e., as if no model were employed.

Another method of assessing the model’s applicability is a fit chart—that is, we wish to see how well the model fits the playoffs data.

Playoff Shot Quality Ken Krzywicki – November 2005 4 30.00

25.00

20.00 t n e

c 15.00 r e P

10.00

5.00

0.00 6 9 2 4 5 2 4 h 4 1 0 6 1 8 7 9 3 g 0 2 i ...... 2 8 6 4 3 2 2 7 4 H ------1 1 1 - 3 5 6 3 5 0 - - 5 0 1 8 7 9 3 0 2 7 0 ...... 7 2 0 . 6 4 3 2 2 0 . . 7 8 4 2 1 1 1 Model Score

AvgPred Actual

Illustration 2: Model Fit Chart – Playoffs Predicted vs. Actual Results by Score Band

As shown above, the average predicted probability of scoring fit nicely with the actual goal rate for each score break.

Since we are satisfied that the model separates playoff goals from saves well enough, and that the model predictions fit the new data, we now wish to see if the model score (predicted probability of a goal) distribution is stable from the regular season to the playoffs. This will also help tell us whether or not the shot quality varied from one time period to the other. A test commonly applied to models is the population stability index, or PSI, which can help answer this. Using the model development data of the 2003-04 regular season as our baseline and the 2003-04 playoffs data as validation, we see the following:

Playoff Shot Quality Ken Krzywicki – November 2005 5 Basline Validation 2003-04 Regular Season 2003-04 Playoffs (B) - (A) (B) / (A) ln(B / A) (B - A) * ln(B/A) Natural Log Pred Percent of Percent of Difference of Probability Number of Total Number of Total Distribution in Proportion Population (%) Records R e c o r d s ( A ) R e c o r ds Records (B) Difference Proportion Diff Divergence 17.05-High 7,092 10.28% 484 10.05% -0.23% 97.77% -0.023 0.0001 14.22-17.04 6,958 10.08% 465 9.66% -0.43% 95.74% -0.044 0.0002 12.07-14.21 6,327 9.17% 409 8.49% -0.68% 92.61% -0.077 0.0005 8.70-12.06 6,894 9.99% 448 9.30% -0.69% 93.10% -0.072 0.0005 6.13-8.69 7,393 10.72% 524 10.88% 0.16% 101.54% 0.015 0.0000 4.85-6.12 5,496 7.97% 394 8.18% 0.22% 102.70% 0.027 0.0001 3.76-4.84 8,534 12.37% 555 11.52% -0.85% 93.17% -0.071 0.0006 2.93-3.75 6,148 8.91% 493 10.24% 1.33% 114.88% 0.139 0.0018 2.35-2.92 7,065 10.24% 427 8.87% -1.37% 86.58% -0.144 0.0020 0.00-2.34 7,087 10.27% 617 12.81% 2.54% 124.72% 0.221 0.0056

Total 68,994 100.00% 4,816 100.00% Population Stability Index 0.0114

Table 2: Population Stability Index – Model Data Compared to 2003-04 Playoffs Data

A few more low-quality shots (0.00 – 2.34 range) were taken during the playoffs (12.81% of the shots vs. 10.27%), but the very low PSI of 0.0114 indicates a stable overall shot quality distribution from one time period to the next.

The fact that the regular season model differentiated goals from saves with roughly the same power (KS test), that the PSI was very low (i.e., the distribution of shot quality was relatively the same for both seasons) and that the predicted probabilities fit the playoffs data well, leads to the conclusion that playoff shot quality did not differ from that of the regular season.

The next item we wish to assess is the model characteristics themselves. Though this is not necessary to support our above conclusion, it is useful to determine which—if any— model characteristics shifted and by how much.

The regular season model contains four characteristics: • Shot distance (in feet) • Shot type • Rebound4 (yes or no) • Situation (even, short or power play)

4 See Glossary for definition of rebound.

Playoff Shot Quality Ken Krzywicki – November 2005 6

When comparing the distribution of the model variables from the 2003-04 regular season, from which the model was developed, to that of the 2003-04 playoffs, we see the following:

2003-04 Regular Season 2003-04 Playoffs

Number in Percent in Number in Percent in Population Population Validation Validation Percent Model Point Weighted Variable Range Standard Standard Sample Sample Difference (A) Values (B) Change (A * B) Shot Distance Less than 10 ft 2,945 4.27% 215 4.46% 0.20% 0.6884 0.0013 10 - 12 ft 4,974 7.21% 343 7.12% -0.09% 0.6374 -0.0006 13 - 14 ft 3,570 5.17% 229 4.75% -0.42% 0.5564 -0.0023 15 - 16 ft 3,040 4.41% 210 4.36% -0.05% 0.5174 -0.0002 17 - 22 ft 7,619 11.04% 498 10.34% -0.70% 0.3654 -0.0026 23 - 31 ft 9,095 13.18% 628 13.04% -0.14% 0.0000 0.0000 32 - 36 ft 4,934 7.15% 330 6.85% -0.30% -0.3805 0.0011 37 - 38 ft 2,150 3.12% 148 3.07% -0.04% -0.4758 0.0002 39 - 44 ft 6,620 9.60% 463 9.61% 0.02% -0.8155 -0.0002 45 - 57 ft 15,078 21.85% 983 20.41% -1.44% -1.0848 0.0157 58 ft or more 8,969 13.00% 769 15.97% 2.97% -1.3824 -0.0410 68,994 100.00% 4,816 100.00% -0.0285

Shot Type Wrap 995 1.44% 78 1.62% 0.18% -0.0742 -0.0001 Slap 25,614 37.12% 1,561 32.41% -4.71% -0.0573 0.0027 Wrist 23,636 34.26% 2,102 43.65% 9.39% 0.0093 0.0009 Snap 10,173 14.74% 466 9.68% -5.07% 0.0130 -0.0007 Backhand 5,051 7.32% 375 7.79% 0.47% 0.0361 0.0002 Tip-In 3,525 5.11% 234 4.86% -0.25% 0.1487 -0.0004 68,994 100.00% 4,816 100.00% 0.0026

Rebound Yes 1,781 2.58% 128 2.66% 0.08% 1.3362 0.0010 No 67,213 97.42% 4,688 97.34% -0.08% 0.0000 0.0000 68,994 100.00% 4,816 100.00% 0.0010

Situation EV 53,624 77.72% 3,792 78.74% 1.01% -0.1244 -0.0013 SH 2,512 3.64% 189 3.92% 0.28% 0.0399 0.0001 PP 12,858 18.64% 835 17.34% -1.30% 0.4007 -0.0052 68,994 100.00% 4,816 100.00% -0.0064

Total Deviation in Logit from Baseline -0.0313

Table 3: Characteristics Analysis – Model Data Compared to Playoffs Data

The total deviation in the logit from the baseline is a small number, -0.0313. The logit is the sum of the model points and comprises the linear portion of the logistic model given by:

1 P(GOAL) = . −ƒpoints 1+ e

Shot distance was distributed relatively the same in the playoffs as the regular season; there was an increase in shots 58 feet or further during the playoffs. Thirteen percent of all shots during the regular season came from this distance, while 15.97% where taken from this distance during the playoffs. This increase can be expected—“shoot the puck and good things will happen” is a common hockey phrase and it would appear that

Playoff Shot Quality Ken Krzywicki – November 2005 7 putting the puck on net from any distance was more prevalent in the playoffs, perhaps hoping for tip-ins and rebound shots, which go in at a higher rate than other shot types. Distance also contributed the most to the change in model score distribution (-0.0285 points).

Shot type distribution varied the most, but this was between slap, wrist and snap shots where the model points where relatively low. That is, even though there was a meaningful change in shot type distribution, it was among those shot types that did not affect the model outcome very much. The greatest change was for wrist shots, which accounted for 34.26% of the shots during the regular season and 43.65% during the playoffs. This difference might be attributed to the fact that, from a distance of 200 feet, these shots look similar to snap shots and may have been reported as such by RTSS. Wrist shots also influenced the model the least of all shot types—this shift in distribution did not materially affect the shot quality from one time period to the other.

The distribution of rebound shots—the most dangerous, according to the model— remained stable from the regular season to the playoffs, where it is more vital to clear rebounds. This may be cause for concern for some teams.

Shots taken during each situation—even strength, short-handed, power play—were distributed roughly the same, though there was some tradeoff between even strength and power play shots on goal. During the regular season, according to the PBP data, there were 14,481 penalties called over the course of 1,230 games (11.77 per game). Over the 88 playoff games for which we have data, 1,030 penalties were called (11.70 per game). Statistically, we fail to reject the hypothesis that the number penalties per game were the same in the 2003-04 regular season as in the playoffs (p ≈ 0.45518). The distribution of penalty type is another matter worthy of study (e.g., fights made up 10.8% of all regular season penalties, while only 3.1% of playoff infractions). However, the defenders “allowed” fewer power play shots against during the playoffs (17.34% vs. 18.64%).

Playoff Shot Quality Ken Krzywicki – November 2005 8 Observations

Given that the regular season model may be safely applied to the 2003-04 playoffs as a legitimate means of classifying shot quality, we may employ it to make some inferences and observations about the playoffs and compare these games to the regular season. Each shot on goal taken during the 2003-04 playoffs was given a predicted probability of being a goal—one minus this being the probability of a save. Tables by team for offense and by goaltender for defense are located in the appendix.

Offense

First let us examine goals per game. During the 2003-04 regular season (1,230 games), an average of 5.14 goals per game (GPG) were scored. This compares to 4.42 for the 88 playoff games for which we have data. Testing the null hypothesis

H 0 : GPGRS − GPGPO = 0 against the alternative hypothesis H A : GPGRS − GPGPO > 0 , we obtained a p-value(p ≈ 0.00038) leading us to reject the null hypothesis. That is, with reasonable certitude, we may conclude that the average goals per game for the regular season and playoffs differed significantly, confirming our intuition.

The actual overall shot percentage during the playoffs was 7.8 compared to a shot quality (i.e., predicted shooting percentage) of 9.05. That is, players took high quality shots, but the goalies were up to the task of stopping them. Colorado had the highest shot quality at 10.8, but fell way short of that mark, scoring at only 8.8 percent. Boston had the lowest shot quality (7.6) and failed to score at even that rate (6.2 actual). Philadelphia and Tampa Bay were the only two teams to outperform their predicted shooting percentages. The Flyers were predicted to score on 9.2 percent of their shots (higher quality than the average of 9.0) and came in at 10.1, or 1.10 times higher than predicted. The Lightning had a lower shot quality than the average (8.8 vs. 9.0), but found a way to the back of the net at a 9.5 percent rate—well over the 7.8 mark set for all playoff teams.

5 Excludes empty net goals.

Playoff Shot Quality Ken Krzywicki – November 2005 9 Also of note are differences in offense from the regular season to the playoffs. Overall regular season shot quality (all 30 teams) was 9.1 and 9.0 in the playoffs. Of the sixteen teams that advanced to the post season, 10 of them had worse shot quality than in the regular season, four had better and two the same. The New York Islanders’ actual playoff shot quality was similar to that of their regular season (8.9 vs. 8.8), however they scored but four goals on 141 shots during the playoffs for a paltry 2.8 shot percentage, compared to 9.7 during the regular season. What caused such a dramatic change in capitalizing on roughly the same quality shots? Could it be that during the regular season, the Islanders faced a variety of goalies and defensive systems, while these were limited to those of the Tampa Bay Lightning, whose goaltender outplayed predictions, during the playoffs?

Defense

Examining the flip side of the game—goalie save percentage—we note an average playoff predicted value of .910. Most teams employed only one goaltender for a majority of the time, with the exception of Vancouver. Manny Legace, Detroit’s backup, also faced a significant number of shots (84) and, while tougher than average, stopped them at exactly the rate expected (.905).

Of the two goalies that went to the Finals, Nikolai Khabibulin faced tougher shots. Khabibulin’s predicted save percentage was .909, slightly tougher than the overall average, while Miikka Kiprusoff was expected to stop shots at a .916 rate (much easier than the overall .910 value). Both goaltenders outperformed expectations—Kiprusoff finished with a .928 save percentage and Khabibulin .933.

Next we compare predicted save percentages from the regular season6 to the playoffs. Of the 27 goalies used throughout the post season, nine faced easier shots than in the regular season, 16 tougher and two the same. The Cup-winning goaltender, Nikolai Khabibulin, had a regular season predicted save percentage of .913 (which, at .910, he failed to meet), and faced tougher shots in the playoffs (predicted .909). While the Lightning may have “allowed” tougher shots in the playoffs (compared to the regular

6 All goalies regular season predicted SV% = .909. See Table 4 for summary and Appendix page iv for details.

Playoff Shot Quality Ken Krzywicki – November 2005 10 season) when Khabibulin played, he significantly outperformed predictions, coming in at .933. The defense for Calgary, the other finalist, “allowed” tougher shots during the regular season when Kiprusoff was in net (.912) and appeared to tighten up during the playoffs, seeing shots at the .916 predicted save level. Kiprusoff outperformed predictions in both seasons.

When looking at playoff and non-playoff goalies during the regular season, we note that the playoff goalies not only faced “easier” shots than their non-playoff counterparts, but they performed better than expected, perhaps one of many reasons their teams made the playoffs.

2003-04 Regular Season 2003-04 Playoffs Actual Pred Actual Pred SA SA SV% SV% SV% SV%

Playoff Goalies 28,598 .915 .910 4,801 .922 .910

Non-Playoff 40,208 .908 .908 Goalies All Goalies 68,806 .911 .909

Table 4: Playoff and Non-Playoff Goalies Performance

We also note that, for the playoff goaltenders, predicted save percentage from the regular season to the playoffs was the same, namely .910. And while they outperformed expectations during the regular season, they truly stepped it up a notch in the playoffs.

Conclusions

After confirming that the model built on 2003-04 regular season data was appropriate for the playoffs of the same season, we noted that, on the whole, shot quality remained consistent in the playoffs. These assertions were supported via a KS comparison, data fit chart, population stability index and characteristics analysis. We also saw that regular season play did not necessarily transfer to the playoffs—witness the New York Islanders.

Playoff Shot Quality Ken Krzywicki – November 2005 11

Actual scoring was indeed lower in the playoffs compared to the regular season, but not due to shot quality, which was roughly the same. Though shot quality was generally the same, implying that team defense was as well, goalies had a much higher playoff save percentage compared to the regular season.

Although the playoffs are more meaningful and generally more exciting, the game unfolded relatively consistently with the regular season regarding shots on goal, which, after all, are the only way to score.

Playoff Shot Quality Ken Krzywicki – November 2005 12 Actual vs. Predicted Shooting Percentages & Shot Quality 2003-04 Playoffs

Actual

Pred Shot Index TEAM SF GF SHOT% SHOT% Quality (Actual/Pred) TOTAL 4,801 374 7.8 9.0 86.67%

BOS 225 14 6.2 7.6 Lower 81.58% CGY 696 58 8.3 8.9 Lower 93.26% COL 284 25 8.8 10.8 Higher 81.48% DAL 156 10 6.4 7.8 Lower 82.05% DET 384 23 6.0 8.0 Lower 75.00% MTL 317 21 6.6 9.4 Higher 70.21% NAS 128 9 7.0 8.6 Lower 81.40% NJD 120 9 7.5 8.3 Lower 90.36% NYI 141 4 2.8 8.9 Lower 31.46% OTT 238 11 4.6 9.7 Higher 47.42% PHI 456 46 10.1 9.2 Higher 109.78% SJS 413 36 8.7 9.9 Higher 87.88% STL 127 8 6.3 8.7 Lower 72.41% TBL 599 57 9.5 8.8 Lower 107.95% TOR 311 27 8.7 9.0 Average 96.67% VAN 206 16 7.8 8.4 Lower 92.86%

• Excludes empty net goals.

• Shot Quality column compares predicted shot percentage for each team to the predicted shot percentage for all playoff teams.

Playoff Shot Quality Ken Krzywicki – November 2005 i Playoffs vs. Regular Season Shot Quality (Shot Percentage by Team) 2003-04 Season

2003-04 Regular Season 2003-04 Playoffs Playoffs vs. Actual Pred Actual Pred Regular TEAM SF GF SHOT% SHOT% SF GF SHOT% SHOT% Season TOTAL 68,806 6,129 8.9 9.1 4,801 374 7.8 9.0 Lower

BOS 2,494 203 8.1 8.1 225 14 6.2 7.6 Lower CGY 2,239 191 8.5 9.0 696 58 8.3 8.9 Lower COL 2,405 230 9.6 9.9 284 25 8.8 10.8 Higher DAL 2,192 189 8.6 8.9 156 10 6.4 7.8 Lower DET 2,475 247 10.0 8.4 384 23 6.0 8.0 Lower MTL 2,255 198 8.8 9.0 317 21 6.6 9.4 Higher NAS 2,223 212 9.5 9.3 128 9 7.0 8.6 Lower NJD 2,428 208 8.6 8.4 120 9 7.5 8.3 Lower NYI 2,329 225 9.7 8.8 141 4 2.8 8.9 Higher OTT 2,409 250 10.4 9.3 238 11 4.6 9.7 Higher PHI 2,401 222 9.2 9.2 456 46 10.1 9.2 Same SJS 2,294 213 9.3 10.1 413 36 8.7 9.9 Lower STL 2,214 182 8.2 9.4 127 8 6.3 8.7 Lower TBL 2,450 235 9.6 9.0 599 57 9.5 8.8 Lower TOR 2,242 231 10.3 9.0 311 27 8.7 9.0 Same VAN 2,375 231 9.7 9.3 206 16 7.8 8.4 Lower

• Excludes empty net goals.

• Playoffs vs. Regular Season column compares playoff predicted shot percentage to that of the regular season.

Playoff Shot Quality Ken Krzywicki – November 2005 ii Actual vs. Predicted Save Percentages & Shot Quality 2003-04 Playoffs

Actuals Shot Index GOALIE TEAM SA GA SV% Pred SV% Quality (Actual/Pred) TOTAL 4,801 374 .922 .910 101.32%

RAYCROFT BOS 210 16 .924 .907 Tougher 101.87% KIPRUSOFF CGY 710 51 .928 .916 Easier 101.31% TUREK CGY 3 0 1.000 .912 Easier 109.65% AEBISCHER COL 295 23 .922 .912 Easier 101.10% SALO COL 7 0 1.000 .892 Tougher 112.11% TURCO DAL 119 18 .849 .883 Tougher 96.15% JOSEPH DET 197 12 .939 .918 Easier 102.29% LEGACE DET 84 8 .905 .905 Tougher 100.00% GARON MTL 6 0 1.000 .897 Tougher 111.48% THEODORE MTL 333 27 .919 .918 Easier 100.11% VOKOUN NAS 197 12 .939 .916 Easier 102.51% BRODEUR NJD 108 10 .907 .907 Tougher 100.00% DIPIETRO NYI 120 11 .908 .910 Average 99.78% LALIME OTT 139 13 .906 .908 Tougher 99.78% PRUSEK OTT 15 1 .933 .943 Easier 98.94% BURKE PHI 9 1 .889 .936 Easier 94.98% ESCHE PHI 463 41 .911 .914 Easier 99.67% NABOKOV SJS 461 30 .935 .910 Average 102.75% DIVIS STL 8 0 1.000 .823 Tougher 121.51% OSGOOD STL 109 12 .890 .898 Tougher 99.11% GRAHAME TBL 17 2 .882 .894 Tougher 98.66% KHABIBULIN TBL 598 40 .933 .909 Tougher 102.64% BELFOUR TOR 379 27 .929 .905 Tougher 102.65% KIDD TOR 11 1 .909 .903 Tougher 100.66% AULD VAN 88 9 .898 .891 Tougher 100.79% CLOUTIER VAN 64 5 .922 .905 Tougher 101.88% HEDBERG VAN 51 4 .922 .922 Easier 100.00%

• Excludes empty net goals.

• Shot Quality column compares predicted shot percentage for each team to the predicted shot percentage for all playoff teams.

Playoff Shot Quality Ken Krzywicki – November 2005 iii Playoffs vs. Regular Season Shot Quality (Save Percentage by Goalie) 2003-04 Season

2003-04 Regular Season 2003-04 Playoffs Playoffs vs. Actual Pred Actual Pred Regular GOALIE TEAM SA SV% SV% SA SV% SV% Season TOTAL 68,806 .911 .909 4,801 .922 .910 Easier

RAYCROFT BOS 1,586 .926 .912 210 .924 .907 Tougher KIPRUSOFF CGY 966 .933 .912 710 .928 .916 Easier TUREK CGY 463 .914 .913 3 1.000 .912 Tougher AEBISCHER COL 1,703 .924 .909 295 .922 .912 Easier SALO COL 136 .912 .917 7 1.000 .892 Tougher TURCO DAL 1,646 .913 .910 119 .849 .883 Tougher JOSEPH DET 744 .909 .904 197 .939 .918 Easier LEGACE DET 1,019 .920 .912 84 .905 .905 Tougher GARON MTL 480 .921 .918 6 1.000 .897 Tougher THEODORE MTL 1,853 .919 .909 333 .919 .918 Easier VOKOUN NAS 1,958 .909 .911 197 .939 .916 Easier BRODEUR NJD 1,845 .917 .915 108 .907 .907 Tougher DIPIETRO NYI 1,261 .911 .912 120 .908 .910 Tougher LALIME OTT 1,334 .905 .908 139 .906 .908 Same PRUSEK OTT 648 .917 .907 15 .933 .943 Easier BURKE PHI 389 .910 .901 9 .889 .936 Easier ESCHE PHI 932 .915 .919 463 .911 .914 Tougher NABOKOV SJS 1,610 .921 .910 461 .935 .910 Same DIVIS STL 291 .900 .892 8 1.000 .823 Tougher OSGOOD STL 1,604 .910 .906 109 .890 .898 Tougher GRAHAME TBL 664 .913 .912 17 .882 .894 Tougher KHABIBULIN TBL 1,414 .910 .913 598 .933 .909 Tougher BELFOUR TOR 1,483 .918 .910 379 .929 .905 Tougher KIDD TOR 388 .876 .895 11 .909 .903 Easier AULD VAN 168 .929 .910 88 .898 .891 Tougher CLOUTIER VAN 1,554 .914 .909 64 .922 .905 Tougher HEDBERG VAN 459 .900 .910 51 .922 .922 Easier

• Excludes empty net goals.

• Playoffs vs. Regular Season column compares playoff predicted shot percentage to that of the regular season

Playoff Shot Quality Ken Krzywicki – November 2005 iv Stanley Cup Playoffs

Western Conference Eastern Conference

DET (1) TBL (1) DET (1) TBL (1) NAS (8) 4 - 2 4 - 1 NYI (8)

CGY (6) TBL (1) VAN (3) 4 - 2 4 - 0 BOS (2) CGY (6) MON (7) CGY (6) 4 - 3 4 - 3 MON (7)

CGY (6) TBL (1) SJS (2) 4 - 2 4 - 3 PHI (3) SJS (2) PHI (3) STL (7) 4 - 1 4 - 1 NJD (6)

SJS (2) PHI (3) COL (4) 4 - 2 4 - 2 TOR (4) COL (4) TOR (4) DAL (5) 4 - 1 TBL (1) 4 - 3 OTT (5) 4 - 3

Illustration 3: 2003-04 Playoffs Results

Playoff Shot Quality Ken Krzywicki – November 2005 v Glossary

• Author Contact: [email protected]

• Original Paper: Shot Quality Model – a logistic regression approach to assessing NHL shots on goal [Krzywicki – January 2005] can be found at Alan Ryder’s web site: http://www.hockeyanalytics.com/Research_files/Shot_Quality_Krzywicki.pdf

• Rebound Shot: A rebound was a shot within two seconds of another shot with no intervening “event” with a distance less than 25 feet. [Ryder]

• Kolmogorov-Smirnov (KS) Report: This is a measure of the model’s effectiveness in rank ordering performance (e.g., goal versus save). The KS statistic is calculated by subtracting the cumulative distribution of one performance category (goal) from another (save). The point where maximum separation occurs is the KS spread. It is desirable to maximize this separation – the higher the KS, the more predictive the score. As a scorecard, or model, matures, the KS statistic can be expected to degrade over time. The KS Report will compare the KS statistic from the development (baseline) population to that of another time period (out-of-time validation).

• Population Stability Index: The population stability index evaluates changes in the scored population over time. This index measures the separation (divergence) between score distributions from time of development and those from a validation period. Over time, changes in the population may result in corresponding changes in scores. A smaller index value is more indicative of a stable population, suggesting that a validation population is similar to the baseline standard population used in scorecard development.

n »≈ ’ ≈ À ¤’ÿ ∆ vi bi ÷ ∆ vi bi ÷ Index = ƒ …∆ − ÷ × ∆lnà ÷ ‹÷Ÿ [E. M. Lewis] i=i …« V B ◊ « ÕV B ›◊⁄Ÿ

where vi is the count of validation records in the i-th score band, V is the total count of the validation records, bi is the count of baseline records in the i-th score band and B is the total count of baseline records. As a rule of thumb, PSI less than or equal to 0.100 indicate similar populations.

• Characteristics Analysis: A characteristics analysis is a comparison of the statistical distribution of counts or percentages of the attributes in the validation population with those in the sample that were originally used to develop the model. Characteristics analysis enables one to identify the cause(s) for any significant changes that a population stability index may detect. From this analysis, the characteristics responsible for the shift in scores can be identified and the contribution to the overall change in the final score can be computed.

Playoff Shot Quality Ken Krzywicki – November 2005 vi