A Wins-Added Salary Evaluation for Pitchers

Do Teams Pay for Wins? A Wins-Added Salary Evaluation for Pitchers

Jared Forbus Advisor: Richard Ball Spring 2010

1. Introduction

According to Michael M. Lewis’s Moneyball, in which Lewis examines how the Oakland

Athletics manage to win so many baseball games despite extremely tight budget constraints, established closers are “systematically overpriced, in large part because of the statistic by which closers [are] judged in the marketplace: ‘saves’”. He contends that teams could “take a slightly above average pitcher and drop him into the closer’s role,” without giving up a significant number of wins (Lewis, 2003). This particular thrust of Lewis’s broader argument that inefficiencies in the baseball labor market allow teams with small budgets to compete with teams with large budgets sticks out because it is the only example he uses that specifically identifies an exploitable inefficiency.

One possible reason behind inefficiency in the labor market for closers is that teams value closers’ contributions incorrectly. One way to examine this possibility is to use a category of baseball statistics called wins added statistics, which represent the number of wins a player contributes to his team above an average replacement player. As opposed to most baseball statistics, wins added statistics represent the effect of a player’s contribution rather than specific aspects of it. They are therefore capable of standing alone as indicators of performance, and so are extremely useful for measuring a player’s value. This paper will use WAD, a measure derived from two wins added statistics—SNVAR for starters, and WXRL for relievers—by using only the value associated with a pitcher’s role for the season in question. These raw statistics, SNVAR and WXRL, are explained in detail below.

By separately examining three types of pitcher—starter, reliever, and closer—this paper seeks to identify an inefficiency in Major League Baseball’s labor market by determining a wins added salary equation for each. I expect to find that teams undervalue wins added statistics in 2 closers’ salaries. I expect this result because of baseball’s over-emphasis on saves as a measure of closers’ contributions to their teams, and I accordingly predict a positive relationship between saves and real salary.

Baseball’s labor market is a popular topic among economists, and there are many articles that deal with player salary and value. This paper differs from most in its focus on wins added statistics; while the more traditional statistics are also examined, they are not the focus, as they are in most other studies. There are also many articles written by baseball analysts that use wins added, but rarely in the context of the labor market, and rarely using data from many years and teams. In a way this paper is an attempt to bridge the divide between economists and baseball analysts with the aim of making sense of a game that often defies explanation.

My analysis shows that teams use wins added statistics inconsistently when evaluating the value of pitchers in the labor market. While wins added has a positive relationship with real salary for starting pitchers, indicating that pitchers who contribute more wins are paid more, closers seem to be valued in the opposite manner. Furthermore, these relationships both decrease over time, signaling that teams increasingly undervalue WAD in both starters and closers. This finding indicates that despite Moneyball’s identification of an inefficiency, the baseball market has not yet corrected itself.

2. Literature Review

Since Moneyball’s publication in 2003 there has been a flurry of research done on possible inefficiencies in baseball’s labor market. Most of this research, though, deals not with pitching but with hitting; Lewis argues that players’ ability to get on base, measured with on- base percentage, was undervalued by the market. This hypothesis has been supported using statistical analysis (Hakes and Sauer, 2006). The paper, using a variety of offensive baseball 3 statistics, shows that before Moneyball’s publication, teams underestimated the importance of getting on base to winning games, and therefore paid players with high slugging percentage too much, underpaying players with high on base percentages. However, they concluded that by

2004, the year after Moneyball, this inefficiency in the market had been corrected. This finding illustrates that baseball’s labor market is capable of quickly adjusting to inefficiencies, once they are pointed out.

This particular test of the hypothesis presented in Moneyball is, though, the only direct response to Lewis’s book. The analysis of the pitching market is very scattered, yet several ideas emerge from an overview of existing work: there are three types of pitcher, and each type must be evaluated separately due to the differences of their roles; pitchers need to be evaluated independently of their teammates because fielding effects balls that are put in play, altering the result of a pitcher’s performance; and comparison with an average pitcher is the most effective way of measuring contribution to winning.

Importantly for the goal of this paper, it has been shown that teams value different qualities for different types of pitcher (Krautmann, et. al., 2003). Using a sample of 180 pitchers, all of whom were potential free agents between the 1990 and 1994 seasons, this study ran a partial logarithmic regression and found that the three classifications of pitcher have very different salary equations based on an array of performance statistics and salary. Whereas it might make sense that all pitchers, regardless of their role, would be rewarded primarily for their ability to record outs and prevent runs, the authors found that the emphasis of the salary process is very different for the three types of pitcher. He finds, for example, that innings pitched is positively related to real salary, but only significant for closers, and that ERA does not affect the salaries of long relievers. Most interestingly, he finds that strikeouts per innings pitched are twice 4

as important for long relievers as they are for starters, and aren’t important for closers

(Krautmann, et. al., 2003). These differences are significant enough to warrant separate analysis

of each type of pitcher.

This finding is supported in a paper that attempts to determine how teams value starters

(Bradbury, 2007). He argues that teams have different valuation schemes for starters and

relievers, because starters pitch many more innings than relievers, and so it is more useful to

value them using statistics that might be volatile when measured for only a few appearances.

Bradbury’s analysis first finds that a pitcher’s ability to prevent runs without the help of his

teammates is a better gauge of ability than ERA, which depends on what his teammates do with

balls put in play. Though ERA does not penalize pitchers for runs scored due to errors, it can still

vary with the quality of defense: fast outfielders can catch more balls than slow ones, for

example, but a ball the defender can’t get to is not counted as an error. The pitcher is therefore liable for his team’s defense. While this finding seems evident, it is important because it opens

the door for the use of statistics that capture real contribution rather than snapshots of specific

qualities.

These findings all support Lackritz’s (1990) assertion that players should be compared

with an average player in the league when determining their value to their team. Lackritz

regresses team winning percentage with hitting, pitching, and fielding statistics to determine the

relative worth of each facet of the game, then compares each player’s statistics with the league average to determine contribution to winning percentage above the average. Based on this relationship, and a relationship he establishes between winning percentage and revenue, he discusses player value in the context of the free agent market, and finds that many players are over-valued. While his method is somewhat roundabout, it uses many of the same principles this 5 paper will use, notably comparison of a player to a league-average player, and the relationship between that comparison and the player’s salary.

Analysts have developed a wide range of tools that can be used to measure a pitcher’s contribution to his team more accurately than is possible using recorded statistics, such as strikeouts and earned run average. Support Neutral Value Above Replacement (SNVAR) and

Win Expectation above Replacement Level (WXRL) are two such statistics. SNVAR and

WXRL were developed separately, but are derived from the same concept, which is that a pitcher has an effect on his team’s likelihood of winning a game in which he appears. These statistics tell the number of wins in the standings a pitcher earns his team, above an average replacement pitcher, a comparison described in detail below. Very good pitchers will have high, positive values, while below average pitcher will have negative values. These statistics are derived as follows.

Wolverton (1993) invented SNVAR to measure the quality of a pitcher’s start. Such a statistic, he says, should depend only on numbers in the game’s box score, and be independent of the pitcher’s support and ballpark. He first maps an algorithm for measuring support-neutral wins and losses, and then transforms them into SNVAR, or “Support-Neutral Value Above

Replacement.” In defining support-neutral wins, he sums among all a pitcher’s starts the product of the probability that his team scores “r” runs in “i” innings and the probability that an average team will hold a “k” run lead for the “i” remaining innings of the game. This product equals the probability that a starter who lasts “i” innings and gives up “r” runs will get a win, assuming an average team is playing behind him.

This method yields a reliable metric for calculating a pitcher’s “fair” win-loss record, but is not particularly useful because a win-loss record is not something that can be used by itself. 6

Wolverton therefore takes his analysis a step further, by evaluating how much the starter’s outing changed his team’s probability of winning the game, which he assumes to be 50% at the beginning. Calculating this metric is very similar to the support-neutral wins metric, with the major difference being that the product being summed is the probability of scoring “r” runs in “i” innings multiplied by the probability a team will win a game with “i” innings left and a given difference between the scores.

The probabilities used to calculate these statistics are determined by what percent of such games were won in the season in question. For example, if a starter leaves after the sixth inning with a 3-0 lead, the probability that his team wins the game is simply the percent of teams that won in that situation that year. If teams won 80% of those games, his value for that game is .3 because he increased his team’s probability of winning by 30%. Thus, SNVAR measures the total change in a team’s probability of winning the games that a pitcher starts, capturing their contribution to wins above a replacement-level pitcher. This statistic is useful on its own because it is measured in games in the standings.

Woolner (2005) invented WXRL as an equivalent statistic to SNVAR, that could be used to evaluate relievers’ contributions to their team’s wins. Like SNVAR, WXRL sums the change in the probability of winning across all a reliever’s appearances. The main difference between the two, then, is that while starters begin their outings with an assumed probability of winning of

0.5, relievers can enter a game in any situation, meaning that they can begin their outing with nearly any starting value.

These two statistics are the focus of much analysis regarding the relationship between them and traditional pitching statistics. Wolverton (2004) discusses the merits of SNVAR in an article that questions the validity of wins and losses as measures of a pitcher’s performance. He 7

argues that these numbers depend too much on the pitcher’s teammates: a starter can lose a game

in which he pitches well and gives up only one run, if his team scores none. Using SNVAR as a

determinant eliminates this concern because he is not punished for his team’s inability to support

him with runs. Furthermore, because SNVAR is based on probability, not on same-game results,

the pitcher in the above situation will benefit from the fact that he gave up only one run, whereas

in a simple win-loss evaluation it does not matter how he pitched in each performance.

Finally, Click (2005) uses a comparison between WXRL and saves to determine whether closers are properly valued. Though he does not use salary data, his findings indicate a gross

misevaluation of closers: in the year the article was written, three closers had negative WXRL,

meaning they cost their team games compared to the average replacement described above.

Furthermore, eleven teams’ closers did not lead their team in WXRL, an alarming figure given

the situations closers generally pitch in (close games in late innings give more opportunity to get

high WXRL than any other relief situation). These bits of information further validate concerns

that closers are not evaluated in the same way as other pitchers, and that this difference in

evaluation results in teams paying their closers too much.

The literature that deals with baseball’s labor market and wins added statistics forms a

picture that indicates a market inefficiency: the economic research points to the superiority of

player analysis using comparisons to average level replacements, while the analysis of baseball

statisticians shows that teams do not use such data to evaluate their closers. This paper will

attempt to bridge the gap between the two groups.

3. Data

My data for this project came from two sources: Baseball Prospectus’s online statistical

database and the Society for American Baseball Research’s (SABR) player salary database. 8

SABR’s data only covers player salaries from the 1985 season through the 2009 season, whereas

Baseball Prospectus’s database includes data from every season with recorded statistics, so I

downloaded individual season data to correspond with the SABR dataset. SABR’s salary figures

do not adjust salary for inflation, so I converted to real salary it using CPI data from the Bureau

of Labor Statistics.

The focus of my analysis is the relationship between WAD and LNRSALARY, but my

data include performance statistics and control variables as well. The performance statistics

include: starts; relief appearances; ERA; saves; wins; and hits, walks, strikeouts, and home runs

allowed per nine innings. The control variables include: year, team, experience, pitcher type, age,

age squared, and a dummy for rookies. Each row of my data represents a pitcher-year, or all the

statistics a pitcher compiled during a year.

Looking at an overview of my full dataset, I found that there were entries that had statistical data but no salary data. The most likely explanation for these entries is that teams call

players up from the minor leagues all the time but send them down before the end of the season.

Players like this receive Minor League rather than Major League compensation, and so would

not be recorded in official Major League salary data. After eliminating most of the discrepancies

by dropping pitchers who appeared in fewer in ten games or pitched fewer than fifteen innings,

most of the problems disappeared, so I dropped the remaining observations without an entry for salary.

I used the following criteria to determine in what category to place each pitcher: if a player appears in more games in relief than as a starter, they should be considered a reliever, and though relievers who are not their team’s closer sometimes get saves, only closers get more than five in a season. Closers are therefore relievers with more than five saves. 9

After incorporating these adjustments I have a data set with 8128 observations. Of these,

3,533 are starters, 3,514 are relievers, and 1,081 are closers, giving me a large enough sample to look at the relationship between RSALARY and WAD. Among starters, the mean value for

WAD is 2.49, with a minimum of -2 and a maximum of 11.2; among relievers, the mean value of

WAD is .53, with a minimum of -2.856 and a maximum of 6.875; finally, among closers the mean value of WAD is 2.35 with a minimum of -3.257 and a maximum of 9.198. Meanwhile, the mean RSALARY is $1.3 million for starters, $0.47 million for relievers, and $1.1 million for closers (Figure 1).

The mean value of EXPERIENCE is 4.95, and the maximum is 25. Experience is an important variable to control for because of the structure of baseball’s collective bargaining agreement; players are eligible to be free agents only after their sixth season in the league. After their third season they can file for salary arbitration with their team, often resulting in pay increases, but this process is not equivalent to a true valuation because the player can only negotiate with one team. Thus, after six years in the league it can be assumed that a player’s salary will reflect the player’s value on the free agent market, and thus represents teams’ valuation of his services. I therefore expect salaries to dramatically increase starting when

EXPERIENCE equals 6. The relationship between experience and salary fits these expectations: there is a significant jump in mean salary when EXPERIENCE equals five, with steady increases until EXPERIENCE equals 10 (Figure 2).

One possible issue with an analysis of salary using WAD is that pitchers with the most innings pitched have the most opportunities to accumulate wins. For starters there does seem to be a positive relationship between innings pitched and WAD (Figure 3). For relievers (Figure 4) and closers (Figure 5), though, that relationship is less significant and not clearly positive. For 10

closers especially there seems to be a fairly narrow window of innings pitched, with WAD values not varying much within that window. This suggests that while starters who pitch a lot might have higher WAD than their counterparts, relievers and closers with the heaviest workloads do not have an advantage over others regarding WAD.

Finally I wanted to get an idea about how the performance statistics I use relate to wins added. Figures 6 through 10 depict these relationships, which are mostly intuitive. Giving up hits, walks, and homeruns, and having a high earned run average, is bad for pitchers and is visible from the graphs, which show the relationship between each performance variable and

WAD, by PITCHERTYPE. Each of the variables, H9, BB9, HR9, and ERA, have general downward patterns for each category of pitcher, though this trend seems to be less pronounced for closers than for other pitchers. Similarly, SO9 has a general upward trend for each type of pitcher, reflecting the fact that striking batters out is the most effective way to prevent runs from scoring. These patterns suggest a correlation between these performance statistics and WAD, a suggestion supported by Table 1.

Figures 11 and 12 look at results statistics for closers and starters, respectively. In Figure

11, a plot of WAD and SV, or saves, there appears to be a general upward pattern, but most of the observations hover around the cutoff of five saves. Furthermore, most of the observed upward slope is between WAD values of 0 and 3, suggesting more that these values are not related to saves than that there is a meaningful positive relationship. In Figure 12 there is a strong positive association between WAD and W, showing that the two measure fairly comparable aspects of the pitchers’ performance. Because of this comparability, I do not use W in my analysis.

4. Analysis and Results

With a clear grasp of patterns within the data and the relationships between my

performance data and WAD, I move on to my analysis of the relationship between WAD and

salary. To get an initial grasp of this relationship, I broke down LNRSAL into ten subsections to

get an idea of how higher salaried players perform relative to other players. I then created a box

plot showing WAD and each subsection of LNRSALARY for each type of pitcher (Figure 13).

For starters, there is a positive, increasing relationship between WAD and LNRSALARY, shown

in the first panel. The same is true of relievers in the second panel, though the increases are less

drastic. What stands out, though, is the lack of a trend in the third panel, indicating that higher- paid closers do not have higher WAD.

This pattern is further shown by looking at a bar graph of average WAD for each subsection of RSALARY, for each type of pitcher (Figure 14). For starters and relievers, again, there is a positive trend, though the highest-paid relievers have lower average wins added. For closers, there is only a positive trend beginning in the top four RSALARY brackets, and only the highest two have higher average WAD than the fourth subsection.

To illustrate these results quantitatively, the mean of VALUE is positive for starters and relievers, but negative for closers (Table 2). The mean for closers, in fact, is nearly the opposite of the value for starters; on average, teams pay closers the same amount for every fewer game they win that they pay starters for their wins. This effect suggests a strong misevaluation of

closers’ worth. In order to more precisely evaluate teams’ valuation of pitchers, I do regression

analysis on LNRSALARY and WAD, employing five models, labeled A, B, C, D, and E. Each

model has three regressions, for each value of PITCHERTYPE. 12

For these models I dropped most performance variables because of high correlation with

WAD (Table 1). Thus, my simplest model, model A, uses as independent variables WAD,

ROOKIE, AGE, AGE2, EXPERIENCE, and YEAR, with SV included for closers only. In this

model the coefficient of WAD is significant for starters and closers, and all other coefficients are

significant on all three pitcher types. The high R-squared values for this model indicate that it has significant explanatory power. Predictably, ROOKIE has a negative coefficient, reflecting the structure of baseball’s bargaining agreement that pays rookies minor league salaries.

Experience has a high positive coefficient, probably reflecting the jump in salary once a player becomes a free agent. Year, too, has a positive coefficient, reflecting annual increases in real salary. The coefficient on WAD for starters indicates that an extra win implies a 10.6% increase in real salary, all else constant, while for closers an extra win results in a 7.98% decrease in real salary. The coefficient on WAD for relievers was not significant, and remained insignificant in subsequent models, likely a result of the many different roles those pitchers perform and the resulting unreliability of a metric that is influenced strongly by how much a player pitches. For closers, saves has a positive coefficient that implies five extra saves raises real salary by thirteen percent (Table 3).

This preliminary model shows a positive relationship between WAD and RSALARY for starters, but a negative one for closers. This result means that teams are essentially getting better value out of their closers than their starters; closers with high WAD have low salaries, implying that teams that pay their closers high salaries pay more per extra win than other teams. In the remaining models, I control for combinations of team and year, and ultimately introduce an interaction term for WAD and year, in an attempt to determine whether the difference in the

WAD coefficients for starters and closers is due to some factor other than the type of pitcher. 13

Controlling for team alone slightly reduced coefficients on WAD for both starters and

closers, but the sign of the effects was unchanged along with the significance (Table 4).

Controlling for year alone left starters’ WAD coefficient unchanged while slightly increasing the

coefficient for closers, again without changing the sign or significance (Table 5). Including both

team and year controls produced a coefficient for starters equivalent to the result in model B and

a coefficient for closers slightly higher than the coefficient in model B. Model D has higher R-

squared values than the previous models, so while the coefficients are not very different, they are

more reliable (Table 6).

Therefore, it is model D to which I add an interaction term between WAD and YEAR, in

an effort to control the coefficient on WAD for the yearly increases in RSALARY (Table 7). The

resulting coefficients on WAD and YR_WAD are significant when considered together (Table

8). Because in this model the coefficient on WAD varies with year, there is a range of values

rather than a single value:

LNRSALARY=β0 + [βWAD + βYR_WAD(YEAR)] [WAD]

Thus, the range for starters is all positive, and the range for closers is negative (Table 9); and both decrease as YEAR increases. In model E the coefficients on ROOKIE, EXPERIENCE, and SV are the same as for the previous models, and significant, and the R-squared is the same as for model D. Model E therefore has as much or more explanatory power as the previous models while more accurately representing the relationship between LNSALARY and WAD. It is therefore this model that I use to examine the pricing formulas for Major League pitchers.

The pricing equation for pitchers derived in model E shows very different pictures for starters and closers. Using the coefficients of WAD and YR_WAD from model E to derive a

year-dependent coefficient for WAD, we see that in 2009 each additional WAD increased

starters’ salaries 5.72% but decreased closers’ salaries 10.95% (Table 9). This result for closers 14

supports the hypothesis of this paper because it means that closers with the highest WAD are

paid the least. Teams that pay closers high salaries are, on average, reducing their win totals relative to what other, cheaper, closers could produce.

Furthermore, these coefficients for 2009 are sharp decreases from the coefficients for

1985. For starters, they make 7% less per extra win in 2009 than in 1985; closers make 3% less.

The negative trend indicates that for starters, teams decreasingly reward WAD, and for closers, teams increasingly undervalue WAD while consistently paying a 2.79% higher salary for each

additional save. This steady decline of the year-dependent coefficient on WAD is especially

jarring given that they are paying starters and closers 6.46% and 4.54% more per year. So, a

pitcher with consistently high WAD sees his salary increase slower than the salaries of pitchers

who contribute fewer wins to their team.

5. Conclusion

These results support Lewis’s hypothesis in Moneyball, that there is an exploitable

inefficiency in the market for closers, and indicate that while some teams are exploiting that

inefficiency, many are either unaware of it or do not care. From the coefficient on saves it is

clear that teams value them very highly; one theory as to why is that teams desire a sense of

security at the end of games, a feeling that saves are strongly associated with. However, though a

team may feel more secure with an expensive closer in its bullpen, this analysis shows that they

would do better over the long haul with an inexpensive closer, if everything else remained

constant. In addition to supporting Lewis, these conclusions support the hypothesis of this paper,

that teams undervalue wins added in determining salary equations for closers.

Appendix A: Figures and Tables Figure 1: 1.5e+06 1.0e+06 mean of rsalary 500000 0 012

Figure 2: 1.0e+07 8.0e+06 6.0e+06 RSALARY 4.0e+06 2.0e+06 0

012345678910111213141516171819202122232425

Figure 3: 300 200 IP START 100 0

-5 0 5 10 WAD

Figure 4: 150 100 IP RELIEF 50 0

-2 0 2 4 6 WAD

Figure 5: 150 100 IP RELIEF 50 0

-5 0 5 10 WAD

Figure 6:

0 1 20 15 10 5

-5 0 5 10

H9 2 20 15 10 5