Measuring Umpire Consistency David J. Hunter January 11, 2018 Assumptions/Motivations I Consistency is more important than conformity. I A consistent zone need not be rectangular, but should be convex. I Consistency within a game is important. I Different zones for LH and RH batters are OK. I One egregiously bad call is worse/as bad as several marginally bad calls. Assumptions/Motivations I Consistency is more important than conformity. I A consistent zone need not be rectangular, but should be convex. I Consistency within a game is important. I Different zones for LH and RH batters are OK. I One egregiously bad call is worse/as bad as several marginally bad calls. Assumptions/Motivations I Consistency is more important than conformity. I A consistent zone need not be rectangular, but should be convex. I Consistency within a game is important. I Different zones for LH and RH batters are OK. I One egregiously bad call is worse/as bad as several marginally bad calls. Assumptions/Motivations I Consistency is more important than conformity. I A consistent zone need not be rectangular, but should be convex. I Consistency within a game is important. I Different zones for LH and RH batters are OK. I One egregiously bad call is worse/as bad as several marginally bad calls. Assumptions/Motivations I Consistency is more important than conformity. I A consistent zone need not be rectangular, but should be convex. I Consistency within a game is important. I Different zones for LH and RH batters are OK. I One egregiously bad call is worse/as bad as several marginally bad calls. Define established ball and strike zones gid_2017_07_19_sdnmlb_colmlb_1 gid_2017_07_19_sdnmlb_colmlb_1 vs. right−handed batters vs. left−handed batters 5 5 4 4 3 3 normedpz normedpz 2 2 1 1 0 0 −2 −1 0 1 2 −2 −1 0 1 2 px px Established Strike Zone: Convex Hull Let Hl denote the open half-plane bounded by the line l. The convex hull of a set of points P is the set \ c S = Hl {Hl |Hl ∩P=∅} When P contains the locations of all called strikes, S is the established strike zone. Established Strike Zone: Convex Hull gid_2017_07_19_sdnmlb_colmlb_1 gid_2017_07_19_sdnmlb_colmlb_1 vs. right−handed batters vs. left−handed batters 5 5 4 4 3 3 normedpz normedpz 2 2 1 1 0 0 −2 −1 0 1 2 −2 −1 0 1 2 px px Established Ball Zone: α-Shape? 2 For α > 0, let Bx,α denote the open ball in R of radius α centered 2 2 at the point x. Given a set of points P ⊂ R , two points p1, p2 ∈ R are α-neighbors if p1 and p2 lie on the boundary of some Bx,α such that Bx,α ∩ P = ∅. The α-shape is the straight line graph formed by drawing line segments between α-neighbors. I α-neighbors are close together but next to big empty regions. I The α-shape is the outline of the points P. I Does not need to be convex or even simply-connected. I For large α, the α-shape is the boundary of the convex hull. Established Ball Zone: α-Shape? gid_2017_07_19_sdnmlb_colmlb_1 gid_2017_07_19_sdnmlb_colmlb_1 vs. right−handed batters vs. left−handed batters 5 5 4 4 3 3 normedpz normedpz 2 2 1 1 0 0 −2 −1 0 1 2 −2 −1 0 1 2 px px Established Ball Zone: α-Hull 2 For α > 0, let Bx,α denote the open ball in R of radius α centered at the point x. Given a set of points P ⊂ R2, the α-hull of P is the set \ c X = Bx,α {Bx,α|Bx,α∩P=∅} When P contains the locations of all called balls, X is the established ball zone. T c I Approaches convex hull S = {Hl |Hl ∩P=∅} Hl as α → ∞. I Intuitively: hole punch. I May not be simply connected for smallish α. Established Ball Zone: α-Hull gid_2017_07_19_sdnmlb_colmlb_1 gid_2017_07_19_sdnmlb_colmlb_1 vs. right−handed batters vs. left−handed batters 5 5 4 4 3 3 normedpz normedpz 2 2 1 1 0 0 −2 −1 0 1 2 −2 −1 0 1 2 px px Inconsistency Index Metric S = established strike zone (convex hull of strikes) X = established ball zone (α-hull of balls) balls in S + strikes in X inconsistency = total calls I This is a per game metric. I Compute separately for right-hand batters and left-hand batters and add. Inconsistency Index Metric S = established strike zone (convex hull of strikes) X = established ball zone (α-hull of balls) balls in S + strikes in X inconsistency = total calls I This is a per game metric. I Compute separately for right-hand batters and left-hand batters and add. Choosing α I α too small: Ball zone has lots of components I α too large: Seems unfair to umpires I α way too large: Ball zone becomes simply connected I α = 0.5539: 1/3 rule-book zone width Correlations: observations Games ZoneSize Accuracy BBrate Krate Incon Games 1.00 -0.05 0.05 -0.03 0.30 -0.02 ZoneSize -0.05 1.00 -0.49 -0.39 0.09 0.34 Accuracy 0.05 -0.49 1.00 -0.01 0.27 -0.65 BBrate -0.03 -0.39 -0.01 1.00 -0.24 0.09 Krate 0.30 0.09 0.27 -0.24 1.00 -0.14 Incon -0.02 0.34 -0.65 0.09 -0.14 1.00 I Walk rate uncorrelated to inconsistency, accuracy. I Smaller zones tend to be more accurate and consistent. I Inconsistency is moderately correlated with accuracy. Correlations: observations Games ZoneSize Accuracy BBrate Krate Incon Games 1.00 -0.05 0.05 -0.03 0.30 -0.02 ZoneSize -0.05 1.00 -0.49 -0.39 0.09 0.34 Accuracy 0.05 -0.49 1.00 -0.01 0.27 -0.65 BBrate -0.03 -0.39 -0.01 1.00 -0.24 0.09 Krate 0.30 0.09 0.27 -0.24 1.00 -0.14 Incon -0.02 0.34 -0.65 0.09 -0.14 1.00 I Walk rate uncorrelated to inconsistency, accuracy. I Smaller zones tend to be more accurate and consistent. I Inconsistency is moderately correlated with accuracy. Correlations: observations Games ZoneSize Accuracy BBrate Krate Incon Games 1.00 -0.05 0.05 -0.03 0.30 -0.02 ZoneSize -0.05 1.00 -0.49 -0.39 0.09 0.34 Accuracy 0.05 -0.49 1.00 -0.01 0.27 -0.65 BBrate -0.03 -0.39 -0.01 1.00 -0.24 0.09 Krate 0.30 0.09 0.27 -0.24 1.00 -0.14 Incon -0.02 0.34 -0.65 0.09 -0.14 1.00 I Walk rate uncorrelated to inconsistency, accuracy. I Smaller zones tend to be more accurate and consistent. I Inconsistency is moderately correlated with accuracy. Accuracy vs. Inconsistency (r = −0.65) Pat Hoberg● Eric Cooper● 0.92 Sean Barber Vic Carapazza● ● Ben ●May Chris ●Segal Adam Hamari● Tom Woodring● Jim ●Wolf John Tumpane● Alan Porter● Jordan● Baker Mark Ripperger● Dan Bellino● Chris Guccione● Quinn WolcottLance Barksdale● Todd Tichenor● Mark Wegner ● 0.91 Gabe● Morales Will Little ● ● Stu Scheurwater Manny Gonzalez Phil Cuzzi ● Chris Conroy ● ChadGarySam Whitson HolbrookCederstromTripp Gibson● Brian● III O'Nora ● ● Mark● ● CarlsonD.J. Reyburn● Chad Fairchild ● ● ● Scott ●Barry Mike Everitt Nic Lentz ● ● Mike Muchlinski Tony Randazzo ● Carlos● Torres Jeff Kellogg● Lance ●Barrett ● Bruce Dreckman Ryan Blakney Brian Gorman● ● ● David Rackley● GregBill Gibson Welke James Hoye ● ● Jerry Meals Jim Reynolds ● Jeff Nelson ● ● Alfonso Marquez ● Tim Timmons● Adrian● Johnson● CoryBill Blaser● Miller● Ted BarrettMike● Estabrook Paul Emmel● ● Ed Hickox● Tom Hallion Doug Eddings● ● Angel Hernandez Marty ●Foster 0.90 Ron Kulpa● ● Ramon De● Jesus Dana DeMuth Gerry Davis Marvin Hudson ● ● ● CB Bucknor● Fieldin Culbreth● Laz ●Diaz Paul Nauert● Clint Fagan● Dan Iassogna ● Andy Fletcher● Consensus Rule Book Accuracy Hunter Wendelstedt● Jerry ●Layne Larry Vanover● Mike Winters● Joe West● 0.89 Kerwin ●Danley Rob Drake● 0.03 0.04 0.05 Average Inconsistency Index Principal Component Analysis Standard deviations (1, .., p=5): [1] 1.43 1.21 0.87 0.66 0.54 Rotation (n x k) = (5 x 5): PC1 PC2 PC3 PC4 PC5 ZoneSize 0.495 -0.43 0.101 -0.642 0.39 Accuracy -0.626 -0.13 0.025 0.066 0.77 BBrate -0.089 0.68 -0.478 -0.536 0.10 Krate -0.179 -0.56 -0.784 -0.019 -0.21 Incon 0.568 0.16 -0.383 0.543 0.46 Plot of Components ● Andy Fletcher ● Alfonso Marquez 0.2 ● Sean Barber ● Jeff Kellogg ● Tony Randazzo ● ● James Hoye ● Dana DeMuthFieldin Culbreth ● Paul Nauert ● Bill Welke ● ● Pat Hoberg Larry Vanover ● Sam● Jordan Holbrook● D.J. Baker Reyburn ● Clint Fagan ● Jerry Meals ● ● ● ● David Rackley ● CarlosMark Torres Wegner Tom Hallion ● Jim Reynolds ● Paul EmmelMarvin Hudson ● Alan Porter ● Joe West ● ● Marty Foster ● Rob Drake ● Hunter Wendelstedt ● ● CB Bucknor Brian Gorman ● Lance Barksdale● Jerry Layne ● Manny Gonzalez ● John Tumpane ● Mike Winters ● ● Greg Gibson Will Little● Chad Whitson● MarkStu Carlson Scheurwater 0.0 ● ● ● Chris Conroy Bruce Dreckman Tom Woodring ●Jim Wolf ● Quinn Wolcott ● Tim Timmons ● Kerwin Danley ● Dan Bellino ● ● ● Gerry Davis ● ● Gary Cederstrom Brian O'Nora Ron Kulpa ● Gabe Morales ● Mike Everitt Todd Tichenor ● Dan Iassogna ● ● Chad● Fairchild ● ● Angel Hernandez ● Scott Barry Ed Hickox Jeff Nelson● Chris Guccione ● Tripp Gibson III Laz Diaz ● ● Nic Lentz ● Vic Carapazza ● Mark Ripperger Ramon De Jesus ●Ben May Hitter Friendliness ● ● ● ● Chris Segal Phil● Cuzzi Mike MuchlinskiLance Barrett ● Ryan Blakney ● ● Eric Cooper ● Adam Hamari Adrian Johnson Ted Barrett ● Cory Blaser −0.2 ●Bill Miller ● Mike Estabrook ● Doug Eddings −0.2 −0.1 0.0 0.1 0.2 Strike Zone Sloppiness References I Data from MLBAM.