140 Resampling: The New Statistics CHAPTER On Variability in 9 Sampling Variability and Small Samples Regression to the Mean Summary and Conclusion [Debra said]: “I’ve had such good luck with Japanese cars and poor luck with American...” The ’65 Ford Mustang: “It was fun, but I had to put two new transmissions in it.” The Ford Torino: “That got two transmissions too. That finished me with Ford.” The Plymouth Horizon: “The disaster of all disasters. That should’ve been painted bright yellow. What a lemon.” (Washington Post Magazine, May 17, 1992, p. 19) Does the headnote above convince you that Japanese cars are better than American? Has Debra got enough evidence to reach the conclusion she now holds? That sort of question, and the reasoning we use to address it , is the subject of this chapter. More generally, how should one go about using the available data to test the hypothesis that Japanese cars are better? That is an example of the questions that are the subject of statistics. Variability and small samples Perhaps the most important idea for sound statistical infer- ence—the section of the book we are now beginning, in con- trast to problems in probability, which we have studied in the previous chapters—is recognition of the presence of variability in the results of small samples. The fatal error of relying on too-small samples is all too common among economic fore- casters, journalists, and others who deal with trends and pub- lic opinion. Athletes, sports coaches, sportswriters, and fans too frequently disregard this principle both in their decisions and in their discussion. Chapter 9—On Variability in Sampling 141 Our intuitions often carry us far astray when the results vary from situation to situation—that is, when there is variability in outcomes—and when we have only a small sample of out- comes to look at. To motivate the discussion, I’ll tell you something that almost no American sports fan will believe: There is no such thing as a slump in baseball batting. That is, a batter often goes an alarming number of at-bats without getting a hit, and every- one—the manager, the sportswriters, and the batter himself— assumes that something has changed, and the probability of the batter getting a hit is now lower than it was before the slump. It is common for the manager to replace the player for a while, and for the player and coaches to change the player’s hitting style so as to remedy the defect. But the chance of a given batter getting a hit is just the same after he has gone many at-bats without a hit as when he has been hitting well. A belief in slumps causes managers to play line-ups which may not be their best. By “slump” I mean that a player’s probability of getting a hit in a given at-bat is lower during a period than during average periods. And when I say there is no such thing as a slump, I mean that the chances of getting a hit after any sequence of at-bats without a hit is not different than the long-run aver- age. The “hot hand” in basketball is another illusion. The hot hand does not exist! The chance of a shooter scoring is the same af- ter he has just missed a flock of shots as when he has just sunk a long string. That is, the chance of scoring a basket is no higher after a run of successes than after a run of failures. But even professional teams choose plays on the basis of who suppos- edly has a hot hand. Managers who substitute for the “slumping” or “cold-handed” players with other players who, in the long run, have lower batting averages, or set up plays for the shooter who suppos- edly has a hot hand, make a mistake. The supposed hot hand in basketball, and the slump in baseball, are illusions because the observed long runs of outs, or of baskets, are statistical artifacts, due to ordinary random variability. The identifica- tion of slumps and hot hands is superstitious behavior, classic cases of the assignment of pattern to a series of events when there really is no pattern. How do statisticians ascertain that slumps and hot hands do not exist? In brief, in baseball we simulate a hitter with a given average—say .250—and compare the results with actual hit- 142 Resampling: The New Statistics ters of that average, to see whether they have “slumps” longer than the computer. The method of investigation is roughly as follows. You program a computer or other machine to behave the way a player would, given the player’s long-run average, on the assumption that each trial is a random drawing. For example, if a player has a .250 season-long batting average, the machine is programmed like an urn containing three black balls and one white ball. Then for each simulated at bat, the machine shuffles the “balls” and draws one; it then records whether the result is black or white, after which the ball is re- placed in the urn. To study a season with four hundred at-bats, a simulated ball is drawn four hundred times. The records of the player’s real season and the simulated sea- son are then compared. If there really is such a thing as a non-random slump or streak, there will be fewer but longer “runs” of hits or outs in the real record than in the simulated record. On the other hand, if performance is independent from at-bat trial to at-bat trial, the actual record will change from hit to out and from out to hit as often as does the random simu- lated record. I suggested this sort of test for the existence of slumps in my 1969 book that first set forth the resampling method, a predecessor of this book. For example, Table 9-1 shows the results of one 400 at-bat sea- son for a simulated .250 hitter. (1 = hit, 0 = out, sequential at- bats ordered vertically) Note the “slump”—1 for 24—in col- umns 7 & 8 (in bold). Chapter 9—On Variability in Sampling 143 Table 9-1 A Rookie Season (400 at-bats) 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 1 1 1 0 1 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 1 1 0 1 0 0 1 0 0 0 1 0 0 0 0 1 0 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 1 1 0 0 0 0 1 0 1 0 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 1 1 0 1 0 0 0 1 0 0 0 0 1 1 0 0 1 0 0 0 0 1 1 0 0 0 0 0 1 1 1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 1 1 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 1 0 1 0 0 Harry Roberts investigated the batting records of a sample of major leaguers. He compared players’ season-long records against the behavior of random-number drawings. If slumps existed rather than being a fiction of the imagination, the real players’ records would shift from a string of hits to a string of outs less frequently than would the random-number se- quences. But in fact the number of shifts, and the average lengths of strings of hits and outs, are on average the same for players as for player-simulating random-number devices. Over long periods, averages may vary systematically, as Ty Cobb’s annual batting averages varied non-randomly from season to season, Roberts found. But in the short run, most individual and team performances have shown results simi- lar to the outcomes that a lottery-type random number ma- chine would produce.