Survey Research and Self-Defense Gun Use: an Explanation of Extreme Overestimates David Hemenway
Total Page:16
File Type:pdf, Size:1020Kb
Journal of Criminal Law and Criminology Volume 87 Article 5 Issue 4 Summer Summer 1997 Survey Research and Self-Defense Gun Use: An Explanation of Extreme Overestimates David Hemenway Follow this and additional works at: https://scholarlycommons.law.northwestern.edu/jclc Part of the Criminal Law Commons, Criminology Commons, and the Criminology and Criminal Justice Commons Recommended Citation David Hemenway, Survey Research and Self-Defense Gun Use: An Explanation of Extreme Overestimates, 87 J. Crim. L. & Criminology 1430 (1996-1997) This Criminology is brought to you for free and open access by Northwestern University School of Law Scholarly Commons. It has been accepted for inclusion in Journal of Criminal Law and Criminology by an authorized editor of Northwestern University School of Law Scholarly Commons. 0091-4169/97/8704-1430 THEJOURNAL oF CRIMINAL LAw & CRIMINOLOGY Vol. 87, No. 4 Copyright © 1997 by Northwestern University, School of Law Printed in USA. POLICY AND PERSPECTIVES Editor's note: In Fall 1995, theJournalpublished an articleby Professors Gary Kleck and Marc Gertz, Armed Resistance to Crime: The Prevalence and Nature of Self-Defense with a Gun, 86J. CGlM. & CRIMINOLOGY 150 (1995). As part of its Policy and Perspectives section, the Journalnow publishes the views of ProfessorDavid Hemenway on the Kleck-Gertz paper, a reply by Professors Kleck and Gertz, and the views of Professor Tom Smith on both the Hemenway and Kleck-Gertz papers. As always, the views expressed here are those of the authors. SURVEY RESEARCH AND SELF-DEFENSE GUN USE: AN EXPLANATION OF EXTREME OVERESTIMATES DAVID I{EMENWAY* I. INTRODUCTION AND SUMMARY Gary Kleck and Marc Gertz conducted a survey of civilian defen- sive gun use in 1992. In 1993, Kleck began publicizing the estimate that civilians use guns in self-defense against offenders up to 2.5 mil- lion times each year.1 This figure has been widely used by the Na- tional Rifle Association and by gun advocates. It is also often cited in the media2 and even in Congress.3 The Kleck and Gertz (K-G) paper has now been published.4 It is clear, however, that its conclusions can- not be accepted as valid. * Professor of Health Policy, Harvard School of Public Health. The author gives spe- cial thanks to Dimitrios Trichopoulos, Sue Swartz, Phil Cook, Deb Azrael, Eric Rimm, Frank Zimring, Jon Vernick, Daniel Webster, Tom Cole, and Al Biderman. 1 J.N. Schulman, Guns, Crimes and Self-defense, ORANGE CouNTr REG., Sept. 19, 1993, at 3 (editorial section); Talk of the Nation: Interview with Gary Kleck and Arthur Kellermann (Na- tional Public Radio broadcast, Aug. 8, 1994). 2 See Gordon Witkin, The Great Debate: Should You Own a Gun, U.S. NEws & WORLD REP., Aug. 15, 1994, at 24-31. 3 Cong. Res. Service, Gun Control in CRS IssuE BRIEF (Keith Bea ed., 1995). 4 Gary Kleck & Marc Gertz, Armed Resistance to Crime: The Prevalence and Nature of Self- defense with a Gun, 86J. CRiM. L. & CIMINOLOc 150 (1995). 1430 1997] SURVEY RESEARCH & SELF-DEFENSE GUN USE 1431 Two aspects of the K-G survey combine to create severe misesti- mation. The first is the likelihood of positive social desirability re- sponse, sometimes referred to as personal presentation bias. An individual who purchases a gun for self-defense and then uses it suc- cessfully to ward off a criminal is displaying the wisdom of his precau- tions and his capability in protecting himself, his loved ones, and his property. His action is to be commended and admired. Some positive social desirability response bias, by itself, might not lead to serious overestimation. However, combined with a second as- pect of the survey-the attempt to estimate a very rare event-it does. The search for a "needle in a haystack" has major methodological dangers, especially where researchers try to extrapolate the findings to society as a whole. Until the K-G study, no one had estimated that even as many as 1% of adult civilians had used a gun in self-defense in the past year. Nevertheless, assume that the actual incidence is 1%. On average, for every 100 individuals asked a 'Yes/No" question about the event, ninety-nine respondents will have a chance to be misclassified as a false positive. In ninety-nine answers there is the possibility of positive social desirability response bias. However, on average only one re- spondent-the one who actually did use a gun in self defense-could possibly be misclassified as a false negative (e.g., if she forgot about the event). Even if the chance of forgetting is high, as long as there is any possibility of positive response bias, it is very likely that the survey finding will be an overestimate. The fact that the survey is trying to estimate a low probability event also means that a small percentage bias, when extrapolated, can lead to extreme overestimates. Consider a survey finding which con- tains a 1% overestimate of positive responses. If the true incidence of the event is 60%, estimating it at 61% would not be a problem. But if the true incidence is 1%, measuring it as 2% would be a doubling of the true rate; and if the true incidence is 0.1%, measuring it at 1.1% would be an eleven-fold overestimate. The K-G survey design contains a huge overestimation bias. The authors do little to reduce the bias or to validate their findings by external measures. All checks for external validity of the Kleck-Gertz finding confirm that their estimate is highly exaggerated. II. BACKGROUND Previous data on self-defense gun use came from two sources- the large National Crime Victimization surveys (NCVS), and smaller private surveys (principally random-digit-dial telephone surveys). These two sources produce markedly different results. 1432 DAVID HEMENWAY [Vol. 87 The NCVS employs a multistage design with a probability sample of some 50,000 housing units in the United States (e.g., in 1994 there were 47,600 housing units and 90,560 persons) .5 The survey is con- ducted by the Census Bureau for the Bureau of Justice Statistics. Housing units remain in the NCVS for three years and residents are interviewed at six month intervals. Initial surveys are in-person, while subsequent ones are typically by telephone. Respondents who report a threatened, attempted or completed victimization for six crimes- rape, robbery, assault, burglary, non-business larceny and motor vehi- cle theft-are asked detailed questions about the incident. NCVS results indicate that, nationally, victims use guns against offenders approximately 65,000 times per year. 6 Kleck believes peo- ple under-report to the government NCVS interviewers, especially since the surveys are not anonymous. He also finds fault with the NCVS survey for asking about self-defense gun use only for individuals who have been victimized. 7 Interestingly, it is this latter feature of the NCVS which dramatically reduces the overestimation bias found in the private surveys. Based on eight national surveys, undertaken between 1976 and 1990, Eleck estimates that guns are used approximately 700,000 times per year in self-defense. 8 However, all eight surveys have very serious limitations. Compared to the NCVS, the sample size of each of these surveys is small (600 - 1500) and interviewers typically asked only one vague question about gun use in self-defense (e.g., "Have you used a gun in self-defense in the previous five years?") with no follow-up questions.9 Only one of the surveys meets the minimum criteria of drawing from a representative national population, asking about a specific time frame, distinguishing civilian use from military or police uses, and distinguishing uses against humans from uses against animals.10 A review of Kleck's analysis argued that "Kleck's conclusions rest on limited data and strong assumptions. Small changes in the proce- dure produce large differences in the findings. The estimates are 5 UNITED STATES DEP'T OF JUSTICE, CRIMINAL VICTIMIZATION IN THE UNITED STATES, 1994: A NATIONAL CRIME VICTIMIZATION SURVEY REPORT (1997). 6 PhilipJ. Cook, The Technology ofPersonal Violence, in 14 CRIME AND JUSTICE: AN ANNUAL REVIEW OF RESEARCH (Michael Tonry ed., 1991); see also David McDowall & Brian Wier- sema, The Incidence of Defensive Firearm Use by U.S. Crime Victims, 1987 through 1990, 84 AM. J. PUBLIC HEALTH 1982-84 (1994); Philip J. Cook &Jens Ludwig, Guns in America: Results of a Comprehensive National Survey on Firearms Ownership and Use, Police Foundation (1996). 7 GARY KLECK, POINT BLANK: GUNS AND VIOLENCE IN AMERICA 110 (1991). 8 See id. at 105-07. 9 Kleck & Gertz, supra note 4, at 157. 10 See id. at 155. 1997] SURVEY RESEARCH & SELF-DEFENSE GUN USE 1433 questionable, and it appears unwise to place much weight on them."" A National Research Council report also finds that Kleck's estimates appear exaggerated and says that it is almost certain that "some of what respondents designate as their own self-defense would be con- strued as aggression by others."' 2 III. Tim KLECK-GERTZ SURVEY In 1992, Kleck and Gertz conducted a national random-digit-dial survey of five thousand dwelling units, asking detailed questions about self-defense gun use. 13 Their estimates of civilian self-defense gun use range from 1 million to 2.5 million times per year.14 The 2.5 million figure is the one they believe to be most accurate and the one Kleck has publicized, so that figure will be discussed in this paper. K-G derive their 2.5 million estimate from the fact that 1.33% of the individuals surveyed reported that they themselves used a gun in self-defense during the past year;15 in other words, about 66 people out of 5000 reported such a use.