Nonsampling Errors in Surveys * By
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Downloads Automatically
VIRGINIA LAW REVIEW ASSOCIATION VIRGINIA LAW REVIEW ONLINE VOLUME 103 DECEMBER 2017 94–102 ESSAY ACT-SAMPLING BIAS AND THE SHROUDING OF REPEAT OFFENDING Ian Ayres, Michael Chwe and Jessica Ladd∗ A college president needs to know how many sexual assaults on her campus are caused by repeat offenders. If repeat offenders are responsible for most sexual assaults, the institutional response should focus on identifying and removing perpetrators. But how many offenders are repeat offenders? Ideally, we would find all offenders and then see how many are repeat offenders. Short of this, we could observe a random sample of offenders. But in real life, we observe a sample of sexual assaults, not a sample of offenders. In this paper, we explain how drawing naive conclusions from “act sampling”—sampling people’s actions instead of sampling the population—can make us grossly underestimate the proportion of repeat actors. We call this “act-sampling bias.” This bias is especially severe when the sample of known acts is small, as in sexual assault, which is among the least likely of crimes to be reported.1 In general, act sampling can bias our conclusions in unexpected ways. For example, if we use act sampling to collect a set of people, and then these people truthfully report whether they are repeat actors, we can overestimate the proportion of repeat actors. * Ayres is the William K. Townsend Professor at Yale Law School; Chwe is Professor of Political Science at the University of California, Los Angeles; and Ladd is the Founder and CEO of Callisto. 1 David Cantor et al., Report on the AAU Campus Climate Survey on Sexual Assault and Sexual Misconduct, Westat, at iv (2015) [hereinafter AAU Survey], (noting that a relatively small percentage of the most serious offenses are reported) https://perma.cc/5BX7-GQPU; Nat’ Inst. -
The Effect of Sampling Error on the Time Series Behavior of Consumption Data*
Journal of Econometrics 55 (1993) 235-265. North-Holland The effect of sampling error on the time series behavior of consumption data* William R. Bell U.S.Bureau of the Census, Washington, DC 20233, USA David W. Wilcox Board of Governors of the Federal Reserve System, Washington, DC 20551, USA Much empirical economic research today involves estimation of tightly specified time series models that derive from theoretical optimization problems. Resulting conclusions about underly- ing theoretical parameters may be sensitive to imperfections in the data. We illustrate this fact by considering sampling error in data from the Census Bureau’s Retail Trade Survey. We find that parameter estimates in seasonal time series models for retail sales are sensitive to whether a sampling error component is included in the model. We conclude that sampling error should be taken seriously in attempts to derive economic implications by modeling time series data from repeated surveys. 1. Introduction The rational expectations revolution has transformed the methodology of macroeconometric research. In the new style, the researcher typically begins by specifying a dynamic optimization problem faced by agents in the model economy. Then the researcher derives the solution to the optimization problem, expressed as a stochastic model for some observable economic variable(s). A trademark of research in this tradition is that each of the parameters in the model for the observable variables is endowed with a specific economic interpretation. The last step in the research program is the application of the model to actual economic data to see whether the model *This paper reports the general results of research undertaken by Census Bureau and Federal Reserve Board staff. -
9Th GESIS Summer School in Survey Methodology August 2020
9th GESIS Summer School in Survey Methodology August 2020 Syllabus for short course C: “Applied Systematic Review and Meta-Analysis” Lecturers: Dr. Jessica Daikeler Sonila Dardha E-mail: [email protected] [email protected] Homepage: https://www.gesis.org/en/institute/ https://www.researchgate.net/profile/Sonila_Dard /staff/person/Jessica.wengrzik ha Date: 12-14 August 2020 Time: 10:00-16:00 Time zone: CEST, course starts on Wednesday at 10:00 Venue: Online via Zoom About the Lecturers: Jessica is a survey methodologist and works at GESIS in the Survey Operations and Survey Statistics teams in the Survey Design and Methodology department. Jessica wrote her dissertation on "The Application of Evidence- Based Methods in Survey Methodology" with Prof. Michael Bosnjak (University of Trier & ZPID) and Prof. Florian Keusch (University of Mannheim). At GESIS she is involved in the application of evidence-based methods, in par- ticular experiments, systematic reviews and meta-analyses. She has lots of experience with different systematic review and meta-analysis projects. Her research is currently focused on data quality in web and mobile surveys, link of nonresponse and measurement errors, data linkage and, of course, methods for the accumulation of evi- dence. Sonila is a survey methodologist researching interviewer effects, response patterns, nonresponse error and 3MC surveys, and a survey practitioner coordinating cross-country comparative projects. She is currently pursuing her PhD in Survey Research Methodology at City, University of London, the home institution of the European Social Survey (ESS ERIC). Sonila has previously obtained three Master degrees, one of which in M.Sc in Statistics (Quan- titative Analysis for the Social Sciences) from KU Leuven, Belgium. -
13 Collecting Statistical Data
13 Collecting Statistical Data 13.1 The Population 13.2 Sampling 13.3 Random Sampling 1.1 - 1 • Polls, studies, surveys and other data collecting tools collect data from a small part of a larger group so that we can learn something about the larger group. • This is a common and important goal of statistics: Learn about a large group by examining data from some of its members. 1.1 - 2 Data collections of observations (such as measurements, genders, survey responses) 1.1 - 3 Statistics is the science of planning studies and experiments, obtaining data, and then organizing, summarizing, presenting, analyzing, interpreting, and drawing conclusions based on the data 1.1 - 4 Population the complete collection of all individuals (scores, people, measurements, and so on) to be studied; the collection is complete in the sense that it includes all of the individuals to be studied 1.1 - 5 Census Collection of data from every member of a population Sample Subcollection of members selected from a population 1.1 - 6 A Survey • The practical alternative to a census is to collect data only from some members of the population and use that data to draw conclusions and make inferences about the entire population. • Statisticians call this approach a survey (or a poll when the data collection is done by asking questions). • The subgroup chosen to provide the data is called the sample, and the act of selecting a sample is called sampling. 1.1 - 7 A Survey • The first important step in a survey is to distinguish the population for which the survey applies (the target population) and the actual subset of the population from which the sample will be drawn, called the sampling frame. -
American Community Survey Accuracy of the Data (2018)
American Community Survey Accuracy of the Data (2018) INTRODUCTION This document describes the accuracy of the 2018 American Community Survey (ACS) 1-year estimates. The data contained in these data products are based on the sample interviewed from January 1, 2018 through December 31, 2018. The ACS sample is selected from all counties and county-equivalents in the United States. In 2006, the ACS began collecting data from sampled persons in group quarters (GQs) – for example, military barracks, college dormitories, nursing homes, and correctional facilities. Persons in sample in (GQs) and persons in sample in housing units (HUs) are included in all 2018 ACS estimates that are based on the total population. All ACS population estimates from years prior to 2006 include only persons in housing units. The ACS, like any other sample survey, is subject to error. The purpose of this document is to provide data users with a basic understanding of the ACS sample design, estimation methodology, and the accuracy of the ACS data. The ACS is sponsored by the U.S. Census Bureau, and is part of the Decennial Census Program. For additional information on the design and methodology of the ACS, including data collection and processing, visit: https://www.census.gov/programs-surveys/acs/methodology.html. To access other accuracy of the data documents, including the 2018 PRCS Accuracy of the Data document and the 2014-2018 ACS Accuracy of the Data document1, visit: https://www.census.gov/programs-surveys/acs/technical-documentation/code-lists.html. 1 The 2014-2018 Accuracy of the Data document will be available after the release of the 5-year products in December 2019. -
Journal of Econometrics an Empirical Total Survey Error Decomposition
Journal of Econometrics xxx (xxxx) xxx Contents lists available at ScienceDirect Journal of Econometrics journal homepage: www.elsevier.com/locate/jeconom An empirical total survey error decomposition using data combination ∗ Bruce D. Meyer a,b,c, Nikolas Mittag d, a Harris School of Public Policy Studies, University of Chicago, 1307 E. 60th Street, Chicago, IL 60637, United States of America b NBER, United States of America c AEI, United States of America d CERGE-EI, joint workplace of Charles University and the Economics Institute of the Czech Academy of Sciences, Politických v¥z¬· 7, Prague 1, 110 00, Czech Republic article info a b s t r a c t Article history: Survey error is known to be pervasive and to bias even simple, but important, estimates Received 16 July 2019 of means, rates, and totals, such as the poverty and the unemployment rate. In order Received in revised form 16 July 2019 to summarize and analyze the extent, sources, and consequences of survey error, we Accepted 5 March 2020 define empirical counterparts of key components of the Total Survey Error Framework Available online xxxx that can be estimated using data combination. Specifically, we estimate total survey Keywords: error and decompose it into three high level sources of error: generalized coverage Total survey error error, item non-response error and measurement error. We further decompose these Administrative data sources into lower level sources such as failure to report a positive amount and errors Measurement error in amounts conditional on reporting a positive value. For errors in dollars paid by two Linked data large government transfer programs, we use administrative records on the universe Data combination of program payments in New York State linked to three major household surveys to estimate the error components previously defined. -
Human Dimensions of Wildlife the Fallacy of Online Surveys: No Data Are Better Than Bad Data
Human Dimensions of Wildlife, 15:55–64, 2010 Copyright © Taylor & Francis Group, LLC ISSN: 1087-1209 print / 1533-158X online DOI: 10.1080/10871200903244250 UHDW1087-12091533-158XHuman Dimensions of WildlifeWildlife, Vol.The 15, No. 1, November 2009: pp. 0–0 Fallacy of Online Surveys: No Data Are Better Than Bad Data TheM. D. Fallacy Duda ofand Online J. L. Nobile Surveys MARK DAMIAN DUDA AND JOANNE L. NOBILE Responsive Management, Harrisonburg, Virginia, USA Internet or online surveys have become attractive to fish and wildlife agencies as an economical way to measure constituents’ opinions and attitudes on a variety of issues. Online surveys, however, can have several drawbacks that affect the scientific validity of the data. We describe four basic problems that online surveys currently present to researchers and then discuss three research projects conducted in collaboration with state fish and wildlife agencies that illustrate these drawbacks. Each research project involved an online survey and/or a corresponding random telephone survey or non- response bias analysis. Systematic elimination of portions of the sample population in the online survey is demonstrated in each research project (i.e., the definition of bias). One research project involved a closed population, which enabled a direct comparison of telephone and online results with the total population. Keywords Internet surveys, sample validity, SLOP surveys, public opinion, non- response bias Introduction Fish and wildlife and outdoor recreation professionals use public opinion and attitude sur- veys to facilitate understanding their constituents. When the surveys are scientifically valid and unbiased, this information is useful for organizational planning. Survey research, however, costs money. -
American Community Survey Design and Methodology (January 2014) Chapter 15: Improving Data Quality by Reducing Non-Sampling Erro
American Community Survey Design and Methodology (January 2014) Chapter 15: Improving Data Quality by Reducing Non-Sampling Error Version 2.0 January 30, 2014 ACS Design and Methodology (January 2014) – Improving Data Quality Page ii [This page intentionally left blank] Version 2.0 January 30, 2014 ACS Design and Methodology (January 2014) – Improving Data Quality Page iii Table of Contents Chapter 15: Improving Data Quality by Reducing Non-Sampling Error ............................... 1 15.1 Overview .......................................................................................................................... 1 15.2 Coverage Error ................................................................................................................. 2 15.3 Nonresponse Error............................................................................................................ 3 15.4 Measurement Error ........................................................................................................... 6 15.5 Processing Error ............................................................................................................... 7 15.6 Census Bureau Statistical Quality Standards ................................................................... 8 15.7 References ........................................................................................................................ 9 Version 2.0 January 30, 2014 ACS Design and Methodology (January 2014) – Improving Data Quality Page iv [This page intentionally left -
Levy, Marc A., “Sampling Bias Does Not Exaggerate Climate-Conflict Claims,” Nature Climate Change 8,6 (442)
Levy, Marc A., “Sampling bias does not exaggerate climate-conflict claims,” Nature Climate Change 8,6 (442) https://doi.org/10.1038/s41558-018-0170-5 Final pre-publication text To the Editor – In a recent Letter, Adams et al1 argue that claims regarding climate-conflict links are overstated because of sampling bias. However, this conclusion rests on logical fallacies and conceptual misunderstanding. There is some sampling bias, but it does not have the claimed effect. Suggesting that a more representative literature would generate a lower estimate of climate- conflict links is a case of begging the question. It only make sense if one already accepts the conclusion that the links are overstated. Otherwise it is possible that more representative cases might lead to stronger estimates. In fact, correcting sampling bias generally does tend to increase effect estimates2,3. The authors’ claim that the literature’s disproportionate focus on Africa undermines sustainable development and climate adaptation rests on the same fallacy. What if the climate-conflict links are as strong as people think? It is far from obvious that acting as if they were not would somehow enhance development and adaptation. The authors offer no reasoning to support such a claim, and the notion that security and development are best addressed in concert is consistent with much political theory and practice4,5,6. Conceptually, the authors apply a curious kind of “piling on” perspective in which each new paper somehow ratchets up the consensus view of a country’s climate-conflict links, without regard to methods or findings. Consider the papers cited as examples of how selecting cases on the conflict variable exaggerates the link. -
10. Sample Bias, Bias of Selection and Double-Blind
SEC 4 Page 1 of 8 10. SAMPLE BIAS, BIAS OF SELECTION AND DOUBLE-BLIND 10.1 SAMPLE BIAS: In statistics, sampling bias is a bias in which a sample is collected in such a way that some members of the intended population are less likely to be included than others. It results in abiased sample, a non-random sample[1] of a population (or non-human factors) in which all individuals, or instances, were not equally likely to have been selected.[2] If this is not accounted for, results can be erroneously attributed to the phenomenon under study rather than to the method of sampling. Medical sources sometimes refer to sampling bias as ascertainment bias.[3][4] Ascertainment bias has basically the same definition,[5][6] but is still sometimes classified as a separate type of bias Types of sampling bias Selection from a specific real area. For example, a survey of high school students to measure teenage use of illegal drugs will be a biased sample because it does not include home-schooled students or dropouts. A sample is also biased if certain members are underrepresented or overrepresented relative to others in the population. For example, a "man on the street" interview which selects people who walk by a certain location is going to have an overrepresentation of healthy individuals who are more likely to be out of the home than individuals with a chronic illness. This may be an extreme form of biased sampling, because certain members of the population are totally excluded from the sample (that is, they have zero probability of being selected). -
The Evidence from World Values Survey Data
Munich Personal RePEc Archive The return of religious Antisemitism? The evidence from World Values Survey data Tausch, Arno Innsbruck University and Corvinus University 17 November 2018 Online at https://mpra.ub.uni-muenchen.de/90093/ MPRA Paper No. 90093, posted 18 Nov 2018 03:28 UTC The return of religious Antisemitism? The evidence from World Values Survey data Arno Tausch Abstract 1) Background: This paper addresses the return of religious Antisemitism by a multivariate analysis of global opinion data from 28 countries. 2) Methods: For the lack of any available alternative we used the World Values Survey (WVS) Antisemitism study item: rejection of Jewish neighbors. It is closely correlated with the recent ADL-100 Index of Antisemitism for more than 100 countries. To test the combined effects of religion and background variables like gender, age, education, income and life satisfaction on Antisemitism, we applied the full range of multivariate analysis including promax factor analysis and multiple OLS regression. 3) Results: Although religion as such still seems to be connected with the phenomenon of Antisemitism, intervening variables such as restrictive attitudes on gender and the religion-state relationship play an important role. Western Evangelical and Oriental Christianity, Islam, Hinduism and Buddhism are performing badly on this account, and there is also a clear global North-South divide for these phenomena. 4) Conclusions: Challenging patriarchic gender ideologies and fundamentalist conceptions of the relationship between religion and state, which are important drivers of Antisemitism, will be an important task in the future. Multiculturalism must be aware of prejudice, patriarchy and religious fundamentalism in the global South. -
Observational Studies and Bias in Epidemiology
The Young Epidemiology Scholars Program (YES) is supported by The Robert Wood Johnson Foundation and administered by the College Board. Observational Studies and Bias in Epidemiology Manuel Bayona Department of Epidemiology School of Public Health University of North Texas Fort Worth, Texas and Chris Olsen Mathematics Department George Washington High School Cedar Rapids, Iowa Observational Studies and Bias in Epidemiology Contents Lesson Plan . 3 The Logic of Inference in Science . 8 The Logic of Observational Studies and the Problem of Bias . 15 Characteristics of the Relative Risk When Random Sampling . and Not . 19 Types of Bias . 20 Selection Bias . 21 Information Bias . 23 Conclusion . 24 Take-Home, Open-Book Quiz (Student Version) . 25 Take-Home, Open-Book Quiz (Teacher’s Answer Key) . 27 In-Class Exercise (Student Version) . 30 In-Class Exercise (Teacher’s Answer Key) . 32 Bias in Epidemiologic Research (Examination) (Student Version) . 33 Bias in Epidemiologic Research (Examination with Answers) (Teacher’s Answer Key) . 35 Copyright © 2004 by College Entrance Examination Board. All rights reserved. College Board, SAT and the acorn logo are registered trademarks of the College Entrance Examination Board. Other products and services may be trademarks of their respective owners. Visit College Board on the Web: www.collegeboard.com. Copyright © 2004. All rights reserved. 2 Observational Studies and Bias in Epidemiology Lesson Plan TITLE: Observational Studies and Bias in Epidemiology SUBJECT AREA: Biology, mathematics, statistics, environmental and health sciences GOAL: To identify and appreciate the effects of bias in epidemiologic research OBJECTIVES: 1. Introduce students to the principles and methods for interpreting the results of epidemio- logic research and bias 2.