DOCSLIB.ORG

Survey Experiments

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Survey Experiments IU Workshop in Methods – 2019 Survey Experiments Testing Causality in Diverse Samples Trenton D. Mize Department of Sociology & Advanced Methodologies (AMAP) Purdue University Survey Experiments Page 1 Survey Experiments Page 2 Contents INTRODUCTION ............................................................................................................................................................................ 8 Overview .............................................................................................................................................................................. 8 What is a survey experiment? .................................................................................................................................... 9 What is an experiment?.............................................................................................................................................. 10 Independent and dependent variables ................................................................................................................. 11 Experimental Conditions ............................................................................................................................................. 12 WHY CONDUCT A SURVEY EXPERIMENT? ........................................................................................................................... 13 Internal, external, and construct validity ........................................................................................................... 13 Construct validity ........................................................................................................................................................... 14 External validity ............................................................................................................................................................ 15 Sampling ............................................................................................................................................................................ 16 Sampling for expertise ................................................................................................................................................. 17 Sampling for range ........................................................................................................................................................ 18 Sampling for representation ..................................................................................................................................... 19 Exploratory vs confirmatory approach ............................................................................................................... 21 Why conduct a survey experiment? ..................................................................................................................... 22 CAUSALITY ................................................................................................................................................................................. 23 Requirements for Causality ...................................................................................................................................... 23 Causal inference in observational studies .......................................................................................................... 24 What observational studies can (and cannot) do ........................................................................................... 25 What experiments can do ........................................................................................................................................... 26 Isolating the effects of an independent variable ............................................................................................. 27 Survey Experiments Page 3 Example: control variables for modeling the gender wage gap .............................................................. 28 Control variables in models of observational data ........................................................................................ 29 Random assignment .................................................................................................................................................... 30 Random assignment to condition ........................................................................................................................... 39 Why conduct experimental studies? ...................................................................................................................... 40 GOOD EXPERIMENTAL DESIGN IS HARD ............................................................................................................................. 43 Effective Experimental Design Is Difficult ......................................................................................................... 44 Survey Design ................................................................................................................................................................. 48 EXAMPLES OF – AND BEST PRACTICES FOR – SURVEY EXPERIMENTS .......................................................................... 49 Overview ........................................................................................................................................................................... 49 Measurement Experiments ...................................................................................................................................... 50 Measurement experiment example: voter turnout ........................................................................................ 51 Control / Baseline Conditions ................................................................................................................................... 53 List Treatment Experiments .................................................................................................................................... 54 Social Desirability ........................................................................................................................................................... 54 Social desirability and studying sensitive topics ............................................................................................. 55 The list treatment experimental design............................................................................................................... 56 List treatment experiments to combat social desirability .......................................................................... 57 General advice for list treatment experiments ................................................................................................. 60 Stereotype Content Studies ...................................................................................................................................... 61 Designing continuous dependent variables ....................................................................................................... 64 Using scales for dependent variables .................................................................................................................... 65 Between vs Within-Subjects Designs ..................................................................................................................... 68 Pros and cons of between and within subjects designs ................................................................................ 69 Survey Experiments Page 4 Pros and cons of between and within subjects designs ................................................................................ 70 Group Position Experiments .................................................................................................................................... 71 Designing Effective IV Manipulations ................................................................................................................... 73 Ethics and Deception .................................................................................................................................................... 74 Addressing and justifying deception ..................................................................................................................... 75 Hiring Studies ................................................................................................................................................................. 76 Race, sexual orientation, and hiring (Pedulla 2014 SPQ) ........................................................................... 76 Measuring continuous dependent variables ...................................................................................................... 79 Putting dependent measures in context .............................................................................................................. 80 Using names to manipulate race ............................................................................................................................. 81 Vignette Studies ............................................................................................................................................................. 83 Example vignette study: Mize and Manago 2018 ASR.................................................................................. 84 Assessing categorization and certainty as a DV .............................................................................................
Load more

Recommended publications
  • Statistical Inference: How Reliable Is a Survey?
    Math 203, Fall 2008: Statistical Inference: How reliable is a survey? Consider a survey with a single question, to which respondents are asked to give an answer of yes or no. Suppose you pick a random sample of n people, and you find that the proportion that answered yes isp ˆ. Question: How close isp ˆ to the actual proportion p of people in the whole population who would have answered yes? In order for there to be a reliable answer to this question, the sample size, n, must be big enough so that the sample distribution is close to a bell shaped curve (i.e., close to a normal distribution). But even if n is big enough that the distribution is close to a normal distribution, usually you need to make n even bigger in order to make sure your margin of error is reasonably small. Thus the first thing to do is to be sure n is big enough for the sample distribution to be close to normal. The industry standard for being close enough is for n to be big enough so that 1 − p 1 − p n > 9 and n > 9 p p both hold. When p is about 50%, n can be as small as 10, but when p gets close to 0 or close to 1, the sample size n needs to get bigger. If p is 1% or 99%, then n must be at least 892, for example. (Note also that n here depends on p but not on the size of the whole population.) See Figures 1 and 2 showing frequency histograms for the number of yes respondents if p = 1% when the sample size n is 10 versus 1000 (this data was obtained by running a computer simulation taking 10000 samples).
    [Show full text]
  • On Becoming a Pragmatic Researcher: the Importance of Combining Quantitative and Qualitative Research Methodologies
    DOCUMENT RESUME ED 482 462 TM 035 389 AUTHOR Onwuegbuzie, Anthony J.; Leech, Nancy L. TITLE On Becoming a Pragmatic Researcher: The Importance of Combining Quantitative and Qualitative Research Methodologies. PUB DATE 2003-11-00 NOTE 25p.; Paper presented at the Annual Meeting of the Mid-South Educational Research Association (Biloxi, MS, November 5-7, 2003). PUB TYPE Reports Descriptive (141) Speeches/Meeting Papers (150) EDRS PRICE EDRS Price MF01/PCO2 Plus Postage. DESCRIPTORS *Pragmatics; *Qualitative Research; *Research Methodology; *Researchers ABSTRACT The last 100 years have witnessed a fervent debate in the United States about quantitative and qualitative research paradigms. Unfortunately, this has led to a great divide between quantitative and qualitative researchers, who often view themselves in competition with each other. Clearly, this polarization has promoted purists, i.e., researchers who restrict themselves exclusively to either quantitative or qualitative research methods. Mono-method research is the biggest threat to the advancement of the social sciences. As long as researchers stay polarized in research they cannot expect stakeholders who rely on their research findings to take their work seriously. The purpose of this paper is to explore how the debate between quantitative and qualitative is divisive, and thus counterproductive for advancing the social and behavioral science field. This paper advocates that all graduate students learn to use and appreciate both quantitative and qualitative research. In so doing, students will develop into what is termed "pragmatic researchers." (Contains 41 references.) (Author/SLD) Reproductions supplied by EDRS are the best that can be made from the original document. On Becoming a Pragmatic Researcher 1 Running head: ON BECOMING A PRAGMATIC RESEARCHER U.S.
    [Show full text]
  • Those Missing Values in Questionnaires
    Those Missing Values in Questionnaires John R. Gerlach, Maxim Group, Plymouth Meeting, PA Cindy Garra, IMS HEALTH; Plymouth Meeting, PA Abstract alternative values are not bona fide SAS missing values; consequently, a SAS procedure, expectedly, will include Questionnaires are notorious for containing these alternative values, thereby producing bogus results. responses that later become missing values during processing and analyses. Contrary to a non-response that Typically, you can re-code the original data, so results in a typical, bona fide, missing value, questions that the missing values become ordinary missing values, might allow several alternative responses, necessarily thus allowing SAS to process only appropriate values. Of outside the usual range of appropriate responses. course, there's a loss of information since the original Whether a question represents continuous or categorical missing values did not constitute a non-response. Also, data, a good questionnaire offers meaningful alternatives, such pre-processing might include a myriad of IF THEN / such as: "Refused to Answer" and, of course, the ELSE statements, which can be very tedious and time- quintessential "Don't Know." Traditionally, these consuming to write, and difficult to maintain. Thus, given alternative responses have numeric values such as 97, a questionnaire that has over a hundred variables with 998, or 9999 and, therefore, pose problems when trying varying levels of missing values, the task of re-coding to distinguish them from normal responses, especially these variables becomes very time consuming at best. when multiple missing values exist. This paper discusses Even worse, the permanent re-coding of alternative missing values in SAS and techniques that facilitate the responses to ordinary missing numeric values in SAS process of dealing with multi-valued, meaningful missing precludes categorical analysis that requires the inclusion values often found in questionnaires.
    [Show full text]
  • 10 Questions Opinion Polls
    Questions you might have on 10opinion polls 1. What is an opinion poll? An opinion poll is a survey carried out to measure views on a certain topic within a specific group of people. For example, the topic may relate to who Kenyans support in the presidential race, in which case, the group of people interviewed will be registered voters. 2. How are interviewees for an opinion poll selected? The group of people interviewed for an opinion poll is called a sample. As the name suggests, a sample is a group of people that represents the total population whose opinion is being surveyed. In a scientific opinion poll, everyone has an equal chance of being interviewed. 3. So how come I have never been interviewed for an opinion poll? You have the same chance of being polled as anyone else living in Kenya. However, chances of this are very small and are estimated at about 1 in 14,000. This is because there are approximately 14 million registered voters in Kenya and, for practical and cost reasons, usually only between 1,000 and 2,000 people are interviewed for each survey carried out. 4. How can such a small group be representative of the entire population? In order to ensure that the sample/survey group is representative of the population, the surveyors must ensure that the group reflects the characteristics of the whole. For instance, to get a general idea of who might win the Kenyan presidential election, only the views of registered voters in Kenya will be surveyed as these are the people who will be able to influence the election.
    [Show full text]
  • SAMPLING DESIGN & WEIGHTING in the Original
    Appendix A 2096 APPENDIX A: SAMPLING DESIGN & WEIGHTING In the original National Science Foundation grant, support was given for a modified probability sample. Samples for the 1972 through 1974 surveys followed this design. This modified probability design, described below, introduces the quota element at the block level. The NSF renewal grant, awarded for the 1975-1977 surveys, provided funds for a full probability sample design, a design which is acknowledged to be superior. Thus, having the wherewithal to shift to a full probability sample with predesignated respondents, the 1975 and 1976 studies were conducted with a transitional sample design, viz., one-half full probability and one-half block quota. The sample was divided into two parts for several reasons: 1) to provide data for possibly interesting methodological comparisons; and 2) on the chance that there are some differences over time, that it would be possible to assign these differences to either shifts in sample designs, or changes in response patterns. For example, if the percentage of respondents who indicated that they were "very happy" increased by 10 percent between 1974 and 1976, it would be possible to determine whether it was due to changes in sample design, or an actual increase in happiness. There is considerable controversy and ambiguity about the merits of these two samples. Text book tests of significance assume full rather than modified probability samples, and simple random rather than clustered random samples. In general, the question of what to do with a mixture of samples is no easier solved than the question of what to do with the "pure" types.
    [Show full text]
  • Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System
    Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System Release year: 2006 Authors: G. Brancato, S. Macchia, M. Murgia, M. Signore, G. Simeoni - Italian National Institute of Statistics, ISTAT K. Blanke, T. Körner, A. Nimmergut - Federal Statistical Office Germany, FSO P. Lima, R. Paulino - National Statistical Institute of Portugal, INE J.H.P. Hoffmeyer-Zlotnik - German Center for Survey Research and Methodology, ZUMA Version 1 Acknowledgements We are grateful to the experts from the network countries who supported us in all relevant stages of the work: Anja Ahola, Dirkjan Beukenhorst, Trine Dale, Gustav Haraldsen. We also thank all colleagues from European and overseas NSIs who helped us in understanding the current practices and in the review of the draft version of the handbook. Executive summary Executive Summary Questionnaires constitute the basis of every survey-based statistical measurement. They are by far the most important measurement instruments statisticians use to grasp the phenomena to be measured. Errors due to an insufficient questionnaire can hardly be compensated at later stages of the data collection process. Therefore, having systematic questionnaire design and testing procedures in place is vital for data quality, particularly for a minimisation of the measurement error. Against this background, the Directors General of the members of the European Statistical System (ESS) stressed the importance of questionnaire design and testing in the European Statistics Code of Practice, endorsed in February 2005. Principle 8 of the Code states that “appropriate statistical procedures, implemented from data collection to data validation, must underpin quality statistics.” One of the indicators referring to this principle requires that “questionnaires are systematically tested prior to the data collection.” Taking the Code of Practice as a starting point, this Recommended Practice Manual aims at further specifying the requirements of the Code of Practice.
    [Show full text]
  • Meta-Analysis of the Relationships Between Different Leadership Practices and Organizational, Teaming, Leader, and Employee Outcomes*
    Journal of International Education and Leadership Volume 8 Issue 2 Fall 2018 http://www.jielusa.org/ ISSN: 2161-7252 Meta-Analysis of the Relationships Between Different Leadership Practices and Organizational, Teaming, Leader, and Employee Outcomes* Carl J. Dunst Orelena Hawks Puckett Institute Mary Beth Bruder University of Connecticut Health Center Deborah W. Hamby, Robin Howse, and Helen Wilkie Orelena Hawks Puckett Institute * This study was supported, in part, by funding from the U.S. Department of Education, Office of Special Education Programs (No. 325B120004) for the Early Childhood Personnel Center, University of Connecticut Health Center. The contents and opinions expressed, however, are those of the authors and do not necessarily reflect the policy or official position of either the Department or Office and no endorsement should be inferred or implied. The authors report no conflicts of interest. The meta-analysis described in this paper evaluated the relationships between 11 types of leadership practices and 7 organizational, teaming, leader, and employee outcomes. A main focus of analysis was whether the leadership practices were differentially related to the study outcomes. Studies were eligible for inclusion if the correlations between leadership subscale measures (rather than global measures of leadership) and outcomes of interest were reported. The random effects weighted average correlations between the independent and dependent measures were used as the sizes of effects for evaluating the influences of the leadership practices on the outcome measures. One hundred and twelve studies met the inclusion criteria and included 39,433 participants. The studies were conducted in 31 countries in different kinds of programs, organizations, companies, and businesses.
    [Show full text]
  • Summary of Human Subjects Protection Issues Related to Large Sample Surveys
    Summary of Human Subjects Protection Issues Related to Large Sample Surveys U.S. Department of Justice Bureau of Justice Statistics Joan E. Sieber June 2001, NCJ 187692 U.S. Department of Justice Office of Justice Programs John Ashcroft Attorney General Bureau of Justice Statistics Lawrence A. Greenfeld Acting Director Report of work performed under a BJS purchase order to Joan E. Sieber, Department of Psychology, California State University at Hayward, Hayward, California 94542, (510) 538-5424, e-mail [email protected]. The author acknowledges the assistance of Caroline Wolf Harlow, BJS Statistician and project monitor. Ellen Goldberg edited the document. Contents of this report do not necessarily reflect the views or policies of the Bureau of Justice Statistics or the Department of Justice. This report and others from the Bureau of Justice Statistics are available through the Internet — http://www.ojp.usdoj.gov/bjs Table of Contents 1. Introduction 2 Limitations of the Common Rule with respect to survey research 2 2. Risks and benefits of participation in sample surveys 5 Standard risk issues, researcher responses, and IRB requirements 5 Long-term consequences 6 Background issues 6 3. Procedures to protect privacy and maintain confidentiality 9 Standard issues and problems 9 Confidentiality assurances and their consequences 21 Emerging issues of privacy and confidentiality 22 4. Other procedures for minimizing risks and promoting benefits 23 Identifying and minimizing risks 23 Identifying and maximizing possible benefits 26 5. Procedures for responding to requests for help or assistance 28 Standard procedures 28 Background considerations 28 A specific recommendation: An experiment within the survey 32 6.
    [Show full text]
  • Evaluating Survey Questions Question
    What Respondents Do to Answer a Evaluating Survey Questions Question • Comprehend Question • Retrieve Information from Memory Chase H. Harrison Ph.D. • Summarize Information Program on Survey Research • Report an Answer Harvard University Problems in Answering Survey Problems in Answering Survey Questions Questions – Failure to comprehend – Failure to recall • If respondents don’t understand question, they • Questions assume respondents have information cannot answer it • If respondents never learned something, they • If different respondents understand question cannot provide information about it differently, they end up answering different questions • Problems with researcher putting more emphasis on subject than respondent Problems in Answering Survey Problems in Answering Survey Questions Questions – Problems Summarizing – Problems Reporting Answers • If respondents are thinking about a lot of things, • Confusing or vague answer formats lead to they can inconsistently summarize variability • If the way the respondent remembers something • Interactions with interviewers or technology can doesn’t readily correspond to the question, they lead to problems (sensitive or embarrassing may be inconsistemt responses) 1 Evaluating Survey Questions Focus Groups • Early stage • Qualitative research tool – Focus groups to understand topics or dimensions of measures • Used to develop ideas for questionnaires • Pre-Test Stage – Cognitive interviews to understand question meaning • Used to understand scope of issues – Pre-test under typical field
    [Show full text]
  • Lab Experiments Are a Major Source of Knowledge in the Social Sciences
    IZA DP No. 4540 Lab Experiments Are a Major Source of Knowledge in the Social Sciences Armin Falk James J. Heckman October 2009 DISCUSSION PAPER SERIES Forschungsinstitut zur Zukunft der Arbeit Institute for the Study of Labor Lab Experiments Are a Major Source of Knowledge in the Social Sciences Armin Falk University of Bonn, CEPR, CESifo and IZA James J. Heckman University of Chicago, University College Dublin, Yale University, American Bar Foundation and IZA Discussion Paper No. 4540 October 2009 IZA P.O. Box 7240 53072 Bonn Germany Phone: +49-228-3894-0 Fax: +49-228-3894-180 E-mail: [email protected] Any opinions expressed here are those of the author(s) and not those of IZA. Research published in this series may include views on policy, but the institute itself takes no institutional policy positions. The Institute for the Study of Labor (IZA) in Bonn is a local and virtual international research center and a place of communication between science, politics and business. IZA is an independent nonprofit organization supported by Deutsche Post Foundation. The center is associated with the University of Bonn and offers a stimulating research environment through its international network, workshops and conferences, data service, project support, research visits and doctoral program. IZA engages in (i) original and internationally competitive research in all fields of labor economics, (ii) development of policy concepts, and (iii) dissemination of research results and concepts to the interested public. IZA Discussion Papers often represent preliminary work and are circulated to encourage discussion. Citation of such a paper should account for its provisional character.
    [Show full text]
  • Analysis of Variance and Analysis of Variance and Design of Experiments of Experiments-I
    Analysis of Variance and Design of Experimentseriments--II MODULE ––IVIV LECTURE - 19 EXPERIMENTAL DESIGNS AND THEIR ANALYSIS Dr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur 2 Design of experiment means how to design an experiment in the sense that how the observations or measurements should be obtained to answer a qqyuery inavalid, efficient and economical way. The desigggning of experiment and the analysis of obtained data are inseparable. If the experiment is designed properly keeping in mind the question, then the data generated is valid and proper analysis of data provides the valid statistical inferences. If the experiment is not well designed, the validity of the statistical inferences is questionable and may be invalid. It is important to understand first the basic terminologies used in the experimental design. Experimental unit For conducting an experiment, the experimental material is divided into smaller parts and each part is referred to as experimental unit. The experimental unit is randomly assigned to a treatment. The phrase “randomly assigned” is very important in this definition. Experiment A way of getting an answer to a question which the experimenter wants to know. Treatment Different objects or procedures which are to be compared in an experiment are called treatments. Sampling unit The object that is measured in an experiment is called the sampling unit. This may be different from the experimental unit. 3 Factor A factor is a variable defining a categorization. A factor can be fixed or random in nature. • A factor is termed as fixed factor if all the levels of interest are included in the experiment.
    [Show full text]
  • The Politics of Random Assignment: Implementing Studies and Impacting Policy
    The Politics of Random Assignment: Implementing Studies and Impacting Policy Judith M. Gueron Manpower Demonstration Research Corporation (MDRC) As the only nonacademic presenting a paper at this conference, I see it as my charge to focus on the challenge of implementing random assignment in the field. I will not spend time arguing for the methodological strengths of social experiments or advocating for more such field trials. Others have done so eloquently.1 But I will make my biases clear. For 25 years, I and many of my MDRC colleagues have fought to implement random assignment in diverse arenas and to show that this approach is feasible, ethical, uniquely convincing, and superior for answering certain questions. Our organization is widely credited with being one of the pioneers of this approach, and through its use producing results that are trusted across the political spectrum and that have made a powerful difference in social policy and research practice. So, I am a believer, but not, I hope, a blind one. I do not think that random assignment is a panacea or that it can address all the critical policy questions, or substitute for other types of analysis, or is always appropriate. But I do believe that it offers unique power in answering the “Does it make a difference?” question. With random assignment, you can know something with much greater certainty and, as a result, can more confidently separate fact from advocacy. This paper focuses on implementing experiments. In laying out the ingredients of success, I argue that creative and flexible research design skills are essential, but that just as important are operational and political skills, applied both to marketing the experiment in the first place and to helping interpret and promote its findings down the line.
    [Show full text]