IU Workshop in Methods – 2019 Survey Experiments Testing Causality in Diverse Samples Trenton D. Mize Department of Sociology & Advanced Methodologies (AMAP) Purdue University Survey Experiments Page 1 Survey Experiments Page 2 Contents INTRODUCTION ............................................................................................................................................................................ 8 Overview .............................................................................................................................................................................. 8 What is a survey experiment? .................................................................................................................................... 9 What is an experiment?.............................................................................................................................................. 10 Independent and dependent variables ................................................................................................................. 11 Experimental Conditions ............................................................................................................................................. 12 WHY CONDUCT A SURVEY EXPERIMENT? ........................................................................................................................... 13 Internal, external, and construct validity ........................................................................................................... 13 Construct validity ........................................................................................................................................................... 14 External validity ............................................................................................................................................................ 15 Sampling ............................................................................................................................................................................ 16 Sampling for expertise ................................................................................................................................................. 17 Sampling for range ........................................................................................................................................................ 18 Sampling for representation ..................................................................................................................................... 19 Exploratory vs confirmatory approach ............................................................................................................... 21 Why conduct a survey experiment? ..................................................................................................................... 22 CAUSALITY ................................................................................................................................................................................. 23 Requirements for Causality ...................................................................................................................................... 23 Causal inference in observational studies .......................................................................................................... 24 What observational studies can (and cannot) do ........................................................................................... 25 What experiments can do ........................................................................................................................................... 26 Isolating the effects of an independent variable ............................................................................................. 27 Survey Experiments Page 3 Example: control variables for modeling the gender wage gap .............................................................. 28 Control variables in models of observational data ........................................................................................ 29 Random assignment .................................................................................................................................................... 30 Random assignment to condition ........................................................................................................................... 39 Why conduct experimental studies? ...................................................................................................................... 40 GOOD EXPERIMENTAL DESIGN IS HARD ............................................................................................................................. 43 Effective Experimental Design Is Difficult ......................................................................................................... 44 Survey Design ................................................................................................................................................................. 48 EXAMPLES OF – AND BEST PRACTICES FOR – SURVEY EXPERIMENTS .......................................................................... 49 Overview ........................................................................................................................................................................... 49 Measurement Experiments ...................................................................................................................................... 50 Measurement experiment example: voter turnout ........................................................................................ 51 Control / Baseline Conditions ................................................................................................................................... 53 List Treatment Experiments .................................................................................................................................... 54 Social Desirability ........................................................................................................................................................... 54 Social desirability and studying sensitive topics ............................................................................................. 55 The list treatment experimental design............................................................................................................... 56 List treatment experiments to combat social desirability .......................................................................... 57 General advice for list treatment experiments ................................................................................................. 60 Stereotype Content Studies ...................................................................................................................................... 61 Designing continuous dependent variables ....................................................................................................... 64 Using scales for dependent variables .................................................................................................................... 65 Between vs Within-Subjects Designs ..................................................................................................................... 68 Pros and cons of between and within subjects designs ................................................................................ 69 Survey Experiments Page 4 Pros and cons of between and within subjects designs ................................................................................ 70 Group Position Experiments .................................................................................................................................... 71 Designing Effective IV Manipulations ................................................................................................................... 73 Ethics and Deception .................................................................................................................................................... 74 Addressing and justifying deception ..................................................................................................................... 75 Hiring Studies ................................................................................................................................................................. 76 Race, sexual orientation, and hiring (Pedulla 2014 SPQ) ........................................................................... 76 Measuring continuous dependent variables ...................................................................................................... 79 Putting dependent measures in context .............................................................................................................. 80 Using names to manipulate race ............................................................................................................................. 81 Vignette Studies ............................................................................................................................................................. 83 Example vignette study: Mize and Manago 2018 ASR.................................................................................. 84 Assessing categorization and certainty as a DV .............................................................................................