What You Need

Introduction to Design 1. - what to ask The Total Survey Error Approach 2. - who gets it Dino P. Christenson 3. Implementation - how to collect it Department of Political Science Hariri Institute for Computational Science & Engineering 4. Statistics - how to understand it

1 2

Survey Implementation

• Survey houses (academic & private) What do you want to know? • • E.g., , YouGov, ISR at UofM, SSI • Do we already know it? • Academic surveys • Or, have we already asked it? • E.g., CCES, ANES, GSS, TOPS • Second hand data can be nice! • DIY • Paper and pencil If not, then, ask it: • • Door-to-door or mail or telephone • What’s your question? (modes) • Online software (e.g., Qualtrics, • Regardless, requires in-depth knowledge of the lit SurveyGizmo)

3 4

Sample Population Statistics

• Representative of the population! • For some good designs... • Which population? Just need the basics • Convenience samples • • Friends and family (not • PO 502 or some recommended) introduction to Sample applied social science • Students statistics • Opt-in online

5 6 Total Survey Error Good’ish News Approach

• Systematic way to consider tradeoffs in conducting a survey • Lots of potential errors Error • Where to expend resources Cost • But no perfect study! Cost • Time Error • Think about likely errors for your study • Money • TSE helps to minimize specific errors • Ethics by focusing on tradeoffs • In order to minimize survey error

7 8

Sampling Error Types of Error Respondent Selection Coverage Error Issues Two types of error but Unit Level Nonresponse Error • we care more about one Item Level Nonresponse Error of them Response Accuracy Respondent Measurement Error Issues 1. Random Interviewer Measurement Error Survey Post-Survey Error • Occur by chance, Administration Mode Effects without a pattern Issues Equivalence Error Weisberg 2005 2. Systematic Types of Survey Error • Can bias the results

9 10

Respondent Minimize Respondent Measurement Error Measurement Error • Response accuracy problem • Use vetted survey • Simplify each “stage of How many times did you go questions survey response” • Respondent lacks motivation to answer Do youto the support Dr’s office tax last year? correctly credits!Shouldfor the theproduction, government • Conduct pre-tests of the 1. Comprehension collectionWouldspend and you lesstransportation be money more on likely the or questionnaire • Unclear question wording of biomasslessmilitary likelythat istoand usedvote more for onJohn 2. Retrieval • Temporal issues energyMcCain production? education?for president if you • Use “think-aloud protocols” 3. Judgment • Double-barreled knew he had fathered an illegitimate black child? Overly sophisticated • Random half-samples of 4. Reporting • question wording Biased question wording • Tourangeau, Rips and Rasinski 2000

11 12 Interviewer The High & Low Roads Measurement Error Minimize “satisficing” • Solutions • • Interviewer objective: • Responding in order to move on rather than responding after • Time survey • Facilitate carefully thinking through the responses question • Obtain accurate answers Encourage respondent Potential problems • • But they can also introduce error • engagement Random • E.g., very short answers to • open ended questions • Mix up the direction E.g., wrongly records an answer of response options • • E.g., long batteries with same • Systematic response options Break up questions • • E.g., always mispronounces a word Krosnick and Alwin 1987 13 14

Minimize Interviewer Modes Matter Measurement Error • Standardized interviewing • Interviewer error vanishes • Ask the identical question the same exact way • Survey modes Mail to all respondents • • Face-to-face Internet • Do NOT interject own views or supply • extra info • Mail • Costs shrink too! • Conversational interviewing • Telephone • Tradeoff • Help respondents understand the questions • Internet • Response accuracy may • Can clarify meanings to achieve accurate decline, especially on responses open ended questions

15 16

Minimize Item Item Nonresponse Nonresponse • Nonresponse on particular survey questions • Require answering question • E.g., refusals, skipped questions, inadequate • Tradeoff: Respondent drops out response options • Skilled interviewer can encourage answers • Results biased when those who answer are Tradeoff: cost in training interviewer different than those who don’t • • Multiple imputation • E.g., study of income on vote choice, but if higher income vote more conservatively but • Create values for missing values via predicted don’t report income than relationship values from regressions with random error term understated • Requires a lot of data and missing at random Weisberg N.d.

17 18 Minimize Unit Unit Nonresponse Nonresponse • Respondents in sample do not take survey • Tailor interview request as valuable to the respondent • Cannot be contacted • Refuse to take it • Pay respondents to participate • Can bias the sample if those who participate are • E.g., $1-$5 can yield 2%-12% increase with systematically different than those who do not diminishing returns • Refusal rate increasing • Unless very large • Some conservative pundits discourage participation • E.g., ANES 2012 $25-$100 yields 38% Could result in underestimate of Rep vote pre- and 94% post • Cantor, O’Hare, and O’Connor 2008, Singer and Ye 2013 19 20 Weisberg N.d.

Minimize Coverage Coverage Error Error • Address-based uses addresses instead of telephone numbers • Discrepancy between list of In 2012 Republican population and actual Internet surveys initially high coverage population pollsters overstated • Romney’s chances error but decreasing steadily with greater • E.g., sampling from a because cell phone Internet access telephone book which numbers were not misses all those with sampled • Use multiple sampling frames - and weight unlisted telephone numbers Weisberg N.d. those with greater probability of falling into sample

21 22

Minimize Sampling Error • Any time we interview only a sample of the Increase sample size population • • By chance our estimates will be off from the • Tradeoff: can be costly population • Less so for internet • With probability sampling we therefore provide a surveys margin of error • But more is not • Conventional confidence interval is 95% always better Estimate is within 2.5% of true population • • 1936 Literary estimate Digest poll of 2m

23 24 Minimize Sampling Minimize Sampling

Error Samples Error Samples • Stratified sample Subgroups • Stratified sample Subgroups • Take proportions • Take proportions from subcategories, from subcategories, e.g., regions e.g., regions • Cluster sample • Cluster sample • Sample within known • Sample within known clusters, e.g., city clusters, e.g., city blocks blocks

25 26

Sampling & the Internet Convenience Samples

• Internet surveys • Most use opt-in polls • Sampling errors cannot be validly • Difficult to conduct computed • Getting respondents is: probability sampling • Increased risk of selection bias Tough • MechanicalTurk • Email list of the population • Similar to coverage error and of interest? nonresponse biases • Expensive Solution: weight respondents • Option: probability • • Time consuming sample via telephone or • E.g., poststratification mail requesting they take adjustment, sample matching, an online survey propensity score weights

27 28

Convenience Samples Survey Mode Effects

• Crowd source your sample! • How the survey is conducted • Not always appropriate • Face-to-face & telephone Symbolic Racism Scale 1.!! It’s really a matter of some people not trying See previous slide Interviewer effects, esp on sensitive • Just • hard enough; if blacks would only try harder they questions could be just as well off as whites • Mechanical Turk okay for Human • Social desirability bias, appear likable 2.! Irish, Italian, Jewish and many other minorities • Experiments Berinsky, Huber & Lenz 2011 to interviewer overcame prejudice and worked their way up.! Blacks should do the same... When other approaches difficult Solution: phrasing of questions to • • Henry, P. J., & Sears, D. O.!2002! legitimate all responses • Tight panel waves around developing events • Solution: use interviewer-less modes Christenson & Glick 2014

29 30 Survey Mode Costs Post-Survey Error

Junk Mail • Money • Time Spam • Response • Error during the processing and analysis of survey data 1. Face-to-face 1. Face-to-face 1. Internet E.g., coding open-ended questions 2. Telephone 2. Mail (awaiting response) 2. Mail • • Solution: create comprehensive coding 3. Mail 3. Telephone 3. Telephone schemes 4. Internet 4. Internet 4. Face-to-face • Solution: calculate inter-coder reliability Interaction Ritual

31 32

Equivalence Error Sampling Error Respondent Selection Coverage Error Issues • Lack of equivalence of surveys measuring same • Solution: tread Unit Level Nonresponse Error concepts carefully when Item Level Nonresponse Error comparing surveys Response • House effects, survey organization regularly across Accuracy Respondent Measurement Error attaining more of one response than another Issues Interviewer Measurement Error time • Post-Survey Error • Different countries, interpretations differ by Survey culture Administration Mode Effects • countries Issues Equivalence Error Different times, real world conditions change Weisberg 2005 • • survey • E.g., “liberal” and “conservative” different organizations today than in the past Types of Survey Error

33 34

Resources Resources Books Articles • Groves, Robert M. (1989). Survey Errors and Survey Costs. New York: Wiley. • Weisberg, Herbert (N.d.). Total Survey Error. In Atkenson, Lonna and R. Michael Alvarez, editors. Handbook on Polling and Polling Methods. Oxford University Press. • Biemer, P.; Lyberg, L. (2003). Introduction to Survey Quality. Wiley & Sons, Inc. • Krosnick, Jon A., & Duane F. Alwin (1987). An Evaluation of a Cognitive Theory of • Weisberg, Herbert F. (2005). The Total Survey Error Approach: A Response-Order Effects in Survey Measurement. Quarterly 51(2): Guide to the New Science of Survey Research, University of 201-219. Chicago Press Henry, P. J., & Sears, D. O.! (2002).! The symbolic racism 2000 scale. !Political • Asher, Herbert. (2007). Polling and the Public: What Every • Citizen Should Know, 8th Edition. Washington, DC: CQ Press. Psychology, 23, 253-283. Cantor, David, Barbara O’Hare, & Karen O’Connor (2008). “The Use of Monetary • Groves, R.; Fowler, F.; Couper, M.; Lepkowski, J.; Singer, E.; • Tourangeau, R. (2009). (2nd Edition). Wiley Incentives to Reduce Non-Response in Random Digit Dial Telephone Surveys” pp. & Sons, Inc. 471-498 in James M. Lepkowski, Clyde Tucker, J. Michael Brick, Edith D. de Leeuw, • Lohr, Sharon L. (2010). Sampling: Design and Analysis, 2nd ed. Lilli Japec, Paul J. Lavrakas, Michael W. Link, & Roberta L. Sangster (Eds.), Advances Boston: Cengage Learning. In Telephone Survey Methodology, New York: J.W. Wiley and Sons. • Atkenson, Lonna and R. Michael Alvarez (N.d.). Oxford • Singer, Eleanor & Cong Ye (2013). The Use and Effects of Incentives in Surveys. University Press Handbook on Polling and Polling Methods. Annals of the American Academy of Political and Social Science 645 (Jan.): 112-41. • Tourangeau, Roger, Lance J. Rips, & Kenneth Rasinski (2000). The Psychology of Survey Response. Cambridge, UK: Cambridge University Press.

35 36 Resources Thank You POQ Special Edition

• Paul P. Biemer. Total Survey Error: Design, Implementation, and • Olena Kaminska, Allan L. McCutcheon,and Jaak Billiet. Evaluation Public Opin Q (2010) 74 (5): 817-848 doi:10.1093/ Satisficing Among Reluctant Respondents in a Cross-National poq/nfq058 Context Public Opin Q (2010) 74 (5): 956-984 doi:10.1093/ poq/nfq062 • Robert M. Groves and Lars Lyberg. Total Survey Error: Past, Present, and Future Public Opin Q (2010) 74 (5): 849-879 doi: • Wendy D. Hicks,Brad Edwards, Karen Tourangeau, Brett 10.1093/poq/nfq065 McBride, Lauren D. Harris-Kojetin,and Abigail J. Moss. Using Cari Tools To Understand Measurement Error Public Opin Q (2010) 74 (5): 985-1003 doi:10.1093/poq/ I welcome your questions • Frauke Kreuter, Gerrit Müller, and Mark Trappmann. nfq063 • Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data Public Opin Q (2010) 74 (5): 880-906 doi:10.1093/poq/nfq060 • Brady T. West and Kristen Olson. How Much of Interviewer Also via email: [email protected] Variance is Really Nonresponse Error Variance? Public Opin • Q (2010) 74 (5): 1004-1026 doi:10.1093/poq/nfq061 • Joseph W. Sakshaug,Ting Yan,and Roger Tourangeau. Nonresponse Error, Measurement Error, And Mode Of : Tradeoffs in a Multi-mode Survey of Sensitive and • Jorre Vannieuwenhuyze, Geert Loosveldt, and Geert Non-sensitive Items Public Opin Q (2010) 74 (5): 907-933 Molenberghs. A Method for Evaluating Mode Effects in Mixed- doi:10.1093/poq/nfq057 mode Surveys Public Opin Q (2010) 74 (5): 1027-1045 doi: 10.1093/poq/nfq059 • Scott Fricker and Roger Tourangeau. Examining the Relationship Between Nonresponse Propensity and Data Quality in Two National Household Surveys Public Opin Q (2010) 74 (5): 934-955 doi:10.1093/poq/nfq064 37 38