What Respondents Do to Answer a Evaluating Survey Questions Question • Comprehend Question • Retrieve Information from Memory Chase H. Harrison Ph.D. • Summarize Information Program on Survey Research • Report an Answer Harvard University Problems in Answering Survey Problems in Answering Survey Questions Questions – Failure to comprehend – Failure to recall • If respondents don’t understand question, they • Questions assume respondents have information cannot answer it • If respondents never learned something, they • If different respondents understand question cannot provide information about it differently, they end up answering different questions • Problems with researcher putting more emphasis on subject than respondent Problems in Answering Survey Problems in Answering Survey Questions Questions – Problems Summarizing – Problems Reporting Answers • If respondents are thinking about a lot of things, • Confusing or vague answer formats lead to they can inconsistently summarize variability • If the way the respondent remembers something • Interactions with interviewers or technology can doesn’t readily correspond to the question, they lead to problems (sensitive or embarrassing may be inconsistemt responses) 1 Evaluating Survey Questions Focus Groups • Early stage • Qualitative research tool – Focus groups to understand topics or dimensions of measures • Used to develop ideas for questionnaires • Pre-Test Stage – Cognitive interviews to understand question meaning • Used to understand scope of issues – Pre-test under typical field conditions • Used to understand contours of findings • Field and Post Stage – Interviewer evaluations • Used to have group evaluate and critique – Behavior coding questions and ideas – Validation to external data – Randomized experiments Focus Groups for Questionnaire Development Focus Groups • Develop parameters of measures • Small group in structured discussion • Lead by trained moderator • Understand typical language and cultural • Uses 8 – 10 “typical” but talkative conventions respondents • Homogenous or heterogeneous groups • Learn about unanticipated responses Moderating Focus Groups Disadvantages of Focus Groups • Develop structured guide for group • Group dynamics can play key role • Encourage respondents to think aloud and • Moderator needs to be skilled discuss • Results not necessarily replicatable • Written exercises can often be used to start group • Requires numerous groups for success and understanding 2 Cognitive Interviews Cognitive Interviews • Administering draft questionnaires • Collecting additional information about responses • Used to evaluate quality of question • Used to understand whether question gathers intended information Typical Framework for Evaluating Cognitive Interviews Responses • Look at question-answering from respondent’s perspective • Comprehension – Understand cognitive strategies used to • Memory Retrieval answer • Information Summarization – Understand how questions are interpreted – Understand how respondents understand • Answer Reporting and Formatting concepts Two Generally Different Different Approaches for Approaches Interviewers • Think-aloud • Standardized: – Facilitate respondent revealing full thought – Standardized probes process – Neutral probing and approach – Relies on standardized training: no specific knowledge • Active probing • Active: – Identify specific problems and answer specific – Interviewer modifies script based on evaluation of questions answering strategies – Plays more active role – Specialized interviewer functions as investigator 3 Thinking Aloud Thinking Aloud • Protocol analysis based in cognitive labs • Ask respondent to think aloud • Requires respondents to “Think Aloud” • Have respondent give free-form answer • Assumes that respondent thoughts are • “What is going through your mind?” – Available – Reported accurately – Does not change further responses Example: Continuing Survey of Thinking Aloud Food Intakes by Individuals (CSFII) • Often begins with generic question and • Original Structure: listens to respondent process of answering – “Starting with the (first/next) time you ate or • Models questions and questionnaire drank something yesterday….. structure based on respondent thought •Time processes • Name of meal • Food item – Examples: • Quantity • Event dating •Place eaten • Recollection forward rather than backward • Place purchased » DeMaio, Ciochetto, and Davis (1994) Example: Continuing Survey of Example: Continuing Survey of Food Intakes by Individuals (CSFII) Food Intakes by Individuals (CSFII) • Cognitive interviews revealed respondents recalled food • 1991 Revision: items more than occasions – Quick list of everything eaten – Naming of time eaten • Respondents used multiple strategies to recall how – Probing of other foods consumed with quick foods were consumed list • Did you have anything else on….. • Did you have anything else in….. • Did you have anything else with – Did you nibble on anything else…. » DeMaio, Ciochetto, and Davis (1994) – Did you have anything else…… 4 Potential Problems with Interviewing with Probes: Respondents Think Out Loud • Respondents veer off course or onto tangents • Read question and probe responses • Respondents focus more on response process – “What made you say that?” than on stimulus of questions – “Why did you respond that way?” • Process of thinking aloud may change – “What does that mean to you?” answering process – “Please tell me what I was asking in your own • Respondents don’t necessarily provide all types words?” of useful information • Potentially overlooks problems following instructions in self-administered questionnaires Example: Types of Probes • “In the past twelve months, how many times have you seen or talked Proactive Administration Reactive Administration on the telephone about your physical, emotional, or mental health (Initiated by interviewer (Triggered by subject with a family doctor or general practitioner?” or administrator) behavior) • Respondent: “Zero” Standardized Construction (1) Anticipated probes (3) Conditional probes • PROBES FROM COGNITIVE INTERVIEWER reveal several doctor (Constructed prior to visits interview) • “Oh, I thought you said talked to on the telephone…..” Non-Standardized Construction (2) Spontaneous Probes (4) Emergent probes (Constructed during the – Adapted from Beatty (2004) interview) From: Willis (2005) Cognitive Interviewing: A Tool for Improving Questionnaire Design Benefits of Active Probing Standardized Approaches • Makes use of expertise • Potentially can be replicated across • Likely more value from fewer interviews facilities, languages, and cultures • May be useful to generate understanding • Can incorporate experimental of types of problems to be included in manipulations and quantitative more standardized phase comparisons • May be better at elucidating rare problems • Facilitate coding and classification of than standardized interviews problems 5 Examples of Classification: Standardized Approaches • Types of Problems: • Require large number of interviews – Lexical –Temporal – Logical • Potentially replicate early mistakes –etc. • Response Stage – Understanding • Often merge with pilot test phase – Task performance – Response formatting –etc. » Conrad and Blair (1996) Selection of Respondents Pilot Tests • Generally limited to convenience samples • Done using realistic field conditions • Relevant population • Help test interviewer instructions and • Demographic variety protocols • Should represent diverse patterns – skip and usage – of survey questionnaire • Data often intensively recorded and • Extreme cases can help to understand analyzed parameters • Respondent and interviewer debriefing • Best if done in a number of locations often conducted • Often conducted iteratively with sets of 5 – 15 respondents Behavior Coding Paralinguistic Measures • Analyzing responses to survey • Coding responses of terms such as: – Comprehension of response – I think – Adequacy of response – I’m not sure • Request for reformulation – Probably • Interpretation of question – Umm…. • Comments and voluntary observations – [Silence] • Use of “Don’t know” • Refusal or other non-answer 6 Response Latency Respondent Debriefing • Length of time to respond is often • “When I asked you ….. Did you think you negatively correlated with would ….?” – Stability • “Were you still thinking when I asked the – Difficulty next question…?” – Accuracy (Current state of Future behavior) • “Did you loose track….?” • “Were you confused?” • Measures of response latency used to • “Did you feel bored or impatient….?” measure quality of question • “Is there something that is relevant that you didn’t tell me?” Interviewer Debriefing Randomized Experiments • Use of interviewers to provide information • Split samples administered different about responses versions of “same” question • Assessment of respondent comprehension • Analysis of: • Assessment of respondent interest – Differences in responses • Interviewer assessment of problems – Accuracy (compared to external knowledge) – Ease of use – Latencies – Percentages don’t know / confused Example of Measure of Chronic Conditions: Further Readings on Pre-testing and Cognitive Testing Question Sequence A: Question Sequence B: • Beaty, Paul C. and Gorden B. Willis. (2007) Research Synthesis: The Practice of Cognitive Interviewing. " Public Opinion Quarterley 71: 287- Do you now have any physical or In the last 12 months, have you seen a 311. A critical synthesis of current practice. medical conditions