Questionnaire Design
Total Page:16
File Type:pdf, Size:1020Kb
Questionnaire Design David Ashley Jeffrey Henning • Human Capital Data Analytics • President of Division Manager for the U.S. Researchscape International Department of Homeland Security • Past President of the MRII (2009) • Current President of the MRII and and the author of upcoming the editor of the upcoming questionnaire design course questionnaire design course • [email protected] • [email protected] • @jhenning on Twitter JH DA Agenda 1. How respondents think 2. Questionnaire design overview 3. Addressing common mistakes Is a Survey the Right Arrow to Hit the Target? • Sometimes the best survey is to not do a survey at all • Talk to stakeholders who will use the data to understand their wants and needs • Is someone elsewhere in the organization doing a survey on this topic or researching this issue? • Are customers (or employees or …) the only source of this information? • Do your CRM, web analytics or other systems hold data that would address this issue? JH Asking a Lot of the Respondent Literally and Figuratively 1. Interpret the meaning of a question 2. Recall all relevant facts related to question 3. Internally summarize those facts 4. Report summary judgment accurately JH Respondent Behaviors Cognitive Social Survey Behaviors Behaviors Behaviors Satisficing Acquiescence bias Response styles Memory biases Social desirability bias Response substitution Economic behavior Halo error Mode effects Practice effects Panel conditioning JH Weak Satisficing Strong Satisficing • Selecting the first choice that • Endorsing the status quo appears reasonable instead of change • Agreeing with assertions • Failing to differentiate in (“acquiescence response ratings bias”) • Selecting “Don’t know” rather than giving an opinion Source: Krosnick, J. A. (1991). “Response Strategies for Coping with the Cognitive • Randomly choosing Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology, 5, 213-236. JH Memory Biases Choice-supportive bias Generation effect Osborn effect Source Confusion Change bias Illusion-of-truth effect Part-list cueing effect Spacing effect Childhood amnesia Lag effect Peak-end effect Stereotypical bias Consistency bias Leveling and Sharpening Persistence Suffix effect Context effect Levels-of-processing effect Picture superiority effect Suggestibility Cross-race effect List-length effect Positivity effect Telescoping effect Cryptomnesia Misinformation effect Primacy effect Testing effect Egocentric bias Misattribution Processing difficulty effect Tip of the tongue Fading affect bias Modality effect Reminiscence bump Verbatim effect Hindsight bias Mood congruent memory bias Rosy retrospection Von Restorff effect Humor effect Next-in-line effect Self-relevance effect Zeigarnik effect Source: http://en.wikipedia.org/wiki/Memory_bias JH Acquiescence Bias • Some respondents are simply agreeable, and indicate agreement out of politeness • Other respondents expect that the researchers agree with the listed items and defer to their judgment • Most respondents find agreeing takes less effort than carefully weighing each optional level of disagreement and agreement Source: Saris, Krosnick and Shaeffer, 2005 JH Mode Effects • Face-to-face: a “guest” script Social desirability bias highest: • Phone interviews: a “solicitor” 1. Telephone surveys script 2. Face-to-face surveys • IVR interviews: a “voice mail” 3. IVR surveys script 4. Mail surveys • Internet surveys: a “web form” 5. Web surveys script • Mail surveys: a “form” script JH Answers Patterns for Common Response Styles Completely Somewhat Neither agree Somewhat Completely Disagree Agree Response Style disagree disagree nor disagree agree agree Optimal Responding Extreme Response Style (ERS) Response Range (RR) Mild Response Style Truncated Scales Truncated Midpoint Response (MPR) Acquiescence Response Style (ARS) Disacquiescence Response Style (DARS) Social Styles Social - Socially Desirable Responding (SDR) Noncontingent Social/Anti Responding (NCR) JH Response Styles by Country: Informed by Culture Source: Johnson, Kulesa, Cho, Shavitt, 2003; Vovici JH Agenda 1. How respondents think 2. Questionnaire design overview 3. Addressing common mistakes Sample Question Sequence DA Scale Types DA Unidimensional vs. Multidimensional DA Data Levels DA Question Balance and Symmetry DA Skip Patterns and Branching DA Avoid Biases DA Agenda 1. How respondents think 2. Questionnaire design overview 3. Addressing common mistakes Asking Objective Questions • Respondents should not be able to determine where you stand on any topic o Use nonjudgmental wording o Choose neutral terms • Don’t ask leading questions o Not “What do you like about your service?” o But “What, if anything, do you like…?” • Write from the respondent’s perspective not your perspective JH Asking Objective Questions • Remove ambiguity: “What is your favorite drink?” (drink = beverage or drink = alcoholic beverage) • Ask one item at a time ⇒ not: “How would you rate our price and service?” ⇒ not: “How easy to reach someone to help?” • Avoid industry jargon • Specify how you use general terms • Don’t make subtle distinctions • Have others proofread your questions for clarity • Pre-test survey with a segment of your audience JH To Label or Not Label Each Point of a Scale Many Variations Possible JH To Label or Not Label Each Point of a Scale Many Variations Possible Best Practices • Respondents prefer fully labeled scales • Fully labeled scales have greater reliability and validity • Numeric values alter the meaning of labels and should be avoided • 5-point unipolar and 7-point bipolar scales have greatest reliability and validity • Where possible use standard scales rather than write your own Source: Krosnick, J. A., & Fabrigar, L. R. (1997). “Designing rating scales for effective measurement in surveys.” JH Patterns to“He's Use embiggened for Scales that role with his Unipolar Scale (0..100)cromulent Bipolar Scale (-1..0..+1) performance.” • Completely* cromulent • Completely* disembiggened • Mostly disembiggened • Very cromulent • Somewhat disembiggened • Moderately cromulent • Neither embiggened nor • Slightly cromulent disembiggened • Somewhat embiggened • Not at all cromulent • Mostly embiggened *or “Extremely” where appropriate • Completely* embiggened JH Patterns to Use for Scales Unipolar Scale (0..100) Bipolar Scale (-1..0..+1) • Completely* cromulent • Completely* disembiggened • Mostly disembiggened • Very cromulent • Somewhat disembiggened • Moderately cromulent • Neither embiggened nor • Slightly cromulent disembiggened • Somewhat embiggened • Not at all cromulent • Mostly embiggened *or “Extremely” where appropriate • Completely* embiggened JH Patterns to Use for Scales Unipolar Scale (0..100) Examples with Completely ___ acceptable • Completely* cromulent ___ likely ___ probable • Very cromulent ___ satisfied ___ true of me • Moderately cromulent ___ true of what I believe • Slightly cromulent Examples with Extremely ___ aware • Not at all cromulent ___ concerned *or “Extremely” where appropriate ___ easy ___ familiar ___ important ___ influential JH Other Common Unipolar Scales Frequency Always, Often, Sometimes, Rarely, Never Grade A, B, C, D, F Like extremely well, Like quite well, Like moderately, Like Liking slightly, Not like at all Essential, High priority, Medium priority, Low priority, Not a Priority priority Quality Excellent, Good, Fair, Poor, Very Poor (traditional) Quality Excellent, Good, Average, Poor, Terrible (contemporary) Excellent given the price, Good given the price, Average Quality (relative) given the price, Poor given the price, Terrible given the price Quantity All, Most, Half, Some, None Developing Custom Scales Labels Reflect Equal Intervals Recent Client Example 100 How do you feel about advertisements on television? 80 • Love 60 • Like 40 • Neutral • Dislike 20 0 Other scale ideas: • Love, Like, Neutral, Dislike, Hate • Terrible, Poor, Average, Good, Excellent JH Bipolar Scales are Outdated Case Against Bipolar Scales Alternatives • Because bipolar scales contrast two opposites, they require more cognitive • Bipolar satisfaction, importance, and likelihood scales can easily be replaced by effort for respondents to evaluate than unipolar scales. Respondents must unipolar scales that will be more reliable and easier on the respondent decide which of two extremes to select (or the midpoint), then to what degree • Bipolar scales work well for changes in quantity and attitudes about changes in they tend towards that extreme. For a unipolar scale, respondents are just quantity. A bipolar scale works well for predicting positive vs. negative assessing the extent. recommendations of brands (Schneider, Berent, Thomas, and Krosnick). • The midpoints of bipolar scales are a point of confusion, open to different • Malhotra, Krosnick, and Thomas show that breaking bipolar scales into . Sometimes a choice like Neither interpretations by different respondents multiple questions improves criterion validity over using single questions. boring nor interesting is selected as a “don’t know” response, while to other For instance, instead of using a 7-point bipolar scale, use these three questions: respondents it might mean “neither” (“neutral”) or it might mean “boring in some ways and interesting in others.” (This example is from “The Science of Asking 1. “Do you think that the amount of money the federal government spends on Questions” by Nora Schaeffer and Stanley Presser). the U.S. military should be increased, decreased, or neither increased nor decreased?” • Survey