Leading Questions: a Categorization System Scott and Steward
Total Page:16
File Type:pdf, Size:1020Kb
Leading Questions: A Categorization System Scott and Steward LEADING QUESTIONS: A CATEGORIZATION SYSTEM CLIFFORD D. SCOTT, University of Arkansas—Fort Smith MICHELLE D. STEWARD, Wake Forest University Marketing researchers agree that questionnaire design and specifically, leading questions, can be a significant contributor to total error in survey research. In challenges to research, leading questions are often a central issue. Interestingly, marketing literature on leading questions is mostly circular and lacking in a framework to identify and a process to remedy leading questions. This article focuses on: (a.) current treatment of leading questions and pitfalls of existing definitions; (b.) a synthesis of types of leading questions grounded in semiotics literature which leads to; (c.) a categorization of dimensions in which questions can be assessed to identify and remedy any leading nature of the question; and (d.) a new definition of a leading question. The value of the categorization lies in providing a framework allowing marketing researchers to better comprehend, diagnose and treat leading questions. INTRODUCTION oracle of truth, is subject to creating research to suit the interests of well-heeled contributors A leading question, or a question that “gives the (Goldberg, 2002; Crossen, 1994; Soley, 1995). respondent a strong cue or expectation as to People are deeply skeptical of reported research how to answer,” have long been recognized as a results because they fear the researchers are not significant source of error in survey research asking the right questions and may be (Burns & Bush, 2010, p. 309). Surveys are manipulating the wording of questions to get designed to understand the truth about the the responses they want (Mann & Dionne, views and the behaviors of people who can 2003). Why shouldn’t they be skeptical? ultimately improve societal well-being. Beyond the creation of more skeptical Leading questions can misdirect marketing consumers, distortion on a brand’s marketing actions that are based on the results of surveys. strategy can occur due to leading questions that For example, in an attempt to illustrate the misdirect the actions of the brand. trendiness of Levi’s jeans, the company ran an advertisement stating that ninety percent of Years ago, Raymond (1977) noted how college students say Levi’s 501 are “in” on questioning can bias respondents. Indeed, in college campuses. What the ads failed to reveal, what became one of the most controversial FTC however, is that the research question gave cases regarding advertising claims, one of the students a list of things from which to choose significant points made to question the research and Levi’s 501 was the only blue jean listed. supporting the FTC’s position was based upon Other items listed were: 1960’s-inspired leading questions (Jacoby & Szybillo, 1995). clothing, Lycra/spandex clothing and so on Opposing experts in the case each made (Utts, 2004, p. 45), allowing little opportunity credible arguments (Stewart, 1995) but it was to call anything other than Levi’s “in.” While clear there was disagreement, even among short-term gains may result from such an ad, in highly expert researchers, as to whether the the long-run Levi’s misses opportunities to questions were leading. This led Preston better know how consumers perceive their (1992) to write that we need greater clarity in products and to more significantly differentiate our survey methods issues such that at least the brand from competitors. experts may agree on proper/improper methods. It has been well-documented that the media While there may be suspicion that leading themselves and even the ivory-towered questions were used, rarely does a final user of university system, long viewed as the last survey research see the actual questions used to collect the information upon which the user The Marketing Management Journal Volume 28, Issue 2, Pages 80-92 may base his or her decision. In their review of Copyright © 2018, The Marketing Management Association six decades of published research about how All rights of reproduction in any form reserved consumers view advertising, Calfee and Marketing Management Journal, Fall 2018 80 Leading Questions: A Categorization System Scott and Steward Ringold (1994, p. 250) noted that the possibility intend our approach to be used sequentially or of leading questions used in the studies was a as a checklist, we do suggest that researchers limitation on the validity of the published examine all three elements (question structure, research yet there was no way to determine the content and context) on a question-by-question extent of this as the questions themselves are basis. The reality is that it may be efficient for missing from the published research (italics the researcher to approach the examination added). For a specific example, consider the process sequentially, but care should be given following question taken from an actual survey to return through the process if any changes are conducted by the National Tax Limitation made. The process holistically allows the Committee: “Do you feel that limiting taxes by researcher to apply discernment to the question law is an effective way to stop the government using the categorization in situ. In the next from picking your pocket every payday?” In sections, we identify the problems with current their discussion of this single question, definitions of leading questions and present a Churchill and Iacobucci (2005, p. 250) surmise categorization of mechanisms by which leading that “what is especially unfortunate is that it is questions are created. We conclude with a new unlikely that the questions themselves definition of leading questions that resolves the accompanied the report to Congress. Rather, it problems with current definitions. is more likely that the final report provided a summary such as ‘Ninety percent of the citizens PROBLEMS WITH favor limiting taxes.’” While there are clear CURRENT DEFINITIONS directives in the research literature that leading questions are sources of error, as noted above, A review of marketing research and public researchers are given paltry guidance when it opinion-related texts on the subject reveals the comes to identifying leading questions and/or following: most define a leading question in knowing how to remedy potentially leading terms of its impact on the respondent rather questions. than in terms of some aspect of the question itself. In terms of the familiar S-O-R (Stimulus, The purpose of this article is to provide Organism, Response) paradigm, this approach researchers with a tool to assist in this very attempts to define the “S” (question) in terms of task—a categorization of mechanisms by which the interpretation by the “O” (respondent’s questions can be assessed to determine if the reading of questioner’s intent) or in terms of the question is leading. The categorization system “R” (answer given by respondent). Note that allows a systematic review process for the definitions that follow do not state that questionnaires, to help identify, and ultimately some quality of the “S” makes it leading; it is eliminate, leading questions. defined only in terms of the respondent’s interpretation or the eventual answer given. So This categorization system helps researchers to the definitions are circular. Simply put, current identify leading questions before conducting definitions state that a question is leading if it research, thus eliminating many problems. The leads. categorization presented gives guidance to researchers, yielding survey questions that more A review of definitions (see Table 1) illustrates accurately tap into the phenomena of interest to that most fall into one of two groups: (1.) those the researcher, while reducing the systematic that emphasize the organism and (2.) those that bias that comes with leading questions. Our emphasize the response. Those emphasizing categorization strives to pinpoint those the organism (“O”) employ terminology elements within the question itself that cause it referencing human perceptions or other to be leading. The major categories are (1) internal, cognitive responses. For example, the question content, (2) question structure and (3) respondent may interpret the question as a clue question context. These three categories are (Malhotra & Peterson, 2006) or an indication broken into more specific sub-categories, (Churchill & Iacobucci, 2005) as to what allowing the researcher to carefully review their answer is desired. The respondent also may own work to spot leading questions, allowing perceive the question as a suggestion (Aaker, them to identify and eliminate the root cause of Kumar & Day, 2001) or expectation (Burns & the question’s leading quality. While we do not Bush, 2010). Again, it should be noted that 81 Marketing Management Journal, Fall 2018 Leading Questions: A Categorization System Scott and Steward TABLE 1: A Sample of Definitions of Leading Questions DEFINITIONS OF “LEADING” OR SOURCE S-O-R EMPHASIS “LOADED” QUESTIONS McDaniel & Gates, 1991 Influence people to respond in a manner that does R not accurately reflect their positions. Zaltman & Burger, 1975 Make it easier for the respondent to react in a cer- R tain way. Rubenstein, 1995, Influences or cues the respondent to some desired R, O response. Gallup & Rae, 1940 Worded in such a way as to bias the response. R, S included Moser & Kalton, 1972 By its content, structure or wording, leads the re- R, S included spondent in the direction of a certain answer. Gawiser, 1994 Use language and structure to push the respondent R, S included to a specific position. Young, 1994 Phrased in such a way that respondent answers are R, S included influenced by the question wording. Mangione, 1995 Written in such a way as to force people to answer R, S included in one direction or another. Malhotra & Peterson, 2006 One that clues the respondent to what the answer O should be. Churchill & Iacobucci, Framed to give the respondent an indication about O 2005 how he or she ‘should’ answer.