Leading Questions: A Categorization System Scott and Steward

LEADING QUESTIONS: A CATEGORIZATION SYSTEM CLIFFORD D. SCOTT, University of Arkansas—Fort Smith MICHELLE D. STEWARD, Wake Forest University

Marketing researchers agree that questionnaire design and specifically, leading questions, can be a significant contributor to total error in survey research. In challenges to research, leading questions are often a central issue. Interestingly, marketing literature on leading questions is mostly circular and lacking in a framework to identify and a process to remedy leading questions. This article focuses on: (a.) current treatment of leading questions and pitfalls of existing definitions; (b.) a synthesis of types of leading questions grounded in semiotics literature which leads to; (c.) a categorization of dimensions in which questions can be assessed to identify and remedy any leading nature of the question; and (d.) a new definition of a leading question. The value of the categorization in providing a framework allowing marketing researchers to better comprehend, diagnose and treat leading questions.

INTRODUCTION oracle of truth, is subject to creating research to suit the interests of well-heeled contributors A leading question, or a question that “gives the (Goldberg, 2002; Crossen, 1994; Soley, 1995). respondent a strong cue or expectation as to People are deeply skeptical of reported research how to answer,” have long been recognized as a results because they fear the researchers are not significant source of error in survey research asking the right questions and may be (Burns & Bush, 2010, p. 309). Surveys are manipulating the wording of questions to get designed to understand the truth about the the responses they want (Mann & Dionne, views and the behaviors of people who can 2003). Why shouldn’t they be skeptical? ultimately improve societal well-being. Beyond the creation of more skeptical Leading questions can misdirect marketing consumers, distortion on a brand’s marketing actions that are based on the results of surveys. strategy can occur due to leading questions that For example, in an attempt to illustrate the misdirect the actions of the brand. trendiness of Levi’s jeans, the company ran an advertisement stating that ninety percent of Years ago, Raymond (1977) noted how college students say Levi’s 501 are “in” on questioning can bias respondents. Indeed, in college campuses. What the ads failed to reveal, what became one of the most controversial FTC however, is that the research question gave cases regarding advertising claims, one of the students a list of things from which to choose significant points made to question the research and Levi’s 501 was the only blue jean listed. supporting the FTC’s position was based upon Other items listed were: 1960’s-inspired leading questions (Jacoby & Szybillo, 1995). clothing, Lycra/spandex clothing and so on Opposing experts in the case each made (Utts, 2004, p. 45), allowing little opportunity credible arguments (Stewart, 1995) but it was to call anything other than Levi’s “in.” While clear there was disagreement, even among short-term gains may result from such an ad, in highly expert researchers, as to whether the the long-run Levi’s misses opportunities to questions were leading. This led Preston better know how consumers perceive their (1992) to write that we need greater clarity in products and to more significantly differentiate our survey methods issues such that at least the brand from competitors. experts may agree on proper/improper methods.

It has been well-documented that the media While there may be suspicion that leading themselves and even the ivory-towered questions were used, rarely does a final user of university system, long viewed as the last survey research see the actual questions used to collect the information upon which the user The Marketing Management Journal Volume 28, Issue 2, Pages 80-92 may base his or her decision. In their review of Copyright © 2018, The Marketing Management Association six decades of published research about how All rights of reproduction in any form reserved consumers view advertising, Calfee and

Marketing Management Journal, Fall 2018 80 Leading Questions: A Categorization System Scott and Steward

Ringold (1994, p. 250) noted that the possibility intend our approach to be used sequentially or of leading questions used in the studies was a as a checklist, we do suggest that researchers limitation on the validity of the published examine all three elements (question structure, research yet there was no way to determine the content and context) on a question-by-question extent of this as the questions themselves are basis. The reality is that it may be efficient for missing from the published research (italics the researcher to approach the examination added). For a specific example, consider the process sequentially, but care should be given following question taken from an actual survey to return through the process if any changes are conducted by the National Tax Limitation made. The process holistically allows the Committee: “Do you feel that limiting taxes by researcher to apply discernment to the question is an effective way to stop the government using the categorization in situ. In the next from picking your pocket every payday?” In sections, we identify the problems with current their discussion of this single question, definitions of leading questions and present a Churchill and Iacobucci (2005, p. 250) surmise categorization of mechanisms by which leading that “what is especially unfortunate is that it is questions are created. We conclude with a new unlikely that the questions themselves definition of leading questions that resolves the accompanied the report to Congress. Rather, it problems with current definitions. is more likely that the final report provided a summary such as ‘Ninety percent of the citizens PROBLEMS WITH favor limiting taxes.’” While there are clear CURRENT DEFINITIONS directives in the research literature that leading questions are sources of error, as noted above, A review of marketing research and public researchers are given paltry guidance when it opinion-related texts on the subject reveals the comes to identifying leading questions and/or following: most define a leading question in knowing how to remedy potentially leading terms of its impact on the respondent rather questions. than in terms of some aspect of the question itself. In terms of the familiar S-O-R (Stimulus, The purpose of this article is to provide Organism, Response) paradigm, this approach researchers with a tool to assist in this very attempts to define the “S” (question) in terms of task—a categorization of mechanisms by which the interpretation by the “O” (respondent’s questions can be assessed to determine if the reading of questioner’s intent) or in terms of the question is leading. The categorization system “R” (answer given by respondent). Note that allows a systematic review process for the definitions that follow do not state that questionnaires, to help identify, and ultimately some quality of the “S” makes it leading; it is eliminate, leading questions. defined only in terms of the respondent’s interpretation or the eventual answer given. So This categorization system helps researchers to the definitions are circular. Simply put, current identify leading questions before conducting definitions state that a question is leading if it research, thus eliminating many problems. The leads. categorization presented gives guidance to researchers, yielding survey questions that more A review of definitions (see Table 1) illustrates accurately tap into the phenomena of interest to that most fall into one of two groups: (1.) those the researcher, while reducing the systematic that emphasize the organism and (2.) those that bias that comes with leading questions. Our emphasize the response. Those emphasizing categorization strives to pinpoint those the organism (“O”) employ terminology elements within the question itself that cause it referencing human perceptions or other to be leading. The major categories are (1) internal, cognitive responses. For example, the question content, (2) question structure and (3) respondent may interpret the question as a clue question context. These three categories are (Malhotra & Peterson, 2006) or an indication broken into more specific sub-categories, (Churchill & Iacobucci, 2005) as to what allowing the researcher to carefully review their answer is desired. The respondent also may own work to spot leading questions, allowing perceive the question as a suggestion (Aaker, them to identify and eliminate the root cause of Kumar & Day, 2001) or expectation (Burns & the question’s leading quality. While we do not Bush, 2010). Again, it should be noted that

81 Marketing Management Journal, Fall 2018 Leading Questions: A Categorization System Scott and Steward

TABLE 1: A Sample of Definitions of Leading Questions

DEFINITIONS OF “LEADING” OR SOURCE S-O-R EMPHASIS “LOADED” QUESTIONS McDaniel & Gates, 1991 Influence people to respond in a manner that does R not accurately reflect their positions.

Zaltman & Burger, 1975 Make it easier for the respondent to react in a cer- R tain way.

Rubenstein, 1995, Influences or cues the respondent to some desired R, O response.

Gallup & Rae, 1940 Worded in such a way as to bias the response. R, S included

Moser & Kalton, 1972 By its content, structure or wording, leads the re- R, S included spondent in the direction of a certain answer.

Gawiser, 1994 Use language and structure to push the respondent R, S included to a specific position.

Young, 1994 Phrased in such a way that respondent answers are R, S included influenced by the question wording.

Mangione, 1995 Written in such a way as to force people to answer R, S included in one direction or another.

Malhotra & Peterson, 2006 One that clues the respondent to what the answer O should be.

Churchill & Iacobucci, Framed to give the respondent an indication about O 2005 how he or she ‘should’ answer.

Aaker, Kumar & Day, 2001 Clearly suggests the answer to the respondent O

Zikmund & Babin, 2007 Suggests or implies certain answers O

Tull & Hawkins, 1980 Suggest what the answer should be or indicate the O researcher’s own point of view

Burns & Bush, 2010 Worded or structured in such a way as to give the O, S included respondent a strong cue or expectation as how to answer

Warwick & Lininger Content, structure, or wording-push the respondent O,S included 1975 in the direction of a certain answer by implication or suggestion

these definitions of a leading question do not question. These definitions indicate that to be focus on characteristics of the question itself, leading, the question must not only trigger an but, rather, are one step removed from the internal cognition, but also a reaction. Now the question, and place emphasis on the mental question pushes (Gawiser & Will, 1994) or responses to, or interpretations of, the question. influences the respondent (Rubenstein 1995). The subject is led (Moser & Kalton, 1972) or Those definitions emphasizing the response even forced (Mangione, 1995) to a specific (“R”) are even one more step removed from the answer.

Marketing Management Journal, Fall 2018 82 Leading Questions: A Categorization System Scott and Steward

Several definitions do mention the question structured review protocol allowing the itself. We are told that something about the researcher to detect potentially leading wording (Gallup & Rae, 1940) or language questions a priori in a reasonably efficient (Gawiser & Will, 1994) allows the question to fashion. The only practical approach is to lead. Others state that it is the phrasing develop a system for inspection of the “S” (the (Young, 1994) or structure (Moser & Kalton, question itself – its wording and context) that 1972) that moves a question into the “leading” will assist the researcher in isolating the leading category. One of the better definitions tells us aspects of the “S” so as to improve the question that by the question’s content, structure, or prior to conducting the survey. wording the respondent is pushed in the direction of a particular answer (Warwick & TOWARDS A SOLUTION: Lininger, 1975). While these definitions do THEORETICAL FOUNDATIONS raise the issue of question wording or structure, they in no way indicate what wording or which The answer to why leading questions are so structures are culpable. All questions contain prevalent after being scrutinized for years is wording; all questions have structure. Telling deceptively simple: there is not just a single the reader that the problem lies in some pathway for making a question “leading.” We unidentified thing within the structure or are seeking a unidimensional definition of a wording of these particular questions is not multidimensional construct. The question, “You helpful. These definitions acknowledge that the ran the red light, didn’t you?” is leading question is the root of the problem while because it contains (a.) the complete answer simultaneously avoiding any attempt to narrow desired, plus (b.) a direct request for assent. On or delineate the issue so as to aid the researcher the other hand, the question, “Should the state in identifying a leading question. Simply ignore the principles of the Bible, which is the stated, these definitions are not operational: law of God and morality, by legalizing murder they do not specify the problem in such a way by abortion?” provides a self-contained as to facilitate a solution. argument for the answer desired. Both questions lead, but they do so via application of Some would argue that a respondent-based differing techniques. To sort out the problem of approach to the definition of “leading” is “what is a leading question?” is to categorize warranted in which the respondent’s the variety of overarching ways in which a interpretation is central to the definition. While question may lead. this perspective holds theoretical weight, it is fruitless from a research process perspective. The issue of categorization is a common one Informing the researcher that “leadingness” is within any field of scientific inquiry. In fact, it based on perception means only one thing from has been said that “partitioning is the a process point of view: prior to conducting a of analysis” (Kerlinger, 1973, p. study, the researcher should conduct a pre- 137). Others go even further, claiming that study for each question to learn how each “ordering, classification, or other grouping of question is perceived. The results of the pre- the objects or phenomena under investigation” study should be used to revise questions into may be the most important phase of any non-leading questions. At this point, the scientific inquiry (Carper & Snizek, 1980, p. researcher should proceed with traditional 65). We present in the next section such a pretesting of the new questionnaire. After again classification of the phenomena. revising the questions as a result of the pretest, the researcher is ready to collect data. Some The Semiotics of Leading Questions have suggested when “the stakes are high,” (Dillman, Smyth & Christian, 2009) an In the field of semiotics, a well-developed even more laborious method should be applied, system exists for organizing and studying such as those methods reviewed by Presser, et meaning (Mick, 1986). The field of semiotics al. (2004): cognitive interviews, behavior posits there are three distinct functional coding, response latency, vignette analysis, components to a complete symbol system, or formal respondent debriefings, experiments and the way in which language patterns and statistical modeling. What is needed is a experiences are formed to create meaning. The

83 Marketing Management Journal, Fall 2018 Leading Questions: A Categorization System Scott and Steward

components are syntactics (how signs work marketing leads to greater sales. Do you favor together to create meaning), semantics (what higher or lower spending on digital marketing?” the signs mean together) and pragmatics (how In this example, certain facts (or purported is communication affected by context). These facts) are presented in such a way as to make components translate nicely to pinpoint three the question leading. A purported fact – greater broad dimensions of a question that may spending on digital marketing – is used to lever convert a question into a leading question. The the respondent in the direction of favoring question structure – such as beginning with higher spending on digital marketing. Within “Isn’t it true that…” – can convert virtually any our classification system, question content question into a leading question. The actual focuses on the facts, as well as purported facts question content – such as the facts inserted or implications of fact, forming the building into the question – may make a question blocks of any question. Semantics focus on the leading. Additionally, the way in which the meaning of a collection of signs. The facts in question is asked, the question context/delivery the example above are the signs or symbols – such as the tone of voice or the body language containing the elemental kernels of meaning of the questioner that invites agreement – can within the question. Such facts, as the create a leading question. In the following semiotician would say, are the fundamental section, we illustrate the theoretical symbols that contain meaning (Peirce, 1931). underpinning of question structure, content and context using a foundation in semiotics, and we Question Context as Pragmatics provide examples to demonstrate how these three distinct dimensions may be used to create The context of the question may also make it leading questions. leading. Imagine the exact same words delivered in two different contexts. First, Question Structure as Syntactics “greater spending on digital marketing leads to greater sales,” delivered in a business meeting. The structure of the question may cause the This would not even be perceived as a question, question to lead, as in: “Wouldn’t you agree but as a declarative statement. Now, imagine that greater expenditures on digital marketing the exact same words spoken at a meeting lead to greater sales growth?” Notice that this hosted by a major digital marketing firm in question has not introduced facts that may which banners and other signage show the firm affect the respondent. Rather it is the structure as the event sponsor. The context openly asks of the question – the presence of the for agreement. Pragmatics specifically focuses preparatory phrase “Wouldn’t you agree” – that upon intent, occasion and inflections cues the respondent to deliver the desired (Stalnaker, 1972). In semiotics terminology, affirmative response. This category of Leading pragmatics recognizes how the delivery of a Question parallels the semiotic category of message may create meaning not inherent in the “syntactics.” Syntactics explains how speaker’s words alone, but in the intonation, individual signs—in this case, single English volume, expression, pitch and sound quality of words—operate together to form more complex the voice. Unlike semantics, with a focus upon meanings, such as complete questions (Morris, words as they stand in a vacuum, so to speak, 1938). Syntactics represent the formal pragmatics looks at communication in context structural/grammar rules of the entire sign (Fotion, 1995). The parallel of pragmatics system. These rules are such that certain within our categorization is question context, question structures will lead the respondent pointing out that the totality of the questioner’s because they are understood by both questioner delivery, which incorporates the non-verbal and and respondent as a request for a specific general context of the setting, greatly impacts response. the question’s meaning.

Question Content as Semantics There is one more point to be made here, and it is the key to understanding the classification To illustrate how question content may create a system: the three leading mechanisms may be leading question, consider the example: used in any combination; a single question “Studies show that greater spending on digital might leverage all three. Therefore, the

Marketing Management Journal, Fall 2018 84 Leading Questions: A Categorization System Scott and Steward

classification we propose is not a classification commercials featuring Tiger Woods?” This of leading questions: it is a classification of the question tells the respondent that Tiger Woods very mechanisms causing the question to lead. is in Buick commercials. Another variation: “Is Rarely will a leading question make use of only Brand Y the best brand on the market?” In a single mechanism. these questions, the substance of the answer is contained within the question. If this substance Proposed Categorization originates anywhere other than inside the respondent’s head, the question is probably a An inspection of the extant literature combined leading question. with the authors’ own experiences yielded many questions held out as examples of Non-leading versions of these questions “leading.” Our categorization is created by eliminate the “answer,” creating a more open examination of the key dimensions that give question structure. “What color was the light rise to meaning of a question which stem from when you went through the intersection?” semiotics. The result is a categorization “Have you seen any commercials for consisting of three primary mechanisms and a automobiles?” and “In your opinion, what is the total of 12 sub-mechanisms. See Table 2. best brand on the market?” all eliminate the specific, desired response to the question. Question Structure Mechanisms Some would refer to this as a “funneling technique” where respondents are moved from We should note that within the discussion of the general (to eliminate any cognitive cue that question-structure-based leading questions, our could trigger a specific response) to the example questions will often employ facts. specific. For example, relative to automobile The reader may wonder why these questions commercials, a second question may be “Can are not covered in our category of question- you recall any auto brand names advertised in content-based leading questions. The answer is those commercials?” that we wish to give realistic examples and very few questions are fact-free and, as we have (2.) Logic. Rather than coming out and stating noted previously, most leading questions make an answer, it is also possible to steer the use of more than one leading mechanism. respondent to an answer by logic or apparent logic. This question structure includes In the first section of our categorization, we information that leads the respondent towards place the focus on those characteristics of some answer by making one response appear question structures that create leading the most reasonable or by making other questions. We discuss five mechanisms that responses seem foolish. “During the Greyhound may be used to create leading questions based bus drivers’ strike, there was a good deal of up the structure of the question. violence. Do you support unions when strikers (1.) Includes answer. The simplest way to resort to violence?” (Rubenstein, 1995, p. 215.) create a leading question is by including the is an example. Another question structure answer within the question itself. The root of using logic is to make a statement that leads in such a question is a declarative statement. one direction then requests a “choice,” even Then, some twist converts the statement into a though the selection has effectively already question that simply asks for agreement with been made for them. For example, “Studies the declarative. “The product worked well, show that greater spending on digital marketing didn’t it?” is an example of this question form. leads to greater sales growth. Do you favor Almost any question seeking acknowledgement digital marketing spending, or should digital of or agreement with a declarative statement is marketing spending be lowered in spite of this leading. Examples include “isn’t it true,” or fact?” “wouldn’t you agree,” or “isn’t it possible that” or ending with, “correct?” (3.) Presumption. There are many ways to insert a presumed fact into a question, then use There may be many variations on this theme. any reply as of the respondent’s One version purports to provide a memory aid, support of the presumption. If the question is, such as, “Have you seen any Buick “How fast was the car going when it went

85 Marketing Management Journal, Fall 2018 Leading Questions: A Categorization System Scott and Steward

TABLE 2: Leading Question Mechanisms, Elements and Examples

CATEGORY EXPLANATION EXAMPLE Step 1: Common ways to make the FORM of the ques- “Wouldn’t you agree that tion leading include crafting a question that: greater expenditures on Examine digital marketing lead to 1. Includes the answer in the question. greater sales growth?” Question Structure 2. Uses logic or apparent logic to steer the re- spondent to an answer (parallel examples). 3. Presumes the truth of an answer, or the truth of something logically leading to an answer. 4. Avoids allowing the respondent to provide a possible answer; false choice. 5. May be answered only as Yes/No.

Step 2: Common ways to use FACTS to make a question “Studies show that greater leading would be to include: spending on digital market- Review ing leads to greater sales. 1. Unsupported assertions presented as Do you favor higher or low- Question Content facts. er spending on digital mar- 2. Supported facts (or points presented as keting?” supported) presented in an unbalanced fash- ion in which facts that might justify an an- swer are avoided and facts that support oth- ers are provided. 3. Loaded words/broadly held beliefs which generate an emotional/cognitive impetus towards an answer. Step 3: Common ways to use DELIVERY OR CON- Meeting hosted by a major TEXT to make a question leading include: digital marketing firm in Evaluate which the message is deliv- 1. Make respondents aware of the desired out- ered that, “Greater spending Question Context come of, or “purpose” for, the survey. on digital marketing leads 2. Make respondents aware of the sponsor. to greater sales.” 3. Use preceding questions to set up assump- tions in the mind of the respondent. 4. Use non-verbal cues.

Repeat the examination of questions if changes were made at each step.

through the yield sign?” (Loftus, 1979) some (4.) Avoidance. Just as including information respondents will jump to the conclusion that the may bias a question, so may excluding car went through the yield sign, whether they information. But the more common way for witnessed such a thing or not. The earmark of avoidance to bias a question is by disallowing these presumption questions is that the content certain responses. Like one-sided positions is inserted in such a way so as to not draw stated in the question itself, response options attention to itself. Rather than showcasing the may also be leading if they are one sided. content for the respondent’s consideration as in Questions which limit response options in one the above “logic” questions, it is delivered in a direction or another may lead a respondent and fashion that invites the respondent to gloss over yield a biased response. its presence and respond as if the content was already established fact. In a closed ended question, consider: “Which of the following best describes your intention to

Marketing Management Journal, Fall 2018 86 Leading Questions: A Categorization System Scott and Steward

buy product X?” followed by the options, These definitions make it clear that researchers “Very Likely, Somewhat Likely, Neither Likely should always be suspicious of the “yes/no” nor Unlikely, Not Likely to Buy.” By using an question. This type of question invariably unbalanced scale, there are fewer “unlikely” triggers both structure issues and content issues. response options than “likely” categories. The The form of the question is such that it invites, result is that the measured level of intention to or, at the very minimum permits, acquiescence purchase will be higher than the true level to the question. As the Texas court suggests, a (Foddy, 1993). It should be noted that in some single syllable forms a complete response to the special circumstances, such as in customer question. The form is such that all information satisfaction surveys, an unbalanced scale is is supplied by the questioner rather than the often used to develop finer discrimination at the respondent. The respondent does not supply upper end of the scale when there is a priori facts, s/he merely assents or declines. knowledge that most respondents will use the upper end of the scale. Still, many researchers As the definitions above mention, “yes/no” argue, for reasons given above, that “don’t questions invariably include facts or purported know” or “no opinion” options should always facts. These facts may suggest the answer, be provided (Payne, 1951). again inviting yea-saying. This combination of potential yea-saying form plus potentially A variation on the avoidance theme is the “false suggestive facts is always dangerous. The “yes/ choice” question. This form presents limited no” question is structured to ask the respondent options as if they were the only options. A to agree/disagree with specific question content. respondent might be asked, “Should we continue to increase our funding for Medicare However, not every “yes/no” question is annually, or should Medicare be replaced with a leading. Not every “yes/no” question more fiscally sound program?” Many options constitutes an attorney’s attempt to interject his/ are available that are not mentioned within the her version of the facts. “Isn’t it true that a fire question, such as keeping Medicare and not caused by the lithium ion battery from your increasing funding, or keeping Medicare and company’s cell phone injured the plaintiff?” is reducing funding By avoiding any such a “yes/no” question presented in a leading mention, the question is framed to suggest that fashion. However, the “yes/no” question, “Did the two options suggested are the only two any injury occur?” would not be considered available. leading in most circumstances.

(5.) The Special Case of Yes/No. Many non- A “yes/no” question is only objectionable for attorneys, and survey researchers in particular, the exact same reason any other question would are surprised to learn that the “yes/no” question be objectionable: because it involves a biasing is particularly suspect of being leading in a technique. So long as the answer to the court of law. The definition of a “leading question is a function of the ’ question” found in Black’s Law Dictionary knowledge, not a function of the question, it is begins as a researcher might expect: “(a) not a leading question and no objection should question that suggests the answer to the person be sustained (Murphy, 1993). A “yes/no” being interrogated.” Then the definition adds question should be evaluated in the exact same the twist surprising to most researchers: “esp., a manner as any other question to determine if it question that may be answered by a mere ‘yes’ is leading. The mere fact that it is in a “yes/no” or ‘no’” (Black’s Law Dictionary, 2000, p. format does not immediately transform it into 719). The Texas Supreme Court developed its an improper, leading question (Mueller & definition of the leading question in 1891 and Kirkpatrick, 2004). But it should immediately still sees fit to apply that definition today. trigger the researcher’s heightened scrutiny for There, the court stated that a classic leading both structure-based and content-based biases. question is one that contains a “group of facts” and may be answered by a simple “yes” or” Question Content Mechanisms no” (Lott v. King, 1891). (1.) Unsupported Assertions / Hypotheticals. Perhaps the most common approach to leading

87 Marketing Management Journal, Fall 2018 Leading Questions: A Categorization System Scott and Steward

via facts is the use of the unsupported assertion. avoided in a survey instrument. The first ones “Some people say President Jimmy Carter was that come to mind include: “disaster”, incompetent. Do you agree or disagree?” is a “pollution”, “zealot”, “terrorist”, “abortion,” good example because it seems fairly “pedophile” and “gun control.” However, innocuous (Rubenstein, 1995, p. 215). It is many words/phrases have the potential to bias a almost beyond doubt that SOME people think response. “Don’t you think the New Deal that any one of our previous Presidents was dictatorship should stop persecuting incompetent: the statement needs no support. business?” (Gallup & Rae, 1940, p. 96) or “Do Equal weight is placed on both alternatives: the you want union officials, in effect, to decide respondent is asked to agree OR disagree. how many municipal employees you, the However, there is little doubt that we would get taxpayer, must support?” are representative a very different response with this different— examples (Bradburn & Sudman, 1988, p. 63). and equally leading—phrasing of the question: “Some people say President Jimmy Carter was A particular type of loaded word deserving one of our finest Presidents. Do you agree or special attention is the all-inclusive term – all, disagree?” More to the point, a neutral every, totally, etc. Just as standardized test- phrasing, “How would you rate Jimmy Carter takers are wary of all-inclusive terms, many as a President: excellent, above average, survey respondents recoil at all-inclusives as average, below average or poor” would draw a well. A classic example of the use of an all- response different from either of the leading inclusive term is “Do all cats have four versions that include an unsupported assertion. legs?” (Payne, 1951, p. 160). While a correct conceptual answer may be “yes,” a technically A very similar approach substitutes a correct answer is “no.” Very few ideas that may hypothetical for the unsupported fact. “Would be expressed in English words have, literally, you vote for someone who had forced public zero exceptions. Therefore, when a question employees to join a labor union or be fired?” makes use of an all-inclusive term, that shows how the technique is used (Bradburn & question comes with a built-in validity issue. Is Sudman, 1988, p. 63). In that example, a “fact” the respondent giving an answer based on their was not stated; no claim was made in the breadth of experience in the real world? Or is example that any public employees have ever the answer limited to a narrow logical function actually been forced to join a union. It does not dictated by the use of an all-inclusive? The only state the fact, but merely implies it. However, way to resolve the issue is by asking further, the impact is essentially the same. non-leading questions. Clearly, the simpler fix is to avoid the use of all-inclusives in the first (2.) Supported Facts. This is very similar to the place. above category, but goes one step further by inserting some level of support for the Similar to the loaded word is the broadly held purported fact. The example in the section belief. Within a given cultural milieu, it may be above could be modified to: “Recent surveys almost unacceptable to question certain prove that a majority of Americans say conclusions. Questions that include references President Jimmy Carter was incompetent. Do to these cultural touchstones are automatically you agree or disagree?” Here, surveys provide suspect. Here are a few classic examples: support for the assertion about what  Should the state ignore the principles of “Americans say.” This inclusion yields a the Bible, which is the law of God and question even more leading than the morality, by legalizing murder by “unsupported” form. The use of the word abortion? “prove” further exacerbates the problem by  As the Founding Fathers gave us the right signaling that the purported fact is beyond to bear arms, don’t you agree that…? question.  Are you opposed to groups that give comfort to terrorists? (3.) Loaded Words/Broadly Held Beliefs. Certain words evoke such a powerful affective A closely related approach is to substitute a response – at least within a specified time, popular or respected figure for the belief, place and audience– that they would be better implicitly or explicitly stating the figure Marketing Management Journal, Fall 2018 88 Leading Questions: A Categorization System Scott and Steward

supports a particular position. The question, particular geography. When restaurants called “Do you support the star wars technology “Shoney’s Town and Country” opened, Frisch’s backed by former President Ronald Reagan?” objected. They claimed that the Shoney’s name makes use of this approach (Rubenstein, 1995, was so notoriously associated with “Big Boy” p. 216). Both the loaded words and broadly that mere use of “Shoney’s” within Frisch’s held beliefs approaches insert encoded exclusive territory constituted an infringement. arguments into the question. The bias is not made explicit in a denotative fashion, but via a The survey instrument used in the subsequent connotative pathway. The proverbial “man litigation repeatedly used the term “Big Boy” in from Mars”, or a stranger unfamiliar with the the text of the questions. By the time the cultural context, would not recognize such respondent reached the instrument’s critical questions as leading. But those in touch with question, asking if Shoney’s was a “Big Boy” the culture and the mood of the responding restaurant, s/he had heard “Big Boy” repeated population will discern the leading quality of six times within just the last few questions. The such questions. court concluded that this repeated hammering “set up” the respondent to find an association Question Context Mechanisms between “Shoney’s” and “Big Boy.” The verbiage of the preceding questions had (1.) Aware of Desired Outcome. As the predetermined the answer (Frisch’s v. Shoney’s, respondent has little or no stake in the survey, it 1985). may not be uncommon for the respondent to wish to please the interviewer or survey Bartels (2003), referring to context as “framing sponsor. A preface that even in the smallest effects,” illustrates the impact of a preceding way hints at the specific purpose for the survey, question on the response to a subsequent the desired results or intended applications may question. Citing a 1950 study, half of a create a context that will sway some national sample was asked: “Do you think the respondents. The auto service manager who United States should let Communist newspaper prefaces his/her request for a customer reporters from other countries come in here and satisfaction survey by noting that dealers are send back to their papers the news as they see evaluated and rewarded based upon the surveys it?” The other half of the sample was asked the would more likely get favorable results from same question but only after being asked sympathetic customers. whether, “A Communist country like Russia should let American newspaper reporters come (2.) Aware of Sponsor. This is closely related to in and send back to America the news as they the above. The key distinction occurs in the see it.” Without this preceding question, 36% mind of the respondent. S/he must make a of the sample agreed that we should let logical leap from the identity of the sponsor to Communist reporters report the news as they the outcome desired by the sponsor. Once that see it. But, when asked this same question but leap has been made, the impact is quite the with the preceding question, 73% of the sample same. However, it is perfectly possible that agreed (Bartels, 2003). some respondents may reach an unanticipated conclusion and be led in some direction other (4.) Non-Verbal Cues. There is a dense than the one the sponsor anticipates. Survey literature on the use and interpretation of non- research conducted to try to determine verbal cues. (A bibliography of 75 entries may problems or sources of consumer be found at http://www.flc.kyushu-u.ac.jp/ dissatisfaction, for example, would be foiled by ~michel/serv/intcult/lit-nonverbcomm.html.) respondents seeking to please the identified The Journal of Nonverbal Behavior is currently sponsor with positive ratings. in its 32nd volume. This literature will not be reviewed here. Suffice to say that the (3.) Preceding Questions as Set-up. This is questioner’s delivery may influence the illustrated in the widely-referenced case of respondent such that a questioner must be Frisch’s v. Shoney’s (1985). There, the debate cognizant of vocal and physical expression. revolved around the Frisch’s exclusive rights to use the “Big Boy” designation within a

89 Marketing Management Journal, Fall 2018 Leading Questions: A Categorization System Scott and Steward

A New Definition We have been critical of the literature. In no way do we mean to impugn the foundational Having explained our categorization and its classic works such as that by Payne, Sudman, rationale, we present a new definition of a Bradburn, Schumann, Presser and others. Their leading question. As stated previously, existing work is and will continue to be regarded as definitions are circular with a focus on the classic and required reading for researchers. outcome of the respondent’s answer and Rather, our criticism lies in that we, as whether that answer was one in the desired researchers and academics, stopped with the direction of the questioner. It is only by looking classic foundations. We see our role as tying at all three dimensions that one may develop an many of their previous contributions together in all-inclusive approach to leading questions. a unified manner in an effort to move us toward This allows us to propose a more complete a clearer understanding of a significant problem definition of a leading question: A leading in survey research. question is an interrogative expression containing a mechanism potentially capable of In terms of future research, a categorization systematically biasing responses. Those allows us to move past simple eyeballing and mechanisms may operate via any one of, or take a more scientific structured approach combination of, three pathways: biasing towards dealing with leading questions. By structure, biasing content and/or biasing providing a structure, a categorization suggests context. an approach towards research on the effects of leading questions as called for by Presser, et al. While the dimensions of the categorization are (2004). Breaking the broad concept of “leading not intended for a researcher to examine questions” into distinct categories allows for sequentially, it may be beneficial for the systematic investigation. In the tradition of researcher in terms of not knowing where to Schuman and Presser (1981), exploration start to first examine the structure of a question, should be conducted evaluating the impact of second to review the content of the question the three mechanisms identified in our and third to evaluate the context of the categorization. Do they produce differing question. Once this process is complete, it may effects? What methods are best for remedying be necessary, if changes were made, to revisit these mechanisms? Thus, in the short term, a and reevaluate each dimension. categorization gives a disciplined approach to identifying leading questions and a structured DISCUSSION rationale for improving them. In the longer term, it provides a research pathway for We have attempted to take a disparate literature understanding the nuances of leading questions and provide a structure to help us better and eliminating their impact. understand an area that is an everyday problem for survey researchers. In our proposed Some may wish to explore the relative categorization we have identified 12 elements importance of the different mechanisms, of the three dimensions that are used to create wondering if some types of leading questions leading questions. No doubt, there are other may be more problematic than others. The possible dimensions and some of our present current authors do not find this to be a high- categories certainly could be broken down into priority research avenue. Any leading another level of subcategories. The key technique may be so damaging as to render elements provide a description of what we invalid the results of an entire survey. If the found that has been discussed in the literature to question “tips the researcher’s hand,” causing date, coupled with our own experiences. They the respondent to believe she knows the are certainly not the only approaches possible. answers desired, the results are called into Creative people will always be able to craft new question in toto. Other applications of the very approaches. We look forward to others refining same technique may be so slight as to not only not only our categorization but our proposed go unnoticed, but as to have a negligent impact definition and examples. on results. Similarly, any technique may be applied in a fashion so artful as to sway responses while not causing alarm bells to ring

Marketing Management Journal, Fall 2018 90 Leading Questions: A Categorization System Scott and Steward

in any respondent’s head. Any one may also Bartels, L. M. (2003). Is" popular rule" be applied in so ham-fisted a fashion as to bias possible? Polls, political psychology, and no one. It is not the specific technique that is democracy. The Brookings Review, 21 (3), of import, it is the application. While this is 12-15. our current viewpoint, we welcome discussion Black's Law Dictionary, 7th ed., (2000). Saint and exploration on this topic. Paul, MN: West Group. Bradburn, N. M., & Sudman, S. (1988). Polls & Future research should also examine what surveys: understanding what they tell us (Vol. methods are being used by research 41). Jossey-Bass Inc Pub. practitioners to screen for leading questions. Burns, A.C., & Bush, R.F. (2010). Marketing What heuristics are practitioners using to Research, 6th ed., Upper Saddle River, NJ: screen questions? We recommend a “market Pearson Prentice Hall. test” of our categorization will prove useful in Calfee, J. E., & Ringold, D. J. (1994). The 70% both evaluating and revising the categorization. majority: Enduring consumer beliefs about advertising. Journal of Public Policy & Finally, our intent here has not been to provide Marketing, 13 (2), 228-238. the definitive treatise on leading questions. We Carper, W. B., & Snizek, W. E. (1980). The are still far from understanding how to nature and types of organizational taxonomies: properly word questions in survey research. If An overview. Academy of Management nothing else, the classic debate between Jacoby Review, 5 (1), 65-75. and Szybillo (1995) and Stewart (1995) Churchill, G.A., Jr. & Iacobucci, D. (2005). validates this lack of consensus. Rather, our Marketing Research: Methodological intent is to call attention to an area that is at the Foundations, 9th ed., Mason, OH: Thomson core of marketing survey research yet has not South-Western. seen much research effort since the classics Crossen, Cynthia (1994). Tainted Truth: The referred to earlier. Manipulation of Fact in America, New York: Simon & Schuster. In our initial survey about the challenges with Dillman, D. A., Smyth, J. D., & Christian, L. M. leading questions, we asked the degree to (2014). Internet, phone, mail, and mixed-mode which respondents agreed with this statement, surveys: the tailored design method. John “It would be helpful to market Wiley & Sons. researchers if they had a typology of the types Foddy, W. (1994). Constructing questions for of leading questions and suggestions on how to interviews and questionnaires: theory and remedy each.” The highest level of agreement practice in social research. Cambridge across questions was found in this question university press. about leading questions. Our hope is that our Fotion, N. (1995). Pragmatics. In T. Honderich manuscript will stimulate marketing scholars’ (ed.): The Oxford Companion to Philosophy, interests on a topic that is widespread and Oxford: Oxford University Press. troublesome. Our categorization provides a Frisch's Restaurant Inc. v. Shoney's Inc. (1985). useful tool to help avoid, or at least lessen 759 F.2d 1261. experts disagreeing (Sudman, 1995) on a topic Gallup, G. H., & Rae, S. F. (1940). Pulse of where they should not have to do so and also democracy: The Public-Opinion Poll and How reduce what has been called our “scandalous It Works, New York: Simon and Schuster. record of avoidable errors” (Preston, 1992) in Gawiser, S. R., & Witt, G. E. (1994). A our collective efforts at discovery. journalist's guide to public opinion polls. Greenwood Publishing Group. REFERENCES Goldberg, Bernard (2002). Bias, Washington, D.C.: Regnery Publishing, Inc. Aaker, D. A., Kumar, V., & Day, G. S. (2001). Jacoby, J., & Szybillo, G. J. (1995). Consumer Marketing Research, 7th. John Wiley research in FTC versus Kraft (1991): A case Operations Research & Sons, New York, 51 of heads we win, tails you lose? Journal of (4), 509-518. Public Policy & Marketing, 14 (1), 1-14.

91 Marketing Management Journal, Fall 2018 Leading Questions: A Categorization System Scott and Steward

Kerlinger, F. N. (1973). Foundations of Rubenstein, S.M. (1995). Surveying Public Behavioral Research. New York, NY: Holt, Opinion, London: International Thompson Rinehart and Winston. Publishing. Loftus, E. F. (1979). Eyewitness , Schuman, H. & Presser, S. (1981). Questions Cambridge, MA: Harvard University Press. and Answers in Attitude Surveys: Lott v. King (1891). 79 Tex. 292. Experiments on Question Form, Wording and Malhotra, N. K., & Peterson, M. (2009). Basic Context, New York: Academic Press. marketing research: A decision-making Simpson, F.A. & Selden, D.J. (1998). approach. Prentice hall. Objection: Leading Question! Texas Bar Mangione, T. W. (1995). Mail surveys: Journal, 61, 1123-1135. Improving the quality (Vol. 40). Sage. Soley, Lawrence C. (1995). Leasing the Ivory Mann, Thomas E. and E.J. Dionne, Jr. (2003). Tower: The corporate takeover of academia, “Polling & Public Opinion,” The Brookings Boston: South End Press. Review, 21 (3), 2. Stalnaker, R. (1972). Pragmatics. In Semantics McDaniel, C. D., & Gates, R. H. (1991). of natural language, Gilbert Harman and Contemporary marketing research. West Donald Davidson, eds. Dordrecht: D. Reidel, publishing company. 380-397. Mick, D. G. (1986). Consumer research and Stewart, D. W. (1995). Deception, , semiotics: Exploring the morphology of and survey research: Some lessons from signs, symbols, and significance. Journal of Kraft. Journal of Public Policy & Marketing, consumer research, 13 (2), 196-213. 14 (1), 15-28. Morris, C.W. (1938). Foundations of the Sudman, S. (1995). When experts disagree: Theory of Signs. University of Chicago Press; comments on the articles by Jacoby and 1st edition. Szybillo and Stewart. Journal of Public Moser, C.A. & Kalton, G. (1972). Survey Policy & Marketing, 14 (1), 29-34. Methods in Social Investigation, 2nd Edition, Utts, Jessica M. (2004). Seeing through New York: Basic Books, Inc. statistics. Belmont, CA: ThomsonBrooks/ Mueller, C.B. & Kirkpatrick, L.C. (2004). Cole, 45. Evidence under the rules: text, cases, and Warick, D. P. & Lininger, C.A. (1975). The problems, New York: Aspen Publishers. sample survey: Theory and practice. New Murphy, J.F., Jr. (1993). Maryland Evidence York:McGraw-Hill. Handbook, 2nd ed., Charlottesville, VA: Young, M.L. (1994). Dictionary of polling: Lexis Law. The language of contemporary opinion, Payne, S.L. (1951). The Art of Asking Westport, CT: Greenwood Press. Questions, Princeton, NJ: Princeton Zaltman, G., & Burger, P. (1975). Marketing University Press. research: Fundamentals and dynamics. Peirce, C.S. (1931). Collected Writings (8 Dryden Press. Vols.). (Ed. Charles Hartshorne, Paul Weiss Zikmund, W. G., & Babin, B. J. (2007). & Arthur W Burks). Cambridge, MA: Exploring marketing research: Thomson- Harvard University Press. South Western. Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., & Singer, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68 (1), 109-130. Preston, I. L. (1992). The scandalous record of avoidable errors in expert evidence offered in FTC and Lanham Act deceptiveness cases. Journal of Public Policy & Marketing, 11 (2), 57-67. Raymond, C. (1977). What’s wrong with our (misnamed) attitude research? Advertising Age, 48(4), 124-128.

Marketing Management Journal, Fall 2018 92