QUESTIONNAIRE DESIGN for ESTABLISHMENT SURVEYS Allen

QUESTIONNAI RE DESIGN FOR ESTABLISHMENT SURVEYS

Allen R. Gower, Statistics Canada lS-K R.H. Coats Building, Tunney's Pasture, Ottawa, Ontario KIA OT6

KEY WORDS: Questionnaire design, questionnaire primarily during working hours. The respondent's testing, focus groups, cognitive research functional position and level of responsibility within the esU!blishment affects how a questionnaire is Introduction completed. Identifying the most appropriate person to provide the information is critical in establishment A workshop on Questionnaire Design for surveys. Establishment Surveys took place at the International Response burden is a very real COncern for Conference on Establishment Surveys in Buffalo, New respondents. It involves the number of questions that York, on June 27, 1993. The instructor was Allen R. are asked and the time required to complete the Gower, Head of the Questionnaire Design Resource questionnaire. Response burden also includes the Centre at Statistics Canada. effort that respondents put into searching or This paper provides selections from the notes that manipulating other data sources to provide the participants received during the workshop. The half information in the format requested. day workshop focused on the design of survey Establishments vary in size. Large establishments questionnaires for businesses, farms, and institutions. may have employees whose responsibilities include Topics included the use of pretesting, focus groups, completing form s for governments and surveys. In and cognitive research methods in the development of small establishments, the respondent often is the questionnaires. Also covered were types of questions owner or office manager who may not have as much and considerations in question design and wording. time or fl exibility in their schedule. Examples of questionnaires used hy Statistics Canada There are fundamental differences between and by statistical agencies in other countries were establishment surveys and household surveys. For used to illustrate the principles and practices of example, they differ in the type and complexity of the questionnaire design for establishment surveys. information requested. While household respondents answer from memory, data provided by respondents Establishment Survey Questionnaires in establishment surveys typically involves the use of Establishment surveys include surveys of records or other information systems. Questionnaires businesses, fanns, and institutions. An establishment generally contain technical or professional terminology survey may ask for information about the associated with providing filUlncial or administrative establishment itself, or about a more aggregated (e.g., dnro. These are important considerations that should corporation, ag';ncy), or less aggregated (e.g., division, be taken into account when designing questionnaires department) level of organization. for establishment surveys. A business survey may require information about a company's employees, inventories, products or sales. Questionnaire Development and Testing A farm survey may collect information about a farm 's 1. Objectives. Data Requirements. and Analysis Plan operations including data on crops and livestock. A survey of educational institutions may collect data Define the objectives and infol7lUltion require about students, teachers or other employees. A ments. Prepare a document that provides a clear and survey of health institutions may collect information comprehensive statement of objectives, data about employees, patients or facilities. requirements, and the analysis plan. This document Establishment surveys are frequ ently conducted by is a necessary step that leads to the determination of mail. They also can be administered by an interviewer the variables to be measured and, ultimately, the in person or over the telephone. Follow-ups to mail survey questions and response alternatives. surveys often are conducted by telephone. New data Ensure that the questions relate to the survey collection technologies involve cotmputer-assisted objectives and infol7lUltion requirements. Establish interviewing, fax machines, toucbtone self-response, the rationale for each question, how the information and the electronic transmission of data. will be used, and whether the questions will be good EsUlblishment survey respondents answer in their measures of what is required. role as employers or employees of an establishment,

950

2. Consultation Statisti cs Canada often gives focus group parti cipants a copy of a publication that is of interest to them. In formulating objectives and preparing the Focus groups should be conducted at a tim e that analysis plan, consult with clients and data users to is cODvenient to the participants. For busmcss people, understand fully their requirements and expectatIons. focus groups are usually held during working hours. Contact subj ect maller experts for advice and Other times of the day are more appropnate for other guidance. types of respondents. For example, focus groups with If possible, also consult with members of the target farm operators can be held during the evening. population. This will help identify issues and concerns Focus groups vary in size from 6 to 12 persons. that are important to the target populatlOn, and may The optimum size is 7 to 9 persons. Smaller groups affect decisions regarding the content of the with 4 or 5 people are sometimes called mini focus questionnaire. groups or mini groups. . 3. Previous Questionnaires Focus groups are audio-recorded, and are VIewed by observers in an adjoining room behind a one-way Examine questions that were used in other surveys mirror. on the same or a similar topic. This provides a useful starting point in the formulation of ques tion~ .. 5. Considerations in Drafting the Questions In some situations (e.g., for data comparability over Many considerations go into formulating the time), the same questions may b~ used. Ensure that questions and response categories. Primarily these the questions provide valid, consistent, and effectIve considerations reflect the following: measures of what is required. o Objectives and data requirements 4. The Use of Focus Groups in Developing o Analysis plan Questionnaires o Method of data collection • Data processing requirements A focus group is an informal discussion of a o Respondents selected topic involving participants chosen from the o Availability of the data It insi~ts population of interest. provides into the o Response burden attitudes, opinions, concerns, and expenences of the o Complexity of the data to be collected participants. A focus group is led by ~ mod.erator o Conftdentiality and sensitivity of the information who is knowledgeable about group mtervlewmg o Comparability of results with other surveys techniques and the purpose of the discussion. o Data reliability Focus groups provide the opportunity to consult • Nonresponse with the target population, data users, and • Interviewers interviewers. In the early stages of developing a o Administrative requirements questionnaire, focus groups are. used to cl~ t~e survey objectives and data reqlllrements, to Identify 6. The Response Process salient research issues, and to clarify definitions and Responding to a question involves four distinct concepts. processes: Focus groups also are useful in testing and • Understanding evaluating questionnaires (see 7. The Use of Cognitive o Retrieval Methods in Testing Ouestionnaires). They are used o Thinking/judging to evaluate respondents' understanding of the . Communicating an answer language and wording used in qu.estions ~d Respondents must fust understand the question. instructions, and tf) evaluate alternatIve questIon They then search their memories to retrieve the wordings and formats. requested information. After retrieving the infor Participants from establishments present unique mation, they think about what the correct answer to challenges in recruiting. The usual incentive or the question might be and how much of that answer honorarium of $30 to $50 may not be appropriate for they are willing to reveal. Only then do they business people. Assurances of conftdentiality an.d communicate an answer to the question. emphasis on the importance of the survey an~ therr Most establishment surveys involve the use of participation in the study are more meanmgful. Another type of incentive is a donatIon to the non records. Respondents must normally access one or more external sources of information such as profit organization of the participant's choice. administrative or fmancial records.

951

Tbe ability of respondents to retrieve the requested complete sections, and corrections or cbanges made to information will depend upon tbeir familiarity witb responses. and understanding of tbe external source of informa Tbe interviewer then conducts tbe in-depth tion. Tbey also must understand tbe relationship interview and obtains information about the between tbe survey questions and tbe external data respondent's experiences and impressions LD so urce. Multiple sources of information may add to completing the form. The follow- up discussion tbe difficulty or complexity of this task. Furtber typically involves a review of tbe questionnaire, complexities may be introduced if the respondent bas question by question, with the respondent to discuss to consult anotber individual wbo can provide tbe any problems or difficulties that were encountered requested information and who, in turn, may have to while completing the form. The interviewer probes to use onc or morc data sources. see bow respondents interpreted terms and concepts, how and why they cbose responses, and how th ey 7. Tbe Use of Cognitive Metbods in Testing recalled information especially if the question covers Questionnaires a long reference period. Questionnaire testing is essential to developing In the case of an interviewer-administered effective questionnaires tbat collect useful and questionnaire, the questions are first asked by an meaningful data. Cognitive researcb methods, also interviewer eitber in person or by telephone. Tbe in referred to as qualitative testing, are especially useful depth follow-up discussion takes place following this in testing questionnaires. first interview. Cognitive metbods provide the means to examine • Concurrent think-aloud interviews respondents' tbought processes as they answer tbe The respondent is observed and asked to "think survey questions. Tbey are used to ascertain wbetber aloud" while answering the questions, commenting on or not respondents understand wbat questions mean. each question and explaining how the final response In this way, cognitive methods belp assess tbe validity was chosen. The observer may probe the responses of questions and identify potential sources of to get more information about a particular statement measurement error. or to clarify the process through which a response was Cognitive methods provide tbe opportun.ity to cbosen. evaluate the questionnaire from tbe respondent's point The success of the concurrent think-aloud of view. Tbey focus on issues sucb as comprebension interview technique depends on the respondents' and reactions to tbe form. This brings the respond ability and willingness to articulate and express their ent's perspective directly into the questionnaire design thoughts aloud. The observer sometimes may have to process. The use of cognitive methods leads to the belp respondents in this task by gently prompting design of respondent-friendly questionnaires that can them with comments such as: "what question are you be easily and accurately completed. answering now?," "what are you thinking now?," In establishment surveys, cognitive methods are "please explain how you chose the answer," or other used to investigate the relationship between the probes to clarify the respondent's thoughts. When respondent and the external information source as respondents are reluctant to articulate their thoughts, well as the influence that this data source has on the tbe observer may decide that the best approach is to response process. These methods provide the means handle the interview as an in-depth interview and to assess the compatibility of question wording, time proceed accordingly. reference periods, and response categories with tbe Think-aloud interviews are very useful in obtain establishment's record-keeping practices. ing respondents' reactions to questionnaires. They are Cognitive testing methods include: especially helpful in identifying areas of the • In-depth interviews questionnaire where the respondent has difficulty and The technique involves one-on-one interviews in understanding the process through which the (sometimes called retrospective think-aloud interviews). questionnaire is completed. Respondents first complete the questionnaire as they • Focus groups normally would. An interviewer observes the process. Focus groups are used to evaluate respondents' The observer notes areas of the form read, the understanding of the language and wording used in sequence in which the questions are answered, questions and instructions. The questionnaire is reference made to instructions, types of records or usually administered before the focus group session in other persons consulted, the time required to person, over the telephone or on a self-completion basis as appropriate.

952

During the focus group session th e moderator Pretesting is not as effective as cognitive meth ods re -it. "/;. the questi onnaire with the participants and in evaluating respondents' understanding and the discusses any problems or diliiculties that they may difficulty of the response task. Pretesting only have encountered when completing the form. In this indicates that there is a problem. Without furth er way fo cus groups encourage thoughtful analysis of the investigation, it does not identify why there is a questionnaire. They are especially useful in providing problem nor how it can be corrected. suggestions and recomm endations for improvements. Debriefing sessions with interviewers often occur • Paraphrasing in conjunction with a p"retest. Interviewers can Paraphrasing is used in one-an-one interviews and identify important problem areas where the focus groups. Respondents are asked to repeat the questionnaire can be improved. During the redesign question in their own words, or to explain the of existing questionnaires, it is useful to consult with meaning of terms and concepts that are used in th e interviewers to get their input into the redesign survey questions and instructions. process. Interviewers have excellent insights into the Paraphrasing helps determine whether respondents logistics of administering the questionnaire and how it read and understand the instructions and questions affects respondent cooperation. correctly. Paraphrasing is especially helpful in Behavioral coding also can be conducted at the identifying situations where the question wording is time of pretesting. Behavioral coding provides a too com plex or confusing and where respondents do systematic and objective means of examining the not comprehend all the important components of the effectiveness of the questionnaire. The interview is question, such as the recall period. audio-recorded, and the interviewer-respondent interaction is later analyzed. Behavioral coding helps 8. Pretesting identify problem areas such as an interviewer failing to read the question as worded or a respondent asking Pretesting is a fundamental step in the process of for clarification of the question or response task. developing a questionnaire. It involves a relatively small number of fi eld interviews that are carried out 9. Formal Testing Methods to identify problems with a questionnaire. The entire questionnaire or only a portion of it may be tested. Formal testing methods are quantitative in nature. Pretests are useful for discovering poor question They are designed to provide a statistical evaluation of wording or ordering, errors in questionnaire layout or how the questionnaire performs. Pilot studies and instructions, and problems caused by the respondent's split sample testing are two types of formal testing inability or unwillingness to answer the questions. methods. These methods are more suitable for large Pretests also are used to suggest additional response scale and continuing surveys because of the significant categories that can be pre-coded on the questionnaire. cost associated with a large sample size and the Pretests provide a preliminary indication of the analysis of the results. interview length and refusal problems. A pilot study is conducted in order to observe all The pretest sample can range in size from 20 to survey operations working together, including the 100 or more respondents. If the main purpose of the administration of the questionnaire. pretest is to discover wording or sequencing problems, A pilot study is a "dress rehearsal." It duplicates only a minimum number of interviews may be the fmal survey design on a small scale from required. More interviews (50 to 100) are necessary beginning to end, including data processing and to determine pre-coded answer categories based on analysis. It allows the survey researcher to see how open-ended responses obtained in a pretest. well the questionnaire performs in conjunction with Respondents are generally selected purposively rather other phases of the survey. Normally, the than randomly. questionnaire should be thoroughly pretested before The questionnaire should be administered in the a pilot test takes place. A pilot study is usually not same manner as planned for the main survey (e.g., the time to try out new questions or approaches. interviewer-administered in person or by telephone). If previous testing has been carried out, it is A pretest of a mail questionnaire is more effective if unlikely that the pilot study will result in major interviewers are used. Interviewers can be used to changes to the questionnaire. The pilot study, deliver the questionnaire and, afterwards, to discuss however, does provide the opportunity to fme tune the any problems. The questionnaire designers should questionnaire before its administration in the main observe as many pretest interviews as possible. survey.

953

Split sample testing is conducted to determine the Question Design and Wording nbest" of two or morc alternative versions of the questionnaire. Split sample testing also is referred to 1. Considerations in Question Wording as "split ballot" or "split panel" experiments. It ./ Are the words simple, direct and familiar to all involves an experimental sample design that is respondents? incorporated into the data collection process. In a ./ Are the questions as clear and as specific as simple split sample design, half of the sample receives possible? one version of the questionnaire and half, the other. ./ Are the questions applicable to all respondents? ./ Are any of the questions double-barrelled? lO. Review and Revision of the Questionnaire ./ Are the response categories mutually exclusive and The questionnaire should be reviewed by someone exhaustive? outside the project team. Reviewers could include ./ Are any questions leading or loaded? subject matter experts or persons who have experience ./ Do the questions read well? in designing questionnaires. A review can take place at any or all stages of the questionnaire development 2. The questions should be designed to: process. ./ Encourage respondents to complete the The review and testing of the questionnaire lead to questionnaire revisions in the questions and response categories. ./ Facilitate respondents' recall Throughout the whole process of questionnaire ./ Direct respondents to the information source development and testing, changes will continually be ./ Focus on the topic of the survey made to improve the questionnaire. In this way, ./ Follow the logic of the respondent questionnaire design is an iterative process. ./ Flow smoothly from one question to the next Objectives and information requirements are stated, evaluated and decided upon, data users and 3. It is important to ensure that: respondents are consulted, proposed questions are ./ Time reference periods are clear to the respondent drafted and tested, questions are reviewed and revised, " Questions, time reference periods, and response until a final questionnaire is developed. categories are compatible with the establishment's record-keeping practices (this is often difficult to Types of Questions acrueve)

There are two main types of questions: open and 4. The Introduction closed questions. They are sometimes called open ended and closed-ended questions. ./ Provide the title or subject of the survey Open questions are answered in the respondent's ./ Identify the sponsor own .words. An open question allows the respondent ./ Explain the purpose of the survey to interpret the question and answer freely. The ./ Request the respondent's co-operation respondent writes the answer or the interviewer ./ Indicate the degree of confidentiality and any data records verbatim what the respondent says in answer sharing arrangements to the question. Blank spaces are left in the ./ In mail surveys, provide the return address and the questionnaire after the question for the response to be date for return written in. Establishment survey questionnaires Establishments are asked to complete many forms frequently contain open questions with labelled answer and questionnaires. Completing them is not a spaces where respondents or interviewers are asked to priority. Typical reactions include: record data. • "I complete the shortest form first" Closed questions are answered by checking a box • "Is completion mandatory?" or circling the proper response from among those that • "Is there a return deadline?" are provided on the questionnaire. A closed question A respondent once commented that if the answer restricts the respondent or interviewer to select from to these last two questions is "no," then "/ put it [Ihe the answers or response options that are specified. questionnaire 1 in my maybe I'll gel 10 il someday baskel!"

954

Respondents frequently question the value of ./ Illustrations and symbols (such as arrows and information to themselves and to other users. Some circles) should be used to attract attention and like to receive feedback about thc survey. Therefore: guide respondents or interviewers ./ Explain why it is important to complete the ./ The last page or end of the questionnaire should questionnaire provide space for additional comments by ./ Ensure that the value of providing information is respondents made clear to respondents ./ Include an expression of appreciation (thank you). ./ Explain how the survey data will be used ./ Explain how respondents can access the data References 5. The Opening Ouestions Biemer, P., R. Groves, L. Lyberg, N. Mathiowetz and S. Sudman [eds.) (1991), Measurement Errors in ./ The opening questions should relate to the Surveys, John Wiley & Sons, Inc., New York. introduction and the survey ohjectives Bradburn, N., S. Sudman and Associates (1979), ./ They should be applicable to all respondents Improving Interview Method and Ouestionnaire ./ They should be easy and interesting to complete Design, Jossey-Bass Inc. Publishers, San Francisco . ./ If necessary, they should establish that the Bureau, Marcel (1991), "Experience with the Use of respondent is a member of the survey population Cognitive Methods in Designing Business Survey 6. Instructions Questionnaires,1\ Proceedings of the Section on Survey Research Methods, American Statistical Respondents read only what they think is necessary Association, pp. 713-717. to read. They read the boldface print first, and then Butani, Shail, Karen Goldenberg and Polly Phipps decide whether they should read further. (1993), "Response Analysis Surveys for Assessing Respondents rarely read the instructions, and Response Errors in Establishment Surveys; Invited proceed directly to the questions. They only refer to Paper, International Conferenee on Establishment the instructions when they think they need help. As Surveys, Buffalo, New York. a result, respondents may miss important instructions Converse, Jean M. and Stanley Presser (1986), Survey and definitions. Errors in reporting often are due to Questions: Handcrafting the Standardized Ques a lack of instructions or misunderstanding what to tionnaire, Sage University Paper series on include or exclude. Therefore: Quantitative Applications in the Social Sciences, ./ Ensure that instructions are short and clear 07-063, Sage Publications, Beverly Hills, California . ./ Tell the respondent where to frod the instructions DeMaio, T J . [ed.) (1983) , Approaches to Developing ./ Provide deflnitions at the beginning of the ques- Questionnaires, Statistical Policy Working Paper 10, tionnaire or in specific questions as required United States Office of Management and Budget, ./ Use boldface print to emphasize important items Washington, D.C. such as the reference or reporting period DeMaio, T. and C. Jenkins (1991), "Questionnaire ./ Specify "include" or "exclude" in the questions and Research in the Census of Construction Industries," items themselves (not in separate instructions) Proceedings of the Section on Survey Research Methods, American Statistical Association, pp. 496- 7. General Guidelines for the Layout of the 501. Questionnaire Desvousges, William H. and James H. Frey (1989), ./ The questionnaire should appear interesting and "Integrating Focus Groups and Surveys: Examples easy to complete from Environmental Risk Studies; Journal of ./ The questionnaire should be respondent-friendly Official Statistics, Statistics Sweden, Vol. 5, No.4, ./ The cover letter and front cover should create a pp. 349-363 . positive first impression Dutka, Solomon and Lester R. Frankel (1991), ./ The questionnaire should appear professional and "Measurement Errors in Business Surveys," in business-like Measurement Errors in Surveys, P. Biemer et a1. ./ If administered in person or over the telephone, the [eds.), John Wiley & Sons, New York, pp. 113-123 . questionnaire should be interviewer-friendly Edwards, W. Sherman and David Cantor (1991), ./ Provide titles or headings for each section of the 'Toward a Response Model in Establishment questionnaire Surveys," in Measurement Errors in Surveys, Paul P. ./ The instructions and answer spaces should facilitate Biemer et al. [eds.), John Wiley & Sons, New York, proper answering of the questions pp. 211-213.

955

Edwards, W. Sherman, Gregory H. Gaertner and Field Test," Proceedings of the Section on Survey Veronica F. Nieva (1993), "Effects of Organizational Research Methods, American Statistical Associa Context on Survey Response," Invited Paper, tion, pp. 179-184. International Conference on Establishment Surveys, Phipps, Polly A (1990), "Applying Cognitive Theory to Buffalo, New York. an Establishment Mail Survey," Proceedings of the Esposito, James L. Pamela C. Campanelli and Section on Survey Research Methods, American Jennifer Rothgeb (1991), "Determining Which Statistical Association, pp. 608-612. Questions are Best: Methodologies for Evaluating Phipps, Polly A (1991), "Assessing Measurement Survey Questions," Proceedings of the Section on Errors in a Touchtone Recognition Survey," Survey Survey Research Methods, American Statistical Methodology, Statistics Canada, Vol. 17, No. 1, pp. Association, pp. 46-53. 15-26. Goldman, Alfred E. and Susan Schwartz McDonald Phipps, Polly A, Kenneth W. Robertson and Keith G. (1987), The Group Depth Interview: Principles and Keel (1991), "Does Questionnaire Color Affect Practice, Prentice-Hall, Englewood Cliffs, NJ. Survey Response Rates?," Proceedings of the Gower, Allen R. and Ruth Dibbs (1989), "Cognitive Section on Survey Research Methods, American Research: Designing a Respondent-Friendly Ques Statistical Association, pp. 484-488. tionnaire for the 1991 Census," Proceedings of the Phipps, P., S. Butaoi and Y. Chun (1992), "Designing Fifth Annual Research Conference of the U.S. Establishment Survey Questionnaires," Paper Bureau of the Census (ARC V). pp. 73-100. presented at the American Association for Public Gower, Allen R. and Mukund S. Nargundkar (1991), Opinion Research Conference, SI. Petersburg, "Cognitive Aspects of Questionnaire Design: Florida. Business Surveys versus Household Surveys," Preston, John L., Dwight K. French and Lynda T. Proceedings of the 1991 Annual Research Carlson (1993), ''The Role of Focus Groups in the Conference, Washington, DC: United States Development of Establishment Surveys," Invited Bureau of the Census, pp. 299-312. Paper, International Conference on Establishment Gower, Allen R. and Peter D. Zylstra (1990), "The Surveys, Buffalo, New York. Use of Qualitative Methods in the Design of a Stewart, D.W. and P.N. Shamdasaoi (1990), Focus Business Survey Questionnaire", Contributed Paper, Groups: Theory and Practice, Applied Social International Conference on Measurement Errors in Research Methods Series, Vol. 20, Sage Surveys, Tucson, Arizona. Publications, Newbury Park, California. Krueger, Richard (1988), Focus Groups: A Practical Sudman, S. and N.M. Bradburn (1982), Asking Guide for Applied Research, Sage Publications, Questions: A Practical Guide to Questionnaire Newbury Park, California. Design, Jossey-Bass Inc. Publishers, San Francisco. Lawrence, David and Frances Laffey (1993), Tanur, Judith M. [ed.) (1992), Questions about "Qualitative Testing of the Farm Financial Survey Questions: Inquiries into the Cognitive Bases of Questionnaire," Contributed Paper, International Surveys, Russell Sage Foundation, New York. Conference on Establishment Surveys, Buffalo, N.Y. Werking, George S., AJan R. Tupek and Richard L. Morgan, David L. (1988), Focus Groups as Oualitative Clayton (1988), "CAT! and Touchtone Self Research, Qualitative Research Methods, Vol. 16, Response Applications for Establishment Surveys," Sage Publications, Newbury Park, California. Journal of Official Statistics, Statistics Sweden, Vol. Morgan, David L. [ed.) (1993), Successful Focus 4, No.4, pp. 348-362. Groups: Advancing the State of the Art, Sage Werking, George S., AJan R. Tupek, Chester H. Publications, Newbury Park, California. Ponikowski and Richard J. Rosen (1986), "A CAT! Nelson, D.D. (1985), "Informal Testing as a Means of Feasibility Study for a Monthly Establishment Questionnaire Development," Journal of Official Survey," Proceedings of the Section on Survey Statistics, Statistics Sweden, Vol. 1, No.2, p. Research Methods, American Statistical Association, 179-188. pp. 639-643. O'Brien, Kerth (1993), "Improving Survey Werking, George S., and Richard L. Clayton (1991), Questionnaires through Focus Groups," in "Enhancing Data Quality Through the Use of Mixed Successful Focus Groups, David L. Morgan [ed.), Mode Collection," Survey Methodology, Statistics Sage Publications, Newbury Park, California, pp. Canada, Vol. 17, No.1, pp. 3-14. 105-117. Palamisano, M. (1988), ''The Application of Cognitive Survey Methodology to an Establishment Survey

956