FS995 Fact sheet For a comprehensive list of our publications visit www.rce.rutgers.edu A Step-by-Step Guide to Developing Effective Questionnaires and Survey Procedures for Program Evaluation & Research Keith G. Diem, Ph.D., Program Leader in Educational Design urveys can be an effective means to collect data ■ Can I get the information from existing sources needed for research and evaluation. How- instead of conducting a survey? ever, the method is often misused and abused. The S It’s a good idea to start with research questions or objec- challenge is to design a survey that accomplishes its purpose and avoids the following common errors: tives. Here are some examples: ■ Sampling Error (How representative is the group Research Questions: being surveyed?) ■ Frame Error (How accurate is the list from which ■ What is the household income of program partici- respondents are drawn?) pants in the money management course? ■ Selection Error (Does everyone have an equal chance ■ How many farmers use IPM in the county? of being selected to respond?) ■ What public speaking skills does a typical 4-H ■ Measurement Error (Is the questionnaire valid and member possess? reliable?) ■ Non-response Error (How is the generalizability of Research Objectives: findings jeopardized because of subjects who did not ■ To determine the degree of recycling used in house- reply?) holds of children enrolled in a summer youth envi- This fact sheet provides guidance for constructing ques- ronmental program. tionnaires and developing procedures to administer them ■ To determine the average number of acres planted so they achieve valid and reliable results. This is not with no-till methods by dairy farmers attending the difficult if a logical process is followed. Extension course. ■ To determine the average cost per meal served in the household of EFNEP participants. 1. Determine the purpose Questionnaires are typically used for survey research, to 2. Decide what you are measuring determine the current status or “situation.” They are also used to measure the difference in status “before” and As with determining the purpose, this should be based on “after” to determine changes that may be attributed to an the objectives of your educational program and the educational program. Before creating a questionnaire, evaluation of its outcomes and impact. Consider which start by asking yourself a few important questions: of the following you are aiming to measure: ■ What do I need to know? ■ Attitude ■ Why do I need to know it? ■ Knowledge ■ What will happen as a result of this questionnaire? ■ Skills ■ Goals, intentions, aspirations error” is avoided by eliminating duplication from ■ Behaviors and practices these lists. ■ Perceptions of knowledge, skills, or behavior 4. Consider the audience Of course, it’s possible that you might measure more than one . But the questions will be clearly different based on ■ Age the information you are trying to gather. Refer to the RCE ■ Education level fact sheet, “Measuring impact of educational pro- ■ Familiarity with tests & questionnaires grams” (FS869), to learn more about the types of out- ■ Cultural bias/language barrier comes that can be measured. To ensure that the survey instrument you develop is appropriate for your audience, “field test” your ques- 3. Who should be asked? tionnaire with other people similar to your respondents before administering the final version. This will allow ■ What is the appropriate population (group of people/ you to improve unclear questions or procedures and subjects) to be studied or questioned? detect errors beforehand. Following recommendations ■ Should a census or sampling be used? in this guide pertaining to questionnaire design and • A population is the complete set of subjects that wording of questions will reduce systematic “measure- can be studied: people, objects, animals, plants, ment” error, which will improve the internal validity of etc. your study. • A sample is a subset of subjects that can be studied to make the evaluation/research project more man- ageable. 5. Choose an appropriate data • If a large enough random sample is taken, the collection method. results can be statistically similar to taking a census of an entire population - with reduced effort ■ Mailed and cost. ■ Telephone ■ The population sampled from as well as sampling ■ Personal (face-to-face) interview method used affects to whom research findings can ■ Web-based be generalized. In other words, for whom do the results apply? This is an indication of the external See the RCE fact sheet, “Choosing a data collection validity of a study. method for survey research” (FS996), for more informa- ■ There are a variety of ways samples can be taken: tion about the advantages and disadvantages of each • Simple random (pull names from a hat) method. • Systematic random (i.e. every 5th name) • Stratified random (separate samples for each sub- group) 6. Choose a collection procedure: • Cluster sampling (treating intact groups that can- anonymous vs. confidential not be broken up, such as classrooms, as subjects to be sampled) Confidential ■ Whatever population is studied or the sampling ■ Name or other identifiers are used to follow-up with method used, a high percentage of respondents is nonrespondents or match data from pre-test/post- critical to ensure the respondents are truly represen- tests. tative of the population being studied. Nonresponse ■ Individual data are NOT shared with anyone! Infor- error affects the validity of the study, and a plan for mation is not used for any other purpose. dealing with it, should be determined in advance. ■ See the RCE fact sheet, “Maximizing response rate Confidentiality must never be breached! This and controlling nonresponse error in survey re- pledge is crucial in attaining honest, complete an- search” (FS997), for more information about this. swers from respondents. ■ ■ Other possible errors can be avoided with simple Identifying information is destroyed after survey is procedures: “sampling error” is reduced by using a complete. large, random sample or conducting a census; “frame Anonymous error” is minimized by making sure the list of poten- tial subjects is current and accurate; and “selection ■ Name is not asked of respondents. 2 ■ Because no other identifying codes are used, the 11. Use plain language researcher is unable to follow-up with nonrespondents or match data from pre-test/post- ■ Be direct tests. This may not be a problem when doing random ■ Use simplest language necessary interviews (such as exit surveys). ■ Avoid jargon and acronyms ■ Collecting basic descriptive information about re- ■ Include definitions if needed spondents is still useful for comparing respondents with the population. 12. Be brief ■ One possible way to maintain anonymity while also keeping track of nonrespondents is to send a separate ■ Keep the questionnaire as short as possible (without post card with the questionnaire. The respondent can jeopardizing reliability) return it separately, enabling him or her to declare ■ Focus on “need to know” questions and minimize that “John/Mary Doe has returned the question- “nice to know” information naire.” 7. Choose measurement scale and 13. Put most important questions scoring up front Use scales that provide the information needed and are ■ Respondents may get fatigued or hurried by later appropriate for respondents. Some choices are: questions. ■ Include questions about demographic information at ■ Fixed-response: the end so questionnaire is focused on topic at hand. • Yes-No • True-False • Multiple Choice 14. Make sure questions match • Rating Scale/Continuum (such as a Likert-type the measurement scale scale) selected, and answer • Agree-Disagree categories are precise • Rank ordering ■ Open-ended (narrative response) ■ Make sure answer choices correspond to the ques- tions, both in content and grammar. ■ 8. Title the questionnaire Be consistent in arranging the answers. While it is conventional to read English from left to right, and ■ This will let the respondent know what it’s about go from “low” to “high,” the most important rule is ■ Include a brief purpose of the study (one sentence or to explain the “rule” being used with clear instruc- phrase) tions and to apply the rule consistently throughout ■ Consider including a simple graphic that depicts the the questionnaire. ■ purpose of the evaluation or program Use exact numbers when possible (instead of Fre- quently, Rarely) ■ 9. Start with non-threatening Define time frames if necessary. Instead of “re- cently,” ask “last month” or “during August 2001.” questions ■ Make sure answer categories do not overlap. ■ If you are using a continuum scale with numbers to ■ This will make sure the respondent is not intimidated represent concepts, make sure to “anchor” at least the ■ Make the first questions relevant to the title/purpose top and bottom of the scale with terms that describe and easy to answer meanings of the numbers. (For example, 1 = Low, 10 = High). 10. Include simple instructions ■ Balance the “negative” or “low” answer choices (both in number and degree) with “positive” or ■ How to complete each section “high” choices on the scale. For example, don’t give ■ How to mark answers (pen/pencil, circle, check, etc.) only positive answer choices or five degrees of 3 “positive” (i.e. great, excellent, super, fantastic, 19. Provide space to tell more awesome) and only one, extreme “negative” re- sponse choice (i.e. terrible). ■ Give respondent room to comment about individual ■ An even number of answer choices doesn’t give the questions or the survey as a whole. respondent an easy, “middle” choice. If you want to ■ Ask for “Any additional comments or sugges- offer a “neutral” or “no opinion” choice, then do it by tions?” design, not by accident. ■ Determine in advance how questions will be 20. Make sure it looks professional! scored, what to do with missing data, incomplete or unclear responses, etc. ■ Pruuf reed! ■ Consider using a “booklet” format so it stands out 15. Ask only one question at a from just “paper.” ■ Use quality reproduction time ■ Proof read! ■ Avoid “double-barreled” questions that confuse the respondent into not knowing how to answer.