Questionnaire Design Asking Questions with a Purpose
Total Page:16
File Type:pdf, Size:1020Kb
Questionnaire Design Asking Questions with a Purpose Paul Pope Extension Program Specialist—Evaluation Chris Boleman Assistant Professor and Extension Specialist Scott Cummings Assistant Department Head and Program Leader for Extension Education Texas Cooperative Extension The Texas A&M University System Updated and revised from the original Questionnaire Design: Asking Questions with a Purpose, developed by Mary G. Marshall Contents Introduction....................................................................................................4 Questionnaires: Part of the Survey Process (The Big Picture) .........................5 Using a Questionnaire: Pro and Cons..............................................................6 Modes of Delivery ..........................................................................................7 Constructing a Questionnaire: A Step-By-Step Approach ................................8 Starting Point Guidelines ...............................................................................9 Developing the Program Impact Section .......................................................11 Developing the Program Improvement Section .............................................14 Developing the Demographic Section ...........................................................15 Appendix A: Wording the Question ..............................................................20 Appendix B: Types of Questions ...................................................................24 Appendix C: Rating Scales ...........................................................................29 Appendix D: Formatting the Questionnaire ..................................................32 Appendix E: Questionnaire Examples ..........................................................34 Introduction Questionnaires defined efficient—and often the only—means of gather- ing evidence on program impact, gaining insight For many activities in Cooperative Extension, into program improvement and knowing more asking questions of program participants is the about the target audience. primary source of information for evaluation pur- poses. Questionnaires provide a means of asking In Extension, most questionnaires are self- questions of participants in an efficient, systemat- administered (completed by program participants ic and consistent manner. using written instructions and/or minimal verbal instructions). They are usually completed in per- A questionnaire is a tool for getting similar son at the conclusion of a program or delivered to information from people that can be tabulated participants via mail after a program concludes. and discussed numerically. Specifically, a ques- tionnaire is a document that asks the same ques- Participants control the data collection process tions of all individuals in a sample or population. in two ways: Note that a questionnaire is not defined by a •They decide which questions they will particular mode of delivery, although it is most answer or not answer. often associated with a paper document such as a • They record a written response (as opposed mail survey. Questionnaires can be delivered in a to an interviewer recording it) to each ques- variety of ways—mail, phone, Web, in-person, tionnaire item. etc.—and may be self-administered or interview- er-administered. The information gathered from This publication questionnaires may be quantitative or qualitative. Developing a sound questionnaire is not quick or easy. It takes time and focused attention—typ- Questionnaires and evaluation ically involving forethought, peer review, several in Extension drafts and pretests before a final version emerges. A questionnaire is one of many data collection Although the process of developing a question- methods available to Extension faculty. Other naire does not fit into a step-by-step approach as methods include direct observation, individual neatly as other processes do, this publication measurement, testing, focus groups and case attempts to present it as such—with some gener- studies. However, questionnaires have been a al guidelines to help you along the way. major source of information for evaluation pur- The concepts and guidelines in this publication poses and will continue to be so into the forsee- apply to all questionnaires—regardless of how able future. Questionnaires have proved to be an they are delivered. 4 Questionnaire Design: Part of the Survey Process The Big Picture The figure below presents an overview of the survey process and includes elements of sam- It is helpful to put questionnaire design in con- pling, data collection, data processing and data text. Questionnaire design is a small part of the analysis. overall survey process that may be used in an eval- uation. As part of this process, questionnaires are some- times interchangeably referred to as survey instru- ments. 5 Using a Questionnaire: Pros and Cons Like all data collection methods, question- are not face to face with an interviewer or if the naires have both advantages and disadvantages. responses cannot easily be inked back to them. Certain types of questionnaires (such as mail Advantages or Web) allow participants to respond at their own convenience—within a data collection peri- Questionnaires provide an efficient way to col- od that typically last several weeks or months. lect data. Generally, it takes less time and resources to develop and administer a question- naire than it would to directly observe or interview Disadvantages participants individually. Questionnaires become Questionnaires are limited in the number and even more efficient as the number of potential scope of questions they can ask. They cannot eas- respondents increases. ily or spontaneously probe deeper into a partici- For evaluations involving many participants pant’s opinions or experiences on a specific topic (1,000 or more) a questionnaire used in conjunc- as can an in-person interviewer. Space considera- tion with sampling is often the most practical and tions and the risk of survey fatigue tend to limit efficient way to collect data. branching opportunities (asking a more specific set of questions based on a previous response). Questionnaires can be used with any of the three primary evaluation strategies: pre-post, ret- Questionnaires rely on the abillity of partici- rospective post or post only. pants to clearly comprehend what is being asked of them, their willingness to answer, their will- Questionnaires can be administered to allow ingness to consider and reflect, and their honesty participants to provide responses confidentially and accuracy of recall in providing responses. or anonymously. Participants feel more secure in Failure on any of these counts may result in no providing frank and complete information if they response or inaccurate responses. 6 Modes of Delivery As mentioned earlier, questionnaires can be Mixed mode administered in a variety of ways: Mixed mode delivery of questionnaires is •Postal (traditional mail) becoming more popular as a means of increasing • Phone response rates. One form of mixed mode delivery •Web involves sending a questionnaire to participants • E-mail in multiple ways. For example, some participants • In-person may be mailed a questionnaire, whereas other •Mixed mode (using more than one method participants may be administered the question- to deliver a questionnaire) naire by phone or sent an invitation via e-mail to There are several factors to consider in decid- complete a Web-based version of the question- ing on a mode of delivery, including anticipated naire. response rates, ability to follow-up or probe, Another form of mixed mode delivery involves speed of data collection and availability of a sam- giving participants options in completing a ques- pling frame. tionnaire. For example, a survey packet may be If you need the data quickly, and most or all mailed to program participants with instructions participants have access to the Internet, deliver- that allow them to complete and return the paper ing a questionnaire via the Web can be an effec- questionnaire or to access and complete the same tive means of collecting data in a short period. questionnaire on the web. Be aware that the mode of delivery you select If you use mixed mode delivery, be sure that may affect questionnaire design to some extent. the wording is consistent (or at least comparable) For example, if the questionnaire is administered between modes so that the results from all modes by phone, it may need to have fewer and more can be tabulated and analyzed together in a valid concise response categories than those on a writ- and meaningful way. ten questionnaire. Although mixed mode delivery is becoming more common, most questionnaires in Extension are still delivered to participants single mode—in person, by mail or via the Web. 7 Constructing a Questionnaire: A Step-By-Step Approach Below is a step-by-step process for developing require—nominal, ordinal, interval) a questionnaire. The guidelines in this process dictate the structure of questions. are sound practices that apply to any question- Frequencies and percentages are naire. The three suggested sections of a question- available with any question struc- naire—Steps 4, 5 and 6—should be adequate for ture. most evaluations. Step 4. Write a set of draft questions for the Step 1. Review and adhere to the Starting program impact section of the ques- Point Guidelines (next section). This tionnaire (page 11). involves thinking about the ques- Step 5. Write a set of