SURVEY GUIDE SURVEY FUNDAMENTALS A GUIDE TO DESIGNING AND IMPLEMENTING SURVEYS 1 S U R V E Y G UIDE OFFICE OF QUALITY IMPROVEMENT SURVEY FUNDAMENTALS This guide describes in non-technical terms the underlying principles of good survey design and implementation. Clear, simple explanations lead the reader through methodology and logistics decisions, writing effective questions, and drawing conclusions from the results. The guide also provides tips for maximizing the response rate as well as an overview of human subject rules. The material presented here is based on short courses presented by staff of the University of Wisconsin Survey Center in 2008 in honor of the UWSC’s 25th anniversary. The course materials were developed by Nora Cate Schaeffer, Jennifer Dykema, Kelly Elver, and John Stevenson. Nancy Thayer-Hart compiled this guide based on those courses, supplemented with additional material. Version 2.0, December 2010 Copyright 2010 University of Wisconsin System Board of Regents Contributors: Nancy Thayer-Hart, Office of Quality Improvement Jennifer Dykema, Kelly Elver, Nora Cate Schaeffer, John Stevenson, University of Wisconsin Survey Center For additional information contact: Office of Quality Improvement Room 199 Bascom Hall, 500 Lincoln Drive Madison, WI 53706-1380 608-262-6843 FAX: 608-262-9330 [email protected] http://www.quality.wisc.edu University of Wisconsin Survey Center 1800 University Avenue Madison, Wisconsin 53726 608-262-9032 FAX: 608-262-8432 [email protected] http://www.uwsc.wisc.edu S U R V E Y G UIDE CONTENTS Contents ............................................................................................................... 3 Survey Basics ....................................................................................................... 4 Design the Survey Process .................................................................................. 4 What are the goals? .......................................................................................... 4 What is the target population? .......................................................................... 5 What is the timing?............................................................................................ 5 What mode will be used? .................................................................................. 5 Develop Questions ............................................................................................... 6 Reliability ........................................................................................................... 6 Validity............................................................................................................... 6 Tips for Writing Good Questions ....................................................................... 7 Response Format ............................................................................................ 10 Design of Self-Administered Questionnaires .................................................. 11 Checklist for Effective Questionnaires ............................................................ 12 Test and Train .................................................................................................... 12 Collect Data ........................................................................................................ 14 Response Rate ............................................................................................... 14 Follow-up Procedures ..................................................................................... 14 Web Survey Challenges .................................................................................. 15 Analyze Data ...................................................................................................... 15 Coding and Analyzing Data ............................................................................ 15 Drawing Conclusions ...................................................................................... 16 Additional Considerations ................................................................................... 16 Human Subjects Protection............................................................................. 16 A Word About Climate Surveys....................................................................... 17 Getting Help with Your Survey ........................................................................... 17 Terminology ........................................................................................................ 18 References and Additional Resources ............................................................... 19 Works Cited ..................................................................................................... 19 Additional Resources ...................................................................................... 19 3 S U R V E Y G UIDE SURVEY BASICS A survey is often the best way to get information and feedback to use in planning and program improvement. This guide is written to help you achieve the results you need. The principles and practices described here have been recommended by the University of Wisconsin Survey Center. Designing and implementing a survey is a systematic process of gathering information on a specific topic by asking questions of individuals and then generalizing the results to the groups represented by the respondents. The process involves five distinct steps. Figure 1 depicts the process of designing and implementing a survey: Design Survey Develop Test & Train Collect Data Analyze Data Process Questions Figure 1. Survey Process The type of survey you use for a particular purpose will be the result of decisions you make about: 1. Contacting potential respondents (in-person, telephone, mail, email) 2. Presenting the questions (written, interviewer) 3. Recording responses (paper, electronic) Your choices will be influenced by your research goals and timeline, how sensitive or complex the study topic is, the characteristics, abilities and resources of potential respondents (e.g., their access to and experience with technology), and of course, your budget. The choices you make will affect the quality, cost and timeliness of your results. DESIGN THE SURVEY PROCESS WHAT ARE THE GOALS? The first step in planning for your survey is to determine the goal and how you will use the information. The usual goal of a survey is to describe a population, for example, the group of people using your services. Another common goal that may require a very different study design is to make comparisons between groups, such as comparisons between those using your services and those who do not. Often surveys are used simply to get feedback from a specific group of people and/or learn about their wants and needs. 4 S U R V E Y G UIDE WHAT IS THE TARGET POPULATION? Who are the people you’re interested in hearing from? What criteria separate those who should be included in the survey from those who should not? When the target population is small and easy to contact, it is often feasible to survey the entire population. However, when the target population is large or widely dispersed and the survey is part of a rigorous research project, a “sampling” process is typically used to select and survey only a subset of the total population of interest. If a sampling process would enhance the results of your survey, you will want to engage the services of a sampling statistician to guide you. One reason to survey a sample rather than an entire population is that with a smaller sample size, you can include more follow-up contacts to encourage responses. As a result of these efforts, participation and data quality may be improved. In order to reach your target population, you will need current and accurate contact information. Obtaining and verifying this information can be costly and time-consuming, especially if your target population is highly mobile and/or difficult to locate. A service for mass emailing to campus audiences is available through Division of Information Technology (DoIT). See http://www.doit.wisc.edu/lists/massemail/index.asp for additional information. There is a fee for using this service. WHAT IS THE TIMING? Developing a timeline is a crucial step in planning your survey. When is the survey data needed? When would be the best time to contact potential participants? Do you want to survey during a break or at a time when respondents are engaged in another activity with your unit? If your survey is going to a campus population, you may want to avoid coinciding with one of the periodic campus-wide surveys. What other workload issues may affect those involved in sending the survey and collecting and analyzing the results? How long will it take to design and obtain approval for the survey? How much time is needed for collecting responses? The answers to all these questions will help you develop a realistic timeline. WHAT MODE WILL BE USED? As noted above, there are a variety of options for how you conduct your survey. The combination of choices you make about contacting sample members, administering questions, and recording responses is called the “mode.” The mode is influenced by the survey population, the study topic, and how you plan to use the information you gather. 5 S U R V E Y G UIDE DEVELOP QUESTIONS A survey question is a measuring device for things that are not directly observable, in the same way that a bathroom scale is a measuring instrument for weight. In order for your survey