Step-by-Step Guide to Community Surveys

Summary Surveys are great tools to identify and understand trends, expectations and potential problems in your network. This guide will help you plan a membership survey and present the results to your team.

Before you start We invite you to answer a few questions that will help ground your survey. This will ensure a better result.

a) Why do you want to launch a survey? b) What is the goal of your survey? c) What would you like to track with your survey (in other words, what kind of answers would you like to get from your survey questionnaire)? d) What would you like to do with your survey after is finished? e) Who will read your survey results (your team, managers, network members, donors)? f) Why is this survey important for your readers? (Your answer to this question should be in line with your answers to questions a & b.) g) Who is your survey target group (all members of the network, specific sector, country)? h) How many people would you like to interview (it is your sample size)? i) How many responses would you like to get? You should set a minimum number of survey responses based on your survey sample size (minimum of 30 people responses). j) How do you plan to implement the survey (phone, e-mail, face-to-face, on-line)? k) What is the timeline of my survey (days, weeks, a month)?

Remember, grounding your survey is an important first step—your survey questions are a consequence of your survey goals.

Writing questions Take time to think carefully about what you really need out of your survey before writing. Less is more, and shorter is sweeter. There are lots of resources out there for writing good survey questions. What's most important is to avoid biased questions. Check this out for more on smart survey design.

About biased questions This is a story about how biased questions can change your survey results.

In 1992, Ross Perot, an independent contender for the US Presidential election, conducted a mail- in survey to show that the public supported his desire to abolish special interest groups. This is the question he asked: “Should laws be passed to eliminate all possibilities of special interests giving huge sums of money to candidates?” In Perot's mail-in survey, 99% of respondents said "yes" to that question. It seemed as if everyone in America agreed with Perot's stance. Soon after Perot's survey, Yankelovich Partners, an independent market research firm, conducted two interesting follow-up surveys. In the first survey, it used the same question that Perot asked and found that 80% of the population favored passing the law. YP attributed the difference to the fact that it was able to create a more representative sample than Perot.

Interestingly, Yankelovich then conducted a similar survey, but rephrased the question in the following way: “Should laws be passed to prohibit interest groups from contributing to campaigns, or do groups have a right to contribute to the candidates they support?” The response to this question was strikingly different. Only 40% of the sampled population agreed to prohibit contributions. As it turned out, the results of the survey came down to the way the question was phrased1.

Conclusion: For any survey you conduct, it's critical to phrase the question in the most neutral way possible to avoid bias in the sample results.

Avoiding mistakes There are a few simple rules to design a good survey. We would like to highlight four of them: - Target a representative sample of your community (Click here for an online Sample Size Calculator) - Get a high response from your representative sample (it doesn’t make a sense to send a survey to 1000 people and only get 10 answers) - Be Brief - Be Objective - Be Simple - Be Specific (Click here for tips n tricks) - Avoid biasing the response!

Survey Plan and Survey Implementation Now that you know what you want your survey to accomplish, let’s transfer your answers to something called the "survey plan." See Annex 1 for an example survey plan and consider using it as a model.

OK! We are almost ready to launch your survey. But we still have a few more things to do….

1 Harvard Business School – Quantitative Analysis Case, 2011. Step A: Transfer your survey to a data form. If you are planning to use an online survey we would suggest you use Surveymonkey.com. It is an easy and free* service to launch online surveys (*Surveys under 10 questions are free; surveys over 10 questions requires a low, monthly service subscription). The Surveymonkey website provides tutorials and survey tips and is quite simple to customize colors, fonts, and question format. Please click on the link below to see an example of a survey data form using Survey Monkey:

http://www.surveymonkey.com/s.aspx? PREVIEW_MODE=DO_NOT_USE_THIS_LINK_FOR_COLLECTION&sm=0ppaIvIohnGMsBwtJz5 QpRJCNMuh7qGYfUCdxh6Zpgo%3d

There are other great online services available on the Internet. Click on the icons below to see other alternatives for on-line services:

Step B: Set a day and time to launch your survey. Take a look at your Google Analytics to see what day most people are visiting the site. (Click here if you don’t have Google Analytics). The illustration below is an example of how you can best plan the time to launch your survey:

Figure 1

The chart above infers that most of the visitors access an X network on Tuesdays, Wednesdays and Thursdays. Based on this assumption, we will have better responses launching the survey on a Monday night.

Step C: Marketing your survey. Make sure that everybody knows that you are launching a survey. Advertise it: create a banner in your website, or create incentives to attract more responses (raffle a book, or give discounts to a conference you're preparing). Contacting people with a personalized email or phone call is especially effective—the more personal, the better the response.

Step D: Track your results. Use an Excel file to add your results for future analysis. You can also download your survey results in an Excel file using Survey Monkey. See an example in the chart below:

Figure 2

Step E: Read your survey plan again and compare your survey results. Ask yourself: - Do I have a good response rate? - Did I get answers to the right questions ("Before you start," question c)? - Did I reach my goal? - What conclusions I can draw from this survey?

Step F: Prepare a survey report. See Annex 2 for an example. Use it as a model for your own report, but remember to shape your report based on your target audience (team, manager, WBI, all network). The report in Annex 2 was originally intended for WBI staff.

Step G: Give feedback to all survey participants. It is a small action that has a powerful impact to all of the people who helped you in your work. Below is an example of the Global PPP Network survey feedback:

Step H: Review your data and capture lessons. Your survey will always tell you a story about your network and how you can improve it. Make sure you capture these lessons and used them to update your community and fix potential problems (See annex 3). Congratulations! You are ready now not only to set a successful survey, but also to report its results to your team. Annex 1 – Survey Plan

Global PPP Network Survey Developed by Junglim Ham and Kurt Morriesen

1. Goal of the Survey 1. Assess PPP member satisfaction (content, user experience, community function).

2. Identify areas and features to improve the Global PPP Network services.

2. Time to complete the survey Less than 3 minutes

3. Number of questions Pre-survey: 4 Survey: 9

4. Expected day to launch the survey Tuesday is the best day to launch the survey (Tuesday represents 21% of the overall website access over the period of 12/19/2010 to 01/18/2011).

5. Survey type, sample size and representative rate This survey will contact 300 members to answer an online questionnaire. This survey expects to get a 70% of response rate.

6. Pre-survey (Quotas) Based on the social network benchmark, it is important to have a previous survey (quotas) before the main survey is applied. The quotas help to catalog subgroups and subareas, and to enhance the survey’s quality as well.

6.1 Quotas suggested a) Region (Africa, Asia/Pacific, Europe, Middle East, Latin America and Caribbean, North America)

b) How often do you use the internet per day (less than 1 hour, 1 to 3 hours, 3 to 4 hours, 4 to 5 hours, more than 5 hours)

c) Are you (public sector, private sector, bilateral or multilateral organization, non-governmental organization, academia or think tank, other: open question)

7. Survey distribution Survey is divided into three parts: website format, website content, suggestions

7.1 Survey format questions: 1. The user interface / web design allows me to navigate easily and find what I’m looking for. (strongly disagree, disagree, agree, strongly agree ) 2. The content of the website is comprehensive, high quality, and relevant. (strongly disagree, disagree, agree, strongly agree ) 3. The site is clean, simple and visually appealing. (strongly disagree, disagree, agree, strongly agree )

7.2 Survey content

4. Overall, how satisfied are you with the Global PPP Network (very dissatisfied, somewhat dissatisfied, satisfied, very satisfied ) 5. How would you best describe the Global PPP Network for your professional and educational development? (Useless; Not very useful; somewhat useful; very useful). 6. Overall, how would you rate the quality of each of the following activities we engage in the PPP Network? a) PPP Network Groups: (poor, below average, average, above average, outstanding, unable to rate) b) Online Discussions: (poor, below average, average, above average, outstanding, unable to rate) c) eConferences: (poor, below average, average, above average, outstanding, unable to rate) d) Ask-a-question: (poor, below average, average, above average, outstanding, unable to rate) e) Knowledge hub: (e.g. conference materials, reports, publications): (poor, below average, average, above average, outstanding, unable to rate) f) Other. What? (poor, below average, average, above average, outstanding, unable to rate) 7. Overall, What are your primary reasons to visit the Global PPP Network? (Select all that apply) a) Meet people (professionals, potential partners, colleagues) b) Access to the knowledge hub (reports, publications, videos) c) Participate in the eConferences and eDiscussions d) Read about events and upcoming conferences e) Participate in one or more PPP Groups

7.3 Survey suggestions 8. The Global PPP Network would like to improve the quality of its services. Please select multiple items to enhance the Global PPP Network: a) I am satisfied with the Global PPP Network the way it is b) More eConferences and eDiscussions c) PPP news and information about PPP issues in the World Bank and other bilateral/multilateral banks d) Interviews with PPP practitioners from the Global PPP Network and outside of the Network e) Online courses and online training f) More reports and publications about PPPs g) More interaction functions with the PPP members (surveys, quizzes) h) Other? Specify (open question) 9. Comments and Suggestions not covered in this survey. (open question)

8. Access to the online survey http://www.surveymonkey.com/s.aspx? PREVIEW_MODE=DO_NOT_USE_THIS_LINK_FOR_COLLECTION&sm=0ppaIvIohnGMsBwtJz5QpRJCN Muh7qGYfUCdxh6Zpgo%3d Annex 2 – Survey Report

Annex 3 – Data Review

Description of the problem Issue Type Solution “The website should be translated to other System It is possible to translate part of the network content languages: Spanish, Portuguese, Chinese, or (smart charts, interactive maps, and group’s page layout) French” to Spanish and Portuguese. However, translation is an investment for long term. “If it could be more interactive and more System Add more information in the interactive maps and smart information to read” charts. After that, we could develop quizzes about PPP (ex.: design a easy toolbox for “ABC of the PPPs”) or add features of the source book in the Network. “Sometimes I cannot find my own remarks. System This is something that just Ning can solve it. With Hyves, facebook and linked you would expect that these features would be common.” “Have podcasts to download” System It is easy to implement this feature and we could use it for the next eConference. “Flyers on econferences should be distributed eDicussions and The past marketing strategy was successful (Rui’s amongst the members few weeks ahead of eConferences eConference). But, for the next eConferences, we could start date.” use key stakeholders to advertise and promote our discussions in their website (IADB, IFC, AFDB, UNDP,…) “I would like to read more about PPP Knowledge hub and Interactive Maps page will solve this issue. experiences in Latin America, especially in content Peru” “Collaborative work opportunity” Knowledge hub and  Enhance PPP Groups with new features content  Enhance “ask-a-question” feature  Design PPP News and ask to the users to post news.  Design a job page for the users. “Please list reports for services and sectors Knowledge hub and Enhance and add more information in the Interactive where PPP has been successful for us to learn content Maps pages. and get best practices.” “Could you please introduce some credits Knowledge hub and This is a fair suggestion and it is a good tool to improve system to reward members who participate in content ownership and loyalty in the website. online events and activities? Members can use It is easy for us to track motivated members, and we credits to exchange for some qualifications and could create a ranking of the top members in the PPP online benefits.” Network. Also we could highlight motivated members in the PPP News page (it is simple to develop and low cost to maintain).

“There should be introduction of working Knowledge hub and  Design specific eDiscussions for each PPP groups where clusters are develop. and where content Group expert interact and share ideas at no cost.”  Encourage PPP members to be team leaders in their PPP Groups (incentives must be created to support this idea)