Designing Effective Survey Instruments
Total Page:16
File Type:pdf, Size:1020Kb
rms” Designing Definitions: Te me So – “ Effective ut do an Survey Data H Instrumentation Instruments Quantitative and Qualitative Survey and a Questionnaire Thomas M. Archer, Ph.D. Reliability and Validity 2007 Data Collection Methods Qualitative Methods Consider Major Types of Qualitative Methods: Using multiple measures to increase validity Observations Pairing qualitative and quantitative methods to In Depth Interviews obtain a more complete picture of the phenomenon of interest. Focus Group Interviews Using both qualitative and quantitative methods in Documents and Records sequence so results of each method provides Nominal Group Technique information for the next. Key Informants Case Studies Qualitative Methods Quantitative Methods Other Types of Qualitative Data: Major Types of Quantitative Methods: Written Questionnaire with Open Ended Questions Mail Survey Expert Review Telephone Survey Community Forums/ Public Hearings Group Administered Survey Delphi Technique Web Based Survey 1 Surveys Advantages of Surveys: Measures opinions, knowledge, Can complete anonymously attitudes, beliefs, behaviors, Inexpensive to administer reactions, and attributes in response Easy to compare and analyze to specific questions Administer to many people When need to quickly and/or easily get lots of information from people Can get lots of data in a non threatening way Many sample questionnaires already exist Disadvantages of Surveys: Advantages of Mail Surveys: Might not get careful feedback Efficient for volume of information Wording can bias client's responses collected People more likely to provide frank, Impersonal thoughtful, honest info that is tension In surveys, may need sampling expert free Doesn't get full story Gives people more time All respondents receive exactly the same questions in the same way Advantages of Telephone Disadvantages of Mail Surveys: Surveys: Low response rate Response rate generally high (IF, Must be simple & easy to understand actually talk to a person) Need accurate mailing lists Speed Mailing and copy expense Researcher can provide clarification on Privacy, confidentiality, and anonymity unclear questions must be assured More relaxed than with face to face Results may be misleading if do not follow-up with non respondents 2 Disadvantages of Telephone Advantages of Group Surveys: Administered Surveys: Time consuming High response rate Need telephone numbers Easy to clarify items to all respondents Need trained interviewers Provides greatest sense of respondent Interviewer’s voice may bias anonymity Need simple and easy to understand Inexpensive questionnaire Overabundance of telemarketing Disadvantages of Group Advantages of Web Based Administered Surveys: Surveys: May require the cooperation of others; Nearly complete elimination of paper, i.e. to access groups postage, mail out, and data entry costs Time for implementation can be reduced Reach only those in attendance Once electronic data collection system is Group dynamics may influence developed, cost of surveying additional individual responses respondents is much less Display of response data can be simultaneous Opportunity for researcher influence with completion of surveys Reminders and follow-up on non-respondents is relatively easy Disadvantages of Web Based Surveys: Surveying Questions: Not everyone is connected Who to survey? Not all potential respondents are What is the topic? equally computer literate What is the implementation plan? Sampling of e-mail addresses is difficult How many contacts to make? (no directories) Will contacts be personalized? The decision not to respond is likely to What is the interval between contacts be made more quickly How long is the questionnaire? 3 Survey Error Sampling Error Errors in Surveying Coverage Error Measurement Error Non Response Error Sampling Error Coverage Error The result of surveying only some, and The result of not allowing all members of the not all, elements of the population survey population to have an equal or known non zero chance of being sampled for participation in the survey The extent to which the precision of sample survey estimates is limited by When the “list” from which the sample is the number of persons surveyed drawn does not include all elements of the population Measurement Error Non Response Error Respondent’s answer to a survey The result of people who respond to a question is inaccurate, imprecise, or survey being different from sampled cannot be compared in any useful way individuals who did not respond, in a way relevant to the study to other respondents’ answers When a significant number of people in the Results from poor question wording sample do not respond AND have different and questionnaire construction characteristics from those who do respond. 4 Tailored Design Method Survey Procedures that create Tailored Design Method respondent trust and perceptions of increased rewards and reduced costs for being a respondent, which take into account features of the survey Don Dillman situation and have as their goal the overall reduction of survey error. Social Exchange Implementation Process The likelihood of responding accurately Establish Trust is greater when the respondent trusts Increase Rewards that the expected rewards of Reduce Social Costs responding will outweigh the anticipated costs Establish Trust Reduce Social Costs Provide token of appreciation in Avoid subordinate language advance Avoid embarrassment Sponsorship by legitimate authority Avoid inconvenience Make the task appear important Make questionnaire short and easy Invoke other exchange relationships Minimize requests for personal information Emphasize similarity to other requests 5 Overview of Constructing a Tips for Questionnaire Questionnaire Design 1. Write the purpose of the study 2. Make a list of what you want to know 3. Check to see if information is already “Asking Questions with a Purpose” available 4. Only ask questions that you will use 5. Consider how you will use each piece of information 6. View questions through the eyes of the respondent 7. Be selective and realistic Kinds of Information that can be Tips on Wording the Questions: obtained through a Questionnaire: 1. Use simple wording 1. KNOWLEDGE – what people know; 2. Avoid using abbreviations, jargon, or how well they understand something foreign phrases 2. BELIEFS – ATTITUDES – OPINIONS 3. Be specific 3. BEHAVIOR – What people do 4. Use clear wording – do not be vague 4. ATTRIBUTES – What people are; what people have 5. Include all necessary information 5. ASPIRATIONS – What people plan to do Tips on Wording the Questions Tips on Wording the Questions continued: continued: 6. Avoid questions that may be too 11. Avoid bias in questions precise 12. Avoid double barreled questions 7. Phrase personal or potentially incriminating questions in less 13. Make the response categories clear objectionable ways and logical 8. Avoid questions that are too 14. Use complete sentences demanding and time consuming 15. Plan ahead for analysis 9. Use mutually exclusive categories 10.Avoid making assumptions 6 Types of Questions: Types of Questions: cont’d 1. Open ended questions One response pick lists 2. Close ended questions with ordered Multiple response pick lists responses Narrative comments 3. Close ended questions with unordered Short answer response choices Yes/ No 4. Partially close ended questions Ranking Matrix Increase Rewards Formatting the questionnaire: Show positive regard 1. Begin with complete introduction Say “Thank You” 2. The first question should be easy, Ask for advice avoiding controversial topics Support group values Give tangible rewards 3. Address important topics early Make the questionnaire interesting 4. Arrange questions so that they flow Give social validation naturally Communicate scarcity of response 5. Try to use same type of question opportunities throughout a series of questions Formatting the questionnaire Formatting the questionnaire continued: continued: 6. A numbered response should mean 11. Questions and answers are easier to read if the same thing throughout the they flow vertically questionnaire 12. Give clear directions about how to answer 7. Print in an easy to read type face 13. Pre-code as many items as possible to help 8. Place demographic questions at the tabulate/ analyze end of the questionnaire 14. Use transitional statements to enhance 9. Avoid making respondents turn a page continuity in the middle of a question 15. The more “white space”, the better 10. Distinguish between instructions, questions, and answers 7 Pre-testing the Questionnaire Pre-testing the Questionnaire Must answer the following questions: Must answer the following questions: continued 1. Does each question measure what it is 5. Does the questionnaire create a positive intended to measure? impression? 2. Do respondents understand all of the 6. Are the answers respondents can choose words? from, correct? 3. Are questions interpreted similarly by all 7. Does any aspect of the questionnaire respondents? suggest bias on the part of the researcher? 4. Does each close-ended questions have an answer that applies to each respondent? Pre-testing the Questionnaire: 1. Ask colleagues to review the questionnaire Likert Scales: critically 2. Select people as similar to you respondents ISSUES: as possible to pretest Level of Measurement: Nominal, 3. Simulate the actual data collection procedure Ordinal, or Interval/Ratio 4. Obtain feedback about the form and How many points in the scale? content of the questionnaire Should there