rms” Designing Definitions: Te me So – “ Effective ut do an Data H Instrumentation Instruments Quantitative and Qualitative Survey and a Thomas M. Archer, Ph.D. Reliability and Validity 2007

Data Collection Methods Qualitative Methods

Consider Major Types of Qualitative Methods: „ Using multiple measures to increase validity Observations „ Pairing qualitative and quantitative methods to In Depth Interviews obtain a more complete picture of the phenomenon of interest. Focus Group Interviews

„ Using both qualitative and quantitative methods in Documents and Records sequence so results of each method provides Nominal Group Technique information for the next. Key Informants Case Studies

Qualitative Methods Quantitative Methods

Other Types of Qualitative Data: Major Types of Quantitative Methods: Written Questionnaire with Open Ended Questions Mail Survey Expert Review Telephone Survey Community Forums/ Public Hearings Group Administered Survey Delphi Technique Web Based Survey

1 Surveys Advantages of Surveys:

Measures opinions, knowledge, Can complete anonymously attitudes, beliefs, behaviors, Inexpensive to administer reactions, and attributes in response Easy to compare and analyze to specific questions Administer to many people When need to quickly and/or easily get lots of information from people Can get lots of data in a non threatening way Many already exist

Disadvantages of Surveys: Advantages of Mail Surveys:

Might not get careful feedback Efficient for volume of information Wording can bias client's responses collected People more likely to provide frank, Impersonal thoughtful, honest info that is tension In surveys, may need expert free Doesn't get full story Gives people more time All respondents receive exactly the same questions in the same way

Advantages of Telephone Disadvantages of Mail Surveys: Surveys:

Low response rate Response rate generally high (IF, Must be simple & easy to understand actually talk to a person) Need accurate mailing lists Speed Mailing and copy expense Researcher can provide clarification on Privacy, confidentiality, and anonymity unclear questions must be assured More relaxed than with face to face Results may be misleading if do not follow-up with non respondents

2 Disadvantages of Telephone Advantages of Group Surveys: Administered Surveys:

Time consuming High response rate Need telephone numbers Easy to clarify items to all respondents Need trained interviewers Provides greatest sense of respondent Interviewer’s voice may bias anonymity Need simple and easy to understand Inexpensive questionnaire Overabundance of telemarketing

Disadvantages of Group Advantages of Web Based Administered Surveys: Surveys:

May require the cooperation of others; Nearly complete elimination of paper, i.e. to access groups postage, mail out, and data entry costs Time for implementation can be reduced Reach only those in attendance Once electronic system is Group dynamics may influence developed, cost of surveying additional individual responses respondents is much less Display of response data can be simultaneous Opportunity for researcher influence with completion of surveys Reminders and follow-up on non-respondents is relatively easy

Disadvantages of Web Based Surveys: Surveying Questions:

Not everyone is connected Who to survey? Not all potential respondents are What is the topic? equally computer literate What is the implementation plan? Sampling of e-mail addresses is difficult How many contacts to make? (no directories) Will contacts be personalized? The decision not to respond is likely to What is the interval between contacts be made more quickly How long is the questionnaire?

3 Survey Error

Sampling Error Errors in Surveying Coverage Error Measurement Error Non Response Error

Sampling Error Coverage Error

The result of surveying only some, and The result of not allowing all members of the not all, elements of the population survey population to have an equal or known non zero chance of being sampled for participation in the survey The extent to which the precision of sample survey estimates is limited by When the “list” from which the sample is the number of persons surveyed drawn does not include all elements of the population

Measurement Error Non Response Error

Respondent’s answer to a survey The result of people who respond to a question is inaccurate, imprecise, or survey being different from sampled cannot be compared in any useful way individuals who did not respond, in a way relevant to the study to other respondents’ answers

When a significant number of people in the Results from poor question wording sample do not respond AND have different and questionnaire construction characteristics from those who do respond.

4 Tailored Design Method

Survey Procedures that create Tailored Design Method respondent trust and perceptions of increased rewards and reduced costs for being a respondent, which take into account features of the survey Don Dillman situation and have as their goal the overall reduction of survey error.

Social Exchange Implementation Process

The likelihood of responding accurately Establish Trust is greater when the respondent trusts Increase Rewards that the expected rewards of Reduce Social Costs responding will outweigh the anticipated costs

Establish Trust Reduce Social Costs

Provide token of appreciation in Avoid subordinate language advance Avoid embarrassment Sponsorship by legitimate authority Avoid inconvenience Make the task appear important Make questionnaire short and easy Invoke other exchange relationships Minimize requests for personal information Emphasize similarity to other requests

5 Overview of Constructing a Tips for Questionnaire

Questionnaire Design 1. Write the purpose of the study 2. Make a list of what you want to know 3. Check to see if information is already “Asking Questions with a Purpose” available 4. Only ask questions that you will use 5. Consider how you will use each piece of information 6. View questions through the eyes of the respondent 7. Be selective and realistic

Kinds of Information that can be Tips on Wording the Questions: obtained through a Questionnaire:

1. Use simple wording 1. KNOWLEDGE – what people know; 2. Avoid using abbreviations, jargon, or how well they understand something foreign phrases 2. BELIEFS – ATTITUDES – OPINIONS 3. Be specific 3. BEHAVIOR – What people do 4. Use clear wording – do not be vague 4. ATTRIBUTES – What people are; what people have 5. Include all necessary information 5. ASPIRATIONS – What people plan to do

Tips on Wording the Questions Tips on Wording the Questions continued: continued: 6. Avoid questions that may be too 11. Avoid bias in questions precise 12. Avoid double barreled questions 7. Phrase personal or potentially incriminating questions in less 13. Make the response categories clear objectionable ways and logical 8. Avoid questions that are too 14. Use complete sentences demanding and time consuming 15. Plan ahead for analysis 9. Use mutually exclusive categories 10.Avoid making assumptions

6 Types of Questions: Types of Questions: cont’d

1. Open ended questions One response pick lists 2. Close ended questions with ordered Multiple response pick lists responses Narrative comments 3. Close ended questions with unordered Short answer response choices Yes/ No 4. Partially close ended questions Matrix

Increase Rewards Formatting the questionnaire:

Show positive regard 1. Begin with complete introduction Say “Thank You” 2. The first question should be easy, Ask for advice avoiding controversial topics Support group values Give tangible rewards 3. Address important topics early Make the questionnaire interesting 4. Arrange questions so that they flow Give social validation naturally Communicate scarcity of response 5. Try to use same type of question opportunities throughout a series of questions

Formatting the questionnaire Formatting the questionnaire continued: continued:

6. A numbered response should 11. Questions and answers are easier to read if the same thing throughout the they flow vertically questionnaire 12. Give clear directions about how to answer 7. Print in an easy to read type face 13. Pre-code as many items as possible to help 8. Place demographic questions at the tabulate/ analyze end of the questionnaire 14. Use transitional statements to enhance 9. Avoid making respondents turn a page continuity in the middle of a question 15. The more “white space”, the better 10. Distinguish between instructions, questions, and answers

7 Pre-testing the Questionnaire Pre-testing the Questionnaire Must answer the following questions: Must answer the following questions: continued 1. Does each question measure what it is 5. Does the questionnaire create a positive intended to measure? impression? 2. Do respondents understand all of the 6. Are the answers respondents can choose words? from, correct? 3. Are questions interpreted similarly by all 7. Does any aspect of the questionnaire respondents? suggest bias on the part of the researcher? 4. Does each close-ended questions have an answer that applies to each respondent?

Pre-testing the Questionnaire: 1. Ask colleagues to review the questionnaire Likert Scales: critically 2. Select people as similar to you respondents ISSUES: as possible to pretest Level of Measurement: Nominal, 3. Simulate the actual data collection procedure Ordinal, or Interval/Ratio 4. Obtain feedback about the form and How many points in the scale? content of the questionnaire Should there be a mid-point? 5. Assess whether the questions produce the information needed How many items? 6. Try the tabulation and analysis procedures Left to right or right to left? 7. Revise

Sampling: On-Line Instrumentation Probability Sampling

„ Simple Random Resources

„ Systematic Web Resources Available „ Stratified

„ Cluster (Handout) Purposeful Sampling Sample Size

8 Elements for Increasing Response Rate

Survey Implementation Respondent Friendly Questionnaire Number of Contacts Return Envelops Personalization Incentive

Respondent Friendly Constructing Mail Questionnaires

Questions that are clear and easy to Printed as a booklet comprehend No questions on front or back Good question ordering Pages are printed in photographically reduced form (79%) Good questionnaire layout Reproduced on white or off-white paper Lower case for questions; Upper case for Answers Vertical flow; White space

Contacts Envelop

Four Contacts by First Class Mail Use pre-addressed, stamped reply

„ Pre notice letter envelops „ Questionnaire with cover letter (invitation) Real stamp ~ goodwill gesture „ Thank you postcard Š A few days to a week after questionnaire

„ Replacement questionnaire Š 2-4 weeks after previous questionnaire

9 Personalization Token

Use real stationery on high quality Financial Incentives

paper „ Included with questionnaire Use real names instead of pre-printed Material Incentives

salutations „ Pens, gum, tickets „ Dear Resident Î Poor Lotteries, Prizes, etc. „ Dear Mr. Archer Î Good „ Poor alternative Real signatures

Design Guidelines for Web-Based Surveys

Web Based Surveys Utilize a multiple contact strategy Personalize contacts through e-mail if possible Do not require respondents to provide an answer to each question Make it possible for each question to be visible on the screen at one time Usually, 10-14 work days is sufficient from launch to close

Controlling for Non Response Error Ignore non respondents What are Your Compare respondents to the population Survey Issues? Compare respondents to non respondents ts ….. umen Instr Compare early to late respondents Some ritique Let’s C

10 When Extension Faculty Need to be Human Subjects Concerned about Human Subjects:

Research purposes Publish results in Professional Journal Office of Responsible Research Share results at National Meeting Practices Evaluation part of External Grant Protection of human subjects Data collection involves risk Exempt Sensitive topics „ No controversial topics „ No minors Subjects are vulnerable „ Adults who voluntarily participate

What happens if do not get IRB Approval for projects involving “Human Subjects”:

Harm people Questions ???? Jeopardize the reputation of University Disciplinary action against the person(s) conducting the project Sanctions that prohibit University from projects involving human subjects

11