sign in sign up
<<
Home , Survey methodology

Introduction to Methodology

Professor Ron Fricker Naval Postgraduate School Monterey, California

1 Goals for this Lecture

• Introduce professor and course • Define what we by the term “survey” – Characteristics of typical surveys – Distinction between “polls” and “surveys” • Lay out the basic steps in conducting a rigorous research survey • Discuss various survey modes – Pros and cons – Trade-offs • Considerations in making selection

2 Contact Information

• Professor Ron Fricker – Phone • Commercial: 831-656-3048 • DSN: 756-3048 • Fax: 831-656-2595 – E-mail: [email protected]

3 Professional Experience

• Academic credentials – Ph.D. and M.A. in , Yale University – M.S. in Ops Research, The George Washington University – B.S. in Mathematics from the United States Naval Academy • Teaching credentials – Started teaching post-graduate courses in mid-80s – Have taught at NPS, RAND Graduate School, and USC • “Real world” credentials – Former active duty naval officer – Senior specializing in survey and military research at the RAND Corporation • Can find out more at http://faculty.nps.edu/rdfricke/

4 Class is All About Collecting and Analyzing Survey

• In this class you’ll learn: – How to design good survey questions and craft an effective survey instrument – About the trade-offs between various survey modes – The rudiments of for surveys – Some statistical methods useful for analyzing survey data

5 Course Schedule

6 Course Materials

• Lecture notes: Binder with all slides • Textbook: Survey Methodology – Also excerpts from Mail and Internet Surveys and Sampling: Design and Analysis • Software: JMP version 7.0

¾ All on-line at http://faculty.nps.edu/rdfricke/MCOTEA_Short_Course.htm – Readings password protected: MCOTEA

7 What is a Survey?

•A survey is a: – systematic method for gathering information – from (a of) entities – for the purposes of constructing quantitative descriptors – of the attributes of the larger population of which the entities are members • In many ways, a survey is just a form of …

8 A Special Type of Data Collection

• Typically surveys: – Gather information by asking people questions – Collect information by either (1) interviewers asking questions and recording responses or (2) respondents reading and recording their own answers – Collect information from a subset of the population, a sample, rather than from all members

9 Polls vs. Surveys

• No clear distinction between the two terms – “Poll” often used for private sector opinion studies • Use many of the same design features as studies that would be called surveys – “Poll” rarely used to describe government or scientific surveys • To me, the term poll implies either – a commercial or less-scientific study, or – a quick turn-around survey whose results may be of short-term interest

10 Steps in Conducting a Survey (1)

• Clearly state research objective(s) • Decide on survey mode(s) – How will you contact potential respondents? •“Contact mode” – web, e-mail, phone, etc. – In what media will the survey be given? •“Survey or response mode” – web, e-mail, phone, etc. – How will you follow up with non-respondents? •“Follow-up mode” – web, e-mail, phone, etc. • Determine fielding strategy – How to maximize response rates? • Design the survey questions and the survey instrument

11 Steps in Conducting a Survey (2)

• Determine sample size and sampling strategy • Obtain Institutional Review Board (IRB) or other approval as necessary – Are respondents promised confidentiality? – What is the impact if their survey responses become known? • Pre-test, pre-test, pre-test – Give the survey to some test subjects and get their feedback • What works and what doesn’t? • Are you getting correct data/information? – Revise and re-pre-test as necessary • Draw sample and field the survey

12 Steps in Conducting a Survey (3)

• Follow up with non-respondents • Assemble the survey data –Cleandata as necessary • Analyze the data and report results – Summarize the data • Weight as necessary and appropriate to infer back to population • Calculate and report – Evidence of ? •Unitand item non-response • Communicate survey results to respondents if promised

13 One View of the Steps

14 Another View of the Steps

Excerpted from “What is a Survey?” by Fritz Scheuren 15 Examples of Large-scale Surveys (from your textbook)

• National Crime Victimization Survey • National Survey on Drug Use and Health • Survey of Consumers • National Assessment of Educational Progress • Behavioral Risk Factor Surveillance System • Current Employment Statistics Program

16 National Crime Victimization Survey (NCVS)

17 National Survey on Drug Use and Health (NSDUH)

18 Survey of Consumers (SOC)

19 National Assessment of Educational Progress (NAEP)

20 Behavioral Risk Factor Surveillance System (BRFSS)

21 Current Employment Statistics (CES) Program

22 Examples of DoD Surveys

• Annual Active Duty Personnel Survey • Survey of MWR Services and Family Programs • Military Recruiter Quality of Life Survey • Adult Influencer Poll (AIP) • Youth Attitude Tracking Study (now JAMRS Youth Poll)

23 Which Survey Mode Should I Use?

• You are going to conduct a research survey – Goal is to make the best inference possible from sample to population within budget constraints • To important questions: – What is the most appropriate method to choose for a particular research question? – What is the impact of choosing a particular method on survey cost and accuracy?

24 Traditional Survey Modes

•Mail – Paper sent to respondents – Self-administered and mailed back • Telephone – Interviewers call respondents on telephone – Interviewer-assisted • In-person – Interviewers go to respondent’s home or office – Interviewer-assisted

25 Options Have Multiplied in the Age of Computers and Telecom

CATI TDE IVR/T-ACASI Methods CAI CAPI

Text- Audio- Video- CASI CASI CASI

26 Mode Considerations

• Modes differ by: – Degree of interviewer involvement – Degree of with respondent – Degree of respondent privacy – Channels of communication – Technology usage • Considerations: –Cost – Timeliness – Accuracy 27 Broadly speaking…

• Interviewer-assisted surveys more costly than self- administered – Face-to-face most expensive, then telephone, then mail, then web • Interviewer-assisted surveys less prone to various errors and item nonresponse – Particularly compared to self-administered paper survey – More or less compared to self-administered computer- assisted surveys depending on sophistication of program • Self-administered less subject to sensitivity bias – Can combine some self-administered questions into a predominantly interviewer-assisted mode

28 However, timeliness is not just about…

• …the length of time a survey is in the field – Really, it’s from instrument design through analysis • Much of that is not affected by survey mode – Compared to a mail survey, web-based surveys save on two main things: • Mailing time to send out and return surveys • Time required to do data entry, coding, and cleaning for paper surveys • Yet, it can also take a lot of time to code, test, and de-bug very complicated web (more generally, computer-assisted) survey programs 29 And, cost is not just about…

• …manhours – On large mail surveys, printing and postage costs can be significant • 1,500 surveys x 4 mailings at $1.50 each = $9,000! • 1,500 $1 incentives = $1,500 – On telephone surveys must consider fixed costs of phone and CATI equipment + variable costs – On face-to-face interviews, travel costs can be significant (not to mention interviewer time) • But in terms of manhours, don’t forget: (1) instrument design time; (2) in non-electronic modes, data entry and coding time; (3) non-response follow-up efforts; and (4) analysis time 30 And accuracy is not just about…

• … response rates and missing items – : what mode or modes will most decrease all types of error? • Lots of considerations – Frame coverage – Nonresponse rates – Sensitivity bias, etc…

31 32 Cost and Frame Availability Often Drive Mode Choice

• Survey budgets never unlimited – Frequently cost will eliminate one or more modes • And likely drive other survey design decisions – I.e., face-to-face interviews too expensive • On the other hand, surveys using area probability frames usually have to start out using face-to-face interviews • Similarly, if a complete e-mail frame is available, web survey is possible option – But general population surveys eliminate web… 33 Some Modes Logically Group (1)

• Telephone and face-to-face surveys – Telephone often considered an alternative to face- to-face – Both interviewer-assisted and can have similar frame coverage issues – Many longitudinal surveys start out with a face-to- face with telephone follow-ups • Mail sometimes an alternative to telephone – If both telephone and addresses available

34 Some Modes Logically Group (2)

• Web surveys often considered a replacement for mail surveys – Coverage is still an issue for web surveys – Web superior in terms of automation and speed – However, literature consistently finds that nonresponse rates are higher for web surveys

35 Using Multiple Modes

• Reasons to use multiple modes: – Maximize response rates • Reduce/eliminate selection & coverage bias • Allow respondents to use most convenient mode – Reduce costs • Cross-sectional surveys: start with least costly mode and move to more expensive modes as necessary to get response • Longitudinal surveys: Use less expensive modes in follow-on survey rounds

36 Considerations When Using Multiple Modes

• Focus should be on designing instruments to ensure equivalency across modes – Rather than optimizing for a particular mode – E.g., make the follow-on web survey resemble its counterpart paper survey • Ensure operation is set up to track cases across the modes – Must avoid duplications and uncoordinated respondent contact • Must design and field so that mode effects can be disentangled from other sample characteristics – May have to randomize survey mode over a subset

37 Other Data Collection Methods

• Surveys are not the only (nor necessarily the best) way to collect data • Other methods include – Administrative records – Focus groups and qualitative investigations – Randomized • Which is “best” depends on the research question(s) and/or the purpose for which the data will be used

38 What We Have Covered

• Defined the term “survey,” including the – Characteristics of types of surveys this class will focus on – Distinction between “polls” and “surveys” • Discussed the basic steps in conducting a rigorous research survey • Considerations when choosing a survey mode – In various ways, choice likely to affect survey • Timeliness • Accuracy •Cost – And cost may drive mode choice possibilities

39