Differences between ma il and tel eph one interviewing modes for collection of Kessler’s scalfle for N onspec ifiPific Psyc hlihological lDi Distress.
David Cantor Brett McBride Katherine Kaufmann
Westat
FedCasic, March 18, 2010 , d s with K6 affected? p stere i : n n i m diid d -a lf paper se telephone interviewer administered in-person interviewer administered • • • Modes examined •
resentatio Are there mode effects related to Kessler’s K6 scale for nonspecific psychological distress (NPD)? by population groups? differ Do effects How are relationshi
Issues discussed in this p • • • Self-Administered K-6
taken from website for National Co-Morbidity Study, April 26, 2009, at http://www.hcp.med.harvard.edu/ncs/ftpdir/k6/K6+self%20admin-3-05-%20FINAL.pdf K-6 is used on many surveys
In the US
National Health Interview Survey
National Survey of Drug Abuse and Health
Medical Panel Survey
National Co-Morbidity Study
Outside the US
Australia
Canada
Others…. bias
? desirability
social
to
approval subject
be
Depends on judgment of individual respondent Depends on the respondent’s need for social approval • • Impression vs self-deception (Paulhaus, 2003) It can be difficult to predict which items will be subject to social desirability bias
Question: Is K-6 subject to Question: Social Desirability Bias • • - 005) 2 , ,005) (Ravens
w eeu L items
(de n so r health
pe so (de in- e mental lik
y ll
s a r e n related ge
s i e n o h ep research l
eep o e sge e a y e Te Some exceptions where telephone has been found to exhibit less SD bias • • Self vs Interviewer-administered questionnaires (Tourangeau and Smith, 1996) Are there different effects across telephone and in- person surveys? Prior research related mental health items (Ravens Sieberer, et al, 2008)
Social Desirability Bias leads to mode effect • • • telephone
US and
the e mail in the US
+ for
18
items
Adults
6 items for mail and telephone
elephon :Adults18 : - K T : National Cancer Institute
s : Methods to get health information, opic Sponsor T cancer especially about Population Mail survey using national sample of addresses digit dial survey of households with a Random number landline telephone • • • • • Health Information National Trends Survey III Dual Frame, dual mode survey Compare K
Analytic Approach: Mail v • • •
Digit
civilian –
Random
onse Rate: 24% p s non-institutional with a landline telephone Dial Frame: Random Digit Res n of completes = 4081 Coverage . . . . Telephone e – civilian g
HINTS III Result Frame: USPS addresses Response rate: 31% n of completes = 3582 Covera non-institutional Mail Component . . . . s
ustment j ustments for young adults, males, low education healthy adults age, race, education, income cancer status and health insurance j • • • • Both mail and RDD under-represent particular groups (Han and Cantor, 2007) Weights computed for each mode with ad
HINTS III Ad • • Comparison of Mail vs Telephone: Feel restless or fidgety?
40 36.9
35 32.5
30 28.5 29 27.3 26.3 25 Mail 20 Telephon e 15
10 8.5 4. 2 4. 1 5 2.7 0 All Most Some A littl e Non e
All, Most, A little, None significant at p<.01 Comparison of Mail vs Telephone: Feel nervous?
45 40.9
40 36.4 35 31.1 30 26.2 27.1
25 21.2 Mail 20 TlTelep hhon e 15
10 7.9 434.3 5 2.4 2.5 0 All Most Some A little None
Most, Some, A little and None significant at p<.01 Comparison of Mail vs Telephone: Feel so sad that nothing could cheer you up?
70
60 57.2 52. 33 50
40 Mail 30 Telephon e 25.2 23.7
20 15.5 13.7
10 5.5 3.7 1.4 1.7 0 All Most Some A littl e Non e
Most, None significant at p<.05 Comparison of Mail vs Telephone: Feel everything was an effort?
50 45 44.4 40 38.9 35 30 29.9 Mail 25 22. 99 20.5 20 18.5 Telephone 15 8.5 10 6. 1 6. 1 5 4.2 0 All Most Some A littlelittle None
All, Most, Some, A little, None significant at p<.01 Comparison of Mail vs Telephone: Feel hopeless?
90 80 78
70 64.7 60 50 Mail 40 Telephon e 30 20 18.1 10. 44 11. 22 8.2 10 4.7 2.2 1.6 1.1 0 All Most Some A littl e Non e
Most, A little, None significant at p<.001 Comparison of Mail vs Telephone: Feel worthless?
90 81.1 80 70 71.5 60 50 Mail 40 Telephone 30
20 1338.8 8.5 7.4 8.6 10 4.2 2 1.2 1.7 0 All Most Some A littlelittle None
Most, A little, None significant at p<.01 Item Resppyonse Theory: Relationship Among the Items
Which emotions are reported most frequently?
Least frequent may be the most subject to social desirability
Least frequent indicative of most severe behavior (()?)
Are the scales similar with respect to frequency (IRT = item difficulty)? Relationship of Items by Mode: IRT “Difficulties” 1
0.8 0.6
0.4
0.2 Mail 0 Telephone ‐0.2 Restless Nervous Effort Sad Hopeless Worthless
‐0.4
‐0.6 ‐080.8 Relationship of Items by Mode: IRT “Difficulties” 1
0.8 0.6
0.4
0.2 Mail 0 Telephone ‐0.2 Restless Nervous Effort Sad Hopeless Worthless
‐0.4
‐0.6 ‐080.8 Item Resppyonse Theory: How do the scales work?
Does the interpretation of response categories differ by mode?
How do respondents use the 5-point scales for the items
EdEndpoi itnts vs middle of scal es
“None” vs other categories
IRT evaluates whether points on the scale are used in a consistent way across items with different levels of difficulty ICC Graph: “Sad” Question in Mail Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Sad” Question in Mail Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Sad” Question in RDD Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Sad” Question in RDD Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Hopeless” Question in Mail Mode 1 0.9 0.8 se nn 070.7 None 0.6 respo A little of 0.5 yy Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Hopeless” Question in RDD Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of ? 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345
et , (Kessler
SMI ) as
classifying
for
cutoff
is 0 = never….4=all of the time Scale ranges from 0 to 24
al 2003) Add up all items 13 is cutoff for classifying as SMI (Kessler et Mail is significantly higher than telephone on percent with SMI (9.1% vs 5.2%; p<.0001)
Scale Measuring Serious Mental Illness (SMI Distribution of SMI scale by mode of interview
16 14 12 10 Mail
% 8 Telephone 6 4 2 0 0 2 4 6 8 10 12 14 16 18 20 22 24 SMI scal e Distribution of SMI scale by mode of interview Cutoff for Serious Mental 16 Illness 14 12 10 Mail
% 8 Telephone 6 4 2 0 0 2 4 6 8 10 12 14 16 18 20 22 24 SMI scal e Differences by Demographics?
Is the relationship between SMI and demographic characteristics different by mode of interview?
Age
Education
Income
Race & Ethnicity % with Serious Mental Illness by Age and Mode of Interview
14
12
10
I 8 Mail MM Telephone 6
with S with 4 %%
2
0 18 to 24 25 to 29 30 to 44 45 to 5 55 to 64 65 to 74 75+ % with Serious Mental Illness by Education and Mode of Interview
20 18
16
14 12 I Mail
MM 10 Telephone 8 6 with S with 4 %%
2 0 Less than 12 yearsHigh school Some college College grad or
educ. grad/equivalent higher % with Serious Mental Illness by Income and Mode of Interview
25
20
15 I Mail
Telephone 10 with SM with % 5
0 Less than than $20,000 $20,000$20,000 to $74,999 to $74,999$75,000 or more or more Do Mode Effects Differ by Population Group?
Age Yes Gender No Educati on ? Income No Race No Hispanic No Does Relationship between SMI and Health Status Differ by Mode?
Self Health Assessment No BMI Yes NbNumber o ftiiitdf times visited No Provider Fear Illness No Smoking status No Ever had Cancer No % with Serious Mental Illness by BMI and Mode of Interview
12
10
8
I Mail
MM 6 Telephone
4 with S with %%
2
0 Underweight BMI NormalOverweight BMI BMIObese
2% . 52% 5 vs
? 7% . (2 7% one h ep l e t
III t l h
HINTS Includes the K-6 for sample of adults Interviewer administered Response Rate is approximately 70%
Person interviews an
- National Health Interview Survey (NHIS) lower estimates are significantly NHIS K-6 th HINTS estimated to have SMI) n
Are Mode Effects Different for I Distribution of SMI scale for NHIS and HINTS Telephone
60 50 40 NHIS
% 30 HINTS 20
10 0 0 2 4 6 8 10 12 14 16 18 20 22 24 SMI scale How comparable is NHIS to HINTS?
Big difference in response rates
Non-response error on HINTS overestimates health problems
This might lead to higher rates of SMI Different questionnaires
NHIS includes many measures of physical andtlhlthd mental health
Aldworth, et al (2005) found evidence of context effects for the K-6 items Medical Expenditure Panel Survey
Paper self-administered questionnaires to adults 18+ in sampled households
Response rate is approximately 60%
The SAQ includes questions on other health problems, including selected emotional issues . Comparing K-6 Scores
Medical National Health Expenditure Panel Interview Survey Survey Mode Paper SAQ Interviewer Response 70% 60% Rate Prior Health Yes Yes Questions? K-6 Score 5.2* 2.7
*Different from NHIS at p<.001 Comparison of MEPS vs NHIS: Feel so sad that nothing could cheer you up?
90 78 80 76.5
70 60 50 MEPS
NHIS 40 30 20 12.9 912.2 2 10 6.3 6.9 2.2 0.9 0.8 2 0
All Most Som e A little little None Comparison of MEPS vs NHIS: Feel nervous?
80 68.7 70
60 50 43.8 MEPS
40 34.4 NHIS
30 18.1 20 15.8
9.5 10 3.7 1.4 1.5 2.2 0
All Most Som e A little little None Comparison of MEPS vs NHIS: Feel restless or fidgety?
80 70.4 70
60
50 45.5 MEPS 40 31.1 NHIS
30
15.9 20 14.7
10.1 1
10 4.6 1.5 1.9 2.4 0
All Most Som e A little little None Comparison of MEPS vs NHIS: Feel hopeless?
100 89.2 90
80 72.3
70 60 MEPS 50 NHIS 40 30
20 15.1
7.2 10 5.1 2.9 3.7 1.1 0.8 1.1 0
All Most Som e A little little None Comparison of MEPS vs NHIS: Feel worthless?
100 91.5 90 79.3 80
70 60 MEPS 50 NHIS 40 30
20 10.3 10 5.2 2.5 2.9 3.8 1.4 0.8 0.8 0
All Most Som e A little little None Conclusions
Estimates of SMI are significantly higher for self- administered questionnaires Mail survey vs telephone In-person interview vs paper SAQ
For mail vs telephone: Mail survey response scales have better psychometric properties Conclusions – continued
Relationship between SMI and other variables are not dramatically different, except for: Younggper respondents rep ort disp ppyroportionately lower rates on telephone R’s with high BMI report disproportionately lower rates on the telephone Caveats
Comparisons do not entirely control for differential non-response and coverage: HINTS mail vs HINTS telephone Comparison HINTS to NHIS and MEPS
Some comparisons do not account for different questionnaires HINTS vs MEPS vs. NHIS Thank-you