Differences between ma il and tel eph one interviewing modes for collection of Kessler’s scalfle for Nonspec ifiPific Psyc hlihological lDi Distress. David Cantor Brett McBride Katherine Kaufmann Westat FedCasic, March 18, 2010 Issues discussed in this presentation • Are there mode effects related to Kessler’s K6 scale for nonspecific psychological distress (NPD)? • Modes examined: • paper self-adiidministered , • interviewer administered telephone • in-person interviewer administered • Do effects differ by population groups? • How are relationships with K6 affected? Self-Administered K-6 taken from website for National Co-Morbidity Study, April 26, 2009, at http://www.hcp.med.harvard.edu/ncs/ftpdir/k6/K6+self%20admin-3-05-%20FINAL.pdf K-6 is used on many surveys In the US National Health Interview Survey National Survey of Drug Abuse and Health Medical Panel Survey National Co-Morbidity Study Outside the US Australia Canada Others…. Question: Is K-6 subject to Social Desirability Bias? • Impression vs self-deception (Paulhaus, 2003) • It can be difficult to predict which items will be subject to social desirability bias • Depends on judgment of individual respondent • Depends on the respondent’s need for social approval Social Desirability Bias leads to mode effects • Self vs Interviewer-administered questionnaires (Tourangeau and Smith, 1996) • Are there different effects across telephone and in- person surveys? • Teeeplepho ne i sges gen er all y lik e in-per son (de Leeuw,,005) 2005) • Some exceptions where telephone has been found to exhibit less SD bias • Prior research related mental health items (Ravens- Sieberer, et al, 2008) Analytic Approach: Mail vs Telephone • Health Information National Trends Survey III • Sponsor: National Cancer Institute • Topic: Methods to get health information, especially about cancer • Population:Adults18: Adults 18+ in the US • Dual Frame, dual mode survey • Mail survey using national sample of addresses • Random digit dial survey of households with a landline telephone number • Compare K -6 items for mail and telephone HINTS III Results Mail Component Telephone . Frame: USPS . Frame: Random Digit addresses Dial . Response rate: 31% . Response Rate: 24% . n of completes = 3582 . n of completes = 4081 . Coverage – civilian . Coverage – civilian non-institutional non-institutional with a landline telephone HINTS III Adjustments • Both mail and RDD under-represent particular groups (Han and Cantor, 2007) • young adults, males, low education • healthy adults • Weights computed for each mode with adjustments for • age, race, education, income • cancer status and health insurance status Comparison of Mail vs Telephone: Feel restless or fidgety? 40 36.9 35 32.5 28.5 29 30 27.3 26.3 25 Mail 20 Telephone 15 10 8.5 4.2 4.1 5 2.7 0 All Most Some A little None All, Most, A little, None significant at p<.01 Comparison of Mail vs Telephone: Feel nervous? 45 40.9 40 36.4 35 31.1 30 26.2 27.1 25 21.2 Mail 20 TlTelep hone 15 10 7.9 434.3 5 2.4 2.5 0 All Most Some A little None Most, Some, A little and None significant at p<.01 Comparison of Mail vs Telephone: Feel so sad that nothing could cheer you up? 70 60 57.2 52. 3 50 40 Mail 30 Telephone 25.2 23.7 20 15.5 13.7 10 5.5 3.7 1.4 1.7 0 All Most Some A little None Most, None significant at p<.05 Comparison of Mail vs Telephone: Feel everything was an effort? 50 45 44.4 40 38.9 35 30 29.9 Mail 25 22. 9 20.5 Telephone 20 18.5 15 8.5 10 6.1 6.1 5 4.2 0 All Most Some A little None All, Most, Some, A little, None significant at p<.01 Comparison of Mail vs Telephone: Feel hopeless? 90 80 78 70 64.7 60 50 Mail 40 Telephone 30 20 18.1 10. 4 11. 2 8.2 10 4.7 2.2 1.6 1.1 0 All Most Some A little None Most, A little, None significant at p<.001 Comparison of Mail vs Telephone: Feel worthless? 90 81.1 80 71.5 70 60 50 Mail 40 Telephone 30 20 1338.8 8.5 7.4 8.6 10 4.2 2 1.2 1.7 0 All Most Some A little None Most, A little, None significant at p<.01 Item Resppyonse Theory: Relationship Among the Items Which emotions are reported most frequently? Least frequent may be the most subject to social desirability Least frequent indicative of most severe behavior (()?) Are the scales similar with respect to frequency (IRT = item difficulty)? Relationship of Items by Mode: IRT “Difficulties” 1 0.8 0.6 0.4 0.2 Mail 0 Telephone ‐0.2 Restless Nervous Effort Sad Hopeless Worthless ‐0.4 ‐0.6 ‐080.8 Relationship of Items by Mode: IRT “Difficulties” 1 0.8 0.6 0.4 0.2 Mail 0 Telephone ‐0.2 Restless Nervous Effort Sad Hopeless Worthless ‐0.4 ‐0.6 ‐080.8 Item Resppyonse Theory: How do the scales work? Does the interpretation of response categories differ by mode? How do respondents use the 5-point scales for the items EdEndpo itints vs middle of scal es “None” vs other categories IRT evaluates whether points on the scale are used in a consistent way across items with different levels of difficulty ICC Graph: “Sad” Question in Mail Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Sad” Question in Mail Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Sad” Question in RDD Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Sad” Question in RDD Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Hopeless” Question in Mail Mode 1 0.9 0.8 se nn 070.7 None 0.6 respo A little of 0.5 yy Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 ICC Graph: “Hopeless” Question in RDD Mode 1 0.9 0.8 se nn 070.7 0.6 None respo A little of ? 0.5 y Some 0.4 Most 0.3 All Probabilit 020.2 0.1 0 ‐5 ‐4 ‐3 ‐2 ‐1012345 Scale Measuring Serious Mental Illness (SMI) Add up all items 0 = never….4=all of the time Scale ranges from 0 to 24 13 is cutoff for classifying as SMI (Kessler, et al 2003) Mail is significantly higher than telephone on percent with SMI (9.1% vs 5.2%; p<.0001) Distribution of SMI scale by mode of interview 16 14 12 10 Mail % 8 Telephone 6 4 2 0 0 2 4 6 8 10 12 14 16 18 20 22 24 SMI scale Distribution of SMI scale by mode of interview Cutoff for Serious Mental 16 Illness 14 12 10 Mail % 8 Telephone 6 4 2 0 0 2 4 6 8 10 12 14 16 18 20 22 24 SMI scale Differences by Demographics? Is the relationship between SMI and demographic characteristics different by mode of interview? Age Education Income Race & Ethnicity % with Serious Mental Illness by Age and Mode of Interview 14 12 10 I 8 Mail MM Telephone 6 with S with 4 %% 2 0 18 to 24 25 to 29 30 to 44 45 to 5 55 to 64 65 to 74 75+ % with Serious Mental Illness by Education and Mode of Interview 20 18 16 14 12 I Mail MM 10 Telephone 8 6 with S with 4 %% 2 0 Less than 12 years High school Some college College grad or educ. grad/equivalent higher % with Serious Mental Illness by Income and Mode of Interview 25 20 15 I Mail Telephone 10 with SM with % 5 0 Less than $20,000 $20,000 to $74,999 $75,000 or more Do Mode Effects Differ by Population Group? Age Yes Gender No Educati on ? Income No Race No Hispanic No Does Relationship between SMI and Health Status Differ by Mode? Self Health Assessment No BMI Yes NbNumber o ftiiitdf times visited No Provider Fear Illness No Smoking status No Ever had Cancer No % with Serious Mental Illness by BMI and Mode of Interview 12 10 8 I Mail MM 6 Telephone 4 with S with %% 2 0 Underweight BMI Normal BMI Overweight BMI Obese Are Mode Effects Different for In-Person interviews? National Health Interview Survey (NHIS) Includes the K-6 for sample of adults Interviewer administered Response Rate is approximately 70% NHIS K-6 estimates are significantly lower than HINTS III te lep hone (2.7% vs 52%5.2% estimated to have SMI) Distribution of SMIscalefor NHIS andHINTS Telephone % 10 20 30 40 50 60 0 0 2 4 6 8 SMI scaleSMI 10 12 14 16 18 20 22 24 HINTS NHIS How comparable is NHIS to HINTS? Big difference in response rates Non-response error on HINTS overestimates health problems This might lead to higher rates of SMI Different questionnaires NHIS includes many measures of physical andtlhlthd mental health Aldworth, et al (2005) found evidence of context effects for the K-6 items Medical Expenditure Panel Survey Paper self-administered questionnaires to adults 18+ in sampled households Response rate is approximately 60% The SAQ includes questions on other health problems, including selected emotional issues .