Questionnaire validity Joshua Naranjo Stat 5630 Western Michigan University Joshua Naranjo Questionnaire validity Questionnaire theory Recall: Study context research aims design measurement/questionnaire (satisfaction, ease of use, religiosity) analysis Statistical concepts in questionnaire theory reliability and validity intraclass correlation coefficient Cronbach's alpha factor analysis Joshua Naranjo Questionnaire validity Questionnaire theory Reliability and validity of measurements Ex: "Association between intelligence, communication skills, and effective leadership" Validity { does an instrument measure what it claims to measure? Reliability { does it work consistently? Joshua Naranjo Questionnaire validity Questionnaire theory Types of validity: Construct validity { is it valid in theory? Face validity { does it measure what it aims to measure? Convergent validity { does the survey compare well with another measure of the same thing? Discriminant validity { can it tell different groups apart? (Ex: intelligence, communication skills, leadership) Joshua Naranjo Questionnaire validity Questionnaire theory Types of Reliability: Internal reliability { do respondents give consistent answers over similar questions? Test-retest reliability { do respondents give consistent answers over time? Inter-rater reliability { do raters give consistent answers for the same item? Joshua Naranjo Questionnaire validity A: Which student is X and which is Y ? Intraclass correlation coefficient Ex: \How much agreement is there between physicians reading CT scans for disease progression?" CT Scan Physician 1 Physician 2 1 7 5 2 6 9 ::: ::: n 2 2 Q: Pearson correlation? Joshua Naranjo Questionnaire validity Intraclass correlation coefficient Ex: \How much agreement is there between physicians reading CT scans for disease progression?" CT Scan Physician 1 Physician 2 1 7 5 2 6 9 ::: ::: n 2 2 Q: Pearson correlation? A: Which student is X and which is Y ? Joshua Naranjo Questionnaire validity Intraclass correlation coefficient ICC measures agreement between exchangeable variables (i.e. no inherent ordering). Used for test-retest reliability, and inter-rater reliability. Let 1 X x = (x + x ) 2n i1 i2 1 nX X o s2 = (x − x)2 + (x − x)2 2n i1 i2 Then 1 X ICC = (x − x)(x − x) ns2 i1 i2 Joshua Naranjo Questionnaire validity Intraclass correlation coefficient For groups of three, 1 X x = (x + x + x ) 2n i1 i2 i3 1 nX X X o s2 = (x − x)2 + (x − x)2 + (x − x)2 2n i1 i2 i3 Then 1 X ICC = f(x − x)(x − x) + (x − x)(x − x) 3ns2 i1 i2 i1 i3 +(xi2 − x)(xi3 − x)g Joshua Naranjo Questionnaire validity Cronbach alpha Internal Consistency: Hearing aid survey Suppose we want to measure how well the hearing aid \listens in conversation" Q: Do the questions \measure the same thing"? Joshua Naranjo Questionnaire validity Cronbach's alpha: Suppose that T = x1 + ··· + xk is the composite score for a factor or construct. Then k P var(x ) α = 1 − i k − 1 var(T ) Joshua Naranjo Questionnaire validity Cronbach alpha 10 19:022 α = 10−1 1 − 40:6933 = 0:59172 Joshua Naranjo Questionnaire validity It may be easier to understand as follows: k P var(x ) k var(T ) − P var(x ) α = 1 − i = i k − 1 var(T ) k − 1 var(T ) P k cov(xi ; xj ) = i6=j k − 1 var(T ) So α is an average covariance between the xi s. Joshua Naranjo Questionnaire validity Cronbach alpha But this is still just a computing formula. In Cronbach Psychometrika, 1951, Cronbach proposes the statistic as a \....mean of all split-half coefficients resulting from different splittings of a test... therefore an estimate of the correlation between two random samples of items..." Joshua Naranjo Questionnaire validity Examples Published example: Joshua Naranjo Questionnaire validity Factor Analysis Joshua Naranjo Questionnaire validity Factor Analysis Joshua Naranjo Questionnaire validity Factor Analysis Joshua Naranjo Questionnaire validity Factor Analysis Let Yi be the response to the ith question, i = 1;:::; 15. The idea behind factor analysis is that the 15 responses actually depend on only two or three common underlying unobservable factors, i.e. Yi = µi + ai f1 + bi f2 + History of factor analysis Charles Spearman (1904) two-factor theory of intelligence: general and specific Thurstone 7-factor: numerical, verbal comprehension, word fluency, perceptual speed, memory, inductive reasoning, spatial ability Sternberg 3-factor: analytical (problem-solving), creative (new ideas), and practical (everyday logic) Joshua Naranjo Questionnaire validity Factor Analysis Back to article, let 0 1 0 1 0 1 0 1 Y1 µ1 l11 l12 1 B Y2 C B µ2 C B l21 l22 C f1 B 2 C B C = B C + B C + B C @ : A @ : A @ :: A f2 @ : A Y15 µ15 l15;1 l15;2 15 or Y = µ + LF + where µ and L are constants, F and have mean 0 0 1 1 0 ::: 0 1 0 B 0 2 ::: 0 C V (F) = ; V () = Ψ = B C B . C 0 1 @ . A 0 0 : : : 15 Joshua Naranjo Questionnaire validity so that V (Y) = LL0 + Ψ Estimation: p 15 2 0 X 0 ∼ X 0 p p λ1v1 ^^0 S = λi vivi = λi vivi = λ1v1 λ2v2 p 0 = LL λ2v2 i=1 i=1 where λ1 > λ2 > ··· > λ15. The estimated coefficients of the underlying factors are derived from the eigenvalues and eigenvectors of S. Joshua Naranjo Questionnaire validity Factor Analysis: Rotation Let A be any 2 × 2 orthogonal matrix, i.e. AA0 = I. Then Y = µ + LF + = µ + LAA0F + = µ + L∗F∗ + so that there is an ambiguity to the factor analysis model. But this 0 ∗ ∗ ∗ 0 can be exploited, might as well choose A F = (f1 f2 ) so that the factors are more interpretable. In practice, this means we want to rotate so that each of the 15 questions will have high coefficient in only one factor. Joshua Naranjo Questionnaire validity Joshua Naranjo Questionnaire validity Factor Analysis: Explained variance P15 0 ∼ P2 0 Recall that S = i=1 λi vivi = i=1 λi vivi 2 2 2 trace(S) = s1 + s2 + ··· s15 = λ1 + λ2 + ··· + λ15 The proportion of total variation explained by the ith factor is λi λ1 + λ2 + ··· + λ15 Joshua Naranjo Questionnaire validity Joshua Naranjo Questionnaire validity.