Questionnaire Validity

Questionnaire validity

Joshua Naranjo

Stat 5630 Western Michigan University

Joshua Naranjo Questionnaire validity Questionnaire theory

Recall: Study context research aims design measurement/questionnaire (satisfaction, ease of use, religiosity) analysis

Statistical concepts in questionnaire theory reliability and validity intraclass correlation coeﬃcient Cronbach’s alpha factor analysis

Joshua Naranjo Questionnaire validity Questionnaire theory

Reliability and validity of measurements

Ex: ”Association between intelligence, communication skills, and eﬀective leadership” Validity – does an instrument measure what it claims to measure? Reliability – does it work consistently?

Joshua Naranjo Questionnaire validity Questionnaire theory

Types of validity: Construct validity – is it valid in theory? Face validity – does it measure what it aims to measure? Convergent validity – does the survey compare well with another measure of the same thing? Discriminant validity – can it tell diﬀerent groups apart?

(Ex: intelligence, communication skills, leadership)

Joshua Naranjo Questionnaire validity Questionnaire theory

Types of Reliability: Internal reliability – do respondents give consistent answers over similar questions? Test-retest reliability – do respondents give consistent answers over time? Inter-rater reliability – do raters give consistent answers for the same item?

Joshua Naranjo Questionnaire validity A: Which student is X and which is Y ?

Intraclass correlation coeﬃcient

Ex: “How much agreement is there between physicians reading CT scans for disease progression?”

CT Scan Physician 1 Physician 2 1 7 5 2 6 9 ::: ::: n 2 2

Q: Pearson correlation?

Joshua Naranjo Questionnaire validity Intraclass correlation coeﬃcient

Ex: “How much agreement is there between physicians reading CT scans for disease progression?”

CT Scan Physician 1 Physician 2 1 7 5 2 6 9 ::: ::: n 2 2

Q: Pearson correlation? A: Which student is X and which is Y ?

Joshua Naranjo Questionnaire validity Intraclass correlation coeﬃcient

ICC measures agreement between exchangeable variables (i.e. no inherent ordering). Used for test-retest reliability, and inter-rater reliability.

Let 1 X x = (x + x ) 2n i1 i2

1 nX X o s2 = (x − x)2 + (x − x)2 2n i1 i2 Then

1 X ICC = (x − x)(x − x) ns2 i1 i2

Joshua Naranjo Questionnaire validity Intraclass correlation coeﬃcient

For groups of three,

1 X x = (x + x + x ) 2n i1 i2 i3

1 nX X X o s2 = (x − x)2 + (x − x)2 + (x − x)2 2n i1 i2 i3 Then

1 X ICC = {(x − x)(x − x) + (x − x)(x − x) 3ns2 i1 i2 i1 i3

+(xi2 − x)(xi3 − x)}

Joshua Naranjo Questionnaire validity Cronbach alpha

Internal Consistency: Hearing aid survey

Suppose we want to measure how well the hearing aid “listens in conversation”

Q: Do the questions “measure the same thing”?

Joshua Naranjo Questionnaire validity Cronbach’s alpha:

Suppose that T = x1 + ··· + xk is the composite score for a factor or construct. Then

k P var(x ) α = 1 − i k − 1 var(T )

Joshua Naranjo Questionnaire validity Cronbach alpha

10 19.022 α = 10−1 1 − 40.6933 = 0.59172

Joshua Naranjo Questionnaire validity It may be easier to understand as follows:

k P var(x ) k var(T ) − P var(x ) α = 1 − i = i k − 1 var(T ) k − 1 var(T ) P k cov(xi , xj ) = i6=j k − 1 var(T )

So α is an average covariance between the xi s.

Joshua Naranjo Questionnaire validity Cronbach alpha

But this is still just a computing formula. In Cronbach Psychometrika, 1951, Cronbach proposes the statistic as a “....mean of all split-half coeﬃcients resulting from diﬀerent splittings of a test... therefore an estimate of the correlation between two random samples of items...”

Joshua Naranjo Questionnaire validity Examples

Published example:

Joshua Naranjo Questionnaire validity Factor Analysis

Let Yi be the response to the ith question, i = 1,..., 15. The idea behind factor analysis is that the 15 responses actually depend on only two or three common underlying unobservable factors, i.e.

Yi = µi + ai f1 + bi f2 +

History of factor analysis Charles Spearman (1904) two-factor theory of intelligence: general and speciﬁc Thurstone 7-factor: numerical, verbal comprehension, word ﬂuency, perceptual speed, memory, inductive reasoning, spatial ability Sternberg 3-factor: analytical (problem-solving), creative (new ideas), and practical (everyday logic)

Joshua Naranjo Questionnaire validity Factor Analysis

Back to article, let

        Y1 µ1 l11 l12 1  Y2   µ2   l21 l22  f1  2    =   +   +    :   :   ::  f2  :  Y15 µ15 l15,1 l15,2 15 or Y = µ + LF + where µ and L are constants, F and have mean 0   ψ1 0 ... 0 1 0  0 ψ2 ... 0  V (F) = , V () = Ψ =    . . .  0 1  . . .  0 0 . . . ψ15

Joshua Naranjo Questionnaire validity so that V (Y) = LL0 + Ψ Estimation: √ 15 2 0 X 0 ∼ X 0 p p λ1v1 ˆˆ0 S = λi vivi = λi vivi = λ1v1 λ2v2 √ 0 = LL λ2v2 i=1 i=1 where λ1 > λ2 > ··· > λ15. The estimated coeﬃcients of the underlying factors are derived from the eigenvalues and eigenvectors of S.

Joshua Naranjo Questionnaire validity Factor Analysis: Rotation

Let A be any 2 × 2 orthogonal matrix, i.e. AA0 = I. Then

Y = µ + LF + = µ + LAA0F + = µ + L∗F∗ +

so that there is an ambiguity to the factor analysis model. But this 0 ∗ ∗ ∗ 0 can be exploited, might as well choose A F = (f1 f2 ) so that the factors are more interpretable. In practice, this means we want to rotate so that each of the 15 questions will have high coeﬃcient in only one factor.

Joshua Naranjo Questionnaire validity Joshua Naranjo Questionnaire validity Factor Analysis: Explained variance

P15 0 ∼ P2 0 Recall that S = i=1 λi vivi = i=1 λi vivi

2 2 2 trace(S) = s1 + s2 + ··· s15 = λ1 + λ2 + ··· + λ15 The proportion of total variation explained by the ith factor is

λi λ1 + λ2 + ··· + λ15

Joshua Naranjo Questionnaire validity Joshua Naranjo Questionnaire validity