Psychometric Theory
• ‘The character which shapes our conduct is Psychology 405: a definite and durable ‘something’, and Psychometric Theory therefore! ... it is reasonable to attempt to measure it. (Galton, 1884) William Revelle • “Whatever exists at all exists in some amount. To know it thoroughly involves knowing its Northwestern University quantity as well as its!quality” (E.L. Thorndike, 1918) Spring, 2003 http://pmc.psych.nwu.edu/revelle/syllabi/405.html
Psychometric Theory: Goals Psychometric Theory: Goals II 4. To instill an appreciation of and an interest 1. To acquire the fundamental vocabulary and in the principles and methods of logic of psychometric theory. psychometric theory. 2. To develop your capacity for critical 5. This course is not designed to make you judgment of the adequacy of measures into an accomplished psychometist (one purported to assess psychological constructs. who gives tests) nor is it designed to make 3. To acquaint you with some of the relevant you a skilled psychometrician (one who literature in personality assessment, constructs tests), nor will it give you "hands psychometric theory and practice, and on" experience with psychometric methods of observing and measuring affect, computer programs. behavior, cognition and motivation.
Psychometric Theory: Text and Syllabus Requirements • Objective Midterm exam • Nunnally, Jum & Bernstein, Ira (1994) • Objective Final exam Psychometric Theory New York: • Final paper applying principles from the McGraw Hill, 3rd ed.(required: available course to a problem of interest to you. at Norris) • Sporadic applied homework and data sets • Loehlin, John (1998) Latent Variable Models: an introduction to factor, path, and structural analysis . Hillsdale, N.J.: LEA. (recommended)
1 Syllabus: Psychometric Theory: A conceptual Syllabus Overview X1 Y1 X2 L1 Y2 I. Individual Differences and Experimental Psychology X3 L4 II. Models of measurement Y3 III. Test theory X4 Y4 A. Reliability X5 L2 B. Validity (predictive and construct) Y5 C. Structural Models X6 D. Test Construction Y6 X7 L5 IV. Assessment of traits Y7 X8 L3 V. Methods of observation of behavior Y8 X9
Constructs/Latent Variables Theory as organization of constructs
L1 L1
L4 L4
L2 L2
L5 L5
L3 L3
Theories as metaphors and analogies-1 Theory development and testing • Physics – Planetary motion • Theories as organizations of observables • Ptolemy • Constructs, latent variables and observables • Galileo • Einstein – Observables – Springs, pendulums, and electrical circuits • Multiple levels of description and abstraction – The Bohr atom • Multiple levels of inference about observables – Latent Variables • Biology • Latent variables as the common theme of a set of observables – Evolutionary theory • Central tendency across time, space, people, situations – Genetic transmission – Constructs as organizations of latent variables and observed variables
2 Theories as metaphors and analogies-2 Examples of psychological constructs and their operationalization as observables • Business competition and evolutionary theory • Anxiety – Business niche – Trait – State – Adaptation to change in niches • Love • Learning, memory, and cognitive psychology • Conformity – Telephone as an example of wiring of connections • Intelligence – Digital computer as information processor • Learning and memory – Parallel processes as distributed information processor – Procedural - memory for how – Episodic -- memory for what • Implicit • explicit
Observables/measured variables Models and theory X1 Y1 X2 Y2 • Formal models X3 – Mathematical models Y3 X4 – Dynamic models - simulations Y4 X5 • Conceptual models Y5 – As guides to new research X6 Y6 – As ways of telling a story X7 • Organizational devices Y7 X8 • Shared set of assumptions Y8 X9
Psychometric Theory: A conceptual Syllabus A Theory of Data: What can be measured X1 X1 Y1 X2 L1 What is measured? Y2 Individuals X3 L4 Y3 Objects X4 Y4 X5 L2 What kind of measures are taken? Y5 X6 proximity Y6 order X7 L5 Y7 Comparisons are made on: X8 L3 Y8 Single Dyads or Pairs of Dyads X9
3 Scaling: the mapping between observed and Variance, Covariance, and Correlation latent variables X1 X1 Simple correlation L1 X2 Simple regression X3 Y3 X4 Multiple correlation/regression X5 Y5 Observed X6 Y6 Variable Partial correlation X7 Y7 X8 Y8 Latent Variable X9
Classic Reliability Theory: How well do we Types of Validity: What are we measuring measure what ever we are measuring X1 X1 Face Y1 X2 L1 X2 Concurrent Y2 X3 X3 Predictive Y3 X4 Construct Y4 X5 L2 Y5 X6 Convergent Y6 X7 Discriminant L5 Y7 X8 L3 Y8 X9
Techniques of Data Reduction: Structural Equation Modeling: Combining Factors and Components Measurement and Structural Models X1 X1 Y1 Y1 X2 L1 X2 L1 Y2 Y2 X3 L4 X3 L4 Y3 Y3 X4 X4 Y4 Y4 X5 L2 X5 L2 Y5 Y5 X6 X6 Y6 Y6 X7 L5 X7 L5 Y7 Y7 X8 L3 X8 L3 Y8 Y8 X9 X9
4 Scale Construction: practical and theoretical Traits and States: What is measured? X1 X1 Y1 Y1 X2 L1 X2 BAS Y2 Y2 X3 L4 X3 Affect Y3 Y3 X4 X4 Y4 Y4 X5 L2 X5 BIS Y5 Y5 X6 X6 Y6 Y6 X7 L5 X7 Cog Y7 Y7 X8 L3 X8 IQ Y8 Y8 X9 X9
The data box: measurement across time, Psychometric Theory: A conceptual Syllabus situations, items, and people X1 Y1 X2 L1 Y2 P1 X3 L4 Y3 P2 X4 P3 Y4 P4 X5 L2 . Y5 . X6 Pi Y6 Pj X7 L5 Tn Y7 … … T2 X8 L3 Pn Y8 T1 X1 X2 … Xi Xj … Xn X9
Syllabus: Overview Two Disciplines of Psychological Research (Cronbach, 1957, 1975; Eysenck, 1966, 1997) I. Individual Differences and Experimental Psychology A. Two historic approaches to the study of psychology B=f(Personality) B=f(P*E) B=f(Environment) B. Individual differences and general laws C. The two disciplines reconsidered Darwin II. Models of measurement Galton Weber, Fechner A. Theory of Data B. Issues in scaling Binet, Terman Watson, Thorndike C. Variance, Covariance, and Correlation III. Test theory Allport, Burt Lewin Hull, Tolman A. Reliability B. Validity (predictive and construct) Cattell Atkinson, Spence, Skinner C. Structural Models D. Test Construction Eysenck IV. Assessment of traits Epstein Mischel V. Methods of observation of behavior
5 Two Disciplines of Psychological Research B=f(Person) B=f(Environment) Types of Relationships Method/ Correlational Experimental (Vale and Vale, 1969) Model Observational Causal Biological/field Physical/lab • Behavior = f(Situation) Statistics Variance Mean • Behavior = f1(Situation)+ f2(Personality) Dispersion Central Tendency • Behavior = f1(Situation)+f2(Personality)+ Correlation/ Covariance t-test, F test f3(Situation*Personality) Effects Individuals Situations • Behavior = f1(Situation * Personality) Individual Differences General Laws • Behavior = idiosyncratic B=f(P,E) Effect of individual in an environment Multivariate Experimental Psychology
Types of Relationships: Types of Relationships: Behavior = f(Situation) Behavior = f1(Situation)+f2(Person)
High ability Output Output Low ability Behavioral Behavioral
Environmental Input Environmental Input (income)
Neuronal excitation = f(light intensity) Probability of college = f1(income) + f2(ability)
Types of Relationships: Types of Relationships: Behavior = f1(Situation)+f2(Personality)+ f3(Situation*Personality) Behavior = f(Situation*Person)
Low Output Output High Behavioral Behavioral High Low
Environmental Input Avoidance = f1(shock intensity)+f2(anxiety) + f3(shock*anxiety) Environmental Input Reading = f1(sesame street) = f2(ability) + f3(ss * ability) Eating = f(preload * restraint) GRE = f(caffeine * impulsivity)
6 Types of Relationships: Persons, Situations, and Theory Behavior = f(Situation*Person) Observed relationship
Low High Performance
Output External stimulation-> Individual Difference General Law Behavioral Arousal
Environmental Input Performance External stimulation-> Arousal-> GRE = f(caffeine * impulsivity)
Theory and Theory Testing I: Theory and Theory Testing II: Theory Experimental manipulation
Construct 2 Construct 2 Construct 1 Construct 1 rc1c2
rmo
Fm
Manipulation 1 Observation 1
Theory and Theory Testing III: Theory and Theory Testing IV: Correlational inference Correlational inference
Construct X
? ? Construct 1 Construct 2 Construct 1 Construct 2 ? ?
rmo rmo
Observation 1 Observation 2 Observation 1 Observation 2
7 Theory and Theory Testing V: Alternative Explanations Individual differences and general laws
? Impulsivity Construct 1 Construct 2 Attention Reaction Time
Arousal GREs
Working Memory Memory Span Observation 1 Observation 2 Caffeine
Theory and Theory Testing VI: Psychometric Theory: A conceptual Syllabus Eliminate Alternative Explanations X1 Y1 X2 L1 Y2 X3 L4 Y3 Construct 1 Construct 2 X4 Y4 X5 L2 Y5 X6 Y6 X7 L5 Observation 1 Observation 2 Y7 X8 L3 Y8 X9
A Theory of Data: What can be measured X1 Coombs: A theory of Data What is measured? Individuals • O = {Stimulus Objects} S={Subjects}
Objects • O = {o1, o2, …, oi, …, on}
• S = {s1, s2, …, si, …, sm} What kind of measures are taken? • S x O = {(s1, o1 ) ,(si,oj), … , (sm, on)}
proximity • O x O = {(o1, o1 ) ,(oi,oj), … , (on, on)} order • Types of Comparisons:
Comparisons are made on: – Order si 8 Coombs typology of data Metric spaces and the axioms of Single Dyads Pairs of Dyads a distance measure Measurement (S*O) s Pairs – D(X,Y) + D(Y,Z) => D(X,Z) (triangle inequality) Individual differences in Multidimensional Scaling S * (O*O) * (O*O) Scaling of Stimuli (O*O) Thurstonian Scaling of Stimuli • Finding a distance metric for a set of stimuli • What is scale location of objects I and J on an attribute dimension D? – Sports teams (wins and losses) • Assume that object I has mean value mi with some – Severity of crimes (judgments of severity) variability. • Assume that object J has a mean value m – Quality of merchandise (judgments) j • Assume equal and normal variability (Thurston case 5) – Political orientations of judges (history of – Less restrictive assumptions are cases 1-4) decisions -- voting with or against majority) • Observe frequency of (oi Preferential Choice and Unfolding Thurstonian Scaling (S * O) * (S*O) Comparison of the distance of subject to an item versus another subject to another item: Frequency |si -oj | < |sk -ol | Do you like broccoli more than I like spinach? x1 x2 x3 x .5 .3 .1 Or more typically: do you like broccoli more than 1 you like spinach? x2 .7 .5 .3 x3 .9 .7 .5 Preferential choice Unfolding (S*O)*(S*O) Row > Column x1 x2 x3 9 Preferential Choice: I scales Preferential Choice: J scales • Question asked an individual: • Individual preferences can give information – Do you prefer object j to object k? about object to object distances that are true • Model of answer: for multiple people – Something is preferred to something else if if it “closer” in the attribute space or on a particular attribute dimension • Locate people in terms of their I scales – Individual has an “Ideal point” on the attribute. along a common J scale. – Objects have locations along the same attribute – |si -oj | < |si -ok | – The I scale is the individual’s rank ordering of preferences Preferential Choice: free choice Preferential Choice: forced choice • If you had complete freedom of choice, how 1. If you had complete freedom of choice, how many many children would you like to have? _X_ children would you like to have? _X_ 2. If you could not have X, would you rather have • If you could not have that many, what would X+1 or X-1 (Y). your second choice be? _Y_ 3. If could not have X or Y, would you rather have • Third choice? _Z_ (min(X,Y)-1) or max (X,Y)+1. (Z) 4. If you could have X, Y or Z, would you rather • Fourth choice? -W- have min(X,Y,Z)-1 or max (X,Y, Z)+1 • Fifth choice? _V_ 5. Repeat (4) until either 0 or 5 Preferential choice- underlying Alternative J scales model 100 90 • On a scale from 0 to 100, if 0 means having 80 0 children, and 100 means having 5 children, 70 please assign the relative location of 1, 2, 3, 60 Accelerating 50 deaccelerating and 4 children. equal interval scale value • On this same scale, please give your 40 preferences for having 0, 1, 2, 3, 4, or 5 30 children. 20 10 0 A B C D E F categorical choice 10 4 I scales from the accelerating J scale I scales from the deaccelerating J scale 100 100 90 90 80 80 70 70 60 60 50 50 40 40 30 30 20 20 10 10 0 0 A B C D E F A B C D E F I scales from two J scales 100 Unfolding of Preferences 90 80 • Consider the I scale 234105 70 • What information has this person given us? 60 • Unfold to give J scale 50 • Ideal point is closest to 2, furthest from 5. 40 • J scale of 30 • 0 1 2 3 4 5 20 • Critical information: 2|3 occurs after 1|4 10 0 A B C D E F Joint scales, Points and Midpoints I scales and midpoints example 1 • Preference Orders: Midpoints crossed • 0 1 2 3 4 5 (Individual Scales) • 0|1 0|2 0|3 0|4 0|5 0 1 2 3 4 1 0 2 3 4 01 • 1|2 1|3 1|4 1|5 1 2 0 3 4 01 02 1 2 3 0 4 01 02 03 • 2|3 2|4 2|5 1 2 3 4 0 01 02 03 04 2 1 3 4 0 01 02 03 04 12 • 3|4 3|5 2 3 1 4 0 01 02 03 04 12 13 2 3 4 1 0 01 02 03 04 12 13 14 • 4|5 3 2 4 1 0 01 02 03 04 12 13 14 23 3 4 2 1 0 01 02 03 04 12 13 14 23 24 4 3 2 1 0 01 02 03 04 12 13 14 23 24 34 11 Distance information from I scales and Midpoints: Example 2 midpoints • Preference Orders: Midpoints crossed (Individual Scales) • Consider: 0 1 2 3 4 1 0 2 3 4 01 • 1 1|4 4 1 2 0 3 4 01 02 2 1 0 3 4 01 02 12 • 2 2|3 3 vs 2 1 3 0 4 01 02 12 03 2 3 1 0 4 01 02 12 03 13 • 2 2|3 3 3 2 1 0 4 01 02 12 03 13 23 3 2 1 4 0 01 02 12 03 13 23 04 • Midpoint orders imply distance information 3 2 4 1 0 01 02 12 03 13 23 04 14 3 4 2 1 0 01 02 12 03 13 23 04 14 24 • If 2|3<1|4 then (12) < (34) 4 3 2 1 0 01 02 12 03 13 23 04 14 24 34 • If 2|3 >1|4 then (12)> (34) From midpoints to partial orders Measurement (S * O) • Data example 1 • Ordering of abilities: si Measurement: Objects and Subjects Latent and Observed Scores -- Performance as a function of Ability and Test Difficulty The problem of scale 1.0 Much of our research is concerned with 0.8 making inferences about latent (unobservable) scores based upon observed measures. 0.6 Easy Difficult Typically, the relationship between observed and latent scores is monotonic, but not 0.4 necessarily (and probably rarely) linear. This Performance leads to many problems of inference. The 0.2 following examples are abstracted from real studies. The names have been changed to 0.0 protect the guilty. -4 -2 0 2 4 Ability 12 College and Writing Quality of school affects writing Pretest Posttest Change Junior College 1 5 4 Non-selective university • A leading research team in motivational and educational 5 27 22 psychology was interested in the effect that different Selective university teaching techniques at various colleges and universities 27 73 45 have upon their students. They were particularly interested in the effect upon writing performance of From these data, the researchers concluded that the quality of attending a very selective university, a less selective university, or a two year junior college. A writing test was teaching at the very selective university was much better and given to the entering students at three institutions in the that the students there learned a great deal more. They Boston area. After one year, a similar writing test was proposed to study the techniques used there in order to apply given again. Although there was some attrition from each them to the other institutions. sample, the researchers report data only for those who finished one year. The pre and post test scores as well as Are their conclusions justified? Can you think of several the change scores were: reasons that their conclusions could be incorrect? School Quality and Mathematics College Quality and Mathematics • Another research team in motivational and educational Pretest Posttest Change psychology was interested in the effect that different Junior College teaching techniques at various colleges and universities 27 73 45 have upon their students. They were particularly Non-selective university 73 95 22 interested in the effect upon mathematics performance of attending a very selective university, a less selective Selective university 95 99 4 university, or a two year junior college. A math test was given to the entering students at three institutions in the Boston area. After one year, a similar math test was • From these data, the researchers concluded that the quality of given again. Although there was some attrition from each teaching at the very selective university was much worse and that sample, the researchers report data only for those who the students there learned a great deal less than the other finished one year. The pre and post test scores as well as universities. They proposed to study the techniques used at these the change scores were other institutions in order to apply them to the very selective university. • Are their conclusions justified? Can you think of several reasons that their conclusions could be incorrect? 13