Validity: Theoretical Basis Importance of Validity

Home , Criterion validity, Validity (statistics)

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity Validity: Theoretical Basis Importance of Validity

Classic & Contemporary Approaches PSYC3302: Psychological Measurement and Its Trinitarian View Unitary View Applications Test Content Internal Structure Response Processes Mark Hurlstone Associations With Univeristy of Western Australia Other Variables Consequences of Testing

Other Week 5 Perspectives

Reliability & Validity

[email protected] Psychological Measurement Learning Objectives

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity • Introduction to the concept of validity Importance of Validity • Overview of the theoretical basis of validity: Classic & Contemporary 1 Trinitarian view of validity Approaches Trinitarian View 2 Unitary view of validity Unitary View Test Content • Alternative perspectives on validity Internal Structure Response Processes • Contrasting validity and reliability Associations With Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Everyday Usage

Psychological Measurement mark.hurlstone @uwa.edu.au • In everyday language, we say something is valid if it is Validity sound, meaningful, or supported by evidence Importance of Validity • e.g., we may speak of a valid theory, a valid argument,

Classic & or a valid reason Contemporary Approaches • In legal terminology, lawyers say something is valid if it is Trinitarian View Unitary View "executed with the proper formalities"—such as a valid Test Content Internal Structure contract or will Response Processes • Associations With In psychometrics, validity is a term used to refer to the Other Variables Consequences of meaningfulness of a test score—what the test score truly Testing means Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity • A basic definition of validity is "how well a test measures Importance of what it claims to measure" Validity Classic & • This definition is very common, but it is an oversimplification Contemporary Approaches • Trinitarian View A better definition is that validity is "the degree to which Unitary View evidence and theory support the interpretations of test Test Content Internal Structure scores entailed by the proposed uses" of a test Response Processes Associations With • This definition has at least four important implications Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Importance of Validity 1 Validity concerns interpretations and uses of scores Classic & 2 Validity is not a property of the test itself Contemporary Approaches Trinitarian View 3 Validity is a matter of degree Unitary View Test Content 4 Validity is based on theory and evidence Internal Structure Response Processes Associations With Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 1. Validity Concerns Interpretations and Uses of Scores

Psychological Measurement mark.hurlstone @uwa.edu.au • A measure itself is neither valid nor invalid Validity • The issue of validity concerns the interpretations and uses of Importance of Validity a measures scores Classic & Contemporary • For example, a persons’s Operation Span can be validly Approaches Trinitarian View interpreted as a measure of her or his working memory Unitary View Test Content • It would be less valid to interpret a persons’s Operation Span Internal Structure Response as a measure of her or his short-term memory Processes Associations With Other Variables • It would be totally invalid to interpret a person’s Operation Consequences of Testing Span as a measure of her or his self-esteem Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 2. Validity Is Not a Property of The Test Itself

Psychological Measurement mark.hurlstone @uwa.edu.au • As a short-hand, test users sometimes refer to a particular Validity test as a "valid test" Importance of Validity • For example, someone might say that the "Operation Span Classic & task is valid" Contemporary Approaches • Trinitarian View However, what is really meant is that the test has been Unitary View shown to be valid for a particular use, with a particular Test Content Internal Structure population of people, at a particular time Response Processes Associations With • Validity is not a property of the test itself Other Variables Consequences of Testing • It is a property of the interpretation and uses of test scores Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 2. Validity Is Not a Property of The Test Itself

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity No test is "universally valid" for all time, for all uses, with all

Importance of types of test-taker populations Validity • Rather, tests may be shown to be valid within reasonable Classic & Contemporary boundaries of a contemplated usage Approaches Trinitarian View • Unitary View If those boundaries are exceeded, the validity of the test may Test Content be called into question Internal Structure Response Processes • The validity of a test may have to be re-established with the Associations With Other Variables passage of time or changes in culture Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 3. Validity is A Matter of Degree

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity Like reliability, validity is not an all-or-nothing issue—it exists

Importance of on a continuum Validity • Validity is conceived in terms of strong versus weak, rather Classic & Contemporary than simply valid or invalid Approaches Trinitarian View • Unitary View For test users, validity should be a deciding factor in their Test Content choice of psychological test Internal Structure Response Processes • A test should be selected only if there is strong enough Associations With Other Variables evidence supporting its intended use and interpretation Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity in Psychometrics

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 4. Validity is Based on Theory and Evidence

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity is based on theory and empirical evidence Validity • It is not good enough to hear someone say that the test (or Importance of scores) are valid in someone’s experience Validity Classic & • There must be strong objective data supporting the Contemporary Approaches interpretation and use of a test Trinitarian View Unitary View • There are many popular published tests that have little to no Test Content Internal Structure validity e.g., Response Processes Associations With • hand writing analysis as an indicator of someone’s Other Variables Consequences of personality Testing • Other the "Color Quiz" as indicator of someone’s personality Perspectives

Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity is arguably the most important issue in evaluating a

Validity test’s psychometric quality Importance of • Validity Psychological measurement is only meaningful and useful if measurements have acceptable validity for their intended Classic & Contemporary purpose Approaches Trinitarian View • Unitary View Validity is a crucial basis for: Test Content Internal Structure 1 the meaningful interpretation of behavioural research Response Processes 2 Associations With making sound societal decisions based on such Other Variables Consequences of research Testing 3 making informed test-based decisions about individuals Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity is arguably the most important issue in evaluating a

Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity: 1. Interpreting Behavioural Research

Psychological Measurement mark.hurlstone • Test validity is essential to the meaningful interpretation of @uwa.edu.au behavioural research Validity • For example, suppose a social psychologist wants to know if Importance of exposure to violent video games increases a child’s Validity tendency to behave aggressively Classic & Contemporary Approaches • He measures children’s "inclination to behave aggressively" Trinitarian View Unitary View and the amount of hours spent playing violent video games, Test Content ﬁnding a modest positive correlation between the two Internal Structure Response measures Processes Associations With Other Variables • However, any conclusion that exposure to violent video Consequences of Testing games increases the tendency to behave aggressively Other requires that "inclination to behave aggressively" was Perspectives measured with good validity Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity is arguably the most important issue in evaluating a

Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity is arguably the most important issue in evaluating a

Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity: 2. Societal Decision Making

Psychological Measurement mark.hurlstone • Without good test validity, decisions about societal issues @uwa.edu.au could be misinformed, wasteful, or harmful Validity • For example, suppose based on empirical research showing Importance of Validity that exposure to violent video games increases aggressive Classic & behaviour, a decision is made to regulate the level of Contemporary Approaches violence depicted in video games Trinitarian View Unitary View • If the research is characterised by "good" test validity, then Test Content Internal Structure this is a legitimate decision with the potential to beneﬁt Response Processes society Associations With Other Variables Consequences of • However, if the research is characterised by "poor" test Testing validity, then such a course of action would be highly Other Perspectives questionable and potentially wasteful of time and money Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity is arguably the most important issue in evaluating a

Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity is arguably the most important issue in evaluating a

Reliability & Validity

[email protected] Psychological Measurement The Importance of Validity: 3. Test-based Decisions About Individuals

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity is necessary to make appropriate decisions about Validity individuals Importance of Validity • As we have discussed in previous lectures, scores on

Classic & psychological tests are used to make important and Contemporary sometimes life altering decisions Approaches Trinitarian View Unitary View • If those decisions are based on measures with sound validity Test Content Internal Structure they will hopefully beneﬁt test users and test takers Response Processes • Associations With If such decisions are based on poorly validated tests—or the Other Variables Consequences of inappropriate use of tests validated for a different Testing purpose—then test users and test takers may suffer harm Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Classic and Contemporary Approaches

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Importance of Validity • There are two prominent perspectives in psychology and Classic & education for conceiving validity: Contemporary Approaches Trinitarian View 1 Trinitarian view Unitary View Test Content 2 Unitary view Internal Structure Response Processes Associations With Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Classic and Contemporary Approaches

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement The Trinitarian View of Validity

Psychological Measurement • mark.hurlstone Traditionally, validity has been conceptualised in terms of @uwa.edu.au three different measures—the so-called trinitarian view

Validity (Guion, 1980): Importance of 1 Content validity: a measure based on an evaluation of Validity the content covered by items in a test Classic & Contemporary 2 Criterion-related validity: a measure obtained by Approaches Trinitarian View evaluating the relationship of scores obtained on the Unitary View Test Content test with scores on other tests Internal Structure 3 Construct validity: a measure obtained by performing Response Processes an analysis of: Associations With Other Variables Consequences of a how scores on the test relate to other test scores and Testing measures, and Other Perspectives b how scores on the test can be understood within some

Reliability & theoretical framework Validity

[email protected] Psychological Measurement The Trinitarian View of Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • In this view, construct validity is "umbrella validity" because Validity every other variety of validity falls under it Importance of Validity • Trinitarin approaches to validity assessment are not mutually Classic & exclusive Contemporary Approaches • Trinitarian View All the types of validity evidence contribute to a uniﬁed Unitary View picture of a test’s validity Test Content Internal Structure Response • A test user might not need to know about all three Processes Associations With Other Variables • Depending on the use to which a test is being put, one type Consequences of Testing of validity evidence may be more relevant than another Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement The Trinitarian View of Validity

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity • The trinitarian view of validity has been criticised Importance of • Validity Messick (1995) called the approach "fragmented" and

Classic & "incomplete" Contemporary Approaches • He called for a unitary view of validity, one that takes into Trinitarian View Unitary View account several different elements of validity Test Content Internal Structure • This includes a consideration of the implications of test Response Processes scores in terms of societal values and the consequences of Associations With Other Variables test use Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Classic and Contemporary Approaches

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Classic and Contemporary Approaches

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • The contemporary perspective on validity—known as the Validity unitary view—places construct validity as the essential Importance of Validity concept in validity (Messick, 1998)

Classic & Contemporary • As already noted, construct validity is the degree to which Approaches test scores can be interpreted as reﬂecting a particular Trinitarian View Unitary View construct Test Content Internal Structure Response • According to the unitary view, there are ﬁve types of Processes Associations With evidence relevant to establishing the construct validity of test Other Variables Consequences of score interpretations Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Unitary View

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Classic & Contemporary Approaches Construct Trinitarian View Validity Unitary View Test Content Internal Structure Response Consequences Response Processes Processes of Use Associations With Other Variables Consequences of Testing

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Unitary View

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Test Content

Psychological Measurement mark.hurlstone @uwa.edu.au • This is the match between the content of a test and the

Validity content that should be included in the test Importance of • Validity If a test is to be interpreted as a measure of a particular construct, then the content of the test should reﬂect the Classic & Contemporary important facets of that construct Approaches Trinitarian View • Unitary View The description of the nature of the construct should help Test Content deﬁne the appropriate content of the test Internal Structure Response Processes • There are two types of validity relevant to test content: Associations With Other Variables Consequences of 1 Content validity Testing 2 Face validity Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Test Content

Psychological Measurement mark.hurlstone @uwa.edu.au • This is the match between the content of a test and the

Reliability & Validity

[email protected] Psychological Measurement Test Content: 1. Content Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Content validity describes a judgement of how Validity representative a test’s content is of the full range of content Importance of that is relevant to the construct being measured Validity

Classic & • For example, the content covered by the construct Contemporary assertiveness is wide-ranging Approaches Trinitarian View Unitary View • A content-valid test of assertiveness would be one that Test Content Internal Structure contains items that are adequately representative of this Response Processes wide range Associations With Other Variables • Consequences of Such a test might include items sampling from hypothetical Testing situations at home, work, and in social situations Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Test Content: 1. Content Validity

Psychological Measurement mark.hurlstone • @uwa.edu.au In educational achievement tests, a test is content-valid when the proportion of materials covered approximates the Validity proportion of material given in the course Importance of Validity • A ﬁnal exam in introductory statistics would be content-valid Classic & if the proportion and type of introductory statistics problems Contemporary Approaches approximates that presented in the course Trinitarian View Unitary View • For an employment test to be content valid, its content must Test Content Internal Structure be representative of the job-related skills required Response Processes Associations With • This might be achieved by observing successful veterans on Other Variables Consequences of the job, noting the behaviours necessary for success, and Testing designing a test to include a representative sample of those Other Perspectives behaviours Reliability & Validity

[email protected] Psychological Measurement Test Content: 1. Content Validity

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity • There are two key threats to content validity: Importance of Validity 1 Construct-irrelevant content Classic & • Contemporary a test should not include content that is irrelevant to the Approaches construct being measured Trinitarian View Unitary View 2 Construct underrepresentation Test Content Internal Structure • a test should include the full range of content relevant to Response Processes the construct being measured, as much as possible Associations With Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Test Content: 1. Content Validity

Psychological Measurement mark.hurlstone @uwa.edu.au

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Test Content: 1. Content Validity

Psychological Measurement mark.hurlstone @uwa.edu.au

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Test Content: 1. Content Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • In practice, there is a trade-off between the ideal of content Validity validity and the reality of testing Importance of Validity • A test should include an "adequate" sample of construct

Classic & relevant content Contemporary Approaches • However, for practical reasons it might not be possible to Trinitarian View Unitary View include items assessing every aspect of the construct Test Content Internal Structure thoroughly Response Processes • Associations With Constraints on time, respondent fatigue, respondent Other Variables Consequences of attention, and so on, place constraints on the amount of Testing content included in a measure Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Test Content

Psychological Measurement mark.hurlstone @uwa.edu.au • This is the match between the content of a test and the

Reliability & Validity

[email protected] Psychological Measurement Test Content

Psychological Measurement mark.hurlstone @uwa.edu.au • This is the match between the content of a test and the

Reliability & Validity

[email protected] Psychological Measurement Test Content: 2. Face Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Face validity relates to what a test appears to measure to the person being tested, rather than what the test actually Validity measures Importance of Validity • If a test appears to measure what it claims to measure "on Classic & Contemporary the face of it", then it could be high in face validity Approaches Trinitarian View • A test labelled "The Introversion/Extraversion Test", with Unitary View Test Content items that ask people if they have responded in an Internal Structure Response introverted or extraverted way in different situations may Processes Associations With have high face validity Other Variables Consequences of Testing • A personality test in which respondents report what they see Other in inkblots may have low face validity Perspectives

Reliability & Validity

[email protected] Psychological Measurement Test Content: 2. Face Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Face validity is not typically considered important from a

Validity psychometrics perspective Importance of • Validity However, if a test lacks face validity, it could contribute to a

Classic & lack of conﬁdence in the perceived effectiveness of the test Contemporary Approaches • This may result in a decrease in the test taker’s cooperation Trinitarian View Unitary View or motivation to do his or her best Test Content Internal Structure • Lack of face validity might also lead to an unwillingness of Response Processes test users to employ a particular test Associations With Other Variables Consequences of • A test with high face validity might be better received by test Testing

Other takers and test users Perspectives

Reliability & Validity

[email protected] Psychological Measurement Test Content: Content Validity vs. Face Validity

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity • Content validity can be evaluated only by experts in a ﬁeld Importance of Validity • They need to understand the theoretical and empirical Classic & meaning of the psychological construct being assessed by a Contemporary Approaches test Trinitarian View Unitary View • By contrast, face validity must be assessable by non-experts Test Content Internal Structure Response • It is the respondents who are likely to take the test who must Processes Associations With be kept in mind when assessing face validity Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Unitary View

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Unitary View

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Internal Structure of a Test

Psychological Measurement mark.hurlstone @uwa.edu.au • A test’s internal structure is the way the parts of a test are related to each other: Validity

Importance of • some tests include items that are highly correlated with Validity each other, forming a single cluster Classic & Contemporary • other tests include items that fall into two or more Approaches clusters Trinitarian View Unitary View Test Content • The theoretical basis of a construct has implications for the Internal Structure Response internal structure of a measure of that construct Processes Associations With Other Variables • Factorial validity concerns the match between the actual Consequences of Testing internal structure of a test and the structure the test should Other possess Perspectives

Reliability & Validity

[email protected] Psychological Measurement Internal Structure of a Test

Psychological Measurement mark.hurlstone @uwa.edu.au • The Rosenberg Self-Esteem Inventory (RSEI; Rosenberg, 1989) is used to measure a single coherent theoretical Validity construct—namely, global self esteem Importance of Validity • The RSEI includes 10-items, such as "I take a positive Classic & Contemporary attitude toward myself" and "At times I think I am no good at Approaches all" Trinitarian View Unitary View Test Content • The RSEI should therefore have a speciﬁc internal structure Internal Structure Response amongst its 10-items Processes Associations With Other Variables • Since global self-esteem is single coherent theoretical Consequences of Testing construct, all items on the RSEI should correlate strongly Other with each other to form a single cluster Perspectives

Reliability & Validity

[email protected] Psychological Measurement Internal Structure of a Test

Psychological Measurement mark.hurlstone • By contrast, the Multidimensional Self-Esteem Inventory @uwa.edu.au (MSEI; O’Brien & Epstein, 1988) measures global

Validity self-esteem along with 8 components of self-esteem: Importance of • competence, likeability, loveability, personal power, self Validity control, moral self-approval, body appearance, and Classic & Contemporary body functioning Approaches Trinitarian View • If MSEI scores are validly interpreted as measures of these Unitary View Test Content components of self-esteem, responses to the test items Internal Structure Response should exhibit a structure consistent with the theoretical Processes Associations With deﬁnition of the construct Other Variables Consequences of Testing • Speciﬁcally, items should not form one tight cluster, they Other should (more or less) form one cluster for each of the Perspectives different components Reliability & Validity

[email protected] Psychological Measurement Internal Structure of a Test

Psychological Measurement mark.hurlstone @uwa.edu.au • Researchers use a statistical procedure known as factor analysis to evaluate the factorial validity (internal structure) Validity of the scores derived from a test Importance of Validity • Some items on a test might be more strongly correlated with Classic & Contemporary each other than with other items Approaches Trinitarian View • Items that are highly correlated with each other form clusters Unitary View Test Content of items—known as factors Internal Structure Response Processes Associations With Note: Other Variables Consequences of Testing • Next week’s lecture is devoted to a detailed examination of Other factor analysis Perspectives

Reliability & Validity

[email protected] Psychological Measurement Unitary View

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Unitary View

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Response Processes

Psychological Measurement mark.hurlstone @uwa.edu.au • Many psychological tests are based on assumptions about Validity the psychological processes that people use when Importance of Validity completing a measure

Classic & Contemporary • According to the third type of validity evidence, there should Approaches be a close match between the psychological processes that Trinitarian View Unitary View the respondents actually use when completing a measure, Test Content Internal Structure and the process that they should use Response Processes Associations With • You can’t just assume that people are going to do what you Other Variables Consequences of expect them to do Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Response Processes

Psychological Measurement mark.hurlstone • Suppose a researcher administers a test designed to elicit @uwa.edu.au students’ critical evaluative thinking of evidence-based Validity scientiﬁc arguments Importance of Validity • During the test, the students should be engaged in the Classic & cognitive process of examining argument claims and Contemporary Approaches evidence, and the relevance, accuracy, and sufﬁciency of Trinitarian View evidence Unitary View Test Content Internal Structure • To obtain evidence of validity based on response processes, Response Processes the researcher might use "think-aloud" procedures Associations With Other Variables Consequences of • If the think-alouds reveal evidence for the cognitive Testing processes presumed to underlie the task, we have evidence Other Perspectives of validity in terms of response processes Reliability & Validity

[email protected] Psychological Measurement Response Processes

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity • In addition to the think-aloud protocol, some further methods Importance of Validity for obtaining validity evidence of the response processes Classic & include: Contemporary Approaches • cognitive interviews Trinitarian View Unitary View • focus groups Test Content Internal Structure • response times Response Processes • eye movements Associations With Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Associations With Other Variables

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Associations With Other Variables

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Associations With Other Variables

Psychological Measurement mark.hurlstone @uwa.edu.au • This type of validity emphasises the theoretical Validity understanding of the construct we are trying to measure Importance of Validity • We must consider the way in which the construct is

Classic & connected to other relevant psychological variables Contemporary Approaches • Our theoretical understanding of the construct we are trying Trinitarian View Unitary View to measure should lead us to expect a particular pattern of Test Content Internal Structure associations with other variables Response Processes • Associations With This type of validity evidence emphasises the match Other Variables Consequences of between a measures predicted and observed associations Testing with other measures Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Associations With Other Variables

Psychological Measurement mark.hurlstone • For example, to interpret score on the RSEI as reﬂecting @uwa.edu.au global self-esteem, we must theorise about the nature of self Validity esteem Importance of Validity • We might expect self-esteem to be positively associated with Classic & happiness and social motivation, but negatively associated Contemporary Approaches with depression Trinitarian View Unitary View • Further, we might expect there to be no association between Test Content Internal Structure self-esteem and intelligence Response Processes Associations With • If RSEI scores can be validly interpreted as a measure of Other Variables Consequences of self esteem, then the actual associations between RSEI Testing scores and measures of these other constructs should Other Perspectives match the pattern predicted by the theory Reliability & Validity

[email protected] Psychological Measurement Associations With Other Variables

Psychological Measurement mark.hurlstone • Convergent evidence—also known as convergent @uwa.edu.au validity—is the degree to which test scores are correlated

Validity with tests of related constructs Importance of • Validity Suppose the RSEI is positively correlated with measures of

Classic & happiness and social motivation, but negatively correlated Contemporary with a measure of depression Approaches Trinitarian View • Unitary View Given this is what our theory of global self-esteem predicts, Test Content the pattern of associations provides convergent evidence for Internal Structure Response the RSEI as a measure of global self-esteem Processes Associations With Other Variables • Convergent evidence may come not only from correlations Consequences of Testing with tests claiming to measure related constructs but also Other from correlations with tests claiming to measure an identical Perspectives construct Reliability & Validity

[email protected] Psychological Measurement Associations With Other Variables

Psychological Measurement mark.hurlstone @uwa.edu.au • Discriminant evidence—also known as discriminant validity—is the degree to which test scores are uncorrelated Validity with tests of unrelated constructs Importance of Validity • Suppose the RSEI is uncorrelated (or only weakly Classic & Contemporary correlated) with various measures of intelligence Approaches Trinitarian View • Given this is what our theory of global self-esteem predicts, Unitary View Test Content the null associations provide divergent evidence of construct Internal Structure Response validity for the RSEI Processes Associations With Other Variables • By contrast, if we found RSEI scores are positively Consequences of Testing correlated with measures of intelligence, then the RSEI Other would lack discriminant validity Perspectives

Reliability & Validity

[email protected] Psychological Measurement Associations With Other Variables

Psychological Measurement mark.hurlstone • Another distinction relating to this type of evidence is @uwa.edu.au between concurrent validity evidence and predictive Validity validity evidence Importance of Validity • Concurrent validity evidence refers to the degree to which Classic & test scores are correlated with other relevant variables that Contemporary Approaches are measured at the same time as the test undergoing Trinitarian View validation Unitary View Test Content Internal Structure • For example, if we are trying to establish the validity of a new Response Processes intelligence test we might correlate it with a "benchmark Associations With Other Variables measure" of intelligence Consequences of Testing • Concurrent validity does not have to based on measures Other Perspectives administered precisely at the same time Reliability & Validity

[email protected] Psychological Measurement Associations With Other Variables

Psychological Measurement mark.hurlstone @uwa.edu.au • Predictive validity evidence refers to the degree to which

Validity scores on the test undergoing validation are correlated with

Importance of relevant variables that are measured at a future point in time Validity • A typical example concerns intelligence tests Classic & Contemporary Approaches • The validity of such tests is supported by the fact that they Trinitarian View Unitary View can predict performance in high school and at university Test Content even when administered between the ages of 5 and 11 Internal Structure Response Processes • Predictive validity evidence is very impressive Associations With Other Variables Consequences of • However, it is relatively rare because of the time and Testing resources required to keep track of people over time Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Consequences of Testing

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Consequences of Testing

Psychological Measurement mark.hurlstone @uwa.edu.au Internal Structure Associations With Validity Other Variables

Importance of Validity

Other Perspectives Test Content Reliability & Validity

[email protected] Psychological Measurement Consequences of Testing

Psychological Measurement mark.hurlstone @uwa.edu.au • One novel proposition in the unitary view of validity is the Validity concept of consequential validity Importance of Validity • It refers to the social and personal consequences associated Classic & with using a particular test Contemporary Approaches Trinitarian View • For example, suppose two tests are equally predictive of Unitary View Test Content some criterion measure, but one of the tests yields scores Internal Structure that are biased against women Response Processes Associations With • Other Variables We would consider the non-biased test to be associated with Consequences of Testing greater consequential validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Consequences of Testing

Psychological Measurement mark.hurlstone @uwa.edu.au • There is some debate about the importance of consequential validity Validity

Importance of • The debate is not about whether tests should not adversely Validity and unfairly affect some people Classic & Contemporary • The debate is about whether the consequences of testing Approaches Trinitarian View should be considered a part of the scientiﬁc evaluation of the Unitary View Test Content meaning of test scores Internal Structure Response • Processes Some people consider this a "dangerous intrusion of politics Associations With Other Variables into science" Consequences of Testing • Proponents of consequential validity would point out that you Other cannot divorce science from personal and social values Perspectives

Reliability & Validity

[email protected] Psychological Measurement Other Perspectives on Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • With the exception of consequential validity, the validity Validity discussed thus far has been framed within the context of Importance of Validity scores that are linked to a construct that has a clear Classic & theoretical basis Contemporary Approaches • There are three other types of validity that arguably do not ﬁt Trinitarian View Unitary View as strongly within this construct/theory framework: Test Content Internal Structure Response 1 Criterion Validity Processes Associations With 2 Induction-Construct Development Interplay Other Variables Consequences of 3 Measurement as Theory Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Other Perspectives on Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 1. Criterion Validity

Psychological Measurement mark.hurlstone • Criterion validity (mentioned earlier) is a judgement of how @uwa.edu.au adequately a test score can be used to infer an individual’s

Validity standing on some measure of interest—the criterion Importance of • Validity A criterion is the standard against which a test or test score

Classic & is evaluated Contemporary Approaches • For example, we might administer the Beck Depression Trinitarian View Unitary View Inventory to a population of outpatients to see if it can Test Content successfully differentiate patients with depression from those Internal Structure Response without depression (the criterion) Processes Associations With Other Variables • Concurrent validity and predictive validity (discussed earlier) Consequences of Testing are examples of criterion validity—they refer the extent to Other which test scores are related to, or predict, some criterion Perspectives measure Reliability & Validity

[email protected] Psychological Measurement 1. Criterion Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • According to the traditional perspective on criterion validity, Validity the psychological meaning of test scores is relatively Importance of Validity unimportant Classic & • All that matters is a test’s ability to differentiate groups or Contemporary Approaches predict some measure Trinitarian View Unitary View • Test Content From the unitary view, criterion validity on its own is not Internal Structure enough—the meaning of a measure must always be pursued Response Processes Associations With • Other Variables The unitary view suggests criterion validity evidence should Consequences of Testing be subsumed within the broader concept of construct validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Other Perspectives on Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Other Perspectives on Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 2. Induction-Construct Development Interplay

Psychological Measurement mark.hurlstone @uwa.edu.au • There are occasions where a measure is developed solely Validity from an inductive perspective Importance of Validity • For example, you might create a measure of personality by Classic & including all of the "person-descriptive" adjectives in the Contemporary Approaches dictionary (e.g., gregarious, moody, unpredictable) Trinitarian View Unitary View • Test Content People rate the degree to which all of the adjectives describe Internal Structure them Response Processes Associations With • Other Variables Then the researcher would factor analyse all of the Consequences of Testing responses to help uncover the common dimensions

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 2. Induction-Construct Development Interplay

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity • This is how the Five Factor model of personality was

Importance of discovered Validity • Classic & However, there was a lot of reﬁnement in the model along Contemporary the way Approaches Trinitarian View • Unitary View This purely inductive approach to test development led to Test Content some developments in theories of personality Internal Structure Response Processes • And the theory developments have in turn led to Associations With Other Variables measurement reﬁnements Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Other Perspectives on Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Other Perspectives on Validity

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement 3. Measurement As Theory

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity • This approach to validity emphasises the connection Importance of Validity between tests and psychological constructs Classic & • Constructs are a crucial part of validity and they should be Contemporary Approaches the guiding forces in test development and validation Trinitarian View Unitary View • Test Content This approach rejects much of the unitary view except the Internal Structure importance attached to constructs and the theoretically Response Processes based examination of response processes Associations With Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Contrasting Reliability and Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Reliability and validity are related but distinct psychometric Validity characteristics Importance of Validity • Reliability refers to the consistency of a measuring tool Classic & Contemporary • Reliability is the degree to which differences in test scores Approaches Trinitarian View reﬂect differences among people in their levels of the Unitary View Test Content construct that affects test scores, whatever that construct Internal Structure might be Response Processes Associations With • Other Variables We can discuss reliability without being aware of the Consequences of Testing construct being measured by a test

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement Contrasting Reliability and Validity

Psychological Measurement mark.hurlstone @uwa.edu.au • Validity, by contrast, is directly related to the nature of the construct supposedly being assessed by the measure Validity Importance of • validity is a property of test score interpretations Validity (whereas reliability is a property of test scores) Classic & Contemporary • validity is closely tied to psychological theory (whereas Approaches Trinitarian View reliability is not) Unitary View Test Content • Reliability is a necessary—but not sufﬁcient—condition for Internal Structure Response validity Processes Associations With Other Variables • The reverse is not true—a test might have excellent Consequences of Testing reliability, but we may still not interpret scores on the test in a Other valid manner Perspectives

Reliability & Validity

[email protected] Psychological Measurement References

Psychological Measurement mark.hurlstone @uwa.edu.au

Validity

Importance of Validity Classic & Furr, M. R., & Bacharach, V. R. (2014; Chapter 8). Contemporary Approaches Psychometrics: An Introduction (second edition). Sage. Trinitarian View Unitary View Test Content Internal Structure Response Processes Associations With Other Variables Consequences of Testing

Other Perspectives

Reliability & Validity

[email protected] Psychological Measurement