Construct Validation 1 Construct Validation in Organizational
Total Page:16
File Type:pdf, Size:1020Kb

Load more
Recommended publications
-
Assessment Center Structure and Construct Validity: a New Hope
University of Central Florida STARS Electronic Theses and Dissertations, 2004-2019 2015 Assessment Center Structure and Construct Validity: A New Hope Christopher Wiese University of Central Florida Part of the Psychology Commons Find similar works at: https://stars.library.ucf.edu/etd University of Central Florida Libraries http://library.ucf.edu This Doctoral Dissertation (Open Access) is brought to you for free and open access by STARS. It has been accepted for inclusion in Electronic Theses and Dissertations, 2004-2019 by an authorized administrator of STARS. For more information, please contact [email protected]. STARS Citation Wiese, Christopher, "Assessment Center Structure and Construct Validity: A New Hope" (2015). Electronic Theses and Dissertations, 2004-2019. 733. https://stars.library.ucf.edu/etd/733 ASSESSMENT CENTER STRUCTURE AND CONSTRUCT VALIDITY: A NEW HOPE by CHRISTOPHER W. WIESE B.S., University of Central Florida, 2008 A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Psychology in the College of Sciences at the University of Central Florida Orlando, Florida Summer Term 2015 Major Professor: Kimberly Smith-Jentsch © 2015 Christopher Wiese ii ABSTRACT Assessment Centers (ACs) are a fantastic method to measure behavioral indicators of job performance in multiple diverse scenarios. Based upon a thorough job analysis, ACs have traditionally demonstrated very strong content and criterion-related validity. However, researchers have been puzzled for over three decades with the lack of evidence concerning construct validity. ACs are designed to measure critical job dimensions throughout multiple situational exercises. However, research has consistently revealed that different behavioral ratings within these scenarios are more strongly related to one another (exercise effects) than the same dimension rating across scenarios (dimension effects). -
Construct Validity and Reliability of the Work Environment Assessment Instrument WE-10
International Journal of Environmental Research and Public Health Article Construct Validity and Reliability of the Work Environment Assessment Instrument WE-10 Rudy de Barros Ahrens 1,*, Luciana da Silva Lirani 2 and Antonio Carlos de Francisco 3 1 Department of Business, Faculty Sagrada Família (FASF), Ponta Grossa, PR 84010-760, Brazil 2 Department of Health Sciences Center, State University Northern of Paraná (UENP), Jacarezinho, PR 86400-000, Brazil; [email protected] 3 Department of Industrial Engineering and Post-Graduation in Production Engineering, Federal University of Technology—Paraná (UTFPR), Ponta Grossa, PR 84017-220, Brazil; [email protected] * Correspondence: [email protected] Received: 1 September 2020; Accepted: 29 September 2020; Published: 9 October 2020 Abstract: The purpose of this study was to validate the construct and reliability of an instrument to assess the work environment as a single tool based on quality of life (QL), quality of work life (QWL), and organizational climate (OC). The methodology tested the construct validity through Exploratory Factor Analysis (EFA) and reliability through Cronbach’s alpha. The EFA returned a Kaiser–Meyer–Olkin (KMO) value of 0.917; which demonstrated that the data were adequate for the factor analysis; and a significant Bartlett’s test of sphericity (χ2 = 7465.349; Df = 1225; p 0.000). ≤ After the EFA; the varimax rotation method was employed for a factor through commonality analysis; reducing the 14 initial factors to 10. Only question 30 presented commonality lower than 0.5; and the other questions returned values higher than 0.5 in the commonality analysis. Regarding the reliability of the instrument; all of the questions presented reliability as the values varied between 0.953 and 0.956. -
Construct Validity in Psychological Tests
CONSTRUCT VALIDITY IN PSYCHOLOGICAL TF.STS - ----L. J. CRONBACH and P. E. MEEID..----- validity seems only to have stirred the muddy waters. Portions of the distinctions we shall discuss are implicit in Jenkins' paper, "Validity for 'What?" { 33), Gulliksen's "Intrinsic Validity" (27), Goo<lenough's dis tinction between tests as "signs" and "samples" (22), Cronbach's sepa· Construct Validity in Psychological Tests ration of "logical" and "empirical" validity ( 11 ), Guilford's "factorial validity" (25), and Mosier's papers on "face validity" and "validity gen eralization" ( 49, 50). Helen Peak ( 52) comes close to an explicit state ment of construct validity as we shall present it. VALIDATION of psychological tests has not yet been adequately concep· Four Types of Validation tua1ized, as the APA Committee on Psychological Tests learned when TI1e categories into which the Recommendations divide validity it undertook (1950-54) to specify what qualities should be investigated studies are: predictive validity, concurrent validity, content validity, and before a test is published. In order to make coherent recommendations constrnct validity. The first two of these may be considered together the Committee found it necessary to distinguish four types of validity, as criterion-oriented validation procedures. established by different types of research and requiring different interpre· TI1e pattern of a criterion-oriented study is familiar. The investigator tation. The chief innovation in the Committee's report was the term is primarily interested in some criterion which he wishes to predict. lie constmct validity.* This idea was first formulated hy a subcommittee administers the test, obtains an independent criterion measure on the {Meehl and R. -
Questionnaire Validity and Reliability
Questionnaire Validity and reliability Department of Social and Preventive Medicine, Faculty of Medicine Outlines Introduction What is validity and reliability? Types of validity and reliability. How do you measure them? Types of Sampling Methods Sample size calculation G-Power ( Power Analysis) Research • The systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions • In the broadest sense of the word, the research includes gathering of data in order to generate information and establish the facts for the advancement of knowledge. ● Step I: Define the research problem ● Step 2: Developing a research plan & research Design ● Step 3: Define the Variables & Instrument (validity & Reliability) ● Step 4: Sampling & Collecting data ● Step 5: Analysing data ● Step 6: Presenting the findings A questionnaire is • A technique for collecting data in which a respondent provides answers to a series of questions. • The vehicle used to pose the questions that the researcher wants respondents to answer. • The validity of the results depends on the quality of these instruments. • Good questionnaires are difficult to construct; • Bad questionnaires are difficult to analyze. •Identify the goal of your questionnaire •What kind of information do you want to gather with your questionnaire? • What is your main objective? • Is a questionnaire the best way to go about collecting this information? Ch 11 6 How To Obtain Valid Information • Ask purposeful questions • Ask concrete questions • Use time periods -
“Is the Test Valid?” Criterion-Related Validity– 3 Classic Types Criterion-Related Validity– 3 Classic Types
“Is the test valid?” Criterion-related Validity – 3 classic types • a “criterion” is the actual value you’d like to have but can’t… Jum Nunnally (one of the founders of modern psychometrics) • it hasn’t happened yet and you can’t wait until it does claimed this was “silly question”! The point wasn’t that tests shouldn’t be “valid” but that a test’s validity must be assessed • it’s happening now, but you can’t measure it “like you want” relative to… • It happened in the past and you didn’t measure it then • does test correlate with “criterion”? -- has three major types • the specific construct(s) it is intended to measure Measure the Predictive Administer test Criterion • the population for which it is intended (e.g., age, level) Validity • the application for which it is intended (e.g., for classifying Concurrent Measure the Criterion folks into categories vs. assigning them Validity Administer test quantitative values) Postdictive Measure the So, the real question is, “Is this test a valid measure of this Administer test Validity Criterion construct for this population for this application?” That question can be answered! Past Present Future The advantage of criterion-related validity is that it is a relatively Criterion-related Validity – 3 classic types simple statistically based type of validity! • does test correlate with “criterion”? -- has three major types • If the test has the desired correlation with the criterion, then • predictive -- test taken now predicts criterion assessed later you have sufficient evidence for criterion-related -
How to Create Scientifically Valid Social and Behavioral Measures on Gender Equality and Empowerment
April 2020 EMERGE MEASUREMENT GUIDELINES REPORT 2: How to Create Scientifically Valid Social and Behavioral Measures on Gender Equality and Empowerment BACKGROUND ON THE EMERGE PROJECT EMERGE (Evidence-based Measures of Empowerment for Research on Box 1. Defining Gender Equality and Gender Equality), created by the Center on Gender Equity and Health at UC San Diego, is a project focused on the quantitative measurement of gender Empowerment (GE/E) equality and empowerment (GE/E) to monitor and/or evaluate health and Gender equality is a form of social development programs, and national or subnational progress on UN equality in which one’s rights, Sustainable Development Goal (SDG) 5: Achieve Gender Equality and responsibilities, and opportunities are Empower All Girls. For the EMERGE project, we aim to identify, adapt, and not affected by gender considerations. develop reliable and valid quantitative social and behavioral measures of Gender empowerment is a type of social GE/E based on established principles and methodologies of measurement, empowerment geared at improving one’s autonomy and self-determination with a focus on 9 dimensions of GE/E- psychological, social, economic, legal, 1 political, health, household and intra-familial, environment and sustainability, within a particular culture or context. and time use/time poverty.1 Social and behavioral measures across these dimensions largely come from the fields of public health, psychology, economics, political science, and sociology. OBJECTIVE OF THIS REPORT The objective of this report is to provide guidance on the creation and psychometric testing of GE/E scales. There are many good GE/E measures, and we highly recommend using previously validated measures when possible. -
Validity and Reliability in Social Science Research
Education Research and Perspectives, Vol.38, No.1 Validity and Reliability in Social Science Research Ellen A. Drost California State University, Los Angeles Concepts of reliability and validity in social science research are introduced and major methods to assess reliability and validity reviewed with examples from the literature. The thrust of the paper is to provide novice researchers with an understanding of the general problem of validity in social science research and to acquaint them with approaches to developing strong support for the validity of their research. Introduction An important part of social science research is the quantification of human behaviour — that is, using measurement instruments to observe human behaviour. The measurement of human behaviour belongs to the widely accepted positivist view, or empirical- analytic approach, to discern reality (Smallbone & Quinton, 2004). Because most behavioural research takes place within this paradigm, measurement instruments must be valid and reliable. The objective of this paper is to provide insight into these two important concepts, and to introduce the major methods to assess validity and reliability as they relate to behavioural research. The paper has been written for the novice researcher in the social sciences. It presents a broad overview taken from traditional literature, not a critical account of the general problem of validity of research information. The paper is organised as follows. The first section presents what reliability of measurement means and the techniques most frequently used to estimate reliability. Three important questions researchers frequently ask about reliability are discussed: (1) what Address for correspondence: Dr. Ellen Drost, Department of Management, College of Business and Economics, California State University, Los Angeles, 5151 State University Drive, Los Angeles, CA 90032. -
Reliability & Validity
Another bit of jargon… Reliability & Validity In sampling… In measurement… • More is Better We are trying to represent a We are trying to represent a • Properties of a “good measure” population of individuals domain of behaviors – Standardization – Reliability – Validity • Reliability We select participants We select items – Inter-rater – Internal – External • Validity The resulting sample of The resulting scale/test of – Criterion-related participants is intended to items is intended to –Face – Content represent the population represent the domain – Construct For both “more is better” “more give greater representation” Whenever we’ve considered research designs and statistical conclusions, we’ve always been concerned with “sample size” We know that larger samples (more participants) leads to ... • more reliable estimates of mean and std, r, F & X2 • more reliable statistical conclusions • quantified as fewer Type I and II errors The same principle applies to scale construction - “more is better” • but now it applies to the number of items comprising the scale • more (good) items leads to a better scale… • more adequately represent the content/construct domain • provide a more consistent total score (respondent can change more items before total is changed much) Desirable Properties of Psychological Measures Reliability (Agreement or Consistency) Interpretability of Individual and Group Scores Inter-rater or Inter-observers reliability • do multiple observers/coders score an item the same way ? • critical whenever using subjective measures -
Validity of the Patient Health Questionnaire-9 in Assessing Depression Following Traumatic Brain Injury
J Head Trauma Rehabil Vol. 20, No. 6, pp. 501–511 c 2005 Lippincott Williams & Wilkins, Inc. Validity of the Patient Health Questionnaire-9 in Assessing Depression Following Traumatic Brain Injury Jesse R. Fann, MD, MPH; Charles H. Bombardier, PhD; Sureyya Dikmen, PhD; Peter Esselman, MD; Catherine A. Warms, PhD; Erika Pelzer, BS; Holly Rau, BS; Nancy Temkin, PhD Objective: To test the validity and reliability of the Patient Health Questionnaire-9 (PHQ-9) for diag- nosing major depressive disorder (MDD) among persons with traumatic brain injury (TBI). Design: Prospective cohort study. Setting: Level I trauma center. Participants: 135 adults within 1 year of complicated mild, moderate, or severe TBI. Main Outcome Measures: PHQ-9 Depression Scale, Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (SCID). Results: Using a screening criterion of at least 5 PHQ-9 symptoms present at least several days over the last 2 weeks (with one being depressed mood or anhedonia) maximizes sensitivity (0.93) and specificity (0.89) while providing a positive predictive value of 0.63 and a negative predictive value of 0.99 when compared to SCID diagnosis of MDD. Pearson’s correlation between the PHQ-9 scores and other depression measures was 0.90 with the Hopkins Symptom Checklist depression subscale and 0.78 with the Hamilton Rating Scale for Depression. Test-retest reliability of the PHQ-9 was r = 0.76 and κ = 0.46 when using the optimal screening method. Conclusions: The PHQ-9 is a valid and reliable screening tool for detecting MDD in persons with TBI. -
Construct Validity in Psychological Tests – the Case of Implicit Social Cognition
European Journal for Philosophy of Science (2020) 10:4 https://doi.org/10.1007/s13194-019-0270-8 PAPER IN THE PHILOSOPHY OF THE SCIENCES OF MIND AND BRAIN Construct validity in psychological tests – the case of implicit social cognition Uljana Feest1 Received: 17 February 2019 /Accepted: 27 November 2019/ # Springer Nature B.V. 2020 Abstract This paper looks at the question of what it means for a psychological test to have construct validity. I approach this topic by way of an analysis of recent debates about the measure- ment of implicit social cognition. After showing that there is little theoretical agreement about implicit social cognition, and that the predictive validity of implicit tests appears to be low, I turn to a debate about their construct validity. I show that there are two questions at stake: First, what level of detail and precision does a construct have to possess such that a test can in principle be valid relative to it? And second, what kind of evidence needs to be in place such that a test can be regarded as validated relative to a given construct? I argue that construct validity is not an all-or-nothing affair. It can come in degrees, because (a) both our constructs and our knowledge of the explanatory relation between constructs and data can vary in accuracy and level of detail, and (b) a test can fail to measure all of the features associated with a construct. I conclude by arguing in favor of greater philosoph- ical attention to processes of construct development. Keywords Implicit bias . -
Reliability and Validity of the MBTI® Instrument
Reliability and validity of the ® MBTI instrument Reliability and validity of the MBTI® instrument Reliability and validity of the MBTI® instrument When psychologists or practitioners evaluate a psychometric test or questionnaire, there are usually two main questions that they ask: “Is it reliable?” and “Is it valid?”. On both of these criteria, the Myers-Briggs Type Indicator (MBTI) performs well. Reputable psychometric tools have been developed through years of rigorous research by the test publisher, and OPP makes these research findings available via the MBTI Step I Manual and the MBTI Step II Manual, of which all practitioners are given a copy upon qualification. Major findings are also published in data supplements that can be downloaded free online from https://eu.themyersbriggs.com/en/Knowledge-centre/Practitioner-downloads. In addition, there are many articles by independent researchers in established journals. Interested parties can find hundreds of these on a free searchable database published by CAPT: Mary and Isabel’s Library Online (MILO), at www.capt.org/MILO. This document presents a few key examples. Reliability Reliability looks at whether a test or questionnaire gives consistent results, in particular investigating whether it is consistent over time (test–retest reliability), and whether the questions that measure each scale are consistent with each other (internal consistency reliability). By convention, a correlation of 0.7 is often taken as the minimum acceptable value for personality questionnaire scales. The following independent, peer-reviewed study confirms that the MBTI tool performs well on both of these measures: Capraro and Capraro, 2002: Myers-Briggs Type Indicator Score Reliability Across Studies: a Meta-Analytic Reliability Generalization Study Link to study online The MBTI tool was submitted to a descriptive reliability generalisation (RG) analysis to characterise the variability of measurement error in MBTI scores across administrations. -
Handout 5: Establishing the Validity of a Survey Instrument STAT 335 – Fall 2016
Handout 5: Establishing the Validity of a Survey Instrument STAT 335 – Fall 2016 In this handout, we will discuss different types of and methods for establishing validity. Recall that this concept was defined in Handout 3 as follows. Definition Validity – This is the extent to which survey questions measure what they are supposed to measure. In order for survey results to be useful, the survey must demonstrate validity. To better understand this concept, it may help to also consider the concept of operationalization. Wikipedia.org defines this as follows: For example, in the previous handout we considered measuring students’ interest in statistics using the SATS (Survey of Attitudes Toward Statistics). Note that the construct of Interest is a theoretical and rather vague concept – there is no clear or obvious way to measure this. So, the creators of this survey defined their own measure of Interest using these four questions: The resulting sub-score for Interest is their operationalization of this construct. Will the resulting score really measure Interest? Will the resulting scores for the other constructs on the SATS really measure what the researchers intend? These are the questions we seek to answer when establishing the construct validity of this survey. In the remainder of this handout, we will introduce various types of construct validity and briefly discuss how survey instruments are shown to be valid. 1 Handout 5: Establishing the Validity of a Survey Instrument STAT 335 – Fall 2016 TYPES OF CONSTRUCT VALIDITY When designing survey questionnaires, researchers may consider one or more of the following types of construct validity.