Prediction of Study Success - Creation of Magic Zones
Total Page:16
File Type:pdf, Size:1020Kb
Prediction of Study Success - Creation of Magic Zones Voorspelling van Studiesucces - Creatie van Magische Zones (met een samenvatting in het Nederlands) ISBN: 978-90-393-7063-6 Print: ProefschriftMaken || www.proefschriftmaken.nl Design: Linda van der Linden © 2018 Linda van der Linden Prediction of Study Success - Creation of Magic Zones Voorspelling van Studiesucces - Creatie van Magische Zones (met een samenvatting in het Nederlands) Proefschrift ter verkrijging van de graad van doctor aan de Universiteit Utrecht op gezag van de rector magnificus, prof.dr. H.R.B.M. Kummeling, ingevolge het besluit van het college voor promoties in het openbaar te verdedigen op woensdag 19 december 2018 des middags te 12.45 uur door Linda van der Linden geboren op 31 augustus 1977 te IJsselstein Promotoren: Prof.dr. S. F. te Pas Prof.dr. L. Woertman Copromotor: Dr. M. J. van der Smagt Uit liefde voor het vraagteken Overview of abbreviations Different abbreviations are used for the same pre-programme predictors in different chapters. Also, ‘Y1’ - a measure of in-programme academic success - is used to abbreviate two different operationalisations of first year study success. NG-PSF is used differently in chapter 5 than in the preceding chapters. The overview below hopefully assists in keeping track of what is what across the chapters. Chapter 2 – Table 2 and 3 SSGPA = Average secondary school grade, unweighted average of mathematics, English, Dutch, social sciences and biology (if applicable) MS = Matching grade, unweighted average of essay and exam grades NCF = Non-cognitive factors (score on a ten-point scale) SSGPA/MS/NCF = unweighted average of SSGPA, MS and NCF Y1 = whether at least seven of the eight first year courses were completed successfully Chapter 3 – Table 1 and Table A1 (in the Appendix), Figure 2 ACS = MS = Matching grade NCS = NCF = Non-cognitive factors G = SSGPA = Average secondary school grade A = ACS = MS = Matching grade N = NCS = NCF = Non-cognitive factors GAN = SSGPA/ACS/NCS Chapter 4 - Table 2 S = SSGPA = G = Average secondary school grade A = ACS = MS = Matching grade NG-PSF = NCS = NCF = Non-cognitive factors SAN = unweighted average of S, A and NG-PSF = GAN = SSGPA/MS/NCF Y1 = mean grade of all course grades in the first year (NOT equal to Y1 in Chapter 2) Chapter 5 NG-PSF = non-grade based psychosocial factors in general, NOT as operationalised in chapter 2-4 Content Chapter 1 General Introduction ........................................................................................ 2 Chapter 2 Signal Detection Theory as a Tool for Successful Student Selection ............. 10 Chapter 3 A Signal Detection Approach in a Multiple Cohort Study: Different Admission Tools Uniquely Select Different Successful Students ....................................... 34 Chapter 4 The Accuracy of Pre-Programme Predictors Across Time and Types of Academic Success ............................................................................................................... 54 Chapter 5 The (Im)Possibilities of Predicting Academic Achievement - Non-Grade Based Psycho-Social Factors in Admission: A Systematic and Integrative Review ........... 70 Chapter 6 General Discussion ....................................................................................... 100 Chapter 7 Nederlandse Samenvatting ........................................................................... 108 References ......................................................................................................................... 120 Dankwoord ........................................................................................................................ 132 Publications ....................................................................................................................... 134 Curriculum Vitae ............................................................................................................... 135 1 Chapter 1 General Introduction 2 INTRODUCTION Question “Will this person be successful?” An answer to this question requires a definition of success that is sufficient to allow evaluation of the available information on the probability of success. Of course, ‘being successful’ as a person is not binary, but university admission decisions are. What probability of success is required to decide to admit an applicant? In university admission, success is study success, which is commonly operationalised in Grade Point Average (GPA), course grades, retention and graduating in time. Valid instruments are available providing information on the probability of study success, but this information is incomplete, because the instruments measure only part of (the potential for) success. In addition, the predictive validity of the parts that can be quantified is not perfect. Yet, decisions need to be made to allow specific persons access, or not, to specific university programmes. The decision makers have imperfect information and thus will inevitably make mistakes. This dissertation provides an (not the) answer to the question: “How to minimise mistakes in selection?” Context: technicalities and responsibilities At first sight this appears to be a technical question and it is indeed very well possible to answer it in a technical way: determine the currently best possible way or ways to analyse the data and the outcome(s) will provide a current best practice. There are two types of mistakes in selection: admission of students that later drop out and rejection of students who would have become successful. The first mistake is one that becomes evident with time. The second mistake is one that will remain unknown to both institution and applicant. Given the one-sided decision of the institution to reject an applicant, it is also the one-sided responsibility of the institution to ensure rejection decisions are evidence-based. Here lies a responsibility that is not only technical. The decision to reject is uncertain in two, related, ways. Given the rejection itself, it remains forever uncertain whether it was a correct or incorrect decision and this uncertainty is due to the imperfect predictions of academic success. Given this uncertainty, what does it do to a young person who has decided to apply to certain programme at a certain institution when the institution says “No, you cannot enrol, because we think your probability of success is lower than that of others.”? This is not competition for a job in a commercial setting only one person can get. This is education. Of course, it might be that the available number of lecturers and teachers limits the number of students that can be taught or that the available facilities provide room for fewer students than apply. It might also be that the society only needs a certain number of academics with a certain specialty and that graduates will not be able to get a job if all who wish to enrol do so. Even then, it is the responsibility of the university to ensure selection decisions are made in the best possible, evidence-based, way. Not only because of the responsibility for implementing the best technical solution for the selection problem, but also because of the personal consequences for the individuals that are rejected. 3 INTRODUCTION Being rejected access to education for which you are qualified (in The Netherlands access to tertiary education is granted through secondary school qualifications), when you have only just gained the freedom and responsibility to decide on your education, touches upon the need to achieve and to belong and contradicts the recently acquired autonomy (Deci & Ryan, 2008). Compare this situation to a rejection after a job interview. However badly you want or need the job, it is clear and acceptable that the company will not pay two or more people a full-time salary for doing a one-person job. In education, it is harder to defend that no more than so many graduates will be ‘useful’ in a few years. If the limitation is in the available teachers or facilities and the institution admits only the applicants with the highest probability of success, this might be less difficult to accept for the rejected applicants, who are also willing to invest time, energy and money in their education, just as the admitted applicants. Yet, the acceptability of quality assurance given limited resources (people and buildings) does not change the practical and cognitive-emotional consequences for the individual being rejected. If the institution could be sure the probability of success is so low for certain individuals with certain characteristics that it would be wiser for them to change plans than it is to admit and enrol, this would be acceptable. This brings us back to the prediction of success. How well can study success be predicted among qualified applicants? Previously on the prediction of study success A century of investigation into selection of students shows an expansion of the available instruments predicting academic success, just as Bingham (1917) called for (Bingham, 1917; Cattell & Farrand, 1896; Patterson et al., 2016; Richardson, Abraham, & Bond, 2012; Schneider & Preckel, 2017). Bingham (1917) signalled three ways to progress: 1) to bring together and share the available information as yet scattered across labs, 2) to select a subset of instruments all institutions agree upon to use uniformly in order to allow comparison, and 3) to parcel out the most urgent and important research problems across labs. Then he writes: “The completion of such a program is not a matter