THE COMPARABILITY BETWEEN MODULAR AND NON-MODULAR EXAMINATIONS AT GCE ADVANCED LEVEL Elizabeth A Gray PhD INSTITUTE OF EDUCATION UNIVERSITY OF LONDON 1 ABSTRACT The prime concern of this thesis is the comparability of two types of assessment now prevalent in Advanced level GeE examinations. The more conventional linear scheme assesses all candidates terminally, and the only way to improve the grade awarded is to re-take the whole examination. In contrast, the relatively new modular schemes of assessment include testing opportunities throughout the course of study. This not only has formative effects but allows quantifiable improvements in syllabus results through the medium of the resit option. There are obvious differences between the two schemes, but this does not necessarily imply that they are not comparable in their grading standards. It is this standard which the thesis attempts to address by considering the different variabilities of each of the schemes, and how these might impinge upon the outcomes of the grading process as evidenced in the final grade distributions. A key issue is that of legitimate and illegitimate variabilities - the former perhaps allowing an improvement in performance while maintaining grading standards; the latter possibly affecting the grading standard because its effect was not fully taken into account in the awarding process. By looking at a linear and modular syllabus in mathematics, the differences between the two are investigated, and although not fully generalisable, it is clear that many of the worries which were advanced when modular schemes were first introduced are groundless. Most candidates are seen to use the testing flexibility to their advantage, but there is little evidence of over-testing. Perhaps the major finding is a negative one - that there is no clear evidence for any difference in grading standards between modular and linear schemes of assessment, although there are variabilities which go some way to explaining what appears to be enhanced performances by some of the weaker modular candidates. 2 INDEX List of tables 5 List of Figures 6 Acknowledgements 7 CHAPTER 1: The Hunt for Comparability 8 The Relevance of Examinations 9 Two Approaches to Comparability 13 The Problem 17 Methodology 19 Outline for Chapters 21 CHAPTER 2: A More Forcible Word - the Literature on Comparability 26 Definitions of Comparability 27 Comparability Methodologies 41 Factors Influencing Comparability 45 Aggregation and Awarding 47 Discussion 49 CHAPTER 3: It is Ages Ahead of the Fashion - The Modular Context 51 Some History - Pre 1951 52 The Control of Public Examinations 56 Reform 63 The Rise of Modular Examinations 65 Project Examinations - and Mathematics 67 Modular Curriculum and Modular Assessment.. 70 Discussion 74 CHAPTER 4: Enveloped in Absolute Mystery - The Search for Comparability 76 The Three Domains 79 Domain of Behaviour 82 Domain of Assessment 84 Domain of Measurement 95 Generalisability 98 Discussion 99 CHAPTER 5: A Perfect and Absolute Blank - Within Subject Comparability 101 Some Basic Data 105 Patterns of Behaviour within Modules 109 Relationship between Modules 122 Discussion 132 3 CHAPTER 6: Keeping One Principal Object in View - Multi-level Modelling 136 The Variance Components Univariate Model 139 Random Coefficients Multivariate Model 150 Discussion 153 CHAPTER 7: But Much Yet Remained to be Said - Question Paper Analysis 157 Methodology 161 Syllabus Content 162 The Question Papers 165 Question Performance 177 Discrimination and Facility 180 Reliability 188 Populations and Gender 190 Discussion 191 CHAPTER 8: If I Had but the Time - Longitudinal Analysis 195 The Data 197 The Model 199 The Results -1994 201 The Results -1995 216 Combined Data 221 Discussion 225 CHAPTER 9: On to the Last 228 Variability within Modular Schemes 228 Variability of Demand between Syllabuses 232 Variability of Performance 234 Rationale and Generalisability 236 Why Modular? 238 Validity 245 In the End 245 APPENDIX A - Anomalies, UMS and the Regression Allowance 248 APPENDIX B - Drawing the Line 254 APPENDIX C - Modular Syllabus 273 APPENDIX D - Linear Syllabus 290 APPENDIX E - Question Level Data 297 APPENDIX F - Modelling Likelihood Ratios 308 BIBLIOGRAPHy 309 GLOSSARY 317 4 List of Tables Table: 5.1 1994 Grade Distribution for Modular Mathematics 105 5.2 1994 Grade Distribution for Linear Mathematics 106 5.3 Grade Distribution for Single Subject Modular Candidates 109 5.4 Modules with Significantly Different Syllabus Grades 115 5.5 Resit Pattern for Each Module 117 5.6 Grade Distribution by Module Combination 122 5.7 Grade A Mean Marks for Two Module Combinations 123 5.8 Grade A Mean Marks for Two More Module Combinations 124 5.9 Correlation Coefficients between Modules 125 5.10 Mean UMS Differences between Modules 128 5.11 Weightings Achieved by Optional Modules 132 6.1 Module Data for Mathematics 137 7.1 Question Paper Format. 166 7.2 Syllabus Coverage for All Written Papers (Weighted Marks) 170 7.3 Difference in Syllabus Coverage 172 7.4 Comparison of Compulsory Questions 172 7.5 Percentages of Marks Required at Each Grade 176 7.6 Correlation Coefficients for Components 180 7.7 Correlation Coefficients for Modules 181 7.8 Summary of Component and Module Data at Question Level. 187 5 List of Figures Figure: 3.1 The Relationship between Boards, Schools and Universities 57 3.2 The Pace of Reform 64 5.1 Proportion Taking PM1, 2 & 3 by Examination Session 111 5.2 Proportion Taking M1, 2 & 3 by Examination Session 113 5.3 Proportion Taking S1, 2 & 3 by Examination Session 114 5.4 Cumulative Percentage of Modules Resat. 116 5.5 Gains on Resit for Modules 1 to 5 118 5.6 Gains on Resits for Modules 7 to 9 and 13 to 15 119 5.7 Indexed Entry by Gender to Optional Modules 121 5.8 Coefficient Alpha 130 5.9 Achieved Weightings of the First Three Compulsory Components 131 7.1 Percentage of Component Marks Required for Each Grade 175 7.2 Percentage of Module Marks Required for Each Grade - Compulsory Modules .......................... 175 7.3 Percentage of Module Marks Required for Each Grade - Optional Modules. 176 7.4 Facility and Discrimination Differences for Component 1 182 7.5 Facility and Discrimination Differences for Component 2 182 7.6 Facility and Discrimination Differences for Module 1 183 7.7 Facility and Discrimination Differences for Module 2 183 7.8 Facility and Discrimination Differences for Module 3 184 7.9 Facility and Discrimination Differences for Module 7 184 7.10 Facility and Discrimination Differences for Module 8 185 7.11 Facility and Discrimination Differences for Module 13 185 7.12 Facility and Discrimination Differences for Module 14 185 7.13 Summary of Facility and Discrimination Values for Components and Modules ..................................... 186 7.14 Reliability Coefficients for Each Component/Module 189 8.1 Predicted A level Grade from Simple Regression 202 8.2 Predicted A level Grade from Fixed Effects 204 8.3 Standardised Level 1 Residual Plot 205 8.4 Level 1 Variance 206 8.5 Level 2 Variance 208 8.6 Predicted Normalised A level Grade from Fixed Effects 212 8.7 Level 1 Variance 213 8.8 Standardised Level 1 Residuals 214 8.9 Level 2 Variance 214 8.9a Level 2 Variance as a Function of Predicted A level Grade 215 8.10 Results from Simple Regression Analysis 217 8.11 Predicted Normalised A level Grade from Fixed Effects 218 8.12 Standardised Level 1 Residuals 219 8.13 Level 1 Variance 220 8.14 Level 2 Variance 220 8.15 Fixed Effects from Combined Model. 222 8.16 Standardised Plot of Level 1 Residuals 223 6 ACKNOWLEDGEMENTS First of all I must acknowledge my debt to the Oxford and Cambridge Schools Examination Board, in the person of its Secretary Mr H.F. King, who was not only prepared to finance this Ph.D. thesis, but also allowed me to use the data on which this research is based: and to UCLES who took over responsibility for my fees when the two Boards merged. For a woman must have money and a room ofherown ifshe is to write Virginia Woolf 1928 I am also grateful to the Joint Forum for the GCSE and GCE who allowed me to use the data from the 16/18 database. I must also thank my supervisor, Professor Harvey Goldstein, without whose gentle persistence and expertise the completion of this thesis would not have been possible. To Dr Helen Patrick thanks are due not only for her encouragement, but also for reading parts of this thesis and offering intelligent and helpful suggestions. Also to Mr Alastair Pollitt who allowed me time to do my thinking and writing. Finally I have to thank my husband and children who let me get on with it, in a room of my own! To all these, and more, I give my thanks, but any mistakes within this thesis I, of course, acknowledge as entirely my own. 7 CHAPTER 1 The Hunt for Comparability At first glance, the GCE A level examination has remained essentially unchanged since it was introduced in 1951. It is still primarily a terminal examination (though this primacy is rapidly disappearing) which has the dual purpose of summarising attainment in specified subjects and judging the fitness of 17 and 18 year olds for higher education. Syllabuses have been regularly updated and are all intended to provide the basis for a two year course of study, post 0 level/GCSE. However, although it is seen as being the "gold standard" A level has had to undergo some re-thinking of late. The developments in GCSE and introduction of the National Curriculum have meant that preparation for A level courses has changed. For example, the double award science GCSE has led to some misgivings on behalf of candidates who wish to study a single science at A level, and this has led in tum to the introduction of "Science" A levels, often as part of a modular scheme.