Confidential Report on the Part 1 Frcophth Examination October 2007

Examination Report

October 2016 Part 1 FRCOphth Examination

Contents:

Summary page 2

MCQ paper Content page 3 Statistics page 3 Distribution of scores page 3 Quality of questions page 4 Standard setting page 4 Comparison to previous papers page 5

CRQ paper Content page 6 Statistics page 6 Distribution of scores page 6 Marks for each question page 7 Standard setting page 8 Comparison to previous papers page 8

Overall results page 9

Comparison to previous Part 1 examinations page 10 Breakdown of results page 11

Appendix 1 (results by deanery) page 13

Rehna Khan FRCOphth

1 Summary

The 31st Part 1 FRCOphth examination took place on 10 October 2016. 194 candidates sat the examination, of whom 70 (36%) fulfilled the criteria required to pass the examination overall. This pass rate is lower than May 2016 (50%) and higher than January 2016 (34%). The usual pass rate is around 48%.

The pass rate for candidates in OST is 44% compared to a pass rate of 35% for non- trainees however this was not found to be statistically significant. One ST4 candidate sat the exam and passed. The number of foundation year trainees sitting the exam increased to 31 this time, thirteen were successful (42%).

Both examinations were reliable (0.9) and were highly correlated (0.83).

No questions were removed from the MCQ paper. The pass rate for the MCQ paper was 37% which is lower than previous sitting in May (58%) but similar to the January 2016 exam (44%).

One of the Optics questions (Q5) posed the greatest challenge to candidates relating to prism correction of a lens where the average mark was 2.3/10. Other questions in which candidates performed particularly poorly was the brainstem anatomy question and a question relating to ocular tumours. The pass rate for the CRQ was 30% which is a drop on the last two papers (May 16, 41% and Jan 16 32%).

MCQ examination

2 Content (Table 1)

Topic Jan May Oct Jan May Oct Jan May Oct 2014 2014 2014 2015 2015 2015 2016 2016 2016 Anatomy/embryology 24 24 24 24 24 24 24 24 24 Optics 24 23 24 24 24 24 23 24 24 Pathology 23 23 23 23 23 23 23 22 23 Pharmacology & 18 18 18 18 18 18 18 18 18 genetics Physiology 23 23 23 23 23 23 23 23 23 Miscellaneous & 8 8 8 8 8 8 8 8 8 investigations Total 120 119 120 120 120 120 119* 119* 120

MCQ paper statistics

Mean score 71.2 59.3% Median score 71 59% Standard deviation 14.04 11.7% Candidates 194 KR20: (measurement of reliability) 0.88 Standard error of measurement (SEM): 4.78 4% Range of marks 32 – 103 27% - 86% Pass mark derived from Standard Setting 72/120 60% Pass mark – 1 SEM 67/120 56% Pass rate 72/194 37%

Distribution of results (Table 2) Range of scores Distribution Number 1-30 0 31-40 //// 4 41-50 ///// ///// /// 13 51-60 ///// ///// ///// ///// /// 23 61-70 ///// ///// ///// ///// ///// ///// ///// //// 54 / ///// ///// //// 71-80 ///// ///// ///// ///// ///// ///// ///// //// 48 / /// 81-90 ///// ///// ///// ///// ///// //// ///// // 37 91-100 ///// ///// // 12 101-120 /// 3 Total 194 Quality of questions

3 The Speedwell data allows us to identify easy, moderate and difficult questions, and those, which are good, poor or perverse (negative) discriminators. Ideally all questions should be moderate and good.

Table 3 Negative Poor Good < 0 0- 0.249 >0.250 Number % Number % Number % Total % Difficult Facility<25% 3 3 5 4 1 1 9 8 Moderate Facility 25- 75% 2 2 30 3 56 47 88 52 Easy Facility >75% 0 0 14 12 9 8 23 20 Total 5 5 49 19 66 56 120

Standard setting

The pass mark for the paper was agreed using the Ebel method. (Table 4) Difficult Moderate Easy Total Essential 0 33 30 63 Important 4 35 14 53 Supplementary 1 1 2 4 Total 5 69 46 120

The Part 1 FRCOphth Sub-Committee considered the success of a minimally competent candidate in each category as below: (Table 5) Difficult Moderate Easy Essential 0.55 0.65 0.75 Important 0.45 0.5 0.55 Supplementary 0.25 0.25 0.25

(Table 6) Difficult Moderate Easy Total Essential 0 21.45 22.5 43.9 Important 1.8 17.5 7.7 27 Supplementary 0.25 0.25 0.5 1 TOTAL 2.05 39.2 30.7 71.9

The MCQ pass mark = 72/120 (60%)

Comparison of pass marks and rates for last 8 MCQ papers (table 7)

4 Jan May Oct Jan May Oct Jan May Oct 14 14 14 15 15 15 16 16 16 Candidates 77 119 232 89 114 188 107 123 194 Mean score 73 67 72 69 72 68 69 70 71 Reliability (KR 20) 0.9 0.9 0.9 0.86 0.89 0.85 0.9 0.9 0.88 SEM 4.7 4.8 4.8 4.9 4.7 4.9 4.9 4.9 4.8 Standard setting Ebel Ebel Ebel Ebel Ebel Ebel Ebel Ebel Ebel Pass mark 68 69 69 69 68 71 71 71 72 (57%) (58%) (58%) (58%) (57%) (60%) (59%) (60%) (60%) 33% discrimination Negative 10 4 5 9 3 3 3 6 5 Poor (0-0.249) 59 43 40 56 47 59 55 34 49 Good (>0.250) 51 72 75 55 70 58 62 79 66 Facility Difficult (<25%) 12 6 3 9 5 6 6 3 9 Moderate 70 82 94 91 90 90 91 90 88 Easy (>75%) 38 31 23 20 25 24 23 26 23 Questions 120 119 120 120 120 120 119 119 120 Pass number 55 54 144 53 73 79 47 71 72 (rate) (71%) (45%) (62%) (60%) (64%) (42%) (44%) (58%) (37%)

5 The CRQ paper

Question Subject Topic Sub- Data provided sections 1 Anatomy Cranial nerve location in 10 Diagram of brainstem relation to brainstem 2 Pathology Corneal ulcer and systemic 5 Culture plate complications 3 Pathology Ocular tumour 5 Photomicrograph and globe section 4 Optics* Optical aberration 4 None 5 Optics Prism correction of a lens 3** Data 6 Optics Ocular muscle balance 6 Picture of a Maddox rod and a second figure showing three options of a dot in relation to a horizontal or vertical line. 7 Optics Spectacle & Contact lens 5 Data prescription, transposition and spherical equivalence 8 Investigations Myasthenia Gravis 4 None investigations 9 Investigations Sarcoidosis 7 Blood results 10 Investigations Rectus muscle palsies 5 Hess chart 11 Investigations Ultrasound of the eye 6 A scan and B scan 12 Miscellaneous* Inherited eye disease 5 None * Candidates are expected to draw a diagram as part of the answer **One sub-section (worth 3 marks) was removed.

Statistics Mean score 59/117 50% Median score 59/117 50% Standard deviation 33.4 28.5% Candidates 194 Reliability: Cronbach alpha 0.93 Standard error of measurement (SEM): 4 3.4% Range of marks 14 - 95 12% - 81% Pass mark derived from Standard Setting 69/117 59% Pass mark – 1 SEM 65/117 56% Pass rate 59/194 30%

One subsection (worth 3 marks) of Question 5 was removed from the CRQ paper, which was therefore marked out of 117.

6 Distribution of scores (Table 9) Range of marks Distribution Number 0-30 ///// ///// / 11 31-40 ///// ///// ///// / 16 41-50 ///// ///// ///// ///// ///// ///// //// 34 51-60 ///// ///// ///// ///// ///// ///// ///// /// 41 // / 61-70 ///// ///// ///// ///// ///// ///// ///// // 37 71-80 ///// ///// ///// ///// ///// ///// ///// // 37 81-90 ///// ///// ///// / 16 91-100 // 2 101-117 0 Total 194

Two examiners marked each question in the CRQ papers and the average mark from each was used to produce the candidate mark. Each question has maximum possible 10 marks. Candidate performance was variable for each question, with mean, median, minimum and maximum scores (with standard deviations) of:

(Table 10) Results for each question Q Subject Mean Median Min Max SD BCM 1 Anatomy 2.9 2.9 0 10 2.29 6 2 Pathology 7.5 8 0 10 2.29 6 3 Pathology 3.9 4 0 9 2.18 6 4 Optics* 6.9 8 0 10 2.82 6.8 5 Optics 2.3 2 0 7 2.18 4 6 Optics 4.7 5 0 9 2.36 5.5 7 Optics 6.1 6 0 10 2.21 6 8 Investigations 4.1 4 0 10 2.23 6 9 Investigations 4.9 5 0 9 1.8 5 10 Investigations 4.3 4 0 10 2.76 5.5 11 Investigations 4.9 5 0 10 1.2 6 12 Miscellaneous* 6.2 6 0 10 2.23 6

Candidates performed badly in or were particularly ill prepared for questions 1 (cranial nerve anatomy in relation to the brainstem), 3 (pathology) and 5, (optics).

7 Standard setting

The borderline candidate method was used to identify the pass mark for the CRQ. The examiners who marked the CRQ paper were asked to allocate a mark according to the marking scheme provided and, in addition, class the candidate’s performance as a pass, fail or borderline. The sum of each median borderline mark was used to produce the pass mark:

(Table 11) Standard setting by each examiner Examiners A Examiners B Topic No. No. No. Median No. No. No. Median Fail Border Pass Border Fail Border Pass Border 1 Anatomy 153 32 9 6 157 24 13 6 2 Pathology 5 19 170 6 15 48 131 6 3 Pathology 125 40 29 6 121 42 31 6 4 Optics* 60 24 110 6.5 58 19 117 7 5 Optics 123 51 20 4 120 43 31 4 6 Optics 53 55 86 5 123 25 46 6 7 Optics 52 35 107 6 56 31 107 6 8 Investigations 135 44 15 6 121 46 27 6 9 Investigations 78 53 63 5 62 51 81 5 1 Investigations 0 101 46 47 5 97 43 54 6 1 Investigations 1 114 51 29 6 110 43 41 6 1 Miscellaneous* 2 67 43 84 6 67 33 94 6 Total 67.5 70

Comparison to previous years (Table 12) Jan 14 May Oct 14 Jan 15 May Oct 15 Jan 16 May Oct 16 14 15 16 Mean score 47% 53% 50% 58% 51% 48% 48% 51% 50% Median 47% 55% 52% 62% 52% 50% 50% 54% 50% score Reliability 0.9 0.93 0.94 0.92 0.93 0.94 0.94 0.94 0.93 SEM 4.2 4.9 4.3 4.6 4.6 4.3 3 4.5 4 Pass mark 57% 56% 57% 61% 54% 59% 54% 56% 59% Pass rate 25% 48% 38% 56% 49% 28% 32% 41% 30% Correlation 0.79 0.78 0.76 0.77 0.75 0.81 0.80 0.85 0.83 with MCQ

The correlation between the two parts of the examination is acceptable at 0.83

8 Overall Results

To pass the Part 1 FRCOphth examination candidates are required to:

1. Obtain a combined mark from both papers that equals or exceeds the combined pass marks obtained by the standard setting exercise explained above. 2. Obtain a mark in both papers that equals or exceeds the pass mark minus 1 standard error of measurement for each paper.

A candidate is therefore allowed to compensate a poor performance in one paper by a very good performance in the other paper. They cannot compensate for an extremely poor performance in one paper whatever the combined mark.

The minimum mark required in order to meet standard 1 above for this examination was 141/237 (59%). The minimum mark required in each paper (to meet standard 2 above) was 72/120 in the MCQ paper and 69/117 in the CRQ paper.

Seventy candidates (36%) gained a total mark that met standards 1 and 2 above. Two candidates achieved 141/237 or greater (143, 147) but failed to achieve 69/117 in the CRQ paper (63, 58)

70/194 (36%) candidates passed the examination.

Distribution of scores (Table 13) Range of Distribution Number marks <60 /// 3 61-70 // 2 71-80 ///// / 6 81-90 ///// //// 9 91-100 ///// //// 9 101-110 ///// ///// ///// // 17 111-120 ///// ///// ///// ///// ///// /// 28 121-130 ///// ///// ///// ///// ///// 25 131-140 ///// ///// ///// ///// /// 23 141-150 ///// ///// ///// /// 18 151-160 ///// ///// ///// ///// /// 23 161-170 ///// ///// ///// // 17 171-180 ///// / 6 181-190 ///// // 7 191-200 / 1 Total 194 Mean 130/237 (55%) Median 130/237 (55%) Range 49 –191 (21%-81%)

9 Comparison to previous years: (Table 14)

Examination Candidates Passed % Passed MCQ pass CRQ pass examination mark % mark % Oct 2006 33 3 9 58 62 Jan 2007 24 4 16 60 43 May 2007 32 5 15 50 64 Oct 2007 56 13 23 51 59 Jan 2008 73 27 37 56 55 May 2008 66 16 24 57 48 Oct 2008 88 45 51 58 51 Jan 2009 79 37 47 61 57 July 2009 49 33 67 63 58 Oct 2009 101 56 56 62 56 Jan 2010 50 20 40 63 58 May 2010 79 31 39 60 57 Oct 2010 89 34 38 61 54 Jan 2011 62 23 37 59 58 May 2011 95 47 49 54 57 Oct 2011 122 63 52 56 56 Jan 2012 66 20 33 57 54 May 2012 104 53 51 56 58 Oct 2012 150 84 56 56 54 Jan 2013 91 47 52 57 53 May 2013 102 54 53 58 58 Oct 2013 151 65 43 58 60 Jan 2014 77 23 30 57 57 May 2014 119 55 46 58 56 Oct 2014 232 102 44 58 57 Jan 2015 89 50 56 58 61 May 2015 114 62 54 57 54 Oct 2015 188 57 30 59 59 Jan 2016 107 36 34 59 54 May 2016 123 61 50 60 56 Oct 2016 194 70 36 60 59

Cumulative totals Sitting Candidates Number passed Pass rate (%) January 718 287 40 May 834 384 46 October 1371 606 44 All examinations 2923 1277 44

10 Breakdown of results by training number (%) (Table 15) Failed Passed Total In OST 15 12 (44%) 27 Not in OST 109 58 (35%) 167

Total 124 70 (36%) 194 These differences are not statistically significant. (0.46 Student T test)

Breakdown of results by deanery (Table 16) Deanery Failed Passed Total East Midlands 0 0 0 East of England 1 0 1 East of Scotland 0 0 0 KSS 1 0 1 London 2 2 4 Mersey 1 0 1 N Ireland 2 1 3 North of Scotland 0 2 2 North Western 0 0 0 Northern 0 1 1 Oxford 0 1 1 Peninsula 0 0 0 Severn 0 1 1 South East of Scotland 1 0 1 Wales 3 0 3 Wessex 0 0 0 West Midlands 3 1 4 West of Scotland 1 2 3 Yorkshire 0 1 1

Total 15 12 27

Breakdown of results by stage of training (Table 17) Stage Failed Passed Total Foundation 29 13 (42%) 31 ST1 6 6 (50%) 12 ST2 9 5 (36%) 14 ST3 0 0 0 ST4 0 1 (100%) 1 ST5 0 0 0 Total in OST 44 25 (36%) 69

11 Breakdown of results by number of attempts (Table 18) Attempts Failed Passed Total 1 (First) 78 51 129 2 24 11 35 3 10 5 15 4 8 2 12 5 3 1 4 >5 (6) 1 0 1 Any resit 48 19 67 Total 124 70 194

12 Appendix 1 Overall results for each deanery

Result data by deanery has been available since October 2010. The summary results for each deanery are listed below.

Deanery Total candidates passed Total candidates Pass rate % East of Scotland 8 8 100 Yorkshire 46 64 72 Severn 13 19 68 KSS 30 49 61 North Western 29 47 62 South East of 18 31 58 Scotland Oxford 16 26 62 London 108 197 55 East Midlands 35 65 54 Northern 28 53 53 West of Scotland 47 91 52 West Midlands 66 139 47 East of England 40 86 47 Wales 36 80 45 Wessex 42 93 45 North of Scotland 16 35 46 Peninsula 25 58 43 Mersey 39 94 41 N Ireland 25 68 37

Total 667 1303 51