CONFERENCE REPORTS

Report of a conference on examination methods held on 14 and 15 November' 1968 at the Royal College ofPhysicians, London

THE PARTICIPANTS AT THIS CONFERENCE were not only faculty representatives or members of the College concerned with the development of an examination; half were from other Royal Colleges and their extensive experience of organizing examinations was freely given, greatly to our benefit. It was clear that the running of any examination is a serious and difficult undertaking, requiring study, effort, criticism and constant revision. The effort of this College in organizing this conference was impressive to all who participated. There is no longer any doubt that an examination in general practice is possible, but every method or combination of methods has limitations. Correlation between examination results and subsequent perform- ance as a clinician is in any case small. The multiple choice question method has been shown to be reliable; it can cover a wide range of subjects and is least subject to variation due to different examiners. It stood out in this conference as the best singel method available. It was actually sampled by all the 100 participants on the first day. The conference was opened by SIR , in double capacity, as president of the Royal College of and as chairman of the Education Foundation of the Royal Coilege of General Practitioners. The purpose of the conference was to decide, since proper vocational training and the passing of an examination are now necessary for membership of the Royal College of General Practitioners, what form the examination should take. What is to be tested? The speakers in the first session set out to define the ideal general practitioner, his know- ledge, skills and attitudes. DR ANDREW SMITH (Newcastle) gave a word portrait of two types of practitioner, both good, both with faults but greatly differing from each other. The confer- ence had to discover how qualities good and bad for general practice can be identified. DR IAN RICHARDSON (Aberdeen) described the essential skills under the headings, clinical (including skills of personal relationship), organizational, and epidemiological. There are distinctive skills for a general practitioner in those areas; they cannot yet be adequately defined and testing for them is still empirical, but not impossible. He quoted and commented on an examination question ideal for this purpose. DR PATRICK BYRNE (Manchester), discussed attitudes-"feelings for or against". Attitudes can be examined according to several criteria-their favourableness, their intensity, their salience, their consistency, whether they are public or private, individual or common to many. We must provide by our training a consistent and appropriate set of attitudes towards patients, colleagues and professional life. A core of basic attitudes can be stated and agreement is not too difficult to attain. But measurement of individual components of this core presents great problems. In the discussion the reality of the paragon described by the speakers was questioned, but it was agreed that ideals are valuable even if they are never fully attainable. Several speakers thought that an essential quality of general practice was indefinable and therefore not examin- able. Another likened the general practitioners to Prince Rupert who mounted his horse and rode off in all directions. But the speakers stood their ground. Attitudes can be portrayed and agreement obtained about a common core. Knowledge and skills are less difficult to define than attitudes. The essay-type question DR GRAHAM BULL (London) pointed to the many types of error and variation in marking essays. Accuracy of assessment increases with the number of essays required but never reaches J. ROY. COLL. GEN. PRACTIT., 1969, 17, 112 CONFERENCE REPORTS 113 the high level of the multiple-choice-question method. The traditional essay-type question has an educational value but very little as a method of assessment. The oral examination DR H. J. WALTON (Edinburgh) showed from published studies that the oral examination is also unreliable as a method of assessment. It chiefly tests powers of recall and is wide open to good and bad strategies on the student's part. Many of the virtues claimed for it cannot be substantiated. But there were a number of steps for making it more worthwhile-notably having observers to scrutinize the examiners. Multiple-choice questions PROFESSOR GEORGE SMART (Newcastle) showed that this method cuts out variations between examiners. It reproducibly tests knowledge and its manipulation, by providing objective questions. The examiner's opinion no longer matters. A secretary can do the marking. It gives a wide range of tests throughout the field of . But it requires a great deal of work in the preparation and is expensive. It tests recall but not ability to think or express ideas. The determination of the pass-mark is difficult. DR J. D. E. KNOX (Edinburgh) discussed the core of general-practice knowledge from which multiple-choice questions must be drawn. There are difficulties in producing enough good, concise questions from such areas as . DR JOHN STOKES (London) pointed to subjects which multiple choice questions cannot test- will this doctor pull back the bedclothes? Will he hurt the patient he is examining? A bank of questions is essential; it can be drawn on at intervals and must be continually reviewed for its relevance and its ability to discriminate between candidates. One must ask for each question: Is it relevant? Can the answer be agreed? Is it the right level of difficulty? Is the essential clearly stated? Is it as brief as possible? Is each alternative plausible? Is the question internally consistent? Setting the questions requires much work and it is not for amateurs. PROFESSOR B. LENNOX (Glasgow) discussed the problems of deciding how many marks should be allotted for the right answer and the mechanics of getting a total score. Neither computers nor other machinery are essential. Marking by hand can be very quick. Theclinical examination PROFESSOR CLIFFORD WILsoN (London) pointed to the changes expected in undergraduate which would mean that the main clinical assessment will be moved to a later stage than the qualifying examination, i.e., to the end of general professional training. There are four principles. (1) Clinical assessment must be fully integrated with the training. (2) The objectives of clinical assessment must be clearly defined and methods of testing devised to realize them. (3) There must be a high degree of uniformity of standards for comparability. (4) There must be feedback from the examination to both teachers and students and examiners so that it contributes to teaching, learning and future examinations. The objectives of clinical assessment are these: Can the candidate make accurate observations? Is he capable of logical deductions. Can he correctly interpret the total evidence? Marks should be allotted under agreed and detailed headings. Short cases are better than long ones. It is vital that teachers should be examiners so that there is proper integration. In-course assessment MR E. STONES (Birmingham) discussed objective tests. In medicine they should be tests of competence-more like tests for-a driving licence than an arts examination. We need to decide beforehand on concepts and skills required, then devise tests. Precise statement of the objectives is essential and the syllabus should, where possible, state what the candidates must be able to do rather than what they must know. There are ways of classifying objectives which make them more suitable for evaluation. Kinds of learning can be usefully classified (see De Cecco (1965) 'The philosophy of learning-and instruction'). Tests must be valid for the knowledge, skillstand attitudes learned and must be reliable in giving the same scores with the same students on different occasions. Validity is more important than reliability. MAJOR-GENERAL A. G. C. TALBOT (London) discussed subjective assessment. Confidential reports are required for all medical officers undergoing training in the army. Their usefulness is limited because assessors vary. Opportunities for assessment vary and so does the response of the officer to being assessed. It is essential to have a standardized form of reporting. This 114 CONFERENCE REPORTS can well be arranged in a multiple-choice question form. Opinions vary whether the candidate should see the report. PROFESSOR JOHN ANDERSON (London) and MR F. T. C. HAIWS (London) discussed new ideas and techniques, including the place of the computer. The mock examination The most striking feature of this conference was the subjectibn of all the participants, including guests of the College, to a multiple-choice paper which lasted 1 hours. The results being assembled overnight, the examiners (Drs Byrne and Knox) were subjected to a strong blast of comment by all the candidates. This they accepted with good grace, saying that it had been most useful. The trial showed what immense care has to be taken to select and improve the most suitable questions. The examination of the Australian college DR CHARLES ELLIOTT (Brisbane) described the experimental examination used in Australia since 1967. The multiple-choice question method was found to be the best way of discriminating between candidates. The examination has proved an unexpected boost to their college's voca- tional training programmes and to its status with other branches of the profession. Logistics DR GEORGE SWIFT (Winchester). Although the first purpose of the examination is to control the entry to membership ofthe College, it is likely in fact to control the type of men and women becoming principals in the National Health Service. It must be a test of competence, not of excellence. It should not be too competitive or exclusive. Probably it should happen at the end of vocational training and consist of a multiple-choice question paper, an essay and an oral examination. Thirty-two candidates took the examination in November 1968. The numbers may increase rapidly, producing a big administrative problem: 940 new principals are required annually in the National Health Service in England and Wales. There should be four examinations yearly and some of them may have to take place in the provinces. Conclusion The chair was taken in turn by DR JOHN HUNT, president of The Royal College of General Practitioners, LoRD PLATT, DR JAMES CAMERON, chairman of the General Medical Services Committee and PROFESSOR A. S. DUNCAN, professor ofmedical education, Edinburgh University. The organizers, Drs Patrick Byrne and J. D. E. Knox, were warmly thanked at the end of the conference.

Thoughts on the conference Professor A. S. DUNCAN, D.S.C., M.B., Ch.B., F.R.C.S., F.R.C.O.G. Edinburgh THE LARGE ATTENDANCE of colleagues from sister colleges and from universities shows at least three things: First, how carefully the organizers of the conference have prepared the way and laid out the programme. Secondly, how very many friends in other branches of the profession the Royal College of General Practitioners has, and how they have set an example in uniting different interests in a common objective. The third thing that is shown is the great interest that we all have in this extremely topical subject and I think the fact that the youngest of the Royal Colleges has initiated this conference brings great credit to her. Dr Crombie explained to us that there were two purposes in this examination. The first was to give an accolade to the young practitioner and the second was to test whether a man is a good doctor. On this second purpose Lord Platt made the point that by the time a man was sitting an examination for membership of any Royal College it was too late to test whether or