* Via the ERIC Document Reproduction Service (EDRS). UPS Is Not * Responsible for the Quality of the Original Document
Total Page：16
File Type：pdf, Size：1020Kb
DOCUMENT RESUME ED 128 433 TM 005 618 AUTHOR Johnson, Richard T. TITLE Conducting Large-Scale Examinations: A Handbook. PUB DATE Jul 71 NOTE 165p. EDES PRICE hF-$0.83 HC-$8.69 Plus Postage. DESCRIPTORS Administration; *Admission (School); *Competitive Selection; Costs; *Developing Nations; Examiners; Group Tests; *Guidelines; Higher Education; Item Analysis; Personnel Selection; Pretesting; Publicize; Scoring; Secondary Education; Test Construction; *Testing; *Testing Programs; Vocational Education IDENTIFIERS Test Security ABSTRACT In developing countries, for the most part, traditional methods of selecting students for secondary schools, technical and vocational institutions, teacher training coll9ges,and universities are unable to ensure that the most capable applicants receive the limited opportunities available. To remedy this situation, the author has assembled and incorporated into this handbook a collection of testing procedures which he has found useful in large-scale testing programs in developing countries, In his discussion, the term "large scale" describes examinationstaken by at least a thousand candidates on a single schedule date. Nevertheless, most of the chapters are applicable to examination given on a smaller scale. The handbook may also be valuable to school headmasters and teachers who are interested in the workings ofa testing organization, a body that often exerts great influenceon their lives and those of their pupils. The handbook is divided into these sections:(1) Publicity,(2) Registration,(3) Item Preparation,(4) Pretesting,(5) Editing,(6) Reproduction, (7) Distribution/Administration, (8)Scoring/Processing/Reporting, (9) Overall Security, and (10) Costs. (RC) *********************************************************************** Documents ecquired by ERIC include many informal unpublished * materials not available from other sources. ERIC makesevery effort * * to obtain the best copy available. Nevertheless, items of marginal * * reproducibility are often encountered and this affects the quality * * of the microfiche and hardcopy reproductions ERIC makes available * * via the ERIC Document Reproduction Service (EDRS). UPS is not * responsible for the quality of the original document. Reproductions * * supplied by EDRS are the best that can be made from the original. * *********************************************************************** Conducting Lar6eScale Examinations: A Handbook Richard T. Johnson U.S. DEPARTMENT OF HEALTH, EDUCATIONS. WELFARE NATIONAL INSTITUTE OF EDUCATION THIS OOLUMENT mA$ BEEkREPRO. DUCED EXACTLY AS RECEIVE()FROM THE PERSON OR ORGANIZAT,ONORIGIN- ATING IT POINTS OF VIEWOR OPINIONS STATED DO NOT NECESSARILYREPRE SENT OFFICIAL NATIONALINSTITUTE OF EDUCATION POSITION OR POLICY rami 2 J uly 1971 Virginia Polytechnic Instituteand State University 11440 lssac Newton Square,North CZ Reston, Virginia 22090 TABLE OF CONTENTS 0Preface background 0-1 definition of "large-scale" 0-2 assumptions 0-2 usage of the handbook 0-3 1 Publicity brochures 1-2 advertisements 1-3 candidate information booklets 1-3 meetings, lectures 1-4 sample candidate booklet 1-5 sample headmaster booklet 2 Registration candidate forms 2-2 receipts/admission forms 2-3 centers & allocation 2-3 rosters, packing lists, print orders 2-3 3 Item Preparation specifications 3-2 levels of thinking 3-9 training of item writers 3-15 comnissioning items 3-17 notes on item writing 3-21 4 Pretesting rationale 4-2 pretryout 4-4 trial sample 4-6 directions 4-6 format 4-6 timing 4-7 motivation 4-10 security 4-10 administrative arrangements 4-13 answer sheets 4-13 item analysis 4-13 iii 3 5 Editing rewriting 5-2 item file 5-2 arrangement of items 5-3 editing committees 5-6 6 Reproduction genefal methods 6-2 tests 6-Z directions, accessory materials 6-6 answer sheets 6-6 security 6-9 7 Distribution/Administration counting, numbering 7-2 packing 7-2 transport 7-3 administration 7-3 return of materials 7-6 timing 7-7 security 7-8 sample instructions 7-9 8 Scoring/Processing/Reporting prescanning 8-2 scoring 8-2 editing 8-7 cross-footing, combining 8-8 transformations 8-10 listing 8-12 certificates 8-14 distribution of results 8-16 quality control 8-17 statistics, reports 8-18 retention of data. 8-18 9 Overall Security security areas 9-1 safes 9-6 receipt of materials 9-7 iv 4 9 (continued) storage 9-9 packing 9-9 shipping 9-11 test centers 9-13 data processing department 9-15 typing, copying, printing 9-16 confidential files 9-17 security breaches 9-18 10 Costs survey 10-1 each stage 10-1 totals 10-3 FIGURES 0.1 Overall diagram of the conduct of large-scale examinations 0-4 2.1 Entry form for the WAEC/London GCE 2-4 2.2 (reverse side) 2-5 2.3 Sample packing list 2-6 3.1 Example of a weighted blueprint 3-16 3.2 Instructions to item writers for submitting items 3L18 4.1 Edge stripes to prevent loss of single pages 4-12 4.2 Horizontal and vertical formats for can'idate ident. 4-13 4.3 Sample item analysis data 4-19 5.1 Completed item card 5-4 5.2 Plot of item statisrfcs 5-4 6.1 Comparison of fouh: common methods of re duction 6-3 6.2 Precoded answer sheet 6-8 6.3 Answer card 6-8 7.1 Certificate of examination conduct 7-5 7.2 Typical examination hall layout for forty candidates 7-11 8.1 Overprinting a scoring key 8-4 8.2 A computer-printed frequency distribution 8-11 8.3 Results listing 8-13 8.4 Public results listing 8-15 8.5 A Primary School Leaving Certificate 8-16 9.1 Office organization 9-2 9.2 Security area floor plan 9-2 9.3 Stores register 9-4 9.4 STO inventory control 9-4 9.5 Shipping register 9-4 9.6 Access register 9-4 9.7 Key register 9-4 vi PREFACE In Nigeria, a new pottery factory whichwas about to open advertised '00 positions; over 20,000 people applied. When the Polytechnic College of the University of Malawi was ready to begin instruction,applications for the 60 places were received from 1,350 candidates. In West Africa, only ten to eleven per cent of primary school leavers can hope to begin secondary school, and only one per cent of secondary-school-agechildren are actually attending school at any level. The situation is similar in hundreds of countries, and in each,some authority must decide which particular individuals will receivemore training, a job, or promotion. Although such selection is a problem anywhere, the sheer magnitude of the numbers involved in developing nationsmagnifies and compounds it. Traditional methods of selecting students for secondary schools, technical and vocational institutions, teacher training colleges,and universities clearly offer little help, since theyare unable to ensure that the most capable applicants receive the limited opportunit'ies. When these procedures are used in countries where educational facilitiesare serious/y circumscribed, precious resources are squandered. In many African countries, it has been demonstrated that adding objective tests of achievement and aptitude to existing selection procedures can greatly increase tbe number of above-avera performers within the educational system. Tests of aptitude have also proved useful in selecting applicants for training courses, as well as for jobs in public service and in the private sector. Any government or institution which attempts to use modern methods of testklg, however, is faced with many potential pitfalls, unless it hasaccess to snr.e general guidelines devised by people who have worked in similar situations. General guidelines for testing programs are rare, and usually . based on experience in America and Europe, experience which is seldom directly applicable in other cultures. To rcHady this situation, I have assembled and incorporated in this handbook a collection of testing procedures which have been found useful in developing countries, together with experiences which, although in retrospect have their humorous aspects, were agonizing at the time. I have done this in the fervent hope that at least some disasters of the past will be averted in the future. 0-1 The justification for this handbook is that it deals withproblems specific to developing countries, and that it includes materialnot found in commonly available sources. Most of the material assembled here resulted from direct experience in the Pacific, as wellas in West and Southern Africa: Western Samoa, Nigeria, Ghana, Sierra Leone, The Gambia, Malawi, Botwana, Lesotho, and Swaziland. Since 1965 I have worked with the West African Examinations Council and with the Regional Testing Resourceand Training Centre in Southern Africa on problems of test development. In deference to the nations involved, I have seldom specified the locationof a particular faux pas, although the various ministrieswere wise enough to know that publication of these procedures anderrors might assist others. In the following discussion, the term "large-scale" describesexaminations taken by at least a thousand candidates on a single scheduleddate. Nevertheless, most of the chapters are applicable to examinationsgiven on a smaller scale. The handbook may also be valuable to school headmasters and teachers who are interested in the workings of a testing organization,a body that often exerts great influence on their lives and those of their pupils. In order to present a unified discussion, I have assumed that: 1. there exists a central office with the major responsibility for conduct of the examination; in this handbook I have called this office the State Testing Office, abbreviated STO, which abbreviation I have often used as a substitute for the editorial "we." Z. the examinatin consists mainly of multiple choice questions. For pedagogical purposes I refer to three separate papers--A, B, and C--but also deal with a language essay on occasion. In spite of this focus, the handbook 31' uld be useful in those countries where only trarational essay examinatiofts are given, since most of the problems are identical. 3. the major purpose of the examination is to select a proportion (often less than .25) of the possible candidates for further treatment.