Towards a Research Agenda on Computer-Based
Total Page:16
File Type:pdf, Size:1020Kb
TOWARDS A RESEARCH AGENDA ON COMPUTER-BASED ASSESSMENT Challenges and needs for European Educational Measurement Friedrich Scheuermann & Angela Guimarães Pereira (Eds.) EUR 23306 EN - 2008 The Institute for the Protection and Security of the Citizen provides research-based, systems- oriented support to EU policies so as to protect the citizen against economic and technological risk. The Institute maintains and develops its expertise and networks in information, communication, space and engineering technologies in support of its mission. The strong cross- fertilisation between its nuclear and non-nuclear activities strengthens the expertise it can bring to the benefit of customers in both domains. European Commission Joint Research Centre Institute for the Protection and Security of the Citizen Contact information Address: Unit G09, QSI / CRELL TP-361, Via Enrico Fermi, 2749; 21027 Ispra (VA), Italy E-mail: [email protected], [email protected] Tel.: +39-0332-78.6111 Fax: +39-0332-78.5733 http://ipsc.jrc.ec.europa.eu/ http://www.jrc.ec.europa.eu/ Legal Notice Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of this publication. Europe Direct is a service to help you find answers to your questions about the European Union Freephone number (*): 00 800 6 7 8 9 10 11 (*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed. A great deal of additional information on the European Union is available on the Internet. It can be accessed through the Europa server http://europa.eu/ JRC44526 EUR 23306 EN ISSN 1018-5593 Luxembourg: Office for Official Publications of the European Communities © European Communities, 2008 Reproduction is authorised provided the source is acknowledged Printed in Italy Table of Contents Introduction…………………………………………………………………………………………….... 4 Romain Martin: New Possibilities and Challenges for Assessment through the Use of Technology…………………………………………………………………….. 6 Julius Björnsson: Changing Icelandic national testing from traditional paper and pencil based tests to computer based assessment: Some background, challenges and problems to overcome………………………………….. 10 Denise Whitelock: Accelerating the Assessment Agenda: Thinking outside the Black Box……………………………………………………………………. 15 Martin Ripley: Technology in the service of 21st century learning and assessment – a UK perspective………………………………………………….. 22 René Meijer: Stimulating Innovative Item Use in Assessment.............................................. 30 Dave Bartram: Guidelines and Standards for Psychometric Tests and Test Users………. 37 Mark Martinot: Examinations in Dutch secondary education - Experiences with CitoTester as a platform for Computer-based testing………………...… 49 Annika Milbradt: Quality Criteria in Open Source Software for Computer-Based Assessment……………………………………………………………………... 53 Nicola Asuni: Quality Features of TCExam, an Open-Source Computer-Based Assessment Software…………………………………..... 58 Thibaud Latour & Matthieu Farcot: An Open Source and Large-Scale Computer Based Assessment Platform: A real Winner…………………...…………………………………..…… 64 Friedrich Scheuermann & Angela Guimarães Pereira: Which software do we need? Identifying Quality Criteria for Assessing Language Skills at a Comparative Level……… 68 Oliver Wilhelm & Ulrich Schroeders: Computerized Ability Measurement: Some substantive Dos and Don’ts……………………………………………………………….... 76 Jim Ridgway & Sean McCusker: Challenges for Research in e-Assessment……………….. 85 Gerben van Lent: Important Considerations in e-Assessment: An Educational Measurement Perspective on Identifying Items for an European Research Agenda…….. 97 3 Introduction In 2006 the European Parliament and the Furthermore, as already pointed out by Council of Europe have passed research presented to PISA Governing Board, recommendations on key competences for October 2006 , change of cultural habits e.g. in lifelong learning and the use of a common terms of reading from computers vs. printed reference tool to observe and promote material might suggest an on-going change of progress in terms of the achievement of goals assessment forms as well. formulated in the “Lisbon strategy” in March 2000 (revised in 2006, see Future European surveys will introduce new http://ec.europa.eu/growthandjobs/) and its ways of assessing student achievements. follow-up declarations. For those areas which Tests can be calibrated to the specific are not already covered by existing surveys competence level of each student and become measurements (foreign languages and more stimulating, going beyond what can be learning-to-learn skills), indicators for the achieved with traditional multiple choice tests. identification of such skills are needed, as well Simulations provide better means of as effective instruments for carrying out large- contextualising skills to real life situations and scale assessments in Europe. In this context it provide a more complete picture of the actual is hoped that electronic testing could improve competence to be assessed. the effectiveness of the needed assessments, i.e. to improve identification of skills, by To date many tools and applications are being reducing costs of the whole operation (financial developed by commercial enterprises. efforts, human resources etc.). The European Commercial packages, underlying Commission is asked to assist Member States organisational concepts, use codes/algorithms to define the organisational and resource usually unpublished which make it difficult to implications for them of the construction and be adopted in different contexts. It is therefore administration of tests, including looking into important to take a specific look to what is the possibility of adopting e-testing as the available on the market as open source means to administer the tests. software and to reflect about the potential for their implementation in large-scale surveys at a In addition to traditional testing approaches European level. carried out in a paper-pencil mode, there are a variety of aspects needed to be taken into Overall, CBA is a logic follow-up in the account when computer-based assessment sequence of improvements to be achieved in (CBA) is deployed, such as software quality, terms of assessment methodologies, test secure delivery, reliable network (if Internet- development, delivery and valorisation of based), capacities, support, maintenance, results for multi-purposes. However, a variety software costs for development and test of challenges require more research into the delivery, including licences. barriers posed by the use of technologies, e.g. in terms of computer, performance and Any of the delivery modes, whether Paper- security. The Joint Research Centre of the Pencil and/or computer-based, comprises European Commission in Ispra, Italy, is advantages and challenges which can hardly supporting DG Education and Culture in the be compared, especially in relation to preparation of future surveys. estimated costs. The use of CBA includes additional benefits which can be achieved from The “Quality of Scientific Information” Action an organisational, psychological, analytical and (QSI) and the Centre for Research on Lifelong pedagogical perspective. Many experts agree Learning (CRELL) are carrying out a research on the overall added value and advantages of project on quality criteria of Open Source tools e-testing in large scale assessments. for skills assessment. A workshop was 4 organised which brought together European Implementations / delivery: key experts from assessment research and • What are the experiences made with large- practice in order to identify and discuss quality scale surveys in an international setting? criteria relevant for carrying out large-scale What are the challenges identified, e.g. assessments at a European level in terms of when internet-based delivery modes are objective measurements. applied? • What are relevant guidelines and standards As a consequence, due to the high relevance, to consider? What is there and what is still participants agreed to engage on further needed? discussion about issues and needs of computer-based assessment in Europe and to Assessment tools: publish a report on important aspects to take • Which features should be provided by a into account in this field. These activities are software tool, what are the “must” and “nice seen to be the beginning of the necessary to have” components of the tools? steps to establish a European forum that • What are the most important quality criteria ensures effective exchange of information and of software tools for computer-based ultimately increases the quality of assessment? assessments. • Relevance of Open Source: Why Open Source? What is the added value Open The articles presented here consider several Source can provide to assessment? How perspectives on computer-based assessment: can quality and sustainable benefits be ensured? Examples for implementations, Assessment methodologies: experiences made with large-scale • To what extent does CBA improve methods assessments, community support, and for skills assessment? Or is this an security issues. irreversible trend, due to general pervasion of ICT in everything “we do”? Examples of The articles are framed within a broader innovative item types and their potential discussion on developping a research agenda impact. for European computer-based assessments, • Under which circumstances are further highlighting important aspects to