Developing Standards to Evaluate Vocational Education and Training Programmes Wolfgang Beywl; Sandra Speer
Total Page:16
File Type:pdf, Size:1020Kb
Developing standards to evaluate vocational education and training programmes Wolfgang Beywl; Sandra Speer In: Descy, P.; Tessaring, M. (eds) The foundations of evaluation and impact research Third report on vocational training research in Europe: background report. Luxembourg: Office for Official Publications of the European Communities, 2004 (Cedefop Reference series, 3040) Reproduction is authorised provided the source is acknowledged Additional information on Cedefop’s research reports can be found on: http://www.trainingvillage.gr/etv/Projects_Networks/ResearchLab/ For your information: · the background report to the third report on vocational training research in Europe contains original contributions from researchers. They are regrouped in three volumes published separately in English only. A list of contents is on the next page. · A synthesis report based on these contributions and with additional research findings is being published in English, French and German. Bibliographical reference of the English version: Descy, P.; Tessaring, M. Evaluation and impact of education and training: the value of learning. Third report on vocational training research in Europe: synthesis report. Luxembourg: Office for Official Publications of the European Communities (Cedefop Reference series) · In addition, an executive summary in all EU languages will be available. The background and synthesis reports will be available from national EU sales offices or from Cedefop. For further information contact: Cedefop, PO Box 22427, GR-55102 Thessaloniki Tel.: (30)2310 490 111 Fax: (30)2310 490 102 E-mail: [email protected] Homepage: www.cedefop.eu.int Interactive website: www.trainingvillage.gr Contributions to the background report of the third research report Impact of education and training From project to policy evaluation in vocational education and training – possible concepts and tools. Preface Evidence from countries in transition. Evelyn Viertel, Søren P. Nielsen, David L. Parkes, The impact of human capital on economic growth: a Søren Poulsen review Rob A. Wilson, Geoff Briscoe Look, listen and learn: an international evaluation of adult learning Empirical analysis of human capital development and Beatriz Pont and Patrick Werquin economic growth in European regions Hiro Izushi, Robert Huggins Measurement and evaluation of competence Gerald A. Straka Non-material benefits of education, training and skills at a macro level An overarching conceptual framework for assessing Andy Green, John Preston, Lars-Erik Malmberg key competences. Lessons from an interdisciplinary and policy-oriented approach Macroeconometric evaluation of active labour-market Dominique Simone Rychen policy – a case study for Germany Reinhard Hujer, Marco Caliendo, Christopher Zeiss Evaluation of systems and Active policies and measures: impact on integration programmes and reintegration in the labour market and social life Kenneth Walsh and David J. Parsons Preface The impact of human capital and human capital investments on company performance Evidence from Evaluating the impact of reforms of vocational literature and European survey results education and training: examples of practice Bo Hansson, Ulf Johanson, Karl-Heinz Leitner Mike Coles The benefits of education, training and skills from an Evaluating systems’ reform in vocational education individual life-course perspective with a particular and training. Learning from Danish and Dutch cases focus on life-course and biographical research Loek Nieuwenhuis, Hanne Shapiro Maren Heise, Wolfgang Meyer Evaluation of EU and international programmes and initiatives promoting mobility – selected case studies Wolfgang Hellwig, Uwe Lauterbach, The foundations of evaluation and Hermann-Günter Hesse, Sabine Fabriz impact research Consultancy for free? Evaluation practice in the Preface European Union and central and eastern Europe Findings from selected EU programmes Philosophies and types of evaluation research Bernd Baumgartl, Olga Strietska-Ilina, Elliot Stern Gerhard Schaumberger Developing standards to evaluate vocational education Quasi-market reforms in employment and training and training programmes services: first experiences and evaluation results Wolfgang Beywl; Sandra Speer Ludo Struyven, Geert Steurs Methods and limitations of evaluation and impact Evaluation activities in the European Commission research Josep Molsosa Reinhard Hujer, Marco Caliendo, Dubravko Radic Developing standards to evaluate vocational education and training programmes Wolfgang Beywl; Sandra Speer Abstract There have been numerous attempts in evaluation research to develop guidelines and standards. The best known are the US standards for program evaluation, established by the Joint Committee on Standards for Educational Evaluations (JC). These standards originated in the school and university sector. Some illustrative examples relate explicitly to the area of initial and continuing vocational training. The goal of the study was to assess the transferability of US standards or the derivative standards of the German Evaluation Society (DeGEval, 2002) to vocational education and training (VET). The study considers the following initial questions: (a) does the terminology of the standards match the concepts of European initial and continuing vocational training? (b) are any standards not applicable to initial and continuing vocational training? (c) do European evaluation experts understand and accept the key concepts conveyed (e.g. definition of ‘evaluation’, differentiation between ‘formative’ and ‘summative’ evaluation, purpose of evaluation, etc.)? (d) are there specific national differences which should be considered when adapting the groups of standards? The standards of the DeGEval (2002) were chosen as a reference point for the following analysis. Other relevant standards were presented, and reflections on intercultural transferability and applicability to the subject of VET were made. First, VET experts were consulted during further discussions in Germany and Austria. Nobody expressed reservations about the transferability of the standards to VET, and no one proposed adaptation. Second, evaluation experts in widely divergent European countries were sent a questionnaire. The majority of those surveyed have a positive attitude to standards and endorse maximum standards. Pluralistic evaluation appears to be an important quality criterion. The single DeGEval standards are also debated and subject to comment on the basis of criteria found in recent European literature on VET evaluations. 1 Table of contents Table of contents..........................................................................................................................2 List of tables and figures..............................................................................................................5 1. Introduction..........................................................................................................................6 2. General evaluation standards .............................................................................................10 2.1. Background and purpose of evaluation standards ....................................................10 2.2. Philosophy of evaluation standards..........................................................................13 2.2.1. Excursus on the meaning of the word ‘standard’: minimum vs. maximum standards.....................................................................................14 2.3. Standards for evaluations and guiding principles for evaluators .............................17 2.4. Evaluation standards and models.............................................................................19 3. Transferability of standards ...............................................................................................23 3.1. Intercultural transferability of standards ..................................................................23 3.2. Current approaches to development of evaluation standards in Europe...................27 3.3. Transferability of evaluation standards to VET .......................................................32 4. Dialogue with German and Austrian VET experts on evaluation standards .......................35 4.1. Focused events with the German Federal Institute for Vocational Training (BIBB) .....................................................................................................................35 4.2. Focused meeting with the Austrian Federal Institute for Adult Education (BifEb).....................................................................................................................38 4.3. Conclusions from the dialogues ...............................................................................39 5. E-mail survey of evaluation experts in Europe...................................................................43 5.1. Profession and nationality of respondents................................................................44 5.2. Assessment of existing evaluation standards............................................................46 5.3. Further development of evaluation standards...........................................................50 5.4. Summary of survey findings .....................................................................................53 6. Reflections on VET evaluation standards literature ...........................................................55 6.1. Commentary on the utility standards ........................................................................57