Definitions of Syllabus, Curriculum and Evaluation 4

PGCE (PCE)

Curriculum Evaluation

Derek Wise February 2008 Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 2 Content

Assignment 6b 3 Definitions of Syllabus, Curriculum and Evaluation 4 Curriculum Models 5 HOW to Evaluate: A Simple Model for Evaluating Specifics 7 Success Criteria 8 WHAT to Evaluate: the CIPP Model of Programme Evaluation 9 WHAT to Evaluate: the OFSTED of Course Evaluation 12 Curriculum Evaluation – The Stakeholders and Interested Parties 16 The 4 levels of training evaluation 17 Curriculum evaluation – an exam board perspective 19 Competence – a semantic differential 20 Curriculum Evaluation - Less Reputable Evaluations 21 Bennet’s Seven Curriculum Issues 23 Curriculum Evaluation – Gathering Data 24 Change - the five categories of product adopters: 27 Reading List 27

Progress & Presentation of assignment 6b

‘Carry out an evaluation of curriculum, teaching and learning issues in the teaching of your subject specialism; measure the effectiveness of teaching and learning and make informed conclusions and recommendations as to the future development of the programme’

Feb 7: brief informal discussions with tutor on proposed curriculum evaluation

Feb 28: max 30 minute presentation of your proposed curriculum evaluation in 3 or 4 groups of 4/5 course members

Presenters should provide a seminar paper (max 2 A4 sides) to support their presentation – this will probably be bullet points responses to the 6 seminar issues above. Make 3 copies - 1 copy per 2 members and 1copy for tutor. OHTs should NOT be used. NB presenters will be stopped after 30 minutes to ensure equal time for all members

June 5 : max 30 minute final presentations of your completed curriculum evaluation. In 3 or 4 groups of 4/5 course members.

Presenters should provide a seminar paper (max 2 A4 sides) to support their presentation -this will probably be bullet points responses to the 8 seminar issues above (which could of course be an updated version of the earlier seminar paper). Make 3 copies - 1 copy per 2 members and 1copy for tutor. OHTs should NOT be used. NB presenters will be stopped after 30 minutes to ensure equal time for all members

(Curriculum Evaluation (approx 2500words) in a plastic project folder with name clearly indicated should be handed to Derek at the end of the session for marking). A copy of the seminar papers for February and June should be included in the folder

Assignments will be returned within 1 week and any required developments (hopefully there won’t be’) can be emailed to Derek. Thoughtful and carefully presented seminars will be positively taken into account in the light of any weaknesses in the final assignments!

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 3 Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 4 Assignment 6b Evaluation of a Curriculum

Aims and Learning Outcomes

By the end of this assignment you will be able to:  select and describe a course programme in your specialist subject  analyse specific issues raised by the programme specification, course design, teaching, learning or assessment methods in relation to the needs of typical learners of this subject and level  use evaluation techniques to identify and analyse the responses of learners and colleagues to aspects of the curriculum and its delivery  explain and apply models of curriculum to their evaluation  identify strengths and weaknesses of the current programme,  propose and justify improvements to the design or delivery of the programme  justify your analysis and recommendations with reference to current research into teaching and learning, with specific reference to you specialist subject  present your evaluation in a concise, well-written and referenced report

Candidates are required to:

1. carry out an evaluation of curriculum, teaching and learning issues in the teaching of your subject specialism 2. measure the effectiveness of teaching and learning 3. make informed conclusions and recommendations as to the future development of the programme.

Suggested activities could include:

 identifying your current weaknesses in and updating your subject knowledge  gathering information about syllabuses, standards and national developments in your subject  reviewing the resources available for the teaching of your subject  critically reviewing current issues in assessment related to your subject area  undertaking a comparative examination of the teaching of your subject in another establishment  discussion with experienced practitioners on proven effective strategies for planning and delivering teaching and learning.

Word count 2500 maximum, excluding bibliography and appendices

Assessment Criteria This assignment is marked on a pass/refer basis.

In completing this assignment, you should be able to demonstrate evidence of Standards AS1-7, AK1.1-5.1, 6.1-7.3, BK1.1-2.7, 3.3, 4.1-5.2, CS1-4, CK1.1-4.2, DS1-3, DK1.1, 1.3, 2.1, 3.1-3.2, ES1, 5, EK1.1-2.1, 2.3, 3.1-3.2, 5.1-5.2, FS3,

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 5 Definitions of Syllabus, Curriculum and Evaluation

Evaluating Education- A Warning!

‘With the institutionalisation of ‘new wave’ evaluation, I think we have produced a situation in which there is some danger that evaluators create their own establishment and glamorise it as an elite. Let’s keep hold of the idea that it is mostly a matter of common sense and learning from experience. That is not entirely true but it keeps us from going technical or theological; and a little modest oversimplification is better than a lapse into jargon or pretentiousness’ Stenhouse, L. (1975) An Introduction to Curriculum Research and Development Heinemann p116

Syllabus

A syllabus is a list of indicative content that is usually provided by the awarding body. This may come in forms such as competence statements, programmes of study, directives, learning outcomes etc. It is essentially static and in this sense prescribed and is only brought alive through description by the teacher and by the students where it becomes transformed to a curriculum.

Curriculum

A curriculum is dynamic. It incorporates the whole of the learner’s experience, both formal and informal. Curriculum is not the intention or prescription but what happens in real situations. It is about perceiving, understanding and describing what happens

.. a ‘course; especially a regular course of study at a school or university’ (Shorter Oxford Dictionary)

In 1984 Sir Keith Joseph (then Secretary of State for Education) gave the annual North of England speech on the curriculum stating that ‘The curriculum must have relevance to the world of work ... it must be balanced, there must be breadth and differentiation’ 21 years later these principles are still fundamental to any curriculum

Evaluation

‘To judge or assess the worth of…..’ Collins Concise English Dictionary

‘The process of evaluation is essentially the process of determining to what extent the educational objectives are actually being realised by the program of curriculum and instruction. However, since educational objectives are essentially changes in human beings, that is, the objectives aimed at are to produce certain desirable changes in the behaviour patterns of the student, then evaluation is the process of determining the degree to which these changes in behaviour are actually taking place.’ Tyler’ (1949) Basic Principles of Curriculum and Instruction Chicago: University of Chicago (pp. 105-106)

Evaluation is the process of conceiving, obtaining and communicating information for the guidance of educational decision-making with regard to a specified programme (MacDonald, 1975)

One of the important considerations in any systematic evaluation is to make clear what the criteria are for the evaluation.

It is the insistence on the explicitness about criteria which distinguishes evaluation … from the ‘everyday’ use of the term. (Harlen)

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 6 Curriculum Models

There are a number of 'models' of curriculum or instructional systems. Remember a model is an abstraction from reality - it doesn’t exist in itself but can provide a useful ‘ often visual - basis for analysis. Consider a curriculum with which you are currently engaged – does it conform to one or more of the following models?

1 linear curriculum:

Useful for instruction related to sequential topic areas containing objectives of roughly equal weighting.

Knowledge Knowledge Knowledge Required Knowledge Knowledge Required 1a 1b 1c Knowledge 2a 2b knowledge Skill 1a Skill 1b Skill 1c & Skill for Skill 2a Skill 2b & Skill for topic 1 topic area 2

Example of use: particular value: Any inherent problems: Style(s) of teaching/learning best suited by this model:

2 Spiral Curriculum

Topics are covered in enough depth to provide basic understanding and prerequisite knowledge for 2nd time round and then represented in greater depth for 3rd time etc etc

Example of use: particular value: Any inherent problems: Style(s) of teaching/learning best suited by this model:

3 Pyramid Curriculum

Clear move from general (broad base) to particular

Year 3 - options Year 2 - some specialism Year 1 - broad introduction

Example of use: particular value Any inherent problems:

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 7 Style(s) of teaching/learning best suited by this model: 4 Common Core

The common core is often linked to other models (eg pyramid with common core + options) It requires ensuring that the common element is necessary & sufficient

Example of use: particular value Any Inherent problems Style(s) of teaching/learning best suited by this model:

5 Project centred curriculum

Leads out from an initial central theme and provides network of topics as a map to show where the learner should get to over a period of time - can be accredited as accumulated/transferable units. Very popular in primary education and syllabi for adult learners with particular needs

Example of use: particular value Any inherent problems: Style(s) of teaching/learning best suited by this model:

6 Research/Inquiry centred

Centred around substantive issue(s) or question(s) which the student has to inquire into in order to provide a coherent, in depth and analytical research report or dissertation or thesis

Example of use: particular value Any inherent problems: Style(s) of teaching/learning best suited by this model:

If the curriculum with which you engaged does not fit to any of the above models – how would you define it?

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 8 Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 9 Curriculum Evaluation – HOW to Evaluate: A Simple Model for Evaluating Specifics

Evaluation is EXTREMELY important to the teacher as it provides the opportunity to make an informed opinion about something - and helps reflection. Evaluation is concerned with looking at the strengths & weaknesses of something; the efficiency & effectiveness of how it is undertaken and the value to those who are involved. This can be remembered by the acronym SWEET (nb – the acronym SWEET was created by Derek Wise and is not published). Evaluation should be based on evidence rather than just gut feelings.

This evidence can be quantitative (ie quantifiable as assessment results, numbers or physical evidence such as retention on courses). You can observe learner participation and interest, for example the time they spent on-task (doing the work they should be doing) as against being off-task over a period of time and recording your results. Your evaluation can also be qualitative (ie about the quality or feeling or perception of something for example learner’s feelings, tutor’s feelings or the reaction to the learners to the lesson).

Strengths  What were the good things about it – how can they be used in the future?  What worked well? why? – can this be applied to other lessons?  What were the good results? – can they be replicated in the future?

Weaknesses  What were the weak things about it? – how can they be overcome in the future?  What did not work well? - why? – How could they be developed for the future?  What were the poor results? – how can they be improved?

Efficiency Efficiency is concerned with competence; the ability to do something well or achieve a desired result without wasted energy or effort, or the degree to which this ability is used

 What things showed that the lesson was efficient eg (re-usable learning resources, recap of previous work to ensure all learners were ready to start etc). how can this efficiency be applied to other situations?  What things suggested that the lesson was not efficient? – how can this be improved?

Effectiveness Effectiveness is concerned with producing or causing a result; especially the desired or intended result. To be truly effective the result should be striking especially in producing a strong or favourable impression on people

 What things showed that the lesson was effective? eg (learners were involved and appeared to be learning; appropriate learning resources, recap of work at the end of the session which showed learners had learnt) – what lessons can I learn for my teaching?  What things suggested that the lesson was not effective?

The value to those involved (value added)  Sometimes social / personal things like teaching cannot be easily evaluated in terms of efficiency and effectiveness or based on qualitative or quantitative data. The ‘extent to which the lesson (or more probably whole programme of study) has contributed to the growth of the learner although this may not appear in examination results is very important. This is sometimes called ‘value added’

 Evaluation often requires comparison with similar events eg another lesson – it is essential that this evaluation is based on like events.

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 10 Curriculum Evaluation – HOW to Evaluate: Success Criteria

For a valid and reliable evaluation (particularly a course evaluation) it is essential to have a clear idea of the criteria for what would constitute a successful outcomes (ie the success criteria).

Success criteria are statements of what should reasonably be expected as a result of the action taken

For example if the student achievement on a particular course is 70% is this good or bad? If the success criteria established at the outset of the course was that the course should have an 80% student achievement rate (with grade stated if necessary) based on previous years, or on national benchmark figures from other providers, then clearly 70% is unsatisfactory.

Writing success Criteria

It is essential to ensure that the success criteria fair and honestly capture the true situation – not an adaptation to suit the evaluation!

A course evaluation of a GCSE course states that one success criteria will be the % of students who gain a grade B qualification. Results shows that 60% of the course members who took the exam gained a grade B. It appears pretty good!

The course started with 24 students and only 12 completed and took the exam - there was a 50% drop-out! Of these 12 students 60% - ie 7 students - gained a grade B.

So in fact only 30% of the students who started the programme gained a grade B

The success criteria did not make it clear whether it was using the start number or the number taking the exam.

For each of the issues considered in the CIPP or OFSTED models it is important to establish a success criteria based on previous years, or national benchmark figures or on what the course team feels is appropriate. It is important to state how the success criteria has been reached

. Success criteria should be observable and measurable

. Success criteria should be based on what the objective reasonable and legitimately requires

. Success criteria should be attainable – they should be designed to attain success rather than prevent it.

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 11 Curriculum Evaluation – WHAT to Evaluate: the CIPP Model of Programme Evaluation

A useful model for teaching programme evaluation is the Context-Input-Process-Product (CIPP) model (Daniel Stufflebeam 1966, 1967, 1971,1972,2000) which proposes a four part evaluation for any programme and suggests a number of questions which should be asked within those headings

In general, these four parts of an evaluation respectively ask:

1 What needs to be done? (context) 2 How should it be done? (Input) 3 Is it being done? (process) 4 Did it succeed? (product)

Context - The setting of the course or subject ie the aims. This can be seen in terms of the validity of what is being taught External factors which impinge on the programme and which may affect its outcomes might be considered here (eg course is requirement for Government New Deal) .

Input - The input element relates to the factors which contribute to the course, ie the tutor(s), the students, the resources, the environment

Process - The process element concerns the appropriateness of what happens on the course, how the input elements are used to achieve the aims and objectives.

Product - The product concerns the outcomes - what has been gained (ie value added) and what has been achieved, (ie qualifications)

Product element – sub divisions More recently the ‘Did it succeed?’ or product evaluation part is itself divided into

Impact Were the right beneficiaries reached? Was there a possible audience who were not reached – if so why (nb LLUK values of open access)

Effectiveness - Were their needs met? Was there an in-effective prioritization of one aspect of the curriculum over another. Could the programme have been delivered in a more cost efficient way with equal effectiveness? Do the results of the effectiveness evaluation suggest that changes should be made in specific areas?

Sustainability - Were the gains for the beneficiaries sustained? Was there a short term improvement which was not evidence some time after the programme. Possible interview of past course members to ascertain the long-term programme value

Transportability - . Did the processes that produced the gains prove transportable and adaptable for effective use in other settings? How will early adopters be reached? How will the more reticent early and late adopters be persuaded? (see note on Everett Rogers at back of handbook)

Thus the model can be remembered by the acronym CIPPIEST!

For the purposes of general curriculum evaluation the initial CIPP model should suffice although evaluators could consider the 4 product aspects for greater insight.

CIPP in course evaluation

The four CIPP main headings provide a range of questions to be asked (for each one consider – how do I know this (data)?)

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 12 Context (The setting of the course or subject)

Meeting Needs

Are the aims consistent with the needs of the learner, industry or society? Does the curriculum develop links with the world of work? Does the curriculum meet current Government initiative eg basic skills/key skills?

Proposed Course Structure

Is the course structure the best way of achieving the desired outcomes? Are the aims updated regularly/has the course been updated in the light of national developments (eg basic skills/key skills, vocational developments

Encouraging Progression

Do the aims encourage progression in further or higher education, or transferability to other programmes?

Input (The input factors which contribute to the course)

Students

Do the students possess the entry ability required by the course content Did the students original wish to do this course?

Tutors

Are tutors qualified (an area of current interest in post compulsory education) Are the tutors competent in the subject; in current teaching methods and where necessary, in current industrial requirements? Are there adequate tutors for all aspects Do tutors have adequate time for all the requirements of the course eg IV/double marking/tutorials

Delivery

Is there an appropriate amount of time in line with awarding body specifications Are resources appropriate, adequate and available? Is the environment conducive to learning and appropriate to the world of work?

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 13 Process (The appropriateness of what happens on the course,)

Students . Do students have a ‘voice’ (eg staff/student forums) . Is there appropriate course group support and interactivity facilities . Is there an effective student guidance/counselling system

Tutors . Do tutors receive/access, understand and use, all available documents related to . the curriculum, ie information from examining bodies regarding syllabus and assessment requirements . Are there regular and useful course team meetings . Do team members have access to necessary professional development?

Delivery . Are teaching strategies appropriate? . Are appropriate teaching aids and learning resources used? . Is there appropriate/adequate general access to information technology? . Is there appropriate/adequate general access to library and academic support ? . Are course accommodation & timetabling facilities maintained (eg no constant room or timetable changes)

Monitoring & Assessment . Is the monitoring/tutoring/support system appropriate and adequate? . Is the assessment system valid and reliable? . Is there an effective system of feedback to learners ?

Product (what has been gained (ie value added) and achieved)

Students

Attendance rate (and possible comparison to achievement) What is the staying-on rate as a trend (this year compared to the last 2or 3)? What is the qualification achievement rate as a percentage of starters? Has the course contributed to the students ability to meeting the needs of work (eg presentation skills? What is the progression rate to further industry/FE/HE? What was the students response to the course – (eg results of student perception questionnaires

Tutors What staff development was identified and satisfied at individual and course team level? Are tutors motivated by their involvement in the course?

Delivery To what extent has the course been modified in the light on on-going feedback? If appropriate, what do employers feel about the course? Is there an effective system for evaluating and gaining feedback about the programme?

Developed by Derek Wise from Stufflebeam D. (1971). Educational Evaluation and Decision Making. In Stufflebeam, D.L. Madaus, G. F. & Kellaghan, T. (Eds.), (2000) Evaluation models (2nd ed.). (Chapter 16). Boston: Kluwer Academic Publishers.

(note: the CIPP model was developed over many years with specific aspects added at each review and development.)

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 14 Curriculum Evaluation – WHAT to Evaluate: the OFSTED of Course Evaluation

The Office for Standards in Education (Ofsted) is a non-ministerial government department, established in 1993 under the Education (Schools) Act 1992. It is responsible for inspecting the standards of independent and state schools, local education authorities, child day care and childminding in England; it also monitors the work of the Independent Schools Inspectorate. Ofsted is also required to provide independent advice to the United Kingdom government and parliament on matters of policy and to publish an annual report to parliament on the quality of educational provision in England. The legal name is the "Office of Her Majesty's Chief Inspector", reflecting the fact that Ofsted's powers are vested in its Chief Inspector, appointed by the Queen-Council. In September 2005 a new system of short notice inspections came into being. From April 2007 OFSTED will merge with the Adult Learning Inspectorate (ALI) to provide an inspection service which includes all post 16 government funded education

A college of course Evaluation undertaken by Ofsted

‘A college inspection is a process of evidence gathering to evaluate the effectiveness of the college in supporting the learning and achievement of students. The emphasis is on judging the degree to which rigorous self-assessment leads to quality improvement in the college”

The inspectors carry out inspections using the Common Inspection Framework, details of the criteria can be found on the Ofsted website. www.ofsted.gov.uk . Once the inspection has been conducted a report will be published, this will list the names of the qualification reviewed (including retention and achievement data).

OFSTED’s Common Inspection Framework as a model for course evaluation

. What are the school's main strengths? . What are you working on to improve? . How successful are your policies and practice? . Does what you offer help children to reach the highest standards? . Does what you do help to make the children safe and healthy?

Ofsted considers five issues

. Effectiveness of provision . Capacity to improve . Achievements and standards . Quality of provision . Leadership and management

When observing courses and classes they ask 4 questions

How well do learners achieve?

How effective are teaching and learning?

Do activities meet the needs of learners?

How well are learners guided and supported?

(A curriculum evaluation could be based on these four questions)

These are assessed against four standards . Outstanding . Good . Satisfactory . Inadequate

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 15 OFSTED – SCHOOL SELF-EVALUATION

Schools now receive an on-line self evaluation form which is used by Inspectors. This form can be a useful basis for a curriculum evaluation

1. CHARACTERISTICS OF YOUR SCHOOL

1a Please outline the main characteristics of the learners, including:

- their attainment on entry and how you know this; - their social and economic backgrounds, indicating the level of prosperity or deprivation.

1b Please summarise briefly your distinctive aims and describe any special features of your school, for example:

- whether you intend to become a specialist school, or school with special status, and if it is one already, the main changes that have occurred because of this; - whether you are a school with a religious character; - any special units, additional community services or extended provision; - significant partnerships with other providers or agencies (such as shared arrangements for the curriculum, federal arrangements, or partnerships with employers).

1c Please outline specific contextual or other issues that act as aids or barriers to raising performance, for example:

- any difficulties in recruiting and retaining staff, or governors; - recent or impending reorganisation; - mobility of learners; - particularly important facts in your recent history, such as change of leadership.

1e Please outline briefly the main priorities in your improvement/development plan, and how they reflect the context in which you work.

2. VIEWS OF LEARNERS, PARENTS/CARERS AND OTHER STAKEHOLDERS

What are the views of learners, parents/carers and other stakeholders and how do you know?

2a How do you gather the views of learners, parents/carers and other stakeholders, such as those accessing additional services, how often do you do this, and how do you ensure the impartiality of the information?

2b What do the views of learners, parents/carers and other stakeholders tell you about the learners’ standards, personal development and well-being, and the quality of your provision?

2c How do you share with parents/carers and other stakeholders the collated findings about their views?

2d Can you give examples of action you have taken based on the views of learners, parents/carers or other stakeholders, with an evaluation of the effectiveness of what you did?

- Are there examples of actions you decided not to take (with the reasons for this)? - Are there examples of ways in which your stakeholders have influenced the priorities noted in section 1e?

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 16 3. ACHIEVEMENT AND STANDARDS

How well do learners achieve?

In answering the following questions, please make clear the main evidence, such as performance data, assessments and records of learners’ progress, on which your evaluation is based (but please use data selectively, avoiding the copying out of tables of descriptive information).

3a What are learners’ achievement and standards in their work?

- the standards learners reach as indicated by their test and examination results, taking account of: any significant variations between groups of learners, subjects, courses and key stages; trends over time; comparisons with other schools; whether learners reach challenging targets - the standards of learners’ current work in relation to their learning goals (noting any significant differences between current work and recent results) - learners’ progress relative to their starting points and capabilities, with any significant variations between groups of learners (making clear whether there are any groups that are underachieving and could be doing better)

3c On the basis of your evaluation, what are your key priorities for development?

4. PERSONAL DEVELOPMENT AND WELL-BEING

How good is the overall personal development and well-being of the learners? In answering the following questions, please make clear the main evidence on which your evaluation is based.

4a To what extent do learners adopt healthy lifestyles? - whether learners take adequate physical exercise, and eat and drink healthily - learners’ growing understanding of how to live a healthy lifestyle

4b To what extent do learners feel safe and adopt safe practices? - whether learners feel safe from bullying and racist incidents - the extent to which learners have confidence to talk to staff and others when they feel at risk

4c How much do learners enjoy their education? - take account of learners’ attitudes, behaviour and attendance - learners’ spiritual, moral, social, emotional and cultural development

4d How well do learners make a positive contribution to the community? - learners’ growing understanding of their rights and responsibilities, and of those of others - how well learners express their views and take part in communal activities

4e How well do learners prepare for their future economic well-being? - how well learners develop skills and personal qualities that will enable them to achieve future economic well-being - learners’ understanding of career options, and the acquisition of workplace skills

4g On the basis of your evaluation, what are your key priorities for development?

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 17 5. THE QUALITY OF PROVISION

Your evaluation of the quality of provision should take account of the impact on the standards achieved and the personal development and well-being of learners. In answering the following questions, please make clear the main evidence, such as monitoring of teaching, on which your evaluation is based.

5a How good is the quality of teaching and learning?

- how well teaching meets the needs of the full range of learners and course requirements - the suitability and rigour of assessment in planning learning and monitoring learners’ progress - the diagnosis of, and provision for, individual learning needs - the involvement of parents and carers in their children’s learning and development

5b How well do the curriculum and other activities meet the range of needs and interests of learners?

- the extent to which the curriculum or activities match learners’ needs, aspirations and capabilities, building on prior attainment and experience - how far the curriculum meets external requirements and is responsive to local circumstances - the extent to which the provision enables and encourages learners to be healthy and stay safe - the extent to which learners have opportunities to develop enterprise, financial skills and work in teams - the extent to which enrichment activities and, where appropriate, extended services contribute to learners’ enjoyment and achievement - where appropriate, the extent to which employers’ needs are met through developing work- related skills

5c How well are learners guided and supported?

- the care, including as appropriate integrated day care, advice, guidance and other support provided to safeguard welfare, promote personal development and make good progress in their work - the quality and accessibility of information, advice and guidance to learners in relation to courses and programmes, and, where applicable, career progression - the extent to which the school and any additional services contribute to the learners’ capacity to be healthy, including vulnerable groups, such as looked after children

5e On the basis of your evaluation, what are your key priorities for development?

Quality of teaching and learning Quality of the curriculum and other activities Quality of care, guidance and support for learners

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 18 Curriculum Evaluation – The Stakeholders and Interested Parties Formal & Informal Evaluations

The learner – (primarily in terms of value to self) informal but may be involved in formal (through personal reflection and value judgements about the worthwhileness of the course). Primarily qualitative

The tutor - (for monitoring programme benefits) – formative informal but may be involved in summative formal. Probably primarily qualitative

The curriculum/programme team - (through the internal verifier/moderator) Primarily formal and qualitative (standardisation of assessment and possibly quantitative (eg pass rates)

The College / Educational Organisation (through the quality unit) Primarily formal and quantitative (eg pass rates) possibly qualitative (student perceptions of course)

The Awarding Body (through the external verifier/ moderator) Eg 3 main groups (OCR; Edexel; AQA/CGLI) plus ABC Awards; OCN Primarily formal and quantitative (eg pass rates; standards of assessment; portfolios); possibly qualitative (student perceptions of course)

Learning & Skills Council (LSC) / local LSC (LLSC) (Through OFSTED and ALI (ALI ceasing 2006) Primarily formal and quantitative (eg completion/achievement rates; retention rates); possibly qualitative (student perceptions of course gained during inspections)

DfES Primarily formal and quantitative gained from OfStEd reports

The General Public Primarily formal and quantitative gained from OfStEd reports available on-line (open.gov.uk)

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 19 Kirkpatrick's Four Levels of Evaluation in Training

In Kirkpatrick's (1998) four-level model, each successive evaluation level is built on information provided by the lower level. According to this model, evaluation should always begin with level one, and then, as time and budget allows, should move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level's evaluation. Thus, each successive level represents a more precise measure of the effectiveness of the training programme, but at the same time requires a more rigorous and time-consuming analysis.

Level 1 Evaluation - Reactions

Evaluation at this level measures how participants in a training programme react to it. It attempts to answer questions regarding the participants' perceptions - Did they like it? Was the material relevant to their work? This type of evaluation is often called a ‘smilesheet.’ According to Kirkpatrick, every programme should at least be evaluated at this level to provide for the improvement of a training programme. In addition, the participants' reactions have important consequences for learning (level two). Although a positive reaction does not guarantee learning, a negative reaction almost certainly reduces its possibility.

Level 2 Evaluation – Learning

To assess the amount of learning that has occurred due to a training programme, level two evaluations often use tests conducted before training (pretest) and after training (post test).

Assessing at this level moves the evaluation beyond learner satisfaction and attempts to assess the extent students have advanced in skills, knowledge, or attitude. Measurement at this level is more difficult and laborious than level one. Methods range from formal to informal testing to team assessment and self-assessment. If possible, participants take the test or assessment before the training (pretest) and after training (post test) to determine the amount of learning that has occurred.

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 20 Level 3 Evaluation - Transfer

This level measures the transfer that has occurred in learners' behaviour due to the training programme. Evaluating at this level attempts to answer the question - Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner? For many trainers this level represents the truest assessment of a programme's effectiveness. However, measuring at this level is difficult as it is often impossible to predict when the change in behaviour will occur, and thus requires important decisions in terms of when to evaluate, how often to evaluate, and how to evaluate.

Level 4 Evaluation- Results

Level four evaluation attempts to assess training in terms of business results. In this case, sales transactions improved steadily after training for sales staff occurred in April 1997.

Frequently thought of as the bottom line, this level measures the success of the programme in terms that managers and executives can understand -increased production, improved quality, decreased costs, reduced frequency of accidents, increased sales, and even higher profits or return on investment. From a business and organizational perspective, this is the overall reason for a training programme, yet level four results are not typically addressed. Methods for Long-Term Evaluation include:

 Send post-training surveys  Offer ongoing, sequenced training and coaching over a period of time  Conduct follow-up needs assessment  Check metrics (e.g., scrap, re-work, errors, etc.) to measure if participants achieved training objectives  Interview trainees and their managers, or their customer groups (e.g., patients, other departmental staff)

Kirkpatrick, D. (1998) Evaluating Training Programs: The Four Levels.

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 21

Curriculum Evaluation Comparison of the roles of the internal and external moderators/verifiers as a part of curriculum/programme quality control

Internal Verification External Verification Job Specifications Job Specifications Appointed by awarding body

Occupational competence Occupational competence Need `V1' unit for NVQs Need V2 unit issued by awarding body

Can be marker/assessors, but not IV their Not in their familiar area, conflicts of interest must own work be declared

Need knowledge of using current IV Attend regular meetings for currency Suitably systems and procedures and needs of the qualified and experienced in the area awarding body documents and syllabus/standards of award

Responsibilities Responsibilities Provide support, feedback and guidance Be familiar with the syllabus/standards of awards for marker/assessors

Identify and support training needs for Reports to and receives advice from the awarding assessors body (principal verifier or head office)

Set up and complete an IV system in Provide support, feedback and guidance for accordance with the awarding body marker/assessors regulations Check that assessor/markers are working Make approval visits to organisations wishing to to a common standard become an accredited centre

Sample marked work (All learners/ units/ Make visits to approved centres who wish to offer methods) to check standard and to give further qualifications feedback to the assessors Hold standardisation meetings to share Make routine verifier visits to centres to check that ideas of good practice with assessor team, all aspects of the centre delivery is `safe' (H/S, information from the awarding body equal ops, procedures, paperwork, recording systems etc) Check that the recording of all results and Interview candidates, assessors, verifiers, (observe enrolments are completed assessment or IV taking place in NVQs)

Check that any action plans from the EV Discuss ideas for improvement and make SMART are implemented in the timescale agreed. action plans for the centre to work through

Ensure that the equal ops, anti- Maintain records for each centre as required by discriminatory policies are in place and awarding body. upheld. Maintain safe and secure storage of the Check that formative and summative assessor students' work verification is taking place

Record and deal with any appeals from the Check dates for enrolments, certification and learners using centre systems timescale of the, awards

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 22 Competence assessment

1 2 3 4 5 Once a person is competent in an Competence is very short lived and area they are always competent in needs constant practice - you can be that area competent one minute and not competent the next Competence indicates the ability to True competence indicates a high level perform a skill with basic levels of of creative skill which only occurs after safety and achieving a reasonable years of constantly refined practice outcome (if you pass your driving test you are therefore a competent driver) There are different levels of Competence is competence - you are competence - you can be more or either competent or you are not – there less competent are no half measures A competent performance does not A competent performance requires a require a large amount of large amount of underpinning underpinning knowledge knowledge A true standard of performance A true standard of competent which could be called competent performance can be accurately written only exists in the mind of the person down and communicated to the learner assessing the performance and cannot really be communicated to the learner It is possible to teach a person to be True competence can be only be totally competent at a job acquired by the individual practising the activity for themselves Competence can be accurately Competence can be accurately assessed by a few observations assessed by a large number of observations To accurately assess competence To accurately assess competence you you have to be extremely skilled in really only need to know what the the activity yourself performer has to achieve A performance without thorough A performance can be competently understanding of all of the possible carried out even if the performer implications cannot be seen as doesn't really know all of the competent implications True competence requires significant True competence requires the individual creativity performer to repeat the actions which they have been taught accurately Within an occupational area you can To be truly competent you must be be extremely competent in one job able to perform all related jobs to a and totally incompetent in another, high level although the jobs may be related True competence cannot really be True competence can be easily assessed since it requires accurate assessed if performance criteria are performance in all possible determined conditions True competence can be True competence can only be demonstrated in simulated conditions demonstrated in a realistic work if they are appropriately testing environment with all of the pressures that that entails

Based on The Semantic differential Osgood C & Tannenbaum P (1957) The Measurement of Meaning University of illinois press

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 23 Curriculum Evaluation - Less Reputable Evaluations

Consider the following ‘less reputable’ types of evaluation identified by Patton (1981) - do you recognise any of them? Where might these be used in education or training?

Less reputable types of Where they might be used - and prevented!! evaluation

Quick-and-Dirty Evaluation

Weighty Evaluation

Guestimmate Evaluation

Personality-Focused Evaluation

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 24 Pseudo-evaluation’ identified by Suchman (1967)

Pseudo-evaluation Where they might be used - and prevented!!

Eyewash Evaluation

Whitewash Evaluation

Submarine Evaluation

Posture Evaluation

Postponement Evaluation

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 25 Bennet’s Seven Curriculum Issues

Thinking about evaluation

There is a saying in education –‘if it moves teach it; if it doesn’t move evaluate it’. There is a danger of evaluating for evaluation sake. Always consider the following:

1 What is the evaluation trying to achieve – what am I doing it for?

2 What would be appropriate indicators of success? If you are evaluating something then you must have some idea of a standard or benchmark which means that the thing evaluated is successful or not. If you evaluated a new car you would have ‘success criteria’ eg speed, comfort, in-car facilities, petrol consumption, visibility, price, look etc (NB consider these factors related to a car which ones are efficiency factors and which ones are effectiveness factors)

3 How should data be gathered and processed? What evidence will I gather to evaluate a lesson -–observation; discussion with teacher, discussion with learners, homework, later class work etc

4 With what can the results be validly compared? You may not expect the results of say an ‘A’ level session with students who have struggled to gain a couple of GCSEs grades C/D to be the same as the results of an ‘A’ level undertaken with learners who have gained a number of grade A GCSEs.

5 What other information is necessary to put the results into context?

6 What conclusions can legitimately be drawn?

7 What action follows?

Reference Bennet N et al (1994) Improving Educational Management through Research and Consultancy Chapman

Task What do you understand by Bennet’s 7 points? Make notes on each point and think about these when you come to your programme evaluation.

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 26 Curriculum Evaluation – Gathering Data

What evidence can we use in evaluating courses and curricula?

QUANTITATIVE EVIDENCE [ so-called robust evidence]

 enrolments & patterns of enrolments (gender, ethnicity, widening participation etc.)  attendance & partial attendance  course completion and retention  achievements and public outcomes  evidence of progression to further courses  evidence of progression to employment  three year trends for any of the above  institutional comparisons for any of the above (e.g. attendance)  national benchmarks with other institutional sectors (6th-form, tertiary, general FE etc)

How many of these would be useful to you in your course evaluations? Would you be able to get access to this information?

QUALITATIVE  QUANTITATIVE EVIDENCE [evidence from human sources that you can turn into quantities if you chose to do so]

 evidence from the learners themselves  evidence from employers  evidence from parents  evidence from assessors [markers, examination boards, accreditation bodies]  evidence from moderators, internal and external verifiers  evidence from observers of teaching: formal teaching observations [including those by inspectors, consultants and ‘critical friends’] and more informal peer observations and team-teaching  evidence from the course team and the teachers and themselves

How many of these would be useful to you in your course evaluations? Do you use any of these at present? Which, for your purposes, would you consider the most valuable?

Nevo’s 10 Major Issues in Curriculum Evaluation David Nevo (1986) has attempted to clarify the meaning of evaluation by identifying 10 questions that represent the 'major issues addressed by the most prominent evaluation approaches in education'

1 How is evaluation defined? Educational evaluation is a systematic description of educational objects and/or an assessment of their merit or worth.

2 What are the functions of evaluation? Educational evaluation can serve four different functions:

(a) formative (for improvement); (b) summative (for selection and accountability); (c) political (to motivate and gain public support); and (d) administrative (to exercise authority).

3 What are the objects of evaluation? Any entity can be an evaluation object. Typical evaluation objects in education are students, educational and administrative personnel, curriculum, instructional materials, programs, projects, and institutions.

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 27 4 What kinds of information should be collected regarding each object? Four groups of variables should be considered regarding each object. They focus on

(a) the goals of the object; (b) its strategies and plans; (c) its process of implementation; and (d) its outcomes and impacts.

5 What criteria should be used to judge the merit of an object? The following criteria should be considered in judging the merit or worth of an educational object:

(a) responding to identified needs of actual and potential clients; (b) achieving national goals, ideals, or social values; (c) meeting agreed-upon standards and norms; (d) outdoing alternative objects; and (e) achieving important stated goals of the objects.

Multiple criteria should be used for any object.

6 Who should be served by an evaluation? Evaluation should serve the information needs of all actual and potential parties interested in the evaluation object ('stakeholders'). It is the responsibility of the evaluator(s) to delineate the stakeholders of an evaluation and to identify or project their information needs.

7 What is the process of doing an evaluation? Regardless of its method of enquiry, an evaluation process should include the following three activities:

(a) focusing the evaluation problem; (b) collecting and analysing empirical data; and (c) communicating findings to evaluation audiences.

There is more than one appropriate sequence for implementing these activities, and any such sequence can (and sometimes should) be repeated several times during the life span of an evaluation study.

8 What methods of enquiry should be used in evaluation? Being a complex task, evaluation needs to mobilise many alternative methods of enquiry from the behavioural sciences and related fields of study and utilise them according to the nature of a specific evaluation problem. At the present state of the art, an a priori preference for any specific method of enquiry is not warranted.

9 Who should do evaluation? Evaluation should be conducted by individuals or teams possessing

(a) extensive competencies in research methodology and other data analysis techniques; (b) understanding of the social context and the unique substance of the evaluation object; (c) the ability to maintain correct human relations and to develop rapport with individuals and groups involved in the evaluation; and (d) a conceptual framework to integrate the above-mentioned capabilities.

10 By what standards should evaluation be judged? Evaluation should strike for an optimal balance in meeting standards of

(a) utility (to be useful and practical); (b) accuracy (to be technically adequate); (c) feasibility (to be realistic and prudent); and

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 28 (d) propriety (to be conducted legally and ethically).

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 29 Empiricist v Rationalist approaches to the curriculum

For centuries and throughout the world, education has been dominated by either the Empiricist tradition (ie physically – or empirically - observing by the senses) or the Rationalist tradition(recognising humanities intellectual abilities which may not by observable).

England & the USA have had a strong empiricist tradition (eg Hume & Dewey) and curricula have tended towards the empiricist idea with evidence for achievement being extremely important (for examples portfolios of evidence in NVQs

Europe has tended towards the rationalist tradition (eg Voltaire) and therefore baccalaureate type curricula with value placed upon reasoned argument have been strong

Consider a curriculum with which you are involved. Plot it against each of the empiricist- rationalist criteria by ticking a box.

1 2 3 4 5 The Empiricist Tradition The Rationalist Tradition

Proof by evidence of the Proof by logic & senses eg portfolios of understanding eg rationally evidence! argued essays!

Behaviourist Principles Gestalt Principles linking [lessons based on objectives ideas to form patterns and outcomes!] [experiential reflective learning] Constant Reinforcement Stimulus to gain Insight [Extrinsic motivations!] ‘Eureka’ [intrinsic motivations!]

Mastery Learning Conceptual Learning [Competence assessment] [assessment of understanding]

Ability demonstrated by Ability demonstrated by performance explanation/ critical evaluation

Motivation assumed to be Motivation assumed to be External (Extrinsic) Internal (Intrinsic)

Lots of direct instruction (the student centred learning teacher as expert informing the (learners creating their own unknowing learners) understanding

NVQ & GNVQs /key skills type Academic/recreational type programmes programmes

Lots of Internal and external Internal and external verification of assessor’s examinations of students’ marking understanding

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 30 Change - the five categories of product adopters:

Innovators – venturesome, educated, multiple info sources; Early adopters – social leaders, popular, educated; Early majority – deliberate, many informal social contacts; Late majority – skeptical, traditional, lower socio-economic status; Laggards – neighbours and friends are main info sources, fear of debt.

Rogers E (1962) Diffusion of innovations

Rogers stated that adopters of any new innovation or idea could be categorized as innovators (2.5%), early adopters (13.5%), early majority (34%), late majority (34%) laggards (16%)

Reading List

Bennet N, Glatter R and Levacic R (1994) Improving Educational Management through Research and Consultancy Chapman

Hopkins, D. (1993) A Teacher’s Guide to classroom Research OUPress

Kirkpatrick, D. (1998) Evaluating Training Programs: The Four Levels. 2nd ed.

Osgood, C. & Tannenbaum, P. (1957) The Measurement of Meaning University of illinois press

Stenhouse, L. (1975) An Introduction to Curriculum Research and Development Heinemann

Stufflebeam, D. (1971). Educational Evaluation and Decision Making

Stufflebeam, D.L. Madaus, G. F. & Kellaghan, T. (Eds.), (2000) Evaluation models (2nd ed.). Boston: Kluwer Academic Publishers.

Exeter PGCE (PCE) - Curriculum Evaluation Handbook: Page 31