Making Waves A better future for assessment
December 2020
Loic Menzies | Abi Angus | Sam Baars | Will Millard Bart Shaw | Ellie Mulcahy | Kate Bowen-Viner Alix Robertson | Iesha Small | Anna Trethewey Loic Menzies, Abi Angus, Sam Baars, Will Millard, Bart Shaw, Ellie Mulcahy, Kate Bowen-Viner, Alix Robertson, Iesha Small, Anna Trethewey Pearson is the world’s learning company with more than The Centre for Education and Youth (CfEY) is a ‘think and 22,500 employees operating in 70 countries. We provide action-tank’. We believe society should ensure all children content, assessment and digital services to learners, and young people receive the support they need to make a educational institutions, employers, governments and other fulfilling transition to adulthood. partners globally. We provide the evidence and support policy makers and We are committed to helping equip learners with the skills practitioners need to support young people. they need to enhance their employability prospects and to succeed in the changing world of work. We believe that We use our timely and rigorous research to get under the skin wherever learning flourishes so do people. of issues affecting young people in order to shape the public debate, advise the sector and campaign on topical issues. Find out more about Pearson at www.pearson.com/uk. We have a particular interest in issues affecting marginalised young people.
Find out more at https://cfey.org/ or [email protected]. Contents
List of abbreviations 5 2.1.3 Policy-level drivers of innovation 17 3 The wave makers 24 2.1.4 Workforce drivers of innovation 17 Foreword 7 2.2 How do people shape their solution 3.1 Innovation 1: Heathfield Community and decide what to do? 18 College Assessment Innovation Team 25 1 Introduction and methodology 8 2.3 What approaches are being taken? 18 3.2 Innovation 2: Isaac Physics at 2.3.1 Improving the way assessment Rickmansworth School 39 1.1 Why Making Waves? 9 information is used to provide 3.3 Innovation 3: ImpactEd at 1.2 The Making Waves story 10 feedback and inform teaching 19 Bengeworth Primary Academy 43 1.3 Methodology 10 2.3.2 Providing more meaningful or 3.4 Innovation 4: Shine at Eltham Hill School 55 1.3.1 Recruitment 10 valid information 20 3.5 Innovation 5: Eedi at the Academies 1.3.2 Selection 10 2.3.3 Maintaining and enhancing Enterprise Trust (AET) 62 1.3.3 Research framework 11 teacher and pupil autonomy 20 3.6 Innovation 6: Laser conversations at 1.3.4 Fieldwork 12 2.4 How do people put their ideas the Midland Academies Trust 73 1.3.5 Analysis and synthesis 12 into practice? 20 3.7 Innovation 7: Curriculum and assessment 1.3.6 The advisory group 12 2.4.1 Dissemination mechanisms 20 reform in British Columbia, Canada 81 2.4.2 Flexibility and rigidity 21 3.8 Innovations 8 and 9: Online assessment 2 What have we learned? 14 2.4.3 External partnerships 21 resource banks in New Zealand and 2.4.4 Facilitating factors and Victoria, Australia 100 2.1 Why are people making waves in barriers to success 22 assessment? 15 2.5 Reviewing and adapting 23 4 Conclusion 114 2.1.1 Pedagogical drivers of innovation 15 2.1.2 Cultural drivers of innovation 16 Appendix: Research framework 118
4 A better future for assessment Making Waves List of abbreviations
ACER Australian Council for Educational Research MAT Multi-academy trust AET Academies Enterprise Trust NAPLAN National Assessment Program – Literacy and Numeracy AIT Assessment Innovation Team NDP New Democratic Party ARB Assessment resource bank NFER National Foundation for Educational Research ASCL Association of School and College Leaders NZCER New Zealand Council for Educational Research BC British Columbia OECD Organisation for Economic Co-operation and Development BCTF British Columbia Teachers’ Federation PAT Progressive Achievement Test CfEY Centre for Education and Youth PACT Progress and Consistency Tool CPD Continuing professional development PISA Programme for International Student Assessment CPL Collaborative professional learning SEND Special educational needs and disabilities CSP Classroom support professional SIMS School Information Management System DAL Digital Assessment Library SLT Senior leadership team EAL English as an additional language SSAT Schools, students and teachers network EEF Education Endowment Foundation TLR Teaching and Learning Responsibility FNESC First Nations Education Steering Committee VCAA Victorian Curriculum and Assessment Authority GCSE General Certificate of Secondary Education VCE Victorian Certificate of Education GELP Global Education Leaders’ Programme GLA Greater London Authority HoD Head of Department ICT Information and communications technology INSET In-service training IT Information technology
Making Waves A better future for assessment 5 Pearson and The Centre for Education excessive workload and a feeling that support and Youth (CfEY) are delighted to have for learners’ progress has been pushed to once again renewed our partnership, the side. Teachers and school leaders report through the Making Waves study. that this has been exacerbated by an absence of sufficient training on assessment, whether Making Waves is an important piece of as part of initial teacher training or through research that is being published at an historic continuing professional development. time for education assessment. The Covid-19 Foreword pandemic and the closure of schools in ‘Testing the Water’ showed that workload March 2020 resulted in the government is one of the factors that stand in the way taking the radical decision to cancel A Level of staff taking control of and developing and GCSE examinations. Instead, students’ assessment practices. However, as public examination grades were calculated schools reopen after lockdown there through centre (or teacher) assessed grades. is an opportunity for colleagues to join the practitioners featured in this report Many countries already rely much more in re-evaluating school practices that heavily than the UK on teacher assessment at fail to contribute to learning. Many key moments in learners’ educational careers. schools are looking for alternative ways And, whilst it took a global pandemic to force of capturing progress, and deliberate such a radical shift in this country, the merits refining of practices has the potential to of different types of assessment and their usher in a better future for assessment. benefit to learner progress has long been debated. Never fully resolved, the debate has This report encourages colleagues to wrestle resulted in a myriad of assessment theories with the seemingly intractable problems of and practices but as Pearson and CfEY’s 2017 assessment in education. Pearson has always report ‘Testing the Waters’ showed, this been committed to developing teachers’ has left teachers feeling overwhelmed by assessment expertise and ensuring that the
6 A better future for assessment Making Waves learner is front and centre. It is therefore encouraging that the ‘Making Waves’ project has found that teachers are taking the thorny issue of assessment into their own hands and trialling different approaches and models.
We hope that the ‘wave makers’ in this report empower educators to draw on their professionalism and experience in order to experiment with new forms of assessment. We encourage colleagues to draw on Making Waves’ key findings regarding the vital role of taking a structured approach in doing so.
The etymological root of “assess” reaches to the idea of ‘sitting by’ someone as they learn. At its heart, assessment is being, metaphorically, ‘alongside the pupil or student’. What follows from this is on one hand the identification of misconceptions and celebration of insights, and on the other, making judgements about ‘standards’. ‘Making Waves’ provides important Marry Myatt Rod Bristow Chair of the Board President, Pearson UK & insights from practitioners and academics Global Online Learning about moving discussions about both The Centre for of these priorities to the next level. Education and Youth Pearson
Making Waves A better future for assessment 7 1 Introduction and methodology
8 A better future for assessment Making Waves 1.1 Why Making Waves? Making Waves
Workload in schools has spiralled out of control and this is pushing teachers out of the profession at an unprecedented rate. Teachers do not feel confident about their assessment One in fiveclassroom teachers would expertise and this can stand in the way of them not know where to look for information exercising their professional autonomy. on assessment if they needed it. Only one third of classroom teachers feel However, pockets of innovation are springing 'very confident' conducting assessment as up in response to these two challenges and our part of their day-to-day teaching. previous research1 showed that professionals were developing new approaches that could give rise to a better future for assessment. These innovations need to be properly studied so that lessons on two key questions can be learned, and good practice scaled up. The need for training is greatest at Under half of teachers received the chalkface: classroom teachers are training in assessment as part of their These questions are: less likely than their more experienced initial teacher training and teachers colleagues to have access to ongoing access to assesment training over the professional development in assessment course of their careers is far too limited 1. How can we reduce teachers’ assessment workload? 2. How can we make sure teachers have the assessment expertise they need? The Making Waves Project was designed to many of the challenges in the education answer these questions in a consultative, sector are in the minds and classrooms deliberative manner through partnership of professionals. We therefore embarked 1 Millard, W., Small, I. and Menzies, L. (2017) Testing the with the sector. together on a mission to find out what we Water: How assessment can underpin, not undermine, can learn from the teachers, schools and great teaching, London: LKMco and Pearson, available at: www.lkmco.org/wp-content/uploads/2017/11/ The Centre for Education and Youth (CfEY) countries that are already ‘making waves’ Testing-the-Water-Final-Report-WEB.pdf and Pearson believe that the answers to win assessment.
Making Waves A better future for assessment 9 1.2 The Making Waves story
Christmas 2017: The CfEY and Pearson publish Testing the Waters – a landmark report setting out the biggest challenges for assessment in England. The report brings together the views of thousands of teachers, pupils and parents as well as a number of international studies to suggest potential ways forward.
Easter 2018: We decide to focus on two of the biggest challenges to emerge from Testing the Waters, namely workload and teacher expertise. 1.3 Partners, SSAT (the schools, students and teachers network), the Education and Training September 2018: We launch our ‘Are you Methodology Foundation, the Education Endowment Making Waves?’ campaign, asking teachers, Foundation (EEF), Pearson and CfEY. schools and school groups to share what they 1.3.1 Recruitment are doing to tackle these two key issues. Crowd sourced responses were supplemented Potential innovations were ‘crowdsourced’ by with international insight from Alex Beard and November 2018: Having received disseminating information about the study and the International Education Policy Community nominations for over 40 different innovations, speaking to experts in the field, with links to a of Practice (Teach For All), Dylan Wiliam, our expert advisory group helpedselect 10 short survey asking educational institutions and Tim Oates and Sally Brown (Cambridge innovations or ‘waves’ for us to study over the teachers for information about their innovation. Assessment), Joe Hallgarten (Education course of the next year. The crowdsourcing campaign was promoted Development Trust), Sam Freedman (Ark Spring 2019: The first round of through the following networks and International), Ben Durbin and Lesley Duff fieldwork begins. publications: Schools Week, Teacher Toolkit, (National Foundation for Educational Research NAHT, the Association of School and College – NFER), Amelia Peterson (Harvard University) Autumn 2019: The third and final round of Leaders (ASCL), NASUWT, BAMEed, WomenEd, and Lucy Crehan. fieldwork concludes and final analysis begins. the HeadTeachers RoundTable, Challenge
10 A better future for assessment Making Waves 1.3.2 Selection two innovations at each ‘level’ (individual 1.3.3 Research framework classroom or practitioner, school, group of The recruitment phase yielded a long list of schools and country or jurisdiction). This Since the study focuses on the emergence 44 innovations, which were summarised and yielded a final list of ten innovations. Some of and implementation of new approaches, shortlisted based on the following four criteria: these were eventually unable to participate we designed a framework based around a or had to leave the study over the course theoretical framework involving five elements.2 1. Relevance: The proposed ‘wave’ is clearly of the year (for example, where a school’s This framework is typically used to understand focused on addressing the workload associated circumstances or leadership changed or policy processes but applying it to the different with assessment and/or teachers’ assessment where an individual teacher left the school or tiers of innovation (teacher/school/school expertise (their knowledge, understanding or profession). In this scenario, where possible, group/state) provided a unifying structure. skill in relation to assessment). similar alternatives from the shortlist were The five elements were: 2. Newness: The ‘wave’ has recently been selected as substitutes. This process led us to implemented (or is about to be implemented), focus on nine innovations which we describe 1. Agenda setting: What were/are the ‘pain allowing us to study the process of in section 3 as ‘The wave makers’. points’ that this innovation sought to respond implementing it and the changes that are to? Who and what influenced this agenda? • Innovation 1: Heathfield Community taking place. Why was it considered an important agenda College Assessment Innovation Team to pursue? 3. Plausibility: There is a plausible evidence • Innovation 2: Isaac Physics at base or rationale for how the ‘wave’ will lead Rickmansworth School 2. Formulation: How was a solution arrived to the intended change. In particular, is there a • Innovation 3: ImpactEd at Bengeworth at? Who and what fed into shaping it? considered rationale for the approach, that is, a CE Academy 3. Decision making: What solution was clear answer to the question ‘Why do you think • Innovation 4: Shine at Eltham Hill School selected? Who was involved in deciding on this will work?’ Does this answer clearly build on • Innovation 5: Eedi at the Academies this approach to addressing the pain points? or go against an established evidence base? Enterprise Trust (AET) Why were other solutions rejected and this • Innovation 6: Laser conversations at the approach preferred? 4. Balance: The ‘wave’ allows us to study a Midland Academies Trust 4. Implementation: What mechanisms good balanced basket of innovations – that • Innovation 7: Curriculum and assessment have been used to implement the approach? is, it is not too similar to/takes a different reform in British Columbia, Canada approach from other waves. • Innovations 8 and 9: Online assessment 2 These are drawn from the key stages in the policy The advisory group then met to review the resource banks in New Zealand and process (Howlett, M. and Giest, S., 2013, ‘The Victoria, Australia policy-making process’, in Araral, E., Fritzen, S., shortlist. The aim was to secure at least Howlett, M., Ramesh, M. and Wu, X. (eds) Routledge Handbook of Public Policy, London: Routledge).
Making Waves A better future for assessment 11 Who are the different actors involved? To stages of the policy cycle (agenda setting thematic analysis, identifying and combining what extent is the reified solution consistent and formulation), while later visits looked recurring themes and linking these to quotes with the planned/intended solution? What in more detail at the subsequent stages and examples. This provided the analysis that is obstacles had to be overcome as part (such as implementation and evaluation). presented in section 2: ‘What have we learned?’ of implementation? What conditions contributed to or detracted from effective Field researchers used the overarching implementation? framework to tailor semi-structured 1.3.1 The Advisory Group interview scripts and focus-group Membership of the advisory group evolved 5. Evaluation: How is success or failure being discussion guides to each setting. In over the course of the project as members’ characterised? What if any new success criteria doing this they took into account: is the innovation being held to, compared to roles and availability changed. Participants its originally stated goals? What approaches • institutional circumstances and practicalities met four times to review emerging are being used to evaluate the innovation’s • who the actor was, for example whether themes, identify emerging questions and success? Is the innovation believed to be the focus was on an individual practitioner, ensure lessons arising would be useful to achieving its intended objectives? an overarching multi-academy trust a range of audiences and stakeholders. (MAT) or a set of policy makers The following individuals participated: • what the innovation was and 1.3.4 Fieldwork 1. Eleanor Andressen, Senior what it sought to achieve In most cases, fieldwork took place through Researcher Pearson • emerging themes over the course of the three, in-person visits over the course of year (such as identified challenges). 2. Loic Menzies, Chief Exec, The Centre the year, one each term. However, in some for Education and Youth cases, flexibility was needed – for example Wherever possible, pupils were included where a case study (like Rickmansworth among the research participants. Interviews 3. Debra Rutley, Head of Aspire School) was added at a later stage due to and focus groups were recorded and Alternative Provision attrition or, in the case of New Zealand and transcribed where practical. Attributed Victoria, Australia where it was considered quotes were checked with participants. 4. Simon Yates, Special School Head teacher better to conduct two visits combining the two jurisdictions, one at the start of 1.3.5 Analysis and synthesis 5. Amelia Walker, National Director of 2019 and one in the autumn of 2019. Strategy for Ormiston Academies Trust Fieldworkers wrote up their studies using a Each ‘mini-study’ was structured around common structure based on the five-stage 6. , Director of School the same framework (see the Appendix). James Zuccollo cycle. The lead author then conducted Workforce, Education Policy Institute The first visit generally focused on the early
12 A better future for assessment Making Waves 7. Nick Brook, Deputy General Secretary of the NAHT
8. Kate Chhatwal, Chief Executive, Challenge Partners
9. Mark Lehain, Former Director of Parents and Teachers for Excellence
10. Darren Northcott, NASUWT
11. Allana Gay, Deputy Head teacher
12. Leora Cruddas,. Chief Exec, Confederation of School trusts
13. Julian Astle, Former Head of Education at the Royal Society of Arts
14. Tom Middlehurst, Head of Policy and Public Affairs, SSAT
15. Olly Wimborne , Senior Researcher, Royal Society of Arts
16. Clare Sealy , Former Head, St Matthias Primary, Tower Hamlets.
Making Waves A better future for assessment 13 2 What have we learned?
14 A better future for assessment Making Waves A vanguard of teachers, 2.1 “This is not about curriculum and schools, groups of schools and disciplines being the end destination. Why are people making The end destination is the process of countries is working towards waves in assessment? becoming as a human being. It’s about a better future for assessment human development that’s the end Innovation does not happen without reason, destination. The confidence around We have spent a year studying so the first stage in understanding new communication, around critical thinking, their journeys. approaches to assessment is to pin down the creative thinking, around social and reasons for moving beyond the status quo. emotional wellbeing, around personal The road has not always been easy, identity. That’s our destination.” The innovations we studied responded to and the story that emerges is not multiple drivers. Some of the time these were Maureen Dockendorf, one of simple, transferable solutions. framed in positive ways – a desire to find new Superintendent of Literacy and Numeracy, However, we hope that by telling and better pedagogical models or to empower Ministry of Education, British Columbia these wave makers’ stories we will students to learn independently – but in most help educationalists in schools and cases they were more responsive, seeking governments to rediscover the to tackle perceived pain points, such as an This was often linked to dissatisfaction with professional agency needed to take unmanageable workload. summative assessment, which many feel to be back control of assessment and ensure insufficiently focused on monitoring progression it better serves pupils’ needs. The four main drivers of innovation were and mapping out next steps. This is a key agenda pedagogical, cultural, policy-based and in Victoria, Australia and in New Zealand. We hope the challenges taken on by the workforce-driven. teachers and educations we met will In England, teachers, schools and MATs are resonate for professionals across the often dissatisfied with the quality of assessment. system, and provide a taster of some of 2.1.1 Pedagogical drivers Many want to see a closer link between the the avenues they might pursue when of innovation information that assessment provides and tackling assessment challenges. the actions it prompts. This makes some of In the countries we studied outside of the UK, the innovation that is currently taking place a We argue that this challenge involves not assessment innovation was often seen as part of natural continuation of longstanding efforts to just finding ‘the right solution’ off the a shift towards new or alternative pedagogical move towards ‘assessment for learning ’. 1 shelf, but also planning implementation approaches – for example, developing a cross- carefully, reviewing progress regularly curricula approach that prioritises independent 1 Black, P. and Wiliam, D. (2005) Inside the Black and adapting where necessary. learning and ‘holistic’ non-academic outcomes. Box: Raising standards through classroom assessment, London: Granada Learning.
Making Waves A better future for assessment 15 This led some innovators to develop tools for described in the Midlands as “data for data’s Many were looking for a more collaborative rapid feedback on pupils’ misconceptions or sake” and in Victoria as “a lot of time (being) approach with more pupil or parent what they had and had not been remembered, wasted ‘weighing the pig’ and producing data involvement, or with teachers and managers so that they could reshape their teaching based that is not reliable, valid or comparable”. working together. on the information they gleaned. This challenge Of course, none of the innovations we studied 2 was partly linked to teacher expertise in that are the first attempts to solve these problems. The innovators we spoke to often valued some teachers were said to lack the subject In fact, rejection of previous failed initiatives inclusion highly, and wanted to ensure their knowledge, or assessment expertise, needed to often spawned a desire for something new and approach to assessment reflected this. optimise the relationship between assessment drove innovation. Sometimes this was based on an overall desire and the next steps for their teaching. to ensure that assessment helped identify Typical reasons for failure included several of and respond to pupils’ individual needs rather those explored in greater length in this report’s than being a tool for accountability. At other predecessor, Testing the Water,3 such as: “Because children give you the wrong times, teachers wanted to ensure assessment answer but they think it’s the right one... • excessive cost took into account non-academic priorities. A it’s having the subject knowledge as • labour-intensive approaches that provided desire to respond to specific groups’ needs a teacher to think, ‘Why do you think limited information of value was also a key consideration, for example that’s right? What’s going on in your • approaches that tried to serve multiple indigenous groups in British Columbia in head? What’s your misconception competing functions (such as predicting Canada, Victoria in Australia and New Zealand, that we need to unpick?’ And that is performance as well as guiding teaching or pupils with special educational needs and where the poor subject knowledge and intervention). disabilities (SEND). [from teachers] means that that child doesn’t necessarily move forward.” 2.1.2 Cultural Some contexts provide particularly fertile ground for innovation. This could be seen Maths lead drivers of innovation in schools or countries that reported an Innovators were keen to reflect their overall interest in, or appetite for, innovation Lack of assessment expertise combined with classroom, school or country’s culture in their in and of itself. Some schools, for example, extensive data collection could lead to what was approach to assessment. described themselves as particularly engaged in research, and policy makers in British 2 Millard, W., Small, I. and Menzies, L. (2017) Testing the 3 Millard, W., Small, I. and Menzies, L. (2017) Testing the Columbia referred to the province’s desire Water: How assessment can underpin, not undermine, Water: How assessment can underpin, not undermine, great teaching, London: LKMco and Pearson, available great teaching, London: LKMco and Pearson, available to be at the cutting edge of educational at: www.lkmco.org/wp-content/uploads/2017/11/ at: www.lkmco.org/wp-content/uploads/2017/11/ innovation and reform. Testing-the-Water-Final-Report-WEB.pdf Testing-the-Water-Final-Report-WEB.pdf
16 A better future for assessment Making Waves 2.1.3 Policy-level 2.1.4 Workforce “[The innovation needed to be something] that I could embed very drivers of innovation drivers of innovation easily into my practice and not have to then spend any extra time doing National shifts in policy can drive innovation at Our 2017 report, Testing the Water,4 showed anything. I think anything that is going school level; The Assessment Innovation Team that assessment is often linked to excessive to be effective, in the classroom, at Heathfield Community College was partly set workload, and many of the innovations we does need to consider that because up in response to the government’s decision studied responded to an urgent imperative to you can’t maintain it otherwise.” to move to ‘assessment without levels’ whilst reduce teacher workload. In other cases, a key the approach taken by the college’s Biology design principle was that any change should not Jess Gilespie, Heathfield department was shaped by a perceived increase add to workload. Innovators also sometimes in the rigour of the curriculum. Meanwhile, distinguished between ‘worthwhile’ and ‘less Additionally, some schools and MATs selected changes in the Ofsted framework, SEND code worthwhile’ workload, arguing that what their approach in response to struggles of practice and ‘national agenda’ influenced mattered was ensuring teachers spent less time in securing the workforce they needed, Bengeworth CE Academy’s priorities. on unproductive or meaningless activity. This particularly in subjects like maths and physics. sentiment was echoed by our advisory group. Technological solutions might therefore be one way around the assessment challenges “[Previously] I don’t think the focus was staffing difficulties can cause. on inclusion as it is now in the national “In the long term, it’s going to alleviate agenda. I think it’s really become more the workload of teachers because pivotal… inclusion drives the agenda they’re producing those provision maps “The main thing was the lack of stability of the school, its CPD [continuing and going back into them afterwards in the teaching staff, because there professional development], its vision and collating them. So, [the platform were lots of changes, you would have and its values should be inclusive… That’s we use provides] a dropdown menu a lot of people coming in, and you become high importance in Ofsted and and it will produce it for you.” wouldn’t know what assessment had taken place, you wouldn’t know how other branches of national agendas.” Head of Inclusion well the students had done. You had no real overview. So, from leading a department point of view, I just needed Policy drivers are also linked to school funding, 4 Millard, W., Small, I. and Menzies, L. (2017) Testing the which can push schools to pursue value Water: How assessment can underpin, not undermine, to get a handle on good feedback.” for money. great teaching, London: LKMco and Pearson, available at: www.lkmco.org/wp-content/uploads/2017/11/ Head of Physics Testing-the-Water-Final-Report-WEB.pdf
Making Waves A better future for assessment 17 2.2 extended it, or applied it in different contexts Context shapes options in many ways, or cases. Examples of this include building making some innovations opportunistic or How do people shape an ‘Assessment Innovation Team’ based on a ‘bounding’ the options that can be rationally their solution and decide school’s existing successful use of Innovation selected. Online assessment platforms in Teams focused on other themes; or identifying Victoria, Australia and New Zealand, for what to do? a practice within one school in a MAT and example, were shaped by the status quo, implementing it across different settings, as including who owns data, platforms and tools. Once innovators spot an issue with the Midland Academies Trust did. On the other Meanwhile in British Columbia, some argued assessment, they formulate their response and hand, the curriculum lead in one MAT described that decisions were limited by the ‘Overton decide on a solution. A simple, rational model an approach designed to help weigh up Window’, that is, the range of policies that of decision making would be to imagine that alternatives through “thoughtful disagreement”: are politically acceptable at any one time. a range of different options are assembled, weighed up and the best bet selected. Ultimately, decisions are made by individuals However, this is not what happens in practice. “We thrashed it out with the team. and groups of individuals. At Heathfield We always encourage absolute open Community College, the fact that the A range of individuals are involved in debate and disagreement. We call Head of Geography was also the school’s shaping and making decisions and they have it thoughtful disagreement... we ask research lead influenced their approach, contrasting priorities, different lenses to look people to play devil’s advocate… it’s while the Photography teacher’s background at issues through and limited time. Describing not about the loudest voice at winning as a sports coach shaped theirs. This their decisions as ‘non-rational’ would be the arguments, it’s about the most means that who has power – in terms of unfair, but it is clear that their choices are compelling reasoning and logic.” both their degree of autonomy and their not always a case of simple optimisation. background and beliefs – shapes decisions. MAT curriculum lead Alternatives are not always considered and weighed up. When teachers, schools and 2.3 countries innovate in assessment, they often Research and learning from others also played a proceed, step-wise, from one initiative to role in directing innovators to certain solutions, What approaches another, either building on the previous but this was not based on extensive searches are being taken? approach and taking it to the next level, or through a broad range of literature. Instead, trying and rejecting different approaches certain high-profile researchers like Daisy The innovations we studied were chosen from sequentially. This is not necessarily a bad Christodoulou, Dylan Wiliam and Becky Allen a pool of around 40 different initiatives. These thing. In many cases, innovators identified an were repeatedly mentioned, while others drew clustered around four broad and overlapping approach that was already working well and ideas from their close peers. approaches, or types of innovation:
18 A better future for assessment Making Waves 2.3.1 Improving the way “I keep a continual eye on the information that I get from the 1. Reduced or simplified assessment information is website about how many questions approaches to marking used to provide feedback and they’ve attempted, and I use this as 2. Education technology a way of just keeping an eye on how inform teaching much they’re doing in their own 3. New approaches to assessing time, independent learning at home, The innovations we studied did not focus on non-academic or ‘soft’ skills and also as a way of seeing whether accountability or qualifications; instead, they or not they’ve had particular issues 4. Programmes of training and sought to improve teaching and learning. development seeking to enhance with a certain style of question.” The first way they tried to do this was by teachers’ assessment expertise. Head of Physics shortening the feedback loop between gathering assessment information and making adjustments to teaching. This often involved On the other hand, homework completion As we looked at our selected innovations combining technology and homework. rates and technology cannot always be more closely, it became clear that the three depended on so relying on homework and underlying themes behind assessment Gaining immediate information about technology can mean teachers do not get the innovation were: misconceptions was often what teachers information they need to shape their teaching. found most valuable about these approaches, Some schools within the Academies Enterprise since it allowed them to respond straight away. Trust (AET) have therefore diverged from the 1. Improving the way assessment initial plan by conducting assessments in class information is used to provide “What we’re trying to do is basically or on paper. The Biology teacher within the feedback and inform teaching allow teachers to be better informed Assessment Innovation Team at Heathfield so they can provide better instruction, have integrated assessment tools into lesson 2. Providing more meaningful or allow students to understand – not starters as well as homework. valid information just that they’re wrong, but why they’re 3. Maximising teacher and wrong, and for parents to understand Several teachers who submitted ideas to our pupil autonomy. where their child is struggling and crowdsourcing exercise described developing how they can support them.” Google Form surveys that allow them to get auto-marked, at-a-glance information about Eedi central team their class’s understanding through starter activities. This demonstrates how individual
Making Waves A better future for assessment 19 teachers can take the initiative in developing benchmarking progress against these. Meanwhile, another way of maintaining teacher autonomy, solutions, even where they do not have access Rosie Osborne, Lead Practitioner at Eltham Hill in that teachers have the freedom to select to commercial platforms. In contrast, in New School, wanted to give feedback to pupils in her questions themselves. They are therefore Zealand and Victoria, Australia, it is the state after-school group in a way that has real-world designed to support teachers’ use of high- that has taken the initiative in developing validity and which boosts pupils’ confidence. This quality assessment materials, without dictating assessment platforms. These provide feedback is why she uses a non-teacher (a professional teachers’ focus. that can flag pupils’ misconceptions as well artist) to provide pupils with informal feedback. as mapping out progress, while informing 2.4 summative assessment. These roles are 2.3.3 Maintaining and particularly important in these countries’ How do people put their contexts where there is a loose and un- enhancing teacher and pupil ideas into practice? prescriptive curriculum which leads some autonomy teachers to call for further guidance. Implementing an innovation involves taking an Assessment can often be a top-down affair, idea, putting it into practice and (in most cases) 2.3.2 Providing more and teachers and pupils sometimes resent disseminating or rolling it out. Approaches to prescriptive or rigid systems. Many of the doing this vary depending on how the initiative meaningful or valid information innovations we studied were explicitly designed is taken beyond the initiator, how much to avoid this. The photography department flexibility and adaptation there is along the way, Not everyone is convinced of the value of at Heathfield Community College decided to and the role of external partners. assessment that only includes academic use exemplar work, giving pupils and teachers outcomes. The most radical manifestation of guidance on the features of high-quality work this agenda which we studied was in British 2.4.1 Dissemination – without overly shaping what they produce. Columbia. There, pupils self-assess in three Core Meanwhile, the Assessment Innovation Team mechanisms Competencies, and traditional assessment has model at Heathfield is itself designed to largely been done away with. Most approaches to dissemination can be promote and maintain teacher autonomy by characterised as a form of ‘cascading’, in other giving individual teachers and departments the Within England, several innovators introduced words, they start with an initial node – which power to shape assessment according to their initiatives that extended what was assessed might be an individual, group or organisation – priorities and disciplines. and how, but did so alongside more traditional and are then widened out from there. Selecting approaches. Bengeworth CE Academy did this Assessment banks like those used by the and deploying an approach to cascading is using ImpactEd – a platform that provides tools online platform Eedi and the assessment therefore a critical element of implementation. for assessing non-academic outcomes and for resource banks (ARBs) in New Zealand are
20 A better future for assessment Making Waves Both of the MATs we studied had spread an 2.4.2 Flexibility and rigidity ‘golden core’ of their approach so they can approach out from a central team to their establish non-negotiables. schools. In the larger MAT, the central team Innovators did not tend to take their idea and disseminated their new approach through roll it out as a finished product, or expect it 2.4.3 External partnerships conferences, while in the smaller MAT, to be implemented with total fidelity to the dissemination worked in both directions, with original plan. In some cases, flexibility was built Innovators rarely work on their own. Even the ‘laser conversation’ approach originating in in through piloting, for example Bengeworth CE individual teachers implementing small- one school before being adopted by the central Academy trialled ImpactEd with a small number scale initiatives drew on expertise and team and being spread to other schools. The of academic and non-academic interventions resources from beyond the school, whether a context of being a small MAT made this easier before adopting it more fully. In other cases, professional artist in one case or an established and much of the dissemination happened flexibility has been a more organic process technological product combined with a informally, although all schools were expected of evolution and adaption, with challenges network of other teachers brought together to follow a set timetable and heads were identified throughout and modifications made through a WhatsApp group in the case of required to use the new approach in order to in response. While this approach may be well Rickmansworth School. feed information to the central team. suited to developing a new approach, it can make it harder to ‘test’ the effectiveness of an In some cases, external partners provided a tool As we will see below, the degree of flexibility approach, as the Biology teacher at Heathfield or platform whereas in others they provided given to schools as part of this has important Community College explained: expertise and training, or a combination of both. implications and, in both cases, the MATs have since provided further guidance and direction The Victorian and New Zealand governments or plan to provide this in the future. used external specialist agencies to develop “You’ve got to have a bit of time to their assessment platforms and these agencies Where innovations were introduced at measure. So I only did it for a term, also used their high-level expertise to train and school level, cascading happened between which is an extremely small amount of develop teachers so that they could develop a departments (as in the case of Heathfield), while time, but you’ve then got to have the high-quality offer. at state level (in British Columbia) cascading consistency and the patience really, to took place from state to districts and in turn just keep with that, across a year.” AET and Bengeworth CE Academy both to schools. Flexibility was built into British emphasised the importance of a close Columbia’s model with districts’ approaches relationship with their external provider, since Furthermore, a flexible approach can mean that differing in a number of ways. This raises the this allowed them to co-develop the platforms, innovations stray from their original intentions important question of flexibility versus rigidity - adapting and tailoring them to needs and and innovators therefore need to identify the another key factor in implementation. responding quickly to challenges. It also meant
Making Waves A better future for assessment 21 they were able to secure significant support limited assessment expertise or excessive innovation because they could see that and training, which helped overcome many workload often increased individuals’ resistance students were more engaged in their learning of the challenges involved in implementing a to an innovation and made it harder to thanks to their after-school projects. She technological solution. Meanwhile, Heathfield implement the innovation successfully. created the conditions for this recognition by Community College sourced expertise from a maintaining an open-door policy so that senior university and – much like the training provided Innovation can also be expensive, and it leaders could drop into sessions to see her by the New Zealand Council for Educational was clear that several initiatives were highly work in action. Similarly, results from a mini Research (NZCER) and the Victorian vulnerable to changes in resourcing. Ultimately, control-group trial are making it easier for Curriculum and Assessment Authority (VCAA) if an innovation is to be successful, it needs the Biology teacher at Heathfield Community – this empowered teachers by giving them the to be adequately resourced – in terms of College to persuade others of her approach. assessment expertise they needed to become funding, training and freeing up teachers’ time Her decisions about evaluation are therefore assessment innovators. for an extended period. It is for this reason contributing to successful implementation. that AET now plans to award a Teaching and Meanwhile at Bengeworth CE Academy, the 2.4.4 Facilitating factors and Learning Responsibility (TLR) payment for Eedi leadership team considered discontinuing champions and why the Midland Academies their use of ImpactEd rather than rolling it barriers to success Trust is keen to find time-savings in other areas out, but because Rachel Seneque (the Head of (such as ‘tick and flick’ marking) to free up time Inclusion) was able to show long-term potential As explored above, effective dissemination, for laser conversations. for financial savings, they eventually backed getting the right balance between flexibility deeper and broader implementation. and standardisation, relationships with external Alignment providers and the availability of training can As we saw earlier, innovations often gain all affect how successfully an innovation Securing and maintaining backing from traction when ‘the stars align’ and several is implemented. Beyond these factors, a a coalition of supporters makes it easier different agendas, held by different individuals, number of practical considerations can also to ensure an innovation continues to be come together, or when an innovation fits contribute to or detract from successful prioritised by all those involved. Developing particularly well with a specific context. implementation, as can the degree of and making a strong case for an innovation is Innovators therefore found that securing buy- alignment between stakeholders. therefore an important part of implementation. in from leaders at different levels as well as Practicalities from parents and pupils could oil the wheels of In some cases, innovators did this by implementation. In contrast, implementation Innovators had to overcome a number of demonstrating impact; Rosie Osborne (Lead often suffered when one group struggled to practical issues such as homework completion Practitioner at Eltham Hill School) reported understand the innovation or where sceptics rates and access to technology. Additionally, that senior leaders trusted her to run her could not be persuaded of its value.
22 A better future for assessment Making Waves Sometimes alignment was created between • improved use of evidence a formal evaluation, which innovators feared different, otherwise unrelated pedagogical • changes in pedagogy – in particular, better might create rigidity prematurely. Gathering agendas such as parental engagement, use of assessment information feedback was therefore considered a more raising attainment, curriculum and improved • improved non-cognitive skills pressing priority and many innovators favoured assessment. In others it arose from • shifts in mindsets and educational approach informal assessments of whether pupil work complementary political agendas, something • inclusion and engagement were improving through - that could require lobbying or adapting to a • better relationships between what Heathfield’s Photography teacher called changing political context in order to maintain teachers and pupils. - “over-the-shoulder assessment”. long-term support, as happened for a time with the ARBs in New Zealand. Apart from the biology department’s Others decided that usage of a tool was a Assessment Innovation Team at Heathfield, very good proxy for whether an initiative was In contrast, problems can arise where there few of the innovators we studied evaluated effective. For example, in New Zealand, one are clashes between different agendas their initiatives formally. interviewee argued that the ARBs did not or individuals as well as where remits are need to be evaluated because their popularity overlapping or confused. It is also harder to was sufficient evidence of success. In large “I’m a scientist, so maybe it was just my maintain alignment when there are frequent MATs like AET, gauging success required natural instinct to think, ‘well actually I do changes of staff. triangulation between various different sources need something to compare this to’ but of information. These included a usage tracker, I was just in a really fortunate position, book scrutiny by regional curriculum leads, as 2.5 that I had a natural control in the sense well as upward cascading of feedback from that I had the two parallel groups.” Reviewing and adapting schools’ maths leads via regional leads all the Jess Gilespie, way up to the central team through the school We did not generally find a tight link between science mini project engagement lead. There is clearly a compromise the impact innovators originally intended to needed between flexibility and rigorous have and the indicators of success that they approaches to gauging efficacy, but Heathfield’s It might be argued that this means initiatives referred to when describing their approaches approach – with small-scale innovation taking will not lead to their intended changes and to evaluation. Notions of success included: place at departmental level, with planned but that this will go unnoticed. On the other hand, flexible evaluation – may offer a model that many initiatives were at an early stage where • improved academic results and progression balances these two competing priorities. • pupil and parental engagement processes and approaches were still being • uptake and usage of a platform or tool refined. This meant that rapid and flexible • reduced workload or time-savings adaptation was often a greater priority than
Making Waves A better future for assessment 23 3 The wave makers
24 A better future for assessment Making Waves 3.1 confidence relating to assessment, and it was Different AIT ‘mini projects’ have focused on evident that practice was often driven by different combinations of pain points. For Innovation 1: habit rather than evidence of best practice. example, in biology, teacher and member Heathfield Community Information from work scrutiny also suggested of the AIT – Jess Gilespie – was keen to find that workload was an issue, and that teachers techniques that would improve teaching and College Assessment were spending time on assessment that learning without increasing workload, because Innovation Team was not always delivering value due to poor she felt that if an assessment innovation validity, accuracy or utility for students. increased her workload then it would be more difficult to maintain and embed. Although Jess’s 3.1.1 Introducing the Experimentation with Flightpaths showing primary focus was to improve teaching and innovation pupils’ current and anticipated progress learning, she acknowledged that workload was trajectories had not provided a solution to an important consideration: these issues. Caroline Barlow, the school’s Head, Heathfield Community College is a local and Tom Flower, the Deputy Head, therefore authority maintained secondary school in identified assessment as a priority area in the “[The innovation] would be something rural East Sussex, catering for around 1,500 school’s improvement plan and decided to take that I could embed very easily into my pupils in Years 7 through to 11. The college targeted action on assessment by establishing practice and not have to then spend any also has a sixth form with around 400 an Assessment Innovation Team (AIT). students. ‘Innovation Teams’ are a central extra time doing anything. I think anything that is going to be effective, in the feature of how Heathfield delivers its school The AIT set out to tackle four main ‘pain points’: improvement plan and develops its staff. classroom, does need to consider that Teachers are encouraged to participate in 1. The quality and consistency of the data because you can’t maintain it otherwise.” fortnightly working groups that focus on underlying judgements about pupil progress Jess Gilespie, specific areas of the school improvement 2 The consistency and utility of processes for science mini project plan. The Assessment Innovation Team in capturing and reporting on assessment data this study is one of these working groups. 3. The extent to which assessment was driving effective feedback, and 3.1.2 Agenda setting therefore supporting pupils’ learning 4. Teacher workload: work scrutiny A consultation with teachers at Heathfield had revealed overly burdensome revealed gaps in teachers’ knowledge and assessment and feedback policies.
Making Waves A better future for assessment 25 3.1.3 Formulation The school’s research lead drew on evidence from key thinkers like Daisy Christodoulou, Heathfield has used Innovation Teams before The overall aim of the AIT is to make Dylan Wiliam and Becky Allen to identify some to address other areas identified in the college sure there is sufficient capacity principles for a more effective approach to improvement plan such as boys’ achievement and expertise in the school to assessment across the school, and these and literacy. Innovation Teams typically have improve assessment and make it features of effective assessment practice fed a two to three year life-span. They function more effective. It is headed up by into the design of the mini projects. as action learning sets, with reviews of the the Deputy Head and the school’s evidence base followed by pilot work and a research lead, and 3.1.4 Decision making review, which then leads to the formulation of consists of: longer-term school-wide actions. Why and how was the decision made to go with the AIT? Whole school development Innovation Teams are an established structure for driving school improvement at Heathfield. Regular meetings as an AIT to share findings, develop a ‘shared Therefore, once assessment was identified as a understanding’ of assessment, and priority area in the school’s improvement plan, Mini projects formulate a school-wide strategy the Innovation Team approach seemed a logical, Based in individual departments Develop all staff expertise tried-and-trusted route to follow. While the Focused on 1) consistency/ Bette- informed school decision to take this approach was centralised quality of judge ments, 2) decision making within the senior leadership team (SLT), the capture and reporting, 3) pupil outcomes, 4) workload Innovation Team model involves a high degree Individual CPD of staff involvement, with teachers from across the school being invited to participate. For Assessment Lead Programme training with the ‘Evidence Based instance, individual members of staff were Education’ programme (Deputy given a significant degree of autonomy over the Head and Research Lead) formulation of their mini projects. Teachers taking in part in online training Mini projects
The AIT approach involves a series of different ‘sub-innovations’ within different subject areas.
26 A better future for assessment Making Waves Here, we explore the origins and development of some of those mini projects.
Photography: improving the consistency of teacher assessment
Agenda setting Formulation Decision making
• Results in photography not • Informed by blogs, photography teacher Darren’s • Darren’s approach (student-centred, conversational) consistent between teachers personal experience of ‘what works’ in verbal “seen to work” as he secures the best results formative feedback and in sports coaching • Difficulty moderating • Darren and his team jointly agreed the key in an arts subject • More formal approaches, inc. stamps to show areas for improvement in students’ work, “verbal feedback given,” were unworkable which formed the basis of the project • Deputy Head wanted a creative subject • Combined with using student • The entire team shared a desire to standardise their assessment represented on the AIT exemplar work and templates and feedback practices and were keen to be involved
Geography and history: retrieval/quizzing
Agenda setting Formulation Decision making
• Head of Department wanted assessment • Based on evidence e.g. ResearchEd, • Head of Department is the school’s Research in geography to be more diagnostic and Becky Allen to decide which Lead, and had started a programme of innovation more closely tied to the curriculum techniques to try in assesment before the AIT kicked in • Belief that good assessment will help disadvantaged • Also trialling other techniques • Students are aware they’re part of a trial pupils by improving working memory deficits experimentally e.g. rank order assessment and understand why this is taking place
Making Waves A better future for assessment 27 English: comparative judgement
Agenda setting Formulation Decision making
• English marking is subjective, department wanted to • Trialled different methods and read • Tried using comparative judgement to mark mock know that their marking was both accurate and useful a range of research on assessment papers and it was felt to have worked, but was before settling on specific innovation time-consuming to ‘clean’ and prepare the data • Marking takes a lot of time, desire to reduce teacher workload • Attended a seminar and trialled • Decision was taken by the department to use the ‘no more marking’ software comparative judgement in future, but not using the • There had been some cases where exams had not been and saw it as ‘the future’ system/software. Instead to manually compare whole marked accurately and the school appealed the grades given paper scripts in different pairwise combinations
Science: interleaving
Agenda setting Formulation Decision making
• New science GCSE course • Considered using retrieval practice (and is • A relatively junior teacher has been leading this innovation, requires more depth of knowledge using this a little) but wanted to try something with support from the school’s Research Lead than previous course different to other members of the AIT • The teacher leading this innovation has been doing so • Department given marking template last • Relevant and current research on interleaving relatively independently from the rest of their department year, resulting in confusion over what in STEM (science, technology, engineering • The absence of any significant additional workload was being marked and for what purpose and maths) subjects was easier to find was a key deciding factor in selecting this innovation
28 A better future for assessment Making Waves 3.1.5 Implementation confidence, apply this learning in their school, The AIT’s work concluded at the end of the and lead other teachers to do the same. academic year and the team’s work now sits Implementing the AIT within individual departments to take forward. Teachers at Heathfield were given an open The AIT was led by the Deputy Head and the invitation to join the AIT, which met fortnightly Comparing innovations school’s research lead, who undertook the throughout the school year during timetabled We have focused on two mini projects in Assessment Lead Programme delivered by hours. With support from the Deputy Head and detail: using exemplar work and in-person Evidence Based Education. The Assessment Research Lead, teachers identified their mini feedback in photography; and interleaving Lead Programme is a year-long programme projects and developed plans for evaluating and homework in biology. The projects overlap of continuing professional development disseminating their work, for example through in some areas. For instance, both have some (CPD), which supports teachers to develop in-service training (INSET) days. They were degree of focus on raising attainment, and both their assessment knowledge, skills and also provided with online assessment training. appear to have had a positive impact on their Access to expertise and support from intended outcomes. However, the projects senior staff within the AIT was seen as differ in other ways. For instance, the biology vital, particularly in the planning stage: project was driven by a single member of staff, while the photography project was delivered collaboratively by a team. Meanwhile, the biology project has been formally evaluated using an experimental research design, whereas “If you can have discussions with outcomes of the photography project have other staff and people like Mark, who been assessed on a more anecdotal basis. is a research lead, or people like Tom [Deputy Head] who are in those The following table compares the two projects’ positions of expertise… then you implementation side-by-side. This illustrates can plan something that is doable, the different ways assessment innovations can manageable and actually can really be implemented, even when they sit within change the way that you do things.” the same overall programme within the same school. It also exemplifies how teachers can Jess Gilespie, tailor innovations to their own priorities, science mini project expertise and subject areas, when they are given the freedom and support to do so.
Making Waves A better future for assessment 29 Phase Photography: using exemplar work and in-person feedback Biology: interleaving homework
Initial implementation • Realised that creative work can still have a core • Designed as a trial in two parallel Year 9 classes of non-negotiable ‘quality markers’ • One group has been set homework that is • Began by using the best student work to identify these related to the topic they have just studied; the non-negotiables for presentation (e.g. digital sketchbooks); other group has homework based on content what does it need to look like in order to get a Grade 9? they covered two or three topics back • Defined a set of criteria that teachers and students • Question bank software has allowed the teacher to can agree on, but left room for creative ‘flex’ set homework of equivalent difficulty, from the same • Developed resources/guidance for students based exam board, even though they are on different topics almost entirely on examples of excellent student work, and some of the teachers’ own work Evolution • Went on to create additional resources • Focus and approach has not changed since the outset to address other areas of work • Used the resources to cover common questions, reduce the amount of verbal feedback required, and to allow more time to be spent giving face-to-face feedback on questions that are specific to each student’s work Facilitating factors • The whole department is collaborating on it • Has not required any additional staff, time or resources. • The department was already quite far along in One teacher has implemented the project with some developing new schemes of work, so teachers support from the AIT, particularly around evaluation had these schemes of work to work from • Conducting this mini project under the banner • Students have been supportive and a supportive Head of of the AIT has given it more clout; staff are more Department means there is commitment to the project likely to listen to the findings as a result Barriers • It is a creative subject, and the departmental team • Students were not initially very receptive, but this developing the resources are mindful of the need was predicted and has not proven to be a barrier to avoid them becoming too prescriptive • Rolling out across the department in future might encounter barriers because it involves ‘doing things differently’ and some longer-serving teachers may not be keen to change their practice
30 A better future for assessment Making Waves There has been push-back against innovation approach is still helping his team to deliver groups in Year 10 that were a close match in from some quarters due to the difficulties of more reliable assessment judgements. terms of ability and who were being taught changing established practice. As the Deputy the same schemes and lessons by the same Head put it, “once you let the assessment genie 3.1.6 Evaluation teacher. One group was therefore taught out of the bottle, it’s hard to put it away”. In using the innovation and one was used as other words, old habits and received wisdom in Focusing once again on the biology and a control group to assess the impact: assessment die hard and asking staff to revisit, photography innovations, numerous or change, what they have done for a long time differences emerge in relation to evaluation. can be difficult. Despite this, overall the Deputy Broadly speaking, the biology project took “I’m a scientist, so maybe it was just my Head feels that staff have been open-minded a more structured, data-driven approach natural instinct to think ‘well actually I do about the AIT and happy to acknowledge to evaluation. Meanwhile, evaluation of the need something to compare this to’ but gaps in their knowledge and practice. photography project took into account teacher I was just in a really fortunate position, workload and student engagement as well that I had a natural control in the sense In the English department, the Head of English as attainment, whereas the biology project that I had the two parallel groups.” experienced push-back against the work primarily focused on raising attainment. Jess Gilespie, involved in rank-order assessment. Having science mini project trialled comparative judgement to mark mock papers, he and his team felt the approach “If there was some sort of quantitative measure that we could look to, then that increased the reliability and validity of their Jess is keen to explore additional ways of was something that we used as a key assessment judgements, but that these benefits measuring and evaluating impact and there means of evaluating its impact. And other were outweighed by the increased workload is an assessment the students will sit at the than that we were very happy with any involved in systematically comparing a large end of the year, reflecting on the whole year’s qualitative evaluation comments, either volume of work samples, and preparing teaching. She believes this will be a useful way by members of staff or by students.” students’ work samples for analysis. The to see what impact the innovation has had. department decided on a compromise solution Tom Flower, whereby whole student scripts would be After seeing the positive impact that the Deputy Head compared, in different pairwise combinations innovation is having, Jess is considering without using software to compare individual introducing the innovation with the other class, excerpts of work. This retained the underlying Biology no longer using them as a comparison group, logic of rank-order assessment, but applied exemplifying a common ethical consideration the principles of the approach in a less labour- The biology project was evaluated using a when running control-group trials: intensive way. The Head of English feels this control-group design. There were two parallel
Making Waves A better future for assessment 31 “[We’re] not so keen to carry it [the voice exercise, to gauge student feeling on trial] on because it was having such a particular component of the course. a positive impact on the other group. Last year, Darren and his team developed I thought actually knowing that it’s templates to help students with the research working, I don’t want to then, for another term, hold back that other class.” pages in their portfolios. They are now working on a guide to taking good photographs. Jess Gilespie, Darren acknowledges that photography is science mini project a creative process but also feels that the process can be directed – there are some non- negotiables that all students have to grasp. Photography
The photography innovation was assessed in a less formal manner, relying on continuous “It can be very difficult to get the marks conversations with, and observation of, under [the new] very specific guidelines. So I think what we’ve tried to do in a students. This involved photography teacher creative subject is apply a formula or a Darren Causton assessing pupils’ work “over the set of rules and some of those are non- shoulder”. There are no standardised, academic negotiable which seem for a creative tests in photography, so it is harder to assess subject quite controlling but if you initially students’ work using metrics. Furthermore, understand these non-negotiables and Darren’s discursive approach to feedback allows you apply this key formula then success him to assess the quality of students’ work is going to be a lot easier to come by.” and give direct feedback on how it could be improved. It also allows him to “take the pulse Darren Causton, photography mini project of the room” – to see if students are enjoying the work and feeling motivated by the topic. Darren used this type of student feedback to shape the templates he put together As well as acting as a teaching to support students with their portfolios; resource, this template will also act allowing the resources to focus on the areas as an assessment resource, in the students found most challenging. Occasionally, same way as the research guide. Darren undertakes a more structured student
32 A better future for assessment Making Waves Comparing approaches to evaluation and assessments of impact The table below provides a summary of the different ways in which impact is being evaluated in the two mini projects.
Photography Biology
How is success being • Improved grades • Greater-than-expected progress and attainment characterised? • Allowing teachers to make more effective use of the time they spend giving feedback • Developing students’ love and enjoyment of photography: students feel positive about their work • No specific indicators developed for each of these desired outcomes What does • The department is hoping to see the • Designed as a trial in two parallel Year 9 classes evaluation look like? project’s impact in three ways: • Pupils from both classes sat a common 1. Visual comparisons – does the work baseline test on Key Stage 3 material look better than it did before? • After 1.5 terms, pupils sat a progress test 2. Data – student progress and attainment, taken from formal – this was compared with their baseline assessment; for example, they compared students’ marks on an test score to see distance travelled existing coursework project (before the assessment initiative), • They then looked at predicted Key Stage 4 grades to see with those on a recent exam project (after the initiative) if their progress outstripped their predicted progress 3. Student confidence and independence in the creative process • Study design controls for teacher, class • No formal plan for evaluation beyond size and predicted attainment “discussion and making notes” • Plan to add some student voice to the evaluation next year Unexpected benefits • Some students report early and rapid gains • For Jess as a teacher, it has been an opportunity • Impact is being felt most strongly among middle-attaining pupils to reflect critically on her own practice, and to who might not have sought these resources out for themselves experiment with different ways of teaching Downsides • Some students feel constrained/overly directed, • It has added somewhat to Jess’s workload, and but Darren feels they probably would anyway, this has sometimes been difficult to sustain due to the drive within the curriculum to revisit alongside her other responsibilities completed work and continuously improve it
Making Waves A better future for assessment 33 Photography Biology
How effective • Judged “very successful” based on feedback and work seen • The interleaved class have made considerably more has it been? • Believed to have had positive impact on grades, students’ progress than the control-group class in 1.5 terms understanding of what they are being assessed on, teachers’ effectiveness, quality of lesson resources, students’ positivity about their work, and their grades • Currently these claims are based on anecdotal evidence • Awaiting good results in the form of ‘hard data’ on results day in the summer Longer-term impact • Resources are now written into next year’s schemes • Jess has shared what she is doing with her of work in order to embed the approach department, and more widely through whole- • Good results will make the course more appealing/ school collaborative professional learning (CPL) boost the department’s reputation • The mini project in biology has contributed • The resources will make it easier for teachers to the AIT’s overall momentum by making to communicate what is required of students teachers’ practice more research-informed: in terms of presentation and research • In the long run, having these core resources, based on the mark scheme, will give teachers greater flexibility to support individual students in different ways. By covering basic common queries, the resources will allow teachers to focus their face-to-face feedback on more personal, individual elements of students’ work. This, in turn, supports Darren’s focus on getting to know individual students and tailoring support on the basis of that knowledge • This approach has been transferred to other creative subjects, such as art and design
34 A better future for assessment Making Waves Professional impact Jess has found that experimenting with one Photography assessment technique – interleaving – has acted The assessment innovations in photography as a springboard for experimenting with other Darren feels that the innovation has and biology were both felt to have impacted techniques, such as using ‘hinge’ questions. encouraged him to think about how he on the professional practice of the teachers These are diagnostic tools used when students gives feedback to students; rather than leading the innovations. reach a “hinge” point where teachers need to assessing students at the end of a topic know whether or not students have grasped a and then supporting them with areas of Biology key concept before moving on to the next idea. weakness, Darren uses his templates to deal with problems pre-emptively. The Taking part in the AIT has encouraged Jess Students’ responses provide the teacher with process has reinforced in Darren’s mind to think about assessment as a tool for valuable evidence about what their students that assessment is mainly for pupils’ teaching, rather than a separate activity. Jess know, don’t know and “need to do next” There benefit: assessment is the basis of robust has built the principle of interleaving into is also some evidence that the innovation has feedback, and that feedback will help her teaching, beyond assessment activities had a spill-over effect on other members of students to improve specific pieces of such as quizzes and marked homework. For the science department at Heathfield. Jess has work for examination, but also help them instance, she is increasingly making use of presented her innovation’s results to colleagues to develop broader skills in their subject starter activities in lessons that revisit material in the department, and since then they have area. Overall, running the innovation has from previous weeks, terms and even years. made further enquiries and started using provided Darren with an opportunity to interleaving when setting revision homework reflect critically on who assessment is for. for their Year 11 classes. Next, Jess plans to share the approach with teachers from other schools: “I’m more reflective now on the purpose Despite the additional workload involved in of it and the point of it. So if I’ve got, say, creating the templates, they have allowed an assessment that I’m giving my Year 7s “I’ve been asked to present on it Darren to provide more structured feedback on a piece of work, that I’m going to mark at… INSETs, [where] we get all the and to pre-empt common questions. This and give them feedback on, I’m much local schools together. So I’ve been has, in turn, reduced his workload during more interested to know, actually is that asked to do something on it to the lessons. This is particularly the case during a useful assessment? What am I actually science departments, so I guess the initial stages of the course, where the getting from it and the feedback that I [we’re] rolling it out really and templates allow students to solve problems then give to them, is that actually useful?” getting the word out of how simple and develop their skills more independently. Jess Gilespie, science mini project it is but how effective it seems.”
Jess Gilespie, science mini project
Making Waves A better future for assessment 35 “The workload lesson to lesson is the AIT had encouraged staff to reflect “Improving outcomes for assessment definitely improved because you’ve critically with their own practice and that of just by simply tweaking something like given them a starter for what they need their colleagues, and to consider changes to homework was the desired solution, and to do to make that work successful their practice based on research evidence. that seems to so far have been the case.” from the beginning… initially obviously Jess Gilespie, science mini project creating these resources takes a little bit of time but in the long run, yes “I think it’s made me read a lot more I think it’s had a positive impact on research, it’s made me engage with a Photography managing the lesson and the workload lot more of what gurus, if you like, the within the teaching period.” people that know assessment, are saying.” Students have historically achieved strong results in photography at Heathfield. Darren Darren Causton, Jess Gilespie, science mini project feels the new resources he has designed photography mini project to structure his teaching, feedback and formative assessment have nonetheless The AIT mini projects have enjoyed a high contributed positively to students’ coursework profile, are regularly communicated to all Darren feels the resources have also had and final grades, although this is not based staff, and feature in school-wide CPL day. a positive impact on other teachers’ in the on a formal evaluation of impact. department’s workload. Making the assessment This is partly down to the fact that the AIT criteria and mark scheme clearer, and sharing has been anchored in Heathfield’s school During the focus group we conducted, students this with students through exemplar work improvement plan, and that it benefited explained how the formative assessment and and templates, has reduced the amount of from the input of senior leaders,. feedback they receive in photography differs time teachers spend repeating common from what they experience in other subjects. Pupil impact themes in their feedback. Teachers can They felt that Darren’s approach had two key instead focus their efforts on providing Biology components, which worked well together: students with more individualised feedback. 1. Giving initial, written feedback based The interleaved class have made considerably on template work and mark schemes Across the AIT more progress than the control-group to identify areas for improvement class in 1.5 terms, based on a mid-year Beyond specific mini projects, it is clear that 2. Following this up with more detailed summative assessment. Jess is looking the wider Innovation Team-based approach verbal feedback, to help students forward to using end-of-year exams to to assessment innovation at Heathfield has interpret and understand the written assess impact on attainment after a year yielded professional development benefits feedback, and make improvements that of teaching using the innovation. for staff. Jess, Darren and Tom all feel that fit with their own creative objectives.
36 A better future for assessment Making Waves Students felt the verbal component of Darren’s topics so that students revisit older topics. Photography approach made his feedback more detailed, The availability of standardised question descriptive and individualised. This allows banks can help to reduce the workload Last year, Darren fed-back about the work he them to respond to gaps or weaknesses in involved in constructing new assessments. was doing in photography to other Heads of their work that would lose them marks in their Department at Heathfield. Other staff were final assessment, without sacrificing creative Teachers would need to ensure they continue interested in his approach but it is not clear freedom. Crucially, Darren understands that with the innovation for long enough to that other departments have implemented the objective quality markers provided by cover all topics and see the impact. any of these approaches as a result. Darren a mark scheme and exemplar work need voiced scepticism about the transferability of Jess also described how a school-wide to be translated into a different action plan key elements of his approach, such as providing structure such as the AIT helps in cascading for each student, so their work remains continuous ‘over-the-shoulder’ assessments: learning from innovation to other teachers, personal to them. As one student explained: and argues that it also provides high- level backing which encourages other “I always feel like we’re in a really lucky “I feel like if he didn’t have any connection teachers to listen and experiment. position in the way that we can work with the students because it is very unusual with us and he didn’t really know us The Head of the science department has been as a person, it would be much harder now… I can assess students very quickly encouraging and supporting staff to experiment over the shoulder. I don’t have to read an for him to understand where we’re with the approach that Jess has piloted. She coming from... he knows exactly what we essay or go through the workings out and believes that this is a better approach than the answers as in maths; everything takes want to do. It helps with the examples if she tried to drive the change herself: and ideas of other people’s work.” a bit longer in those kind of subjects.”
Photography student Darren Causton, “Some people have done it their way for photography mini project a very long time and don’t necessarily Replicability and transferability find themselves very open to changing what they do. So I think support in that Despite this, Darren feels that a cornerstone Biology sense, from the Head of Department, of his approach – delivering feedback that is is just crucial to… get you across personalised, and based on an understanding Jess believes her approach to interleaving that awkward blurred line of ‘this is of each individual student – should be homework is easily replicable for different something I’d like you to try’ versus ‘this transferable to all subject areas. Meanwhile, teachers and subject areas, as it simply is something we are all going to do’.” his approach to using templates has now involves reordering teaching and assessment been transferred from photography to
Making Waves A better future for assessment 37 art and design, with Darren supporting While the AIT has now concluded its work, 2. Ensure your innovation has SLT colleagues to adapt the system and resources assessment-related objectives remain within backing and that they: that have worked well in his classroom. the school development plan: a. have put structures in place to support you, for instance with research and Across the AIT evaluation expertise, providing additional “It’s absolutely embedded in our non-teaching time to plan and evaluate, Within the AIT more widely, teachers running improvement plan every year now, and establishing forums to discuss and mini projects have shared their work with staff that we’ve got something relating to share practice with other teachers across the school in meetings and colleagues improving assessment that is there every b. see your innovation as part of a wider have shown interest in their findings. The year. So I think that’s its legacy really.” school supports staff to write blog posts and strategic focus and are willing to give the tips that are shared across the wider team. Tom Flower, Deputy Head innovation adequate time to bed in and The approach seems to be softer than a achieve results, for example by mapping it to school-wide rollout of individual innovations, a theme in the school’s development plan. Teachers who did not take part in mini instead allowing teachers to access and take 3. Discuss your work with other projects as part of the work of the AIT are forward ideas they think are relevant. teachers so you do not miss out on therefore nonetheless experimenting with opportunities to share your experiences This dissemination of learning has had success, new assessment approaches, overseen and receive supportive feedback. with the Deputy Head reporting that, across by senior leaders. Tom is currently 4. ‘Innovation’ does not have to mean a the school, staff knowledge of assessment has working with a group of 20 staff from radical departure from existing practice; improved, resulting in more consistent practice: different curriculum areas to develop their understanding of diagnostic questioning, it might just mean focusing more closely and to experiment with hinge questioning. on what is already working well, codifying “It was the case until recently that practice so others can adopt it, and not necessarily all the staff were Next steps establishing systems for teachers to using the same assessments, which share and discuss what they’re doing. Jess and Darren identified four main is problematic if you’re trying to recommendations for teachers considering The Deputy Head also provided five main compare how students have done. piloting an assessment innovation: pieces of advice to other school leaders: So I think we’ve got better at that.”
Tom Flower, Deputy Head 1. Plan carefully, making sure your innovation 1. Make sure your own knowledge will not impact on your workload, as of assessment is solid first; get the this makes it easier to maintain. sequencing of training right.
38 A better future for assessment Making Waves 2. Use the SLT’s solid knowledge of assessment 3.2 • set pupils’ homework and revision tasks to audit where assessment practice is • encourage pupils to take responsibility strongest and weakest in the school. Innovation 2: for their own revision. 3. Give innovation the time it needs Isaac Physics at to bear fruit and for impact to Pupils are given a ‘block of questions’ to be evidenced meaningfully. Rickmansworth School complete and teachers monitor their progress. 4. Be proportionate when you are evaluating 3.2.1 Introducing the (for example, make use of qualitative “I keep a continual eye on the information as well as quantitative evidence). innovation that I get from the website about how 5. Be aware of your biases and be open- many questions they’ve attempted, Rickmansworth School is a secondary school minded in your expectations of and I use this as a way of just keeping for 11- to 18-year-olds in Hertfordshire. It particular innovations or practices. an eye on how much they’re doing in is a partially selective academy, admitting their own time, independent learning a small proportion of pupils based on high at home, and also as a way of seeing performance in music at the age of 11. whether or not they’ve had particular The ‘Isaac Physics assessment programme’ issues with a certain style of question.” is an online assessment tool available to Head of Physics UK schools and pupils. It assesses pupils’ performance in physics. The tool stores thousands of questions related to different Key Stages and topics in physics. Pupils complete 3.2.2 Agenda setting questions and the programme generates details about their performance and progress. What were/are the ‘pain points’ that this innovation sought to respond to? Rickmansworth School is using the tool with GCSE and A-Level pupils in order to improve The Head of Physics at the school introduced pupil performance in physics and to reduce the idea for the innovation shortly after teacher workload. It is being used to: joining the school. He has a considerable degree of autonomy in the school and senior • help teachers assess GCSE and A-Level pupils’ leaders have had limited involvement. He progress throughout the academic year had previous experience with ‘Isaac Physics’
Making Waves A better future for assessment 39 and felt that the assessment tool would be “The main thing was the lack of stability them to understand what topics pupils are a useful way of responding to three ‘pain in the teaching staff, because there studying and what they are struggling with. points’ he recognised in the department: were lots of changes, you would have This helps them to understand how they a lot of people coming in, and you can support their children with physics. 1. Teacher absence and teacher wouldn’t know what assessment had recruitment problems taken place, you wouldn’t know how 3.2.3 Formulation 2. The need to improve pupil outcomes well the students had done. You had 3. Lack of parental engagement. no real overview. So, from leading a How was the solution arrived at? department point of view, I just needed The solution was developed in two stages: Teacher shortages to get a handle on good feedback.” 1. The Head of Physics assessed the Head of Physics 1. Amidst a national shortage of science areas for improvement within the teachers with a physics specialism, the department when he joined the school. school was struggling to recruit specialist Pupil performance He then drew on his previous experience staff in the subject. Furthermore, a number of using ‘Isaac Physics’ to develop a plan of physics teachers had been on long-term 2. ‘Isaac Physics’ also responds to pupil about how this could be used to respond sick leave because of ill-health, including underperformance. The Head of Physics to the ‘pain points’ he had identified. the Head of Physics himself. This means explained that some pupils “did not do well 2. The Head of Physics attended a course that some staff teaching physics were on their end-of-year exams”. The online run by the developers of ‘Isaac Physics’. supply teachers or not subject specialists. assessment tool is therefore used as a revision This course helped him to familiarise Therefore, the Head of Physics decided programme for underperforming pupils himself with the tool and design a plan for to use the online assessment tool to to complete over the summer holidays. how it could be used and implemented to help non-subject specialists to assess tackle issues in physics within the school. pupils and improve the consistency of Parental engagement assessment. It also allows him to review Who and what fed into shaping it? pupils’ assessment data in periods where 3. Although parental engagement was not he is absent from school due to ill-health. problematic before the innovation began, The Head of Physics shaped, and continues the Head of Physics wanted to harness the to shape how ‘Isaac Physics’ is used in the power of parental engagement to enhance department and the training he has received pupils’ progress and their outcomes. ‘Isaac has helped him make decisions. On the Physics’ allows parents to access data ‘Isaac Physics’ course he met the platform about their child’s performance and helps designers and other teachers using the
40 A better future for assessment Making Waves tool and informal discussions with these 3.2.4 Decision making assessment tools and rejected them because people helped him shape the innovation: questions were not challenging enough. Who was involved in deciding on this approach to addressing the pain points? 3.2.5 Implementation “I went on a week-long course and became, if you like, au fait with all the The Head of Physics was the driving force in What mechanisms have been used various aspects and also meeting the deciding to use the ‘Isaac Physics’ approach. to implement the approach? people behind it was very useful and Other teachers in the department support that’s one of the things that I think about using the tool, largely because of the tool’s Implementation took place in four steps: having remote assessment tools or reputation. As one teacher explained: learning platforms is knowing that you’re 1. Attending a training course: The Head going to have good communication of Physics had detailed conversations with the people behind it so you can “The reason why we found it so useful is with the platform’s designers, and query if there’s problems, if there’s because it not only offers the students other teachers on the course, to learn solutions you don’t understand, if instant feedback, but more importantly more about the platform’s features there’s new things you want to add.” I think it’s quite challenging. Other and how to use it most effectively. online platforms that we’ve tried in Head of Physics 2. Trialling the platform with sixth- the past often have been quite simple form classes: The Head of Physics and ask quite closed questions, but piloted the programme with his sixth- the opportunity within physics and New members of staff have recently joined the form classes because the platform with chemistry to a certain extent as department. Although the agenda is led by the is relevant to the A-Level course. Head of Physics, new teachers are starting to well, is the mathematical side, and shape the way the tool is used. For example, the challenge that that provides to 3. Rolling out the platform to Year one physics teacher has suggested using the our quite high-achieving students 10 and Year 11: After seeing ‘Isaac tool in competitions between students to is actually really, really valuable.” Physics’ work well with sixth-formers, the Head of Department introduced improve enthusiasm for the subject. While this Head of Physics plan has not yet come to fruition, the Head his Year 10s and 11s to the platform. of Physics is taking teachers’ feedback into 4. Introducing the platform to other staff: account while thinking about the future of The Head of Physics did not explore The Head of Department gave short training ‘Isaac Physics’ in the department. other solutions. However, the science sessions to other staff. He demonstrated department has previously used other online how to use the platform and asked teachers
Making Waves A better future for assessment 41 to complete questions as if they were “They have hints and tips on each teachers who use a WhatsApp group to share students. He believes this helped them question, but for some questions, it ideas about how to use the platform effectively: to become familiar with the platform. might just be ‘Oh no, I’ve used the wrong equation’ or something like that, “We have a little WhatsApp group and Not all staff have been using the platform where it would have been really useful occasionally somebody will say, ‘How consistently. For example, some new members just to be like, ‘Have you used this?’ If did you get the answer to this one?’ and of staff have not been collecting data on you get it wrong after a certain amount we will have a conversation about the student progress in the same way. The of tries, it stops you from just redoing best approach to the question… it’s a Head of Department is therefore offering it until you get the right answer.” great learning forum for us as well.” additional training to these staff. He is also Pupil having informal conversations with staff to Head of Physics understand any hurdles they are facing. Furthermore, as noted above, not all staff have To what extent is the reified solution been using the platform consistently and some Having new members of staff join the physics consistent with the planned solution? new members of staff have not been collecting department has made implementation slightly data on student progress in the same way more challenging because these staff need The implemented approach is largely that the Head of Department does. The Head to be trained up and monitored to ensure consistent with the intended plan. GCSE and of Department is therefore having informal they are using the platform consistently. A-Level pupils complete questions and receive conversations with staff to understand their feedback from ‘Isaac Physics’. Teachers also experience of using the platform so that he can 3.2.6 Evaluation find the tool useful for assessing pupils. adapt and add to training. However, teachers How is success or failure being have not highlighted any particular difficulties. However, some pupils cheat when answering characterised? online questions. Other pupils simply What conditions contributed to or The physics department currently define answer a set of questions the day before a detracted from effective implementation? success as: deadline rather than more frequently as was Training for the Head of Physics contributed to • pupils completing assessments as revision initially intended. This limits the innovation’s effective implementation because familiarity • pupils making progress in physics. ability to support pupils’ progress. meant he could plan how to implement ‘Isaac Physics’ in his own department. The What approaches are being used to Some pupils also find that the program’s opportunity to meet other professionals using evaluate the innovation’s success? feedback is not helpful and does not help the platform was also valuable and he has them to progress. As one pupil explained: Teachers of physics, and the Head of Physics, remained part of a community of ‘Isaac Physics’ are evaluating the innovation’s success using
42 A better future for assessment Making Waves quantitative data from the ‘Isaac Physics’ 3.3 of Inclusion, explained the origins of program. They are triangulating this with pupil this focus on inclusion and evidence: progress data and informal comments from Innovation 3: pupils. Teachers are using this to understand: ImpactEd at Bengeworth “I think that since 2014 when the • how many questions pupils CE Academy code of practice was brought in and have attempted to answer new recommendations were made, • which topic areas pupils are struggling with there has been a real change and a • areas where pupils are making progress 3.3.1 Introducing the shift towards partnership work and • whether or not success in GCSE mock innovation creating a truly inclusive culture. We’ve exams correlates with answering a large done a huge amount for inclusion. number of ‘Isaac Physics’ questions. Bengeworth CE Academy is an outstanding I’ve worked nationally, with the local three-form entry primary school in authority, developed myself as a The Head of Physics, and other teachers in Worcestershire, with just over 500 pupils. It local leader of SEND… Historically, the department, will use this information took over a second school in the area in 2018 interventions wouldn’t have necessarily to determine how much teaching and so is now part of a small MAT. Approximately been recorded in a systematic, robust learning time they should dedicate to 15% of pupils at Bengeworth have a special way… there was a need to move ‘Isaac Physics’. This will feed into the educational need (SEN) and around 30% towards evidence-based ones, that department’s strategy for the next academic speak English as an additional language. The showed clear outcomes for pupils.” year as the Head of Physics explained: high proportion of pupils with additional needs at the school has led Bengeworth to focus on creating an inclusive culture. The innovation we studied involves “It’s sort of a cost–benefit analysis of how Bengeworth’s partnership with the organisation much time and effort we devote to it, Although interventions also provide ImpactEd. ImpactEd provides schools with a and what we feel the returns are. So, we academic support for pupils, Bengeworth’s data platform, which allows schools to input feel it’s giving us positive returns at the programme of interventions places progress data and make comparisons across moment, but then we’ve got to decide great emphasis on nurture and support pupils and across other schools. ImpactEd how much time we actually devote.” with non-cognitive skills. The leadership provides schools with information about team are keen for their work to be evidence on different interventions they might Head of Physics evidence-informed and scrutinised for wish to use while building the evidence base for effectiveness. Rachel, Bengeworth’s Head interventions using data from across schools.
Making Waves A better future for assessment 43 ImpactEd offers a variety of scales and “If you don’t get nurture-based 2. It was more difficult to secure funds validated tools to measure progress against interventions correct, then there’s no for and resource effective interventions. different outcomes. Bengeworth has worked point in doing any cognitive-based 3. Teachers and classroom support with ImpactEd to build the collection of interventions to improve academic professionals (CSPs) were not able to tools on the platform that measure non- outcomes because… if a child isn’t choose and implement interventions cognitive skills such as metacognition, self- mentally healthy then they are not based on evidence, leading to low awareness and resilience. These include able to take on new learning.” teacher confidence and a lack of both validated tools and Bengeworth’s own accountability for the effectiveness tools relating to tailor-made interventions. of non-cognitive interventions. Therefore, leaders felt that a focus on Bengeworth’s use of ImpactEd originally evidencing non-academic interventions had Leaders and staff commented on a tendency focused on measuring the impact of non- to come before, or at least be developed in primary schools to allow interventions academic interventions, and this study in tandem with, evidencing academic that are ‘well intentioned’ but ineffective to focuses on this element, although as it interventions. The decision to use ImpactEd continue. However, they felt that this culture evolved, Bengeworth began using the to measure the progress pupils make in needed to change and that a move towards platform to record children’s progress in non-academic interventions was borne robust, measured programmes that support all intervention programmes. The school of a number of pain points, primarily: better outcomes was required. Leaders felt that Bengeworth should lead that change. aims to use ImpactEd to identify where and • a lack of evidence (or lack of accessible why interventions are or are not working evidence) about which interventions Bengeworth staff acknowledge that the so it can make evidence-based decisions. develop pupils’ non-cognitive skills, system they have used in the past meant that resulting in decisions about interventions interventions’ effectiveness was not scrutinised 3.3.2 Agenda setting being made without an evidence base, sufficiently. This old system involved rating • high staff workload and inefficient interventions as red, amber or green by making What were/are the ‘pain points’ that systems for recording evidence. notes on each child’s progress in a paper file this innovation sought to address? Leaders felt there was a lack of evidence and manually inputting data into Excel in order to carry out analysis and discussing progress Staff at Bengeworth feel that children who about which non-academic interventions during half-termly pupil progress meetings. are struggling need support to develop the were effective, both in their school and As a result, some interventions that were non-cognitive skills that then allow them to across the sector. The potential impact not effective were left to go on too long. access learning and make progress in academic of this lack of evidence was threefold: areas, as Rachel, the inclusion lead, explains: 1. Ineffective interventions continued On the other hand, where interventions for too long. are effective, it can be difficult to secure
44 A better future for assessment Making Waves funding and resources without sufficient workload for teachers and CSPs. This previous impact of the intervention in the evidence of their impact. This was a pain point was therefore an important school and the wider evidence base. particular problem for interventions with driver behind the intervention. Prioritising value for money and evidence was non-academic outcomes, even though Why was it considered an important also linked to the climate of reduced funding these were sometimes transformative for agenda to pursue? as this has affected which interventions pupils with very high levels of need. continue and also shaped the school’s choice The school’s inclusive culture and values Previously, decisions about interventions were of system. The use of other systems was partly drove the school’s decision to adopt made based on ‘the status quo’ or a specific stopped due to their cost and ineffectiveness; ImpactEd and this culture was, in turn, driven staff member’s knowledge or experience, rather however, Rachel, the inclusion lead, argued by the high proportion of Bengeworth’s than on the evidence of what was most likely for an investment in ImpactEd, believing it pupils who have additional needs and be effective for the pupil. The Head of Inclusion would save money by ensuring that only Ofsted’s increased focus on inclusion. anticipated that using ImpactEd would make effective interventions were continued. teachers more confident in using assessment data and evidence to select interventions, and “[Previously,] I don’t think the focus was 3.3.3 Formulation would make teachers more accountable for on inclusion as it is now in the national How was a solution arrived at? the interventions they choose for pupils: agenda. I think it’s really become more pivotal… inclusion drives the agenda The senior leadership team at Bengeworth first of the school, its CPD, its vision and heard about ImpactEd through a Challenge “My vision is that it would improve its values should be inclusive… That’s Partners conference. They were already aware teacher confidence in terms of making become high importance in Ofsted and of the workload associated with the existing them more accountable for the measures other branches of national agendas.” intervention tracking system but had not that they’re putting in… and monitoring yet found a solution. Bengeworth’s leaders it more rigorously. From my perspective, Rachel, Head of Inclusion feel that partnerships such as Challenge we’re already doing that as a school but Partners have been useful in pointing the it’s the leadership that are doing it… and Inclusive practice requires interventions way towards ‘tested good practice’. we’re filtering that information back that support pupils in developing their down to staff and empowering them.” Who and what fed into shaping it? non-academic skills (as well as academic skills) and the leadership team felt that Bengeworth became one of 10 Challenge Partner schools that formed a ‘co-design Additionally, although the previous ImpactEd would allow teachers and CSPs group’ for ImpactEd. These schools were system of recording interventions was to make intervention decisions based on very early adopters of the platform and ineffective, it nonetheless involved a high evidence, including both evidence of the
Making Waves A better future for assessment 45 therefore fed into shaping ImpactEd’s measurement tools, some of which were a combination of tracking on the School practice and offer to schools. found to be ineffective or inappropriate for Information Management System (SIMS) Bengeworth’s in-house interventions. These and RAG-rating interventions, but this Rachel, Bengeworth’s Head of Inclusion, was measures were then removed and Natalie was not felt to be working effectively. key in driving the innovation forward during worked with ImpactEd to identify different the first stage of implementation. Over the validated measures or Bengeworth-made Leaders felt that sourcing solutions through 2018–19 academic year, Bengeworth piloted measures to better fit intervention outcomes. collaborations such as Challenge Partners ImpactEd with some academic and non- meant that they could be more efficient academic interventions before deciding to 3.3.4 Decision making by choosing solutions that partner schools roll it out across the school with a focus on have already found to be effective. Also, non-academic interventions in 2019–20. Who was involved in deciding on this given that Bengeworth was one of the approach to addressing the pain points? As it was part of the small group of schools co-design pilot schools for ImpactEd, the that received early access to the platform The decision to use ImpactEd was primarily school has concentrated on shaping and to help shape its development, Bengeworth made by the senior leadership team. The Head refining the tool rather than comparing received considerable tailored support from of Inclusion then drove the innovation forward it to other potential solutions. ImpactEd during its pilot phase. ImpactEd throughout the pilot phase, with support from acted on Bengeworth’s suggested updates the Headteacher, the data and assessment and changes to the platform. Although this lead and the senior leadership team. piloting phase may not be needed by all schools As the innovation was rolled out across as the innovation is scaled, Bengeworth’s the school in the autumn term of 2019, experience highlighted the importance all staff contributed to the decision to of a tailored approach. Bengeworth’s expand the use of ImpactEd to track leaders highlighted that without ImpactEd’s the impact of all interventions, both support they would not have been able academic and non-academic. to make the innovation work for them. Why were other solutions rejected and During the pilot phase, Natalie Snowdon, this approach preferred? Bengeworth’s data and assessment lead, shaped the innovation by working with Other approaches to assessing interventions’ ImpactEd to decide which interventions and impact on non-academic skills, beyond measures would be included on the platform. the systems in place beforehand, were not This involved trialling different impact considered. The previous systems involved
46 A better future for assessment Making Waves 3.3.5 Implementation
What mechanisms have been used to implement the approach?
Bengeworth has implemented this innovation in four stages:
1 2 3 4 Preparing the Small-scale Refining the approach Rolling out the platform pilot and preparing platform to all staff for rollout
When? When? When? When?
2018-19 Spring and summer term 2019 Summer term 2019 Autumn term 2019
Who? Who? Who? Who?
Head of Inclusion and Data Head of Inclusion and Data and Head of Inclusion and Data and All teachers, followed and Assessment Lead Assessment Lead Assessment lead by all CSPs
What? What? What? What?
Working alongside ImpactEd Trial of the platform to assess a Adjusted measurements and All staff received training to input interventions and few interventions on a small scale scales to fit Bengeworth’s on the platform form either corresponding measurement interventions. Prepared whole- ImpactEd or the Head of tools and scales onto staff training Inclusion. All staff then used the platform the platform to assess impact of some interventions
Making Waves A better future for assessment 47 Who are the different actors involved? To what extent is the reified solution This has created greater consistency and consistent with the planned solution? may have increased buy-in from staff as they In the first stage, the main people involved prefer using a single system. However, it may in the innovation were the leadership team, In terms of the platform and the also have reduced the focus on non-cognitive primarily Rachel (the Head of Inclusion) and implementation plan, the reified solution outcomes. It also highlights the difficulty of Natalie (the data and assessment lead) as was consistent with the intended measuring progress in non-cognitive skills. they worked with Ella Knight, Partnerships solution throughout planning, piloting Manager at ImpactEd to ready the platform and implementation. However, there ii) Using tailor-made measures and the tools for rollout across the school. were two differences observed between the initial plan during the set-up phase While ImpactEd is designed to allow The next phase of implementation – initial and the reality of implementation: schools to use validated measures, many, rollout – involved all teachers and some lead though not all, of the tools the school is CSPs who were trained to use ImpactEd. They 1. The platform was used to measure using have been developed by the school began to use the platform to input data on the impact of both non-cognitive and alongside their own interventions. interventions in the autumn term in 2019. academic-focused outcomes, rather than Towards the end of the autumn term, the the original plan of mainly focusing on the This flexibility and the responsiveness of lead CSPs began to train other CSPs to use impact of non-cognitive interventions. ImpactEd in uploading new measures are viewed positively by leaders and staff. Staff the platform. This stage of the rollout was 2. Many measurement tools that have also felt strongly that use of tailor-made key in reducing the duplication of systems been used were developed in-house interventions (especially in lower Key Stages) whereby CSPs were using some of the old by Bengeworth rather than primarily was a crucial element of good practice. tracking systems, such as RAG-rating, while using validated tools and scales. teachers were inputting data onto ImpactEd. However, this change also hampers one of Given that most interventions are delivered i) Measuring the impact of academic ImpactEd’s main aims: allowing schools to by CSPs, it was felt that to truly embed the and non-academic interventions compare the impact of their interventions system, they would need to use ImpactEd The decision to broaden the focus from with nationwide datasets. During the initial and take some responsibility for tracking primarily measuring the impact of non- visits to Bengeworth, the Head of Inclusion data, though ultimately the responsibility for cognitive interventions, to include academic highlighted the need to base intervention monitoring the effectiveness of interventions interventions, was taken for two main decisions on evidence and compare and making decisions will remain with teachers. reasons: first, the school felt there was a Bengeworth’s outcomes with those seen in need for greater consistency; and second, other schools as a major benefit of ImpactEd. it was felt to be more straightforward to This will not be possible if the interventions measure impact on academic outcomes. and measures are school specific. In some
48 A better future for assessment Making Waves senses this suggests that ImpactEd may be Teachers explained that while they were, What obstacles had to be overcome being used with less rigour than intended, as to an extent, using measures and tracking as part of implementation? wider datasets cannot inform decision making. pupils’ progress in interventions before using ImpactEd, the time taken with data entry and Over the course of the pilot year and the On the other hand, ImpactEd is providing analysis meant far less time was available for whole-school rollout, Bengeworth faced an opportunity for the school to monitor evaluative conversations and decisions about several obstacles in setting up ImpactEd. the impact of its interventions more closely pupils’ next steps. Therefore, they felt ImpactEd However, most were resolved with relative when there is no prior evidence regarding was allowing them to scrutinise Bengeworth’s ease. For example, identifying problems their effectiveness (unlike some bought- tailor-made interventions more closely and with the platform or the tool or needing to in interventions that have an established make better-informed decisions for pupils. upload a new tool could have represented evidence base). Therefore, using ImpactEd potential barriers to success; however, to some extent ensures decisions are It is also worth noting that the process of changes were quickly made and were becoming more rigorous. There was also adding new measures exemplifies ImpactEd’s therefore no longer considered obstacles. evidence that this approach to using intense and ongoing involvement in helping ImpactEd had fed into decisions: Bengeworth set up the platform. This was In the school year 2018–19, during a review a crucial facilitating factor behind success. of funding and systems, the leadership “There are really two avenues you can Bengeworth’s leaders praised ImpactEd’s team considered discontinuing their use go down with this. We’ll look at this data responsiveness: often ImpactEd would respond of ImpactEd, rather than rolling it out. This and in terms of the children that were to Rachel’s request for additions to be made decision would have been entirely driven in this top-dog intervention, for those to the platform within a day. At the start of the by funding constraints rather than issues children it’s made a difference to their study, ImpactEd had not decided whether it with the innovation. However, Rachel confidence in class… so we are just going would provide this type of intensive support made a strong case for continuing with to monitor them in class now. But for to all schools in future; however, findings from the innovation and the school decided to this child, this hasn’t worked, perhaps this study and of ImpactEd’s own review of invest in the system in the long term. the pilot demonstrate that it is a key element it wasn’t well suited, it wasn’t right, so During the set-up phase, the leadership of success. Therefore, by the end of this study what is better? What will work? And we team experienced some obstacles, mainly ImpactEd had decided to put partnership at will give them that next. All the while, relating to time. During the implementation the heart of its model and continue with an it gives the overview of ‘Generally is phase, the main obstacles were technical. this intervention really doing what we approach that tailors support to schools. In wanted it to do when we designed it?’” future, there may be some challenges to scaling Setting up interventions and tools on the this level of support but ImpactEd recognises Teacher platform was time-consuming. Although the the importance of this part of its model. leadership team responsible for the set-up
Making Waves A better future for assessment 49 managed to get the platform ready for the ImpactEd believes platform development delivering in-house training which was felt September 2019 rollout, this issue highlights since the pilot started means set-up is now a to be more effective. Though this was an that any school wishing to use the innovation smoother and more straightforward process. in-school technical issue, it demonstrates needs to provide a considerable upfront However, an initial time investment is still the potential challenges of online training. time investment from leaders to secure needed to save time in the long term. long-run reductions in workload. Yet this While this obstacle was overcome, a does seem to pay off. Whilst in September During training in September 2019, staff leader with a good understanding of the 2019 teachers reported having to spend experienced some technical issues. The training platform and capacity to deliver training time getting to grips with the platform, was delivered via video link by ImpactEd; was needed. If other schools wish to roll out as early as December 2019, they began to however, staff experiencing technical issues the innovation without a long pilot phase, experience a reduction in data-entry and or confusion were not able to benefit from they will not have this in-house expertise they anticipated that this would continue. the training because they were not able and ImpactEd must consider how to deliver to ask questions. Rachel addressed this by effective training for those schools.
What conditions contributed to or detracted from effective implementation?
During the pilot phase the main factors that contributed to effective set-up were:
• leaders having dedicated time to prepare the innovation to be rolled out across the school – although time was limited, Rachel and Natalie (the data and assessment lead) were given time to develop the platform
• having on-hand support from ImpactEd to shape the platform according to the school’s needs – this continued to be a key factor of success during rollout
• having a member of the senior leadership team leading the innovation.
50 A better future for assessment Making Waves As the innovation was implemented and Special Educational Needs Coordinator “When you introduce anything new ImpactEd was rolled out across the school, (SENCO) was the ideal person to do this. to staff, with teacher workload being other key factors for success emerged: high on the agenda and the Ofsted • A culture of innovation and embracing framework, you are mindful of that… change was also seen as helpful, though • Ongoing and responsive support particularly with data systems, there can not necessarily crucial, to ensuring success. from ImpactEd was, as in the initial be that preconception that it’s going to There was some disagreement among phase, highly important. On a number be hard and really difficult, especially different staff members as to whether ‘any of occasions, Rachel and Natalie asked with IT. But it hasn’t been like that school could do it’ or whether a culture ImpactEd to make changes to the because of the way it was delivered and of innovation, especially when it comes platform in response to staff feedback. In because they can see the impact of it to new technology, was needed in order December 2019, Rachel reflected on how already because we are getting data.” the rollout could have gone differently to implement ImpactEd successfully. had this support not been available: • Strong leadership and a strategic vision for the innovation was identified Rachel and Natalie also highlighted that as a key factor in ensuring the innovation further development of ImpactEd is needed to “It only takes a few times for people to was used consistently. Natalie, the data ensure long-term use and success, specifically have problems that aren’t dealt with and assessment lead, felt that the system with regard to provision mapping. Currently, for them to lose interest. Teachers would be vulnerable to under-use and ImpactEd does not have a feature to report have a lot to think about and if could be treated simply as a data-entry a summary of all interventions that a single something doesn’t work for them it’s platform rather than as a way of comparing child, over the course of their time at school, difficult to motivate them to use it.” school impact to benchmarks and national has received. Schools are required to produce comparisons and as a way of examining provision maps for children with special the evidence base to support intervention educational needs, which summarise this • The availability of an in-school expert decisions. She felt that the set-up and support and its outcomes. Natalie explained: to deliver training and support to staff was implementation of the innovation had been crucial. Rachel and Natalie’s understanding successful because leaders had identified “Now we are using the program we do of the system, having used it for the whole the area of need, wanted to solve the find we want it to do a little bit more… of the previous year, allowed them to issue without increasing workload and had to show these are all the interventions support staff. Rachel created guidebooks driven the change forward throughout: for staff, which also ensured that new a single child has had, these are staff in the future would be able to use the results, what is the impact.” the platform. Leaders highlighted that the
Making Waves A better future for assessment 51 3.3.6 Evaluation the following three key criteria to assess What approaches are being used to whether the rollout was successful: evaluate the innovation’s success? How is success or failure being characterised? 1. All staff understand ImpactEd’s Before implementation, although Bengeworth’s purpose and potential benefits. leadership team had a clear plan to train and The intention behind using ImpactEd is to 2. All staff involved in running interventions support all staff, as well as clear objectives reduce teacher workload and improve the input pupil data into the system consistently. in terms of what would constitute success evidence used to make decisions about during the implementation phase, they did 3. All staff involved in making decisions about interventions. Originally, leaders envisioned not plan to formally evaluate the innovation’s interventions use ImpactEd data to do so. that achieving these goals would ultimately success beyond gathering staff feedback lead to pupils developing their non-cognitive and reviewing progress. However, collecting Leaders also anticipated that once all skills through effective interventions, staff feedback appeared to be an effective interventions were on the platform, if staff met as The Head of Inclusion explained: way of monitoring the innovation during the above criteria, the innovation would result implementation. Feedback from staff during in reduced workload, despite an initial increase our research interviews was consistent, and in workload as teachers familiarised themselves “Success will look like, every pupil is positive, and leaders demonstrated that with a new system. The previously used process making the most progress that they can they were aware of attitudes to ImpactEd. of RAG-rating interventions – making notes and accessing the correct intervention at Rachel and Natalie also monitored the data on each child’s progress in a paper file and the right time, ensuring better outcomes that was being input in order to check that manually inputting data into Excel in order and that will be being closely monitored. the second ‘success criterion’ – consistent to carry out analysis – was extremely time- It won’t look like children have been in data entry – was also being met. interventions that have made no real consuming. Leaders felt ImpactEd would save impact to their performance, whether time, particularly with data entry and analysis. Is there evidence of success or failure? that be pastorally or academically.” In addition, as part of the initial plan to Interviews with teachers about their focus on non-cognitive interventions, experiences and discussions with leaders This vision was then extended to include one success criterion was that staff who had monitored the intervention’s improving pupil progress in academic would view these interventions and the impact suggested it had been successful. interventions too as the school extended its use need to evidence their impact as on a i) Consistent use and decision making of the platform. ‘par’ with academic interventions. All staff we spoke to had a consistent During the implementation phase, understanding of the potential benefits of Bengeworth’s senior leadership team used
52 A better future for assessment Making Waves ImpactEd and its intended use. All teachers had would receive interventions in the new-year. which to expect pupils to make progress; they received training and were using the system However, most teachers felt that they did not planned to review the interventions they were by December 2019. Lead CSPs had also done yet have sufficient data to make decisions running in spring. While this may be wise, and so and were preparing to disseminate this about which interventions would be run, or some interventions such as ‘Talk About’, which training to other CSPs in the near future. stopped. This may have been influenced in part lasts for a year, are planned to last for longer by a lack of urgency as all interventions showed than a term, the decision to review intervention ImpactEd had fed into decisions about an overall trend of positive progress. Where impact after three or four terms stands at odds which interventions would be run. Data some pupils had not made progress as part with leaders’ original intentions to make quicker collected during the pilot phase had of an intervention, despite the group making decisions about what is and is not working. supported discussions about reducing progress overall, it was generally considered the number of interventions in Key Stage that the pupil should receive the intervention Furthermore, there was not yet evidence of 1, which staff had felt was too high: again before the decision was made to try teachers comparing their pupils’ data with something different. Teachers highlighted wider datasets, often because the measures that one term was a short amount of time in were tailor-made for Bengeworth and therefore “We were reflective enough to know through professional conversations and timetabling that we were struggling to fit the interventions in. What ImpactEd did was give us the data result to support our discussion and say we need to not put so many in because the data is showing that hasn’t had as much impact as it could have had.”
Year One Teacher
Since the whole-school rollout, most interventions had not reached their endpoint but, where they had, teachers had entered data and considered the analysis results. Teachers were clear that they could, and would, use this data to support decisions about which children
Making Waves A better future for assessment 53 there was no evidence base available. Staff were rollout, which was sooner than leaders What are the medium- or long- not concerned about this: they felt that the expected. Teachers found it ‘quick and term impacts expected to be? need for tailor-made interventions outweighed easy’ to enter data and reported that the the need for cross-school comparison. analysis outputs were easy to understand: In the longer term, leaders at Bengeworth Nevertheless, this demonstrates a move away feel ImpactEd will allow them to know which from the original vision of what success with interventions best achieve their intended “It’s much easier than what we were ImpactEd would look like. Leaders felt that outcomes and that this will make Bengeworth’s doing before, you just put it in, click a control groups and comparisons would be practice more effective. Although teachers had few buttons and it gives it to you and the next step in the implementation as it had not used ImpactEd to make decisions to cut or then it allows you to be more forensic.” not happened as quickly as originally planned. change interventions based on the platform yet, Leaders recognised that where they were the pilot had helped to support the decision using their own measures they would have to Staff anticipated that workload would be not to do so, and most felt that ImpactEd develop their own control groups internally. further reduced once they were able to stop would support future decision making. using duplicate systems as some interventions ii) Non-cognitive intervention Staff felt that ImpactEd would improve were still being rag-rated as before while the impact and inclusion CSPs’ confidence in their assessments and changeover to ImpactEd was still in progress. teachers’ confidence in their decision- It was clear that staff did not view ImpactEd iv) Leaders’ knowledge and workload making about interventions. In turn, leaders as a tool specifically for non-cognitive and staff felt that the main underlying impact measurement. Teachers referred ImpactEd has also had a positive impact on purpose of using ImpactEd, like all of their equally to academic interventions and leaders’ ability to understand what is being practice, was to have a greater impact on non-cognitive interventions. Therefore, delivered across the school, to pick up on pupils’ progress and they believed that, by while non-academic interventions retained instances where things are not being delivered supporting better decision making, ImpactEd importance and the school continued and to monitor the impact of all interventions, would have a positive impact on pupils. its culture of inclusion, ImpactEd was forming a holistic picture. Rachel feels this has not seen as an innovation focused on been achieved while reducing her workload: evidencing non-cognitive interventions. iii) Teacher workload “From my point of view and strategically across the school to be able to see There was a clear positive impact on what’s being done has been invaluable… teachers’ workload by the end of the At a glance I can have a holistic view autumn term in 2019, three months after of what everybody has been doing.”
54 A better future for assessment Making Waves 3.4 The pupils use this feedback to revisit why the innovation takes such a different and develop their work, creating a new approach. Rosie wanted to ensure pupils Innovation 4: meme to send back to the artist. As this is received feedback in a way that encouraged Shine at Eltham Hill School an extra-curricular activity, it is not linked them and built confidence, hence using to any particular subject or curriculum, real-life feedback from an artist: 3.4.1 Introducing the although its closest links are to art and English. It has been funded by The Shine innovation Trust and the Stepping Stones Fund from “All of the assessment is meant the Greater London Authority (GLA). to be about giving them feedback This innovation, ‘Shine’, takes place at Eltham that will help them to improve, but Hill School, a single-sex local authority Last year, pupils’ work was displayed in art in a way that is not restrictive.” secondary school in Greenwich, London, galleries such as the Hoxton Gallery, Vinyl Lead Practitioner with an intake of approximately 1,100 pupils. Deptford and Lewisham Arthouse. The lead Shine is in its third year and it targets the teacher organised for the pupils to be on the new intake of Year 7s, while still retaining reception desk at the Lewisham Arthouse for a Historically, the school has had difficulty older pupils who have been attending week and for Saturday trips to the galleries for engaging pupil premium eligible students regularly since they started at the school. the pupils and their families to view the work. and those with special educational needs; Shine is an after-school extra-curricular their attendance was poor and there was a activity that is aimed specifically at pupil 3.4.2 Agenda setting wide gap between outcomes for this group premium pupils; pupils with special educational and the rest of the pupils at the school. The What were/are the ‘pain points’ that this needs; and pupils with mental health issues, school had noticed particular difficulties innovation sought to respond to? although access is open to all pupils. It takes during transition from primary school, so place in an information and communications The innovation was set up to engage the they targeted Year 7 pupils for Shine initially. technology (ICT) room so that all pupils target pupil group in their learning and to Rosie, an experienced teacher and one of have access to a computer and email. build their love of school, or as the teacher the longest-standing members of staff, was who set up the project, Rosie Osborne, The innovation involves the pupils creating appointed Lead Practitioner in the school, a puts it, “working with students who can’t memes and sending them to an artist offsite role that allows her to pilot a series of extra- access the curriculum in traditional ways”. via email through their teacher during curricular groups that target pupil premium the session. The artist gives live feedback Rosie believes traditional forms of assessment students, those with special educational needs in the form of direct comments on the can be a barrier to access – and this is and those with mental health difficulties, work or changes to the original meme. to help them re-engage with school.
Making Waves A better future for assessment 55 Rosie wanted to improve the pupils’ confidence “What we called the D-C borderline Rosie reports that senior leaders trust her and willingness to open up, ensuring they students at GCSE level, which is now because the students she works with are often felt free to express themselves in all lessons. four or five, is often based around more engaged in their learning due to the Rosie therefore set up the innovation to that self-belief, and also things around after-school projects. She has an open-door support the target group by helping them to attendance and behaviour, and literacy policy and senior leaders occasionally drop in to be creative and to ensure they were assessed and oracy, reading and writing… So, I sessions. They have also shared their transition in a way that was less “judgemental. think a lot of those projects that I do interventions, of which this is a part, with a wider address those issues, but in a really fun audience at a national conference held by SSAT It should therefore be noted that while this way that they really get engaged with, (the Schools, Students and Teachers network). project uses innovative assessment methods, and music and art and writing, but in a Why was it considered an the key aim of the project is to develop non- really non-pressured environment.” cognitive skills and engagement with school. important agenda to pursue? Lead Practitioner Who and what influenced this agenda? One senior leader talked about the importance Rosie was appointed Lead Practitioner at the of articulating your vision as a school and Rosie has driven the innovation. She is an school due to the successful work she has done then prioritising projects that help to achieve English and drama teacher as well as the in the past on creative projects. For example, that vision. Eltham Hill’ vision is to extend its school’s Lead Practitioner on the leadership she previously devised a project that involved offer beyond the taught curriculum so that team. An important part of her role involves pupils working with writers from the Royal Court students feel valued and that they belong. developing creative projects for pupils who Theatre in London to craft their own plays that Furthermore, many of the tasks in English find it hard to engage with the curriculum, were performed. The pupils involved in this statutory assessments ask you to write for and to seek additional funding for such project became more engaged and confident an specific intended audience and Rosie felt projects. She developed and leads the in the rest of their school life, so the leadership that students found this hard when they had innovation with an existing artist friend. team wanted to formalise her role to give her no actual input from the “outside world”. the time to lead on future creative projects. Her experience with Creative Partnerships She therefore came to believe that it was earlier in her career led her to believe that The senior leadership team are committed important for pupils to have engagement pupils in receipt of the pupil premium lacked to not becoming an ‘exams factory’ and to and feedback from the outside world. confidence and did not always engage with offering something broader than just the Rosie also believed that the target group of school. Her motivation for starting this taught curriculum. They therefore prioritise students would benefit from having time project was to address these two areas: projects like this, which broaden the outside lessons with set criteria, where curriculum experience. They have taken time they could enjoy “freedom outside lesson to articulate their vision and aims as a school.
56 A better future for assessment Making Waves time, [which] opens it up to some really 3.4.3 Formulation basis. She was working with a small group doing exciting and creative responses”. She felt drama, creative writing and art and when they that if they had the opportunity to feel How was a solution arrived at? started to create memes, she got in touch with freer in a school environment, even outside Previously, in an attempt to secure ‘real-world’ another artist friend of hers, to experiment of normal lessons, then they would feel feedback in English lessons, Rosie asked with assessment through live feedback. more engaged with school overall. students to write letters to authors in the One senior leader said that the intervention This innovation is part of a set of extra-curricular hope they would respond, but they rarely did. was influenced by national evidence on poorer solutions Rosie is working on, all partly funded outcomes for pupil premium students and by the GLA’s Stepping Stones Fund and The After delivering her project with the Royal research by colleagues conducting studies for Shine Trust. One of the other projects, for Court Theatre, she decided to create a new a Master’s degree. This research suggested example, targets a group of pupil premium project where students could gain instant that the blockers the target group were pupils to create soundscapes with an artist- feedback on their work from someone experiencing – such as poor organisation and a in-residence, where they have gained informal from the ‘real world’, something that the lack of support at home – could be addressed feedback on their work to make improvements. senior leadership team supported. by increasing their sense of belonging at school: Rosie feels it is important that this type of Rosie had the idea of getting an artist involved innovation happens as an extra-curricular activity after talking with her networks outside of to give teachers the freedom to take risks: school; one of her artist friends, Jeremy Gluck, “Some of the work that Rosie is leading offered to volunteer for a pilot. He reflects and what happens in that group is on his work with the school in the Master’s those children have got almost a “I love experimenting with things, and degree he is undertaking and he is planning to sense of family, a sense of kinship, in that extra-curricular environment, undertake a PhD to explore how schools and it’s developing those friendship skills you have the ability and you have that artists collaborate to improve young people’s and all the other things that come space to do it, because it can only add mental health. Rosie secured funding from alongside that as part of the benefit.” to their experience. If it’s in lessons all Shine and the Stepping Stones Fund for a yearly the time, you never quite know what Senior leader project. This year she is running this project and the impact’s going to be, and then it’s a another, where students create soundscapes bit of a danger, but with this, you really with artists who are paid for their time. feel like you’ve got the freedom.” Who and what fed into shaping it? Rosie initially tested the solution with a small Lead Practitioner Both Rosie and the artist were keen to group of students before seeing the potential provide pupils with feedback in a way that for it to work with a larger group on a regular did not constrain their creativity. They
Making Waves A better future for assessment 57 therefore experimented with the best way to include the new cohort of Year 7 pupils. students sometimes share sensitive material. to give students live feedback. Initially, Some of these older pupils take on the This means an experienced professional with Rosie sent the memes to the artist after role of informal mentors, supporting the links with other professionals in the school who the session, but it was decided that him younger students with their work. They also have pastoral and safeguarding responsibilities offering feedback during the session was a now use internationally significant dates needs to mediate the relationship better way of engaging the students, as they to inspire the students to create themed between the students and the artist. enjoyed receiving an immediate response. memes, such as International Women’s Day.
3.4.4 Decision making 3.4.5 Implementation “You’ve got to have somebody who knows what they’re doing, because of Who was involved in deciding on this What mechanisms have been used to the safeguarding aspect of it. Because approach to addressing the pain points? implement the approach? it’s going through a third party, you’ve got to be there to mediate it, you can’t Rosie, as Lead Practitioner, was the driving Shine is an after-school club: just create a link with somebody that force behind the initiative, with support from you don’t know. The person from the the artist. The artist said how important it was • The students work in the ICT room outside sends the message to you, to trust the professional in certain instances, to create a meme based on stimulus then you send it back to the student. for example around how safeguarding material that Rosie provides. Yes, so that can be problematic. But would work and which students to select • When they feel their work is once you’ve got that set up it’s fine.” for the intervention. The senior leadership ready, they email it to Rosie, who Lead Practitioner team gave Rosie time, autonomy and the emails it on to the artist. creative freedom to develop the project. • The artist offers live feedback in an email Why were other solutions rejected and back to Rosie, who then shares it with all Rosie also includes the parents and carers so this approach preferred? the students on an interactive whiteboard. that the young people have permission to stay after school. For some students in particular, Other solutions were not considered or Who are the different actors involved? she tailors her communication to maximise the weighed up against each other; instead, chances of them engaging with the project: Both Rosie and a senior leader highlighted the the project grew organically. Initially, the importance of clear safeguarding processes project just involved a group of Year 7 pupils, to ensure that students are carrying out the but now, some of those older pupils have work in a safe environment, especially as stayed as the project has grown each year
58 A better future for assessment Making Waves “I contact one student’s mum regularly. I As it is a drop-in session, Rosie does not What approaches are being used to just wrote to her last week and she came know which students will attend from week evaluate the innovation’s success? along to the exhibition. She was so open to week. To overcome this, she makes sure to it, and she was so grateful that she that each session is self-contained and that Rosie uses student attendance, progress and [her daughter] had been kind of caught, participation is not dependent on having attainment data to monitor the intervention’s and channelled, in a way, that meant that attended before. Rosie has worked closely with success and discusses students’ progress with she stayed in school, because there was a the member of staff who oversees the summer Heads of Year and subject teachers in English, danger that she would have left school.” school Eltham runs with students going into drama and art – as these are the subjects Year 7 so they can identify the students who most closely linked to the programme. Lead Practitioner might benefit most from the innovation. It should be noted that many of the students who attend are in Year 7, so while they 3.4.6 Evaluation form part of a group who have historically The partnerships with the gallery developed not engaged with their learning, it is hard because she took the initiative and contacted How is success or failure to ascertain what impact this intervention the galleries directly. She was surprised being characterised? has on these individuals as they start the at how quickly they agreed to exhibit the programme as soon as they enter the school. work and she collaborated with the art Because this project is not part of a particular However, Rosie reports that they improve department in school to help the students subject or syllabus, there are no specific over the course of the year and as they mount their work ready to exhibit. attainment or progress measures. Instead, spend longer on the programme. During this the innovation’s success or failure is gauged academic year, the whole of the Stepping What obstacles had to be overcome according to four loose goals and these have Stones project, of which this is a part, is being as part of implementation? remained the same since the initiative began: evaluated by the independent evaluator. Technological and safeguarding constraints 1. Increase students’ confidence and Rosie has to provide Stepping Stones and mean that Rosie has to submit student ability to take feedback positively. Shine with impact data each half-term, in work via email. There do not tend to be 2. Increase students’ engagement with school. the form of a narrative report with details of technological issues, such as logins not attendance, demographics of students who working, but if there are, Rosie has access 3. Improve students’ attendance and behaviour. attend, attitudes to learning in the form of to the school’s technology support team. 4. Improve students’ literacy skills. behaviour and exclusions and attainment. If she is unable to attend a session due to CPD, the session does not run that week. One senior leader reported that while attainment was important, it was also some of
Making Waves A better future for assessment 59 the more intangible outcomes that were most The student we spoke to explained differences “It boosts your self-confidence, valuable – for example, re-engaging students between how she receives feedback through definitely, because it’s an outside opinion who were at risk of exclusion with their learning Shine, compared to how her work is coming in and giving you that positive – and where they saw the most difference. normally assessed: feedback, where they’re not influenced by anything, they’re just literally seeing Is the innovation believed to be achieving your work… When I’m there [in Shine], its intended objectives? “It’s got round edges, so I mean there’s it’s definitely lifted a lot of my anxiety All those interviewed, including a student, no like box which can be filled in or fitted because school is an anxiety-ridden felt that the initiative is achieving its aim into, it’s all just one wriggly shape with place for me. It’s lifted a lot of anxiety, of giving students – especially those in basic outlines… [It was] so much more like because for most of Year 7, nobody the target group – a chance to receive helpful because then it means there isn’t except teachers and stuff in the school feedback in a non-pressured, freeing a pressure of having these targets to knew about… different disabilities… But environment to develop their confidence. reach… It honestly took everything away then doing Shine, because that was a from normal feedback, which I hated, place where nobody really knew me, so Rosie particularly highlighted one student with and just put everything I loved about I could just completely just go in with special educational needs who had benefited arts, different arts and communication a new coat of paint… since I came into from the opportunity to be creative: into a just wonderful thing.” Year 8, I’d say, I’ve definitely been more open about my disabilities and stuff, Student “It was more a creative element, and because it’s part of me. Why should I having the freedom to explore your bother trying to hide it or cover it up?” feelings, but without the pressure of This student had also created an alter Student being judged against an exam criteria. ego to interact with the artist, through So, that was what was really interesting which she felt she could express herself for me, especially for students with and receive feedback in a more creative Rosie reported that an additional benefit special needs, in terms of some students way. She talked about how working with was finding out the interests and additional who never, ever spoke in lessons were an artist outside of school has helped her needs of the students so that future projects, suddenly free. Like [student] was totally self-esteem and ability to communicate her such as one on gender equality, could be free to suddenly express herself, in special educational needs to other pupils, designed around them. The artist also talked a way, and just flew with it. And still, teachers and others outside of school: about how much he had learned from the after three years, is still flying with it.” students, such as how quick-witted and Lead Practitioner open they were. Finally, one senior leader
60 A better future for assessment Making Waves advocated working with external professionals “I love the idea of using professionals As yet, no other teachers have tried this as it was a way for teachers to develop their again because that raises the aspiration approach in an after-school club or in expertise and form links in the community. and also it gives the students the idea their lessons, although Rosie feels it would that they can do it; getting that instant be a simple and valuable process. While the innovation was not primarily focused feedback and also the possibility of on student attainment, in terms of whole- exhibiting or sharing in a wider public If another school were considering school data, Rosie also said that there had been sphere, which is great for them… then developing this type of innovation, the senior a positive impact on the target group’s results they can say, ‘I’ve been in a gallery in leader interviewed would recommend: in English and drama, with 97% achieving Hoxton and I can do it, I’ve worked with • pinning down your values as a school first, better-than-expected progress in the first a professional musician, I have created and considering how this project fits year and 100% in the second. Attendance for a garage band track, I’ve performed as • finding the pioneers on the staff team and the target group also improved, especially a DJ, I’ve worked with a professional giving them time, encouragement and the with those who previously had poor focus dancer, I’ve worked with a professional creative freedom to try something different and engagement at the school; the Head of artist.’ But I love the way that they Year 7 reported fewer issues with this target don’t see that as a hierarchy; they will • considering what the rest of the group than they have had with similar students talk to these professionals on a level sector can learn from your project. in the target group in previous years. and they are not frightened to.” The Lead Practitioner would recommend: As Rosie develops future iterations Lead Practitioner of the projects, she is committed to • using experienced staff who have a good using external professionals: understanding of safeguarding processes • working with other professionals in the school, such as pastoral and safeguarding leads and technology support • developing a network of external professionals who can offer ‘real-world’ feedback • identifying how the students’ interests and additional needs can feed into future innovations.
Making Waves A better future for assessment 61 3.5 In our first visit, in the autumn term of 2018, In theory, teachers are able to retain AET’s national curriculum lead set out the autonomy in choosing their own questions Innovation 5: trust’s expectations of how Eedi might work from the question bank to fit the topics Eedi at the Academies in practice. In the trust’s schools, teachers they teach. The diagnostic question bank teach a topic and then pupils complete the test and the feedback that the platform provides Enterprise Trust (AET) on the platform as homework. The platform were designed by an experienced maths marks the test automatically, but also gives teacher who is Secondary Maths Advisor 3.5.1 Introducing the feedback on each pupil’s misconceptions so to the Times Educational Supplement who innovation they understand why they made a mistake, co-created ‘diagnosticquestions.com’, a not just what the mistake was. The diagnostic free website hosting the world’s largest Approximately 60 primary, secondary questions reveal pupils’ common mistakes to collection of high-quality diagnostic multiple- and special schools in the Academies teachers so that they can re-teach or explain choice questions and now part of Eedi. Enterprise Trust (AET) – the largest MAT the mistakes. Pupils take a test again three in England – have been using an online weeks later and this is intended to show In AET schools, including those we visited, maths platform called ‘Eedi’. It is being whether the topic has been embedded in Eedi is used to supplement other forms used across different age groups up to their longer-term memory and to provide of assessment and to triangulate the data Year 8, and additionally with Year 11. further information for teachers on pupil they generate. Stop the Clock assessments, understanding. The Eedi team also hope that produced by AET and used across AET schools Eedi consists of multiple-choice, low-stakes, the platform will be used by parents, although at the end of each unit, assess pupils on the formative assessment tests. The questions are this is not yet a part of AET’s use of Eedi: different elements of the maths curriculum designed to be diagnostic, with each potential and their skills in maths. Questions cover answer specially designed to reveal a common the same content as Eedi (relating to the misconception or mistake for that topic. “What we’re trying to do is basically previous unit’s scheme of learning, which allow teachers to be more informed is centrally planned by AET). Questions are For this study we focused on two primary so they can provide better instruction, not multiple choice as they are with Eedi, schools, both serving deprived, one in allow students to understand, not just but are similar in style and the results are Birmingham (400 pupils) and one in that they’re wrong, but why they’re analysed by AET. AET then feedback pupil Burton on Trent (593 pupils). Eedi is wrong, and for parents to understand data from Stop the Clock to schools, to allow being used in every year group from where their child is struggling and teachers to understand pupils’ progress in Year 1 to Year 6 in both schools. how they can support them.” comparison to their age-related level. Eedi central team
62 A better future for assessment Making Waves Our case study schools both used Eedi as i) Eedi team vital in order to benefit as many pupils as well as Stop the Clock. In School One, Eedi possible. Teachers needed something more was used as an interim assessment for each The Eedi team themselves were heavily structured than a question bank; specifically, unit. The school attempted to use results influenced by research on formative they needed to be able to quickly match from Eedi to target interventions before assessment and memory, especially the questions to existing schemes of work. pupils are assessed again on Stop the Clock. work of Robert Bjork and Dylan Wiliam. In theory this would have allowed the school ii) AET’s central team They felt that their platform could address a to act swiftly to support pupil progress in number of ‘pain points’ that they identified AET’s central team felt that assessment maths, rather than relying on termly data. in schools across the country. These were: across their schools was not precise enough However, this was not the experience of in primary and Key Stage 3. This was the case School One, which decided to stop using • Workload: Eedi seeks to save across an estimated 80% of schools across Eedi during the summer term of 2019. teachers time by automating marking AET. Assessment had tended to be based on and producing reports and analyses School Two used Eedi for regular homework. learning objectives for individual lessons rather of pupils’ understanding. This homework was used formatively to help than the granular understanding of the steps • Weaknesses in formative assessment: ensure that lesson planning targeted pupils’ needed to understand a concept in maths. AET Eedi aims to make it easier for teachers misconceptions. The school also used Eedi wanted granular understanding of what it calls to understand why pupils make mistakes, at the start and end of each unit of work as “fluency steps” to be part of maths assessment and what they need to practise or know. a means of assessing progress. The school practices in every school across the trust. also used teacher assessments, which are • A need for more precise reporting iii) The case study schools moderated internally by school staff, and to parents: The Eedi team believed that used to triangulate with the more formal parents would find detailed analysis of what The two case study schools had different termly assessment that AET produces. a pupil can and cannot do more useful and reasons for using Eedi. At School One, the meaningful than generic pupil reports. Headteacher reported that, having been 3.5.2 Agenda setting • Weaknesses in pupil self-assessment: recently appointed, their immediate priority Eedi aims to help pupils understand was to improve teaching and learning What were/are the ‘pain points’ that this why they got something wrong, not following a disappointing Ofsted report, which innovation sought to address? just that they got something wrong. precipitated a change in the senior leadership team (SLT). Interviewees explained that AET’s deployment of Eedi emerged from The co-founder of the Eedi team had originally previously teachers held low expectations and a combination of different pain points acquired the diagnostic question bank for lacked subject knowledge and assessment experienced by different parties. online tutoring, but realised that schools were expertise in maths and some other subjects.
Making Waves A better future for assessment 63 The maths lead on the new SLT felt that 1. It was an opportunity to reduce workload. Why was it considered an pupils lacked knowledge about how to use 2. The platform was already linked to important agenda to pursue? mathematical skills and concepts. This led their existing schemes of learning. to difficulties understanding why they got 3. Eedi was free, in contrast to The AET core curriculum team prioritised questions wrong, a problem exacerbated by the maths homework package improving the quality of formative assessment weaknesses in maths teaching in the early years they had previously used. in order to improve the quality of teaching and gaps in teacher knowledge in relation to and learning and help pupils make more Of these, workload was considered to be progress in maths. The majority of schools key mathematical concepts, particularly when most important. School Two’s maths leads it came to content that was covered in other in AET are ‘sponsor academies’, which felt that Eedi saved time for teachers. Instead had gone into special measures, or been years or Key Stages. This view was shared by of teachers having to match homework the Headteacher: judged to require significant improvement. questions to their schemes of learning and AET recognised a weakness in maths lesson content, and then print out questions outcomes across the majority of its for pupils to take home, Eedi created “[The Head of Maths] and I haven’t schools, and was looking for changes that questions that were pre-matched to lesson been here very long, and that’s the would support pupil progress in maths. content and were accessible online. They kind of legacy that we’ve picked up, were also excited by the potential for Eedi and it’s a challenge in most areas Eedi’s team believed that diagnostic questions to reduce marking, as the platform records of the curriculum but it is also a were a powerful tool to improve formative and shows teachers which pupils have challenge that’s specific to maths.” assessment. They were also keen to help completed each homework, as well as each parents understand how their child is doing. Headteacher: case study school pupil’s score. The maths leads explained that The team believed that they could support workload was something that was considered pupils by helping parents understand in every decision made in the school. their children’s progress in maths. At School Two, interviewees explained that the school itself was not actively looking Thus, while the decision to use Eedi in AET The parental element of Eedi was also for a solution to any issues with maths schools was taken by the AET central team, influenced by the need to monetise the assessment. However, once AET made the innovation tallied with priorities in each platform, without using schools as the source schools aware of Eedi, the two maths leads school. For one of our two case study schools of revenue. The suggestion had been to at School Two decided to incorporate it tallied with the SLT priority to improve the provide parents with some elements for free Eedi into their practice. They felt that quality of maths teaching and assessment, while and others as part of a paid subscription Eedi promised three main benefits: in the other it tallied with a focus on reducing otherwise known as a freemium model. workload. There was therefore alignment between the central and school teams.
64 A better future for assessment Making Waves Interest in diagnostic questions as a linchpin of For School Two, maths leads did not feel that curriculum. This team worked with Eedi formative assessment and the desire to engage teachers needed support in the same way, alongside the MAT’s core curriculum team with parents meant that Eedi and AET had but they wanted Eedi to strengthen formative to write questions that fitted with the MAT’s shared priorities. Schools also shared a belief assessment at the school and improve planning. maths curriculum. At the point when AET that the diagnostic questions would be useful decided to use Eedi (2016/17), Eedi was able 3.5.3 Formulation to provide a bank of diagnostic questions, Schools within AET were introduced to Eedi but not the automated analysis and feedback How was a solution arrived at? through regional conferences, and were that are now a key feature of the platform. expected (although not required) to try to The AET core curriculum team took the Eedi thus evolved as the AET curriculum make use of the platform. Not all schools across decision to use Eedi. They took this decision team worked with the Eedi team over the the MAT chose to use the platform, but the after first deciding to develop a bank of course of the next year (2017/18) in order two schools we visited saw alignment between diagnostic questions to use in schools across to shape a solution that allowed teachers Eedi and their own agendas. For School One, the MAT. For AET, a key reason for using to understand pupils’ misconceptions Eedi appeared to offer a means of upgrading Eedi was that it offered a simpler, less time- across all AET schemes of work in maths. teachers’ knowledge and understanding of consuming solution to the other options the assessing progress in maths. As the maths leads team had considered. Initially, the curriculum In order to match the case study schools’ explained, Eedi allowed teachers to gain an: team considered creating their own diagnostic needs schools’ maths leads have refined assessment resource, and went so far as the deployment of Eedi by working with to begin writing their own multiple-choice classroom teachers to adapt how pupils “Children give you the wrong answer but assessment questions. At the same time, the access the platform. In some cases, questions they think it’s the right one. Nine times team recognised that many teachers were using have been printed out when there is no out of ten they think it’s the right answer, a platform to write, print and administer their access to IT, or when questions need and it’s having the subject knowledge own diagnostic questions. After meeting the to be read out to Early Years pupils. as a teacher to think, ‘Why do you co-founder of Eedi, however, the central team think that’s right? What’s going on in realised that Eedi could offer a ready-made 3.5.4 Decision making your head? What’s your misconception solution, which offered an accessible platform that we need to unpick?’ And that is that might cut teacher workload by reducing Who was involved in deciding on this where the poor subject knowledge the need to administer and mark tests. approach? [from teachers] means that that child doesn’t necessarily move forward.” Who and what fed into shaping it? The decision to use Eedi in AET schools was taken by AET’s core curriculum team. This AET has a team of maths specialists and Maths lead: School One team made the decision to address a particular external experts who work on its maths
Making Waves A better future for assessment 65 pain point (assessment in maths), and to use The AET curriculum lead described Decisions around platform functionality and Eedi as a means of solving this problem. As we the process in the following terms: design were taken by the Eedi team. These will see below, this team also made decisions decisions needed to combine business and around implementation and evaluation. pedagogical perspectives – in particular by “We thrashed it out with the team, deploying research expertise to understand The AET curriculum team have an agreed way we always encourage absolute open how best to address pain points around of working. This team is made up of a national debate and disagreement. We call formative assessment and deciding how lead (who we interviewed) and five regional it thoughtful disagreement... we ask to develop a solution that was financially leads. The team meet fortnightly to lead people to play devil’s advocate… it’s sustainable. For the Eedi team, the question of strategy in a number of areas across the MAT not about the loudest voice at winning how to monetise the platform was a particularly (including assessment). The team’s decision the arguments, it’s about the most thorny issue. The team decided from the making approach is described as “thoughtful compelling reasoning and logic. And it very start to try to stay away from charging disagreement”. This involves the merit of doesn’t matter how deep we went in individual teachers. With AET, the team each idea being evaluated and decisions being our conversation, it just made sense to explored four options for monetisation before taken collectively, rather than being taken by run with this, because of three reasons: deciding to offer the platform to AET for free: the most senior member of the team. Team it’s clear in its precision, it’s easy to use members are therefore encouraged to put and we can embed our own content.” • Charging each school a small amount: forward deliberately contrasting views, in order Eedi and AET decided against this option AET National Curriculum Leader to test the merit of a particular idea or decision. due to pressures on school budgets. • Charging parents: Eedi is keen to explore this in future but reports resistance to this approach from schools. The platform is currently only designed to be used by teachers, but Eedi is working on a parent tool, which it may charge for. • Charging AET: In future, Eedi might consider charging a fee for MATs. This would require Eedi to build an interface that was useful to central teams within a Trust, for example giving insights into strengths and weaknesses of learning
66 A better future for assessment Making Waves across schools within a Trust. With AET, Classroom teachers appeared to have limited solution to problems with assessment in the this option had not been explored as this decision-making power (other than those who school and a sense that it did not have anything interface did not exist, even before raising were also maths leads). However, teachers else and needed something quickly: the question of whether AET might have were able to decide on how they would adapt been willing or able to pay for this service. their planning and teaching as a result of the “Judgements were not secure in most • Charging teachers for resources and feedback they received from the classrooms. We [the new SLT] came in CPD: Another possible future direction Eedi assessments. the middle of AET working on a different for Eedi might be to charge teachers for Why were other solutions rejected and way of primary schools assessing in insights into strengths and weakness of this approach preferred? reading, writing and maths anyway, so their teaching (for example highlighting we didn’t seek Eedi out but we knew we which topics an individual teacher’s pupils AET’s core curriculum team decided to needed a solution and when AET said, struggle with compared to what might be pursue Eedi because they had worked with a ‘This is what we would like schools to do,’ expected). Charging fees for insights to member of the Eedi team in the past on maths we jumped on that quite quickly because help teachers personalise their CPD would assessment. The fact that they could work we didn’t have anything else. So it wasn’t allow the platform to remain free for its with Eedi to design their own questions tying a switch from one thing to another. We original purpose as an assessment platform. into the MAT’s curriculum was particularly said, ‘Okay, that’s the one that AET wants important as the AET national leader for Eedi’s Chief Executive Officer (CEO) reported us to use and we need something really curriculum explained: that these options had been explored quickly. Let’s get that in.’ So we were quite informally, over time, and in discussion with the proactive in pushing that forwards then.” AET curriculum team. “With Eedi we could write our own Senior leader: School One assessments and just embed them In schools, maths leads and the SLT made the into their system. That was invaluable, decision whether or not Eedi would be used because we could write them against Case study School Two already used a different in their school. Maths leads took decisions our own schemes of learning, whereas online maths homework tool. For this school about precisely how assessments would be other products we couldn’t.” the decision to replace the previous platform administered – for example, whether the with Eedi hinged on two factors: assessment was conducted in class or at home, AET National Curriculum Leader on a computer or on paper, at which exact 1. Cost: Eedi was free to use while the point in the unit (guided by the directions given previous platform carried a fee. by AET) and with which classes. Case study School One believed AET’s decision 2. Workload: The school reported that to use Eedi fitted with its own need for a
Making Waves A better future for assessment 67 the previous platform required printing, a large MAT was more top-down, with senior “We pushed it out to heads of marking, planning which assignment leaders or MAT central teams choosing the departments, because by that point to use for a particular homework, and platform, and attempting to apply it across we were clear with our narrative and, choosing questions to match schemes classrooms. For Eedi’s CEO, this was an attempt therefore, the way that we presented it, of work. In Eedi, question choice is to try a new strategy to increase uptake. it would be difficult to say we don’t want automated and linked to AET’s curricula, to use this, because it was so powerful.” and feedback is also automated. Eedi’s CEO did not see this decision as totally successful. While there was enthusiasm at AET National Curriculum Leader For the Eedi team, an important design MAT level, Eedi believed that this had not principle was to build a platform that is free always translated to enthusiasm at classroom AET organised regional ‘Eedi conferences’ at for schools to use and which prioritises level. This was borne out by the experience which maths leads were shown how to use simplicity. The Eedi team felt that these of our case study School One. Where there the platform. Schools were encouraged to features would give them a competitive had been successes in the uptake of Eedi in use Eedi, through the regional conferences, advantage in a crowded market for online AET schools, this was seen as being driven offering support to schools that expressed formative assessment solutions. They argued by one teacher within the school who had an interest, and highlighting successes in that in contrast, other educational technology bought into the idea and taken steps to make schools with the highest usage of Eedi (Edtech) assessment solutions focused on it work, and shared that with other teachers. through a series of case studies. These case keeping ahead of competitors by developing studies were in development at the time of more complex technology. Instead, Eedi 3.5.5 Implementation our research. In addition, in one of our case kept the technology simple and focused study schools a trainer came in to show on bridging the gap to existing practice. What mechanisms have been used to individual members of staff how to use Eedi. This means that Eedi sought to develop a implement the approach? platform that was easily adaptable and could AET recognised that Eedi formed one part fit into teachers’ current ways of working. For AET, it was important not to coerce of a bigger picture of improvement for schools into using the platform. Once the AET many schools. AET has a series of minimum The decision to work with AET, a large MAT, central team had decided to use Eedi across expectations known as ‘non-negotiables’ in was seen as a risk. Eedi’s approach to marketing their schools, they worked with maths leads terms of how it expects schools to operate. had previouslybeen to work with teachers in schools across the MAT to explain what the These minimum expectations are based directly. For example, a teacher would find solution was, and how it would benefit around various aspects of schools’ practice, Eedi through their own research, and as a maths assessment: such as attendance, financial systems and result would already buy into the aims and basic principles of teaching and learning. While methods used by the platform. Working with
68 A better future for assessment Making Waves Eedi, as a formative assessment model, fits Broadly, there was one individual, the CEO, “85%, possibly 90%, of our users use into one aspect of those, the ‘non-negotiables’ who shaped the business model and overall the content in class by projecting it are not so granular that they include Eedi. strategy. Another individual designed the on a whiteboard and then basically platform. Eedi also has an individual who telling the kids to use their hands to AET is planning to create a series of Eedi provides liaison with schools and MATs. respond or whiteboards to respond.” champions with leadership responsibilities, Beyond this central team, there is also a team Eedi central team with additional pay attached (a TLR payment). providing training for groups of schools, This is a response to a perception at AET that individual schools and individual teachers. its approach to implementation had been As a result, Eedi wants to remove the too ‘laissez-faire’, and that building in some To what extent is the reified technology from the solution so that schools accountability for schools’ use of Eedi, through solution consistent with the have an option to avoid the “disruption” TLRs and appraisals on the implementation of planned/intended solution? that can occur with the use of computers Eedi in each school, would help drive uptake. in schools, such as lost passwords or For Eedi, the platform itself has remained lack of up-to-date IT infrastructure. Who are the different actors involved? similar to its plans. The CEO reported that the experience of working with AET had Across AET, the solution has remained Implementation flowed from the core led to approaches from other MATs, where consistent, although there are planned curriculum team, through maths leaders in implementation would proceed differently. In changes to the way in which it will be schools, to classroom teachers. In one school, those MATs, the Eedi CEO described a more implemented in 2020. AET has also been the key member of staff driving the Eedi prescriptive approach, with Eedi being used flexible in responding to teachers’ use of initiative was a senior leader who was also the as a formal “non-negotiable” assessment. Eedi the platform. Originally intended for use as maths lead. In the other, a pair of middle leaders is also working on developing assessments homework, with teachers using feedback to shared the role of maths lead, and drove that do not require technology. One of adapt future lesson plans, the AET curriculum implementation. In both schools, individual these developments is taking place through lead now describes a “best practice” classroom teachers were then responsible pilots in India in schools where pupils do not model whereby pupils take the assessment for direct implementation with pupils. have access to computers. This may lead independently in class, and are then given to changes in the UK too, as Eedi attempts instant feedback on common misconceptions. Meanwhile, Eedi’s team include a range of to find ways teachers can record data from individuals, ranging from those involved assessments that take place in class, without One way that the case study schools’ in business strategy, to those developing computers. Eedi’s CEO estimates that: implementation of Eedi deviated from the questions, building the software or building original intention was that they now use the relationships and offering support to schools. platform with pupils in Key Stage 1 as well as
Making Waves A better future for assessment 69 those in Key Stage 2. Eedi was designed for use teaching and learning strategy based on 2. Teacher mistrust: Some teachers are with older pupils (Key Stage 2). It was not clear assessment through homework impossible. sceptical about multiple-choice questions. what had led schools to make the decision to Both case study schools reported difficulties The AET curriculum team identified use Eedi with younger pupils, and in the case using Eedi for homework, albeit to different this as a problem at the formulation of one of the case study schools, this led to levels. For School One, at the stage of stage. In part, the decision to have Eedi problems, set out below. planning how best to use Eedi, the maths present at two AET national conferences lead decided not to use the platform for was a response to this scepticism. In the case study schools, there had been homework. There were two reasons for this: contrasting trajectories for the use of Eedi 3. Teacher knowledge: As noted above, during our fieldwork, indeed one school • pupils lacking technology at home responding to misconceptions in one has decided not to continue using Eedi. For • Key Stage 1 pupils’ difficulties school. Whilst Eedi shows where pupils this school, the decision was based on the in navigating the platform. have misconceptions it does not provide feedback for pupils and teachers on difficulties it faced persuading teachers of the Instead, School One used Eedi in class, with tool’s value. In this case teachers were believed what their misconceptions are based paper printouts. While this allowed all pupils on. Teachers therefore need the to be unable to identify pupils’ misconceptions to access the assessment, it also created themselves, and while the platform highlighted knowledge to understand and correct additional workload for teachers in printing misconceptions In one school, not all where pupils made mistakes, teachers needed the papers and marking pupils’ answers. more guidance to understand how this related teachers were secure enough in their to pupils’ understanding of topics. Logistical For School Two, maths leads identified some subject knowledge for this to take place. barriers also played a part. Teachers found difficulties linked to parental involvement at an early stage. Parents were used to Both case study schools highlighted difficulties that, with Key Stage 1 pupils in particular, their using the assessment with Key Stage 1 classes. workload increased as assessments had to be the homework platform that the school used previously. The maths leads had to Although the AET national curriculum lead printed out and inputted onto the platform or told us that Key Stage 2 was the primary focus marked by hand. spend time inviting parents into school to show them Eedi, communicating with for Eedi, this message did not appear to have What obstacles had to be overcome as parents via letter to explain the change in filtered down to our case study schools. For part of implementation? homework system, and speaking about Eedi them, younger pupils struggled to navigate in parents’ evenings. For a small number of the platform, were more likely to guess at A number of challenges have had to be pupils, non-completion was also an issue, multiple-choice answers, and were less likely overcome. In particular: but maths leads believed that this was to to complete Eedi assessments for homework. be expected in any homework system. This was a major factor behind one school’s 1. Low homework completion rates: If pupils decision to discontinue its use of Eedi in 2019. do not complete homework it makes a
70 A better future for assessment Making Waves 3.5.6 Evaluation What approaches are being used to Lessons learned evaluate the innovation’s success? How is success or failure 1. ‘Bottom-up’ development of an being characterised? AET is part way through its second year of innovation by maths teachers versus using the Eedi platform. There has not yet ‘top-down’ implementation: Both Eedi For Eedi, success is characterised by uptake been any formal evaluation. Instead, AET has and AET acknowledge that uptake and among schools. In the longer term, success been gathering feedback from maths leads and impact have been greatest where individual would also involve monetising the platform. triangulating with learning walks where leaders teachers have bought into the ideas behind visit classrooms and through AET’s termly the solution. For the case study schools, For AET there are three success criteria: summative assessment. Eedi has also developed the ‘top-down’ approach has not resulted in a usage tracker to identify which schools are success; one of the schools continues to use 1. Use of Eedi across all schools: Over making more and less use of the platform. the platform, while the other has decided the year, the AET curriculum team have not to use Eedi in the immediate future. modified this aim due to a slow take- The AET central team are using a number of up. Instead, they are now focusing on forms of evaluation: In 2019/20, AET used an approach to a small number of schools using the implementation that was more ‘carrot’ than platform well, in order to provide case • Regional curriculum leaders are ‘stick’. Schools were encouraged to use Eedi, studies to encourage uptake across looking at pupil books to track but there were no consequences for schools more AET schools. In those schools AET changes in the quality of learning. that chose not to use the platform, or that aims to see the majority of pupils in all • Regional leads also gather feedback from have different levels of use across different schools complete 15 units a year, with maths leads in schools (sometimes also classrooms. It seems that this strategy follow-up assessments taking place three called Eedi co-ordinators) on how they chimes with the ‘ground-up’ approach taken weeks after the initial assessment. are using the platform, and whether thus far by the Eedi team. The approach 2. Increased attainment as a result of they perceive positive impact. allows individual teachers to decide whether using Eedi: In schools using Eedi well, • Once every half-term, this information or not the platform works for them. AET expects to see a 75% pass rate in is fed back to Eedi through Eedi’s While this approach benefits Eedi, and the the three-week follow-up assessment. lead on school engagement. individual teacher, it creates a problem for the MAT. What do the central team do 3. Better learners: Less tangibly, AET • The team are using the usage about schools that need most support with reports a success factor as “pupils who tracker, outlined above. maths assessment, but which do not engage are better learners, because teaching • The team are using summative data with solutions such as Eedi? In the future, is focused on their specific gaps”. to show how maths attainment is it seems that AET intends to build in an changing across AET schools.
Making Waves A better future for assessment 71 element of accountability for schools’ use 3. Matching solutions to problems across of Eedi by paying staff to take responsibility several different schools:At the AET/ for how the platform is used in schools. trust level, assessment in maths was a common problem. AET’s response was 2. Decision making at different levels: to make Eedi available to all schools. An One feature of this innovation has been alternative approach might have been to the involvement of several layers of target specific schools for whom maths decision makers. One layer has involved assessment was the most pressing issue the Eedi team making decisions about and to tailor the solution around them. In the platform itself. This team decided this study, we found that, paradoxically, the on functionality, design and a strategy to school that reported the greatest problems market the platform to schools. The next with maths assessment did not choose layer is the MAT central team. They have to use the platform, whereas the school some limited input into Eedi’s decisions that appeared more confident with maths about functionality and design. The assessment found Eedi useful. For this main focus of decision making here is to school, the benefit of Eedi solved a different address pain points. School management problem – that of teacher workload. – whether senior leaders or maths leaders, provide a third layer. This layer decides ultimately whether the platform will be used or not, and how it should be used in classrooms. Finally, teachers have limited decision-making power, but can feed back, indirectly regarding the design and functionality of the platform. This layer can also feedback on how successfully the solution is addressing the problem it was intended to tackle. Teachers can also make decisions about the teaching that occurs as a result of the feedback they receive from pupil assessments.
72 A better future for assessment Making Waves 3.6 Innovation 6: Laser conversations at the Midland Academies Trust Senior leaders discuss plans for improving pupils' progress with middle 3.6.1 Introducing leaders (e.g. interventions) and provide support. the innovation
‘Laser conversations’ are an assessment practice used in all four of the secondary schools that are part of the Midland Academies Trust in the Midlands. Middle leaders discuss next steps with classroom teachers Laser conversations are intended to make for improving pupils' progress assessment more meaningful. Rather than (e.g. interventions for pupils who are not making enough regularly entering pupil performance data progress). Middle leaders provide onto a system, teachers have conversations support to classroom teachers. with their line managers about how pupils are Middle leaders then discuss progressing and what steps they are taking to Classroom teachers pupil progress across a year support pupils who are underperforming. The assess pupils and RAG group with their senior leader. process involves classroom teachers, middle rate their progress. leaders and senior leaders, as set out in the Teachers have a conversation diagram below. with their middle leader about pupil progress.
Making Waves A better future for assessment 73 Laser conversations are timetabled regularly “For example, let’s say they’re very clear these has varied over time and among for individual year groups. For example, about the changes that took place during different stakeholder groups: in the autumn term, all Year 11 teachers the Industrial Revolution, but as a cohort might meet with their line manager to they are less clear about the reasons 1. Developing more valuable, active discuss their class. Information about pupil that those changes occurred. We would responses to assessment information performance and progress can then be then want to have a conversation 2. Increasing professional dialogue fed up to senior leaders and trustees. with the teacher about what is being between teachers done about that. So, we would want 3. Reducing teacher workload. Laser conversations are intended to be a more to know that learning sequences pupil-focused assessment practice compared are being adjusted or tweaked.” Previously, the central trust team collected to traditional data collections. lots of data about pupil progress at different School senior leader, points in the year through ‘data drops’. Like Midland Academies Trust in many schools and MATs, this was time- “It’s literally you sit down with whoever consuming but had limited impact, since it did you’re working with and they talk 3.6.2 Agenda setting not necessarily lead to teachers adjusting their about where your kids are at. It’s practice in response to assessment information. uplifting, because you’re talking about What were/are the ‘pain points’ that this Both teachers and the MAT’s former Executive your students and you’re talking innovation sought to respond to? Principal recognised this and referred to it as about how they’re doing and their “assessment for data’s sake”, “or assessment Former Executive Principal of the trust, progress, so it’s less about the data as an end itself”. One teacher also argued that Ros McMullen, explained that: and more about the student. It has it ended up interrupting the flow of teaching. the student at the heart of it.” “The first main reason was the Teacher, “Two years ago we had a huge amount workload – we were seeing a culture Midland Academies Trust of data but we were not taking that and where teachers were inputting vast doing something meaningful with it. amounts of data, but we felt it was Like lots of schools we were falling into having little impact in the classroom.” They are also intended to be a more accurate the trap of generating lots of numbers… way of assessing pupil needs and deciding next Former Executive Principal it should be about the students and steps, compared to traditional data collections. teaching and what we can do about it.” Within this, three agendas can be School improvement lead drawn out and the relative emphasis of
74 A better future for assessment Making Waves Thus, the trust found that multiple data “All data in the trust was live. So, as The MAT therefore decided to abolish data drops did not result in improvements in pupil a teacher you would be constantly drops altogether in the hope that this would progress or attainment despite creating updating your data as you needed to, as support school leaders, middle leaders and workload. Its innovation was therefore it was relevant to you. You’d be asking classroom teachers to make improvements to designed to provide a springboard towards yourself the question ‘Have they learnt assessment and teaching and learning: professional discussion and action. what I taught, how do we think they’re doing, what do we need to do about it?’” The approach was also intended to “We decided that we would stop ensure that peers came together to MAT central team teachers inputting data completely into spot issues for pupils across the school the system because this was getting in since “none of that gets captured when the way. We decided we would replace However, the MAT realised this approach individuals sit on their own with data”. it with conversations which would was not having its intended effect because improve the way our middle leaders teachers did not keep data updated Who and what influenced this agenda? were able to line manage and quality regularly and workload remained high. assure and improve the performance in Laser conversations were not the MAT’s first their departments. It would also mean attempt to address assessment challenges. “Teachers were keeping rich information that the principals were more effectively Previously, the central team had sought in their mark books, for example, but able to challenge what was happening to improve pupil outcomes and reduce asking them to go into another system in each department and that at the end workload by asking teachers to keep ‘live and keep it continually updated was of that process, there would be data data’ on the centralised computer system an increase in workload without a real produced which was really meaningful.” (that is, they asked them to update data impact on the quality of teaching.” whenever there was a change or development MAT central team in pupil progress rather than in onerous MAT central team half-termly data drops). They hoped this Why was it considered an important would help teachers use assessment data agenda to pursue? to improve teaching and learning: Furthermore, the central team noticed that this type of data collection gave teachers Some central team members were the wrong message – that assessments were particularly motivated by a desire to reduce about holding teachers to account rather teacher workload, particularly where this than helping pupils to make progress. was unproductive. They believed that introducing laser conversations would
Making Waves A better future for assessment 75 lead to a combination of reduced teacher 3.6.4 Decision making 3.6.5 Implementation workload, and more effective assessment. Who was involved in deciding on this What mechanisms have been used to The MAT also prioritised this new approach approach to addressing the pain points? implement the approach? because it felt that traditional data collection was unreliable and unhelpful in supporting Although the laser conversation model was Laser conversations have been implemented pupils to make progress. This was partly first developed in one school, the central team across the trust. Implementation has involved basedy on feedback from teachers about were the driving force behind the MAT-wide the central MAT team, senior leaders in the traditional data collections, and partly on approach, with the Executive Principal and secondary schools and middle leaders. pupil progress tracking, which showed that the strategic lead for academy improvement Different parties are involved in laser pupils were not progressing sufficiently playing leading roles. conversations at different levels to allow despite the ‘live data’ initiative. The MAT’s trustees were initially cautious for information and decision making to be shared between different levels Additionally, the approach reflected the because they were used to the previous of management in the school. MAT’s culture, which is said to emphasise assessment system and wanted to keep responding to specific pupils’ needs, rather track of pupil progress by reviewing data i) The central team role has been to: than prioritising accountability measures. regularly. The MAT central team therefore had to work closely with trustees to show • communicate the plan and provide 3.6.3 Formulation how laser conversations could produce instructions to teachers across the MAT more valid and useful pupil progress data. – they did this by cascading information One school had started using laser out via head teachers and emails conversations to review progress in certain Why were other solutions rejected and • create an annual schedule year groups and the central team therefore this approach preferred? for laser conversations decided to experiment by rolling the approach The real-time and live-data approaches were • adjust instructions and timetables out to other schools and year groups. rejected because data was not up to date in response to staff feedback • give all teaching staff and school leaders The MAT’s school improvement lead argues or accurate and did not result in improved a guidebook about laser conversations. that the opportunity to take promising outcomes for pupils – despite involving practice from one school, learn from it considerable workload. Rather than a range ii) The role of senior leaders and take it to other schools is one of the of different options being weighed up in in schools has been to: important benefits of being a small MAT. parallel, the new approach was therefore selected, cascaded out and gradually refined. • explain the new approach to school staff during meetings
76 A better future for assessment Making Waves • invite middle leaders to training iv) Teachers’ role has been to: pupils, areas for development and plans for on laser conversations improving pupil performance. • conduct formative assessments • monitor middle leaders’ and teachers’ of pupils’ work In turn, Heads and school leaders meet with use of laser conversations through MAT leaders to discuss findings and this is meetings with middle leaders to discuss • use information from formative used to hold school leadership teams to the outcomes of the conversations. assessments to make judgements about how pupils are performing account and to develop MAT-wide strategies Senior leaders’ implementation methods • talk to middle leaders during laser for improving pupil performance. Trusties helped to ensure that middle leaders conversations about which pupils are receive summaries of the information understood the basic expectations for laser underperforming and what steps they that has been gathered and scrutinise the conversations in their school context. However, are taking to improve outcomes. steps being taken to improve outcomes. training differed slightly between schools, and implementation was not uniform and In some cases, teachers sit down with their To what extent is the reified solution homogeneous across schools. For example, the Head of Department (HoD) to discuss pupil consistent with the planned/ instructions that senior leaders gave middle performance without any performance data intended solution? leaders left space to adapt the innovation written down. In these cases, the HoD records The implemented approach is constantly to their own teams. This had important data in a spreadsheet. In other cases, teachers evolving and has moved on since the original consequences, which we return to later. enter data into the SIMS/a spreadsheet in plan as a result of the flexibility individual advance (for example, “working above expected iii) Middle leaders’ role has been to: schools and staff members were given standard”). They then talk about their classes’ regarding how they conduct the conversations. • explain how the approach works data with middle leaders and develop an action The MAT’s school improvement lead points to classroom teachers plan to improve pupils’ performance. out that this is part and parcel of trying and • set expectations for how the conversations Middle leaders discuss conversation outcomes developing something new. would work within their departments with senior school leaders and produce The biggest divergence from the original plan • schedule the conversations in line department-wide or school-wide action has been that in many cases teachers are with the MAT’s timetable plans. Like with teacher to middle leader recording data in spreadsheets. Although the • structure the conversations conversations, the structure of the discussion MAT’s former Executive Principal emphasised according to their preferences varies. In general, middle leaders come to to staff that “if it isn’t making a difference to • identify how pupil outcomes can meetings with data on all pupils in a year group student progress and your effectiveness as a be improved and produce action (for example, Year 9). Senior leaders then ask teacher, don’t do it”, the MAT now believes that plans where necessary. middle leaders about what is going well for some teachers and leaders prefer to structure
Making Waves A better future for assessment 77 laser conversations around recorded data and Our visits to schools suggested that as the consequences and illustrates the trade- that it can prompt valuable conversations. On approach had evolved, it had been hard to offs innovators need to make between the one hand, this is time-consuming, with maintain a shared understanding across flexibility – in order to preserve professional some teachers arguing that it is not a good use all schools and the central team regarding autonomy (as well as responsiveness of time, which could be focused on working expectations for conversations, in particular to context) – and consistency. with pupils to improve their learning. However, whether recording data was appropriate or as the MAT’s school improvement lead explains: not. For this reason, the central MAT team Another important challenge has been the have recently reviewed practice and have fact that laser conversations are sometimes adjusted their guidelines. This continued time-consuming for middle leaders managing “Whilst we initially envisaged the process iteration is seen as an important element of large staff teams. This may be one factor would work better without any numbers developing and refining the initiative. The new that prompted the return to recording data at all, our teachers showed us that they guidance has been written under the tagline in advance of laser conversations since already held information which was ‘Accelerating Progress, Reducing Workload’, it allows middle leaders to ask classroom meaningful and would add value.” re-emphasising the dual focus on progress teachers about specific pupils who are and teacher workload and pointing out that underperforming. Some senior leaders have MAT's school improvement lead ‘we assess to improve our curriculum and our taken a similar approach by asking middle teaching’ and that data is a ‘by-product’ of this. leaders to share data about pupil performance Thus, the MAT emphasises that its priority in a year group before the meeting. is now for data to be used as a stepping It goes on to state that ‘we understand that Although this approach might be considered to stone towards pupil-centred, professional teachers will have their own records to be in tension with the agenda around reducing conversations. As one middle leader put it: help them assess and plan, and to aid their conversations; however, we do not ask nor teachers’ workload, it remains consistent with expect any teacher to enter data on SIMS. If one of the initiative’s initial goals – reducing “Everything starts with ‘What are it doesn’t accelerate student progress and ‘unhelpful’ or ‘wasteful’ workload – since the we doing to improve individual reduce teacher workload it isn’t worth doing’. data has now become a prompt to action, pupil outcomes?’ And actually if we rather than ‘data for data’s sake’. Furthermore, do that for each pupil the headline What obstacles had to be overcome as the MAT hopes that time can be re-allocated figures should improve as a result part of implementation? from less valuable, low-return workload such as of the work that we’ve done.” ‘tick and flick marking’ because “that’s not what The flexibility given to schools, leaders and matters”, resulting in a net saving in workload. Middle leader teachers in terms of how they implemented laser conversations has had important
78 A better future for assessment Making Waves What conditions contributed to or 3.6.6 Evaluation benefit of being a small ‘family’ of schools detracted from effective implementation? is that it enables constant discussion and How is success or failure being feedback. Because of this, and the fact that Training for middle leaders helped ensure characterised? the MAT sees the innovation as a constantly laser conversations were implemented across evolving and iterative process, it has decided the trust. Training also provided space for As noted in section 3.6.2, the innovation’s not to conduct a formal, structured evaluation. middle leaders to discuss how they would original goals were to: On the other hand, the decision not to implement the innovation within their teams. conduct a systematic evaluation as part of the • develop more valuable, active responses innovation may have made it harder to spot The MAT believes that implementation has to assessment information some of the emerging challenges early on. been made easier by the fact that it is small, • increase professional dialogue because rather than designing an approach between teachers Nonetheless, informal feedback has led and then rolling it out, the approach has • reduce teacher workload. the trust to make adjustments to the been more co-created, with best practice innovation, for example in relation to The central trust team currently shared and adapted across schools. timetabling. Furthermore, it has reviewed define success as: guidance and set out the following However, as noted above, there is a trade- principles for laser conversations. off between consistency and flexibility. The • staff taking a pupil-centred heterogeneous approach to how recorded data approach to assessment was used during laser conversations meant • reducing the amount of unnecessary, that some teachers reported being unclear or ‘pointless’, work for teachers about expectations and some did not see the • improving outcomes for pupils. reductions in workload that they had initially The innovation’s goals have therefore evolved expected. This challenge may be linked to an over time and, for now, there remains some underestimate of how time-consuming it would variation in what goals different individuals be to conduct all the conversations in the larger emphasise. This leads to some variation in schools. This may have made it harder for staff people’s assessments of success. to invest the necessary time in developing and implementing effective action plans and the What approaches are being used to MAT has therefore adjusted timelines for the evaluate the innovation’s success? conversations by spreading them out more. The trust is gathering staff feedback by email and the central team believe that an important
Making Waves A better future for assessment 79 What they are What they are not
A reflection on our: • curriculum • organisation (of students) Data reporting • delivery • resources.
A focus on the progress students are making A summative report based on current attainment
An opportunity to decide what needs to be done to accelerate progress A definitive statement of expectation
An opportunity to share strategies and request advice An opportunity to defend and justify
Assistance in making improvements to practice based on experience A focus entirely based on targets and attainment
A focus on individuals, which provides the opportunity to identify patterns A conversation purely about key groups of students
A professional dialogue that is supportive of teachers and students An extension of performance management
A summary of the conversation will produce actions for the teacher/Director of Learning (DoL) A judgement about the quality of teaching
A summary of the conversation will produce meaningful data for DoLs/the SLT The data production, but the process which led to it
Working hard Not The MAT is also looking to roll out certain practices across schools in response to working hard the strengths and weaknesses of different approaches. For example, following a pilot in maths and science, the MAT is now trialling the use of four quadrants to provide Achieving highly a consistent structure for conversations in certain subjects (see below). The latest guidance also includes examples of questions that staff at different levels might want to Not achieving well ask as part of the conversations.
80 A better future for assessment Making Waves 3.7 in the 21st century. This includes placing • revised formats for Foundational Skills three Core Competencies (‘Communication’, Assessments in Grades 4 and 7. Innovation 7: ‘Thinking’ and ‘Personal and Social’) at the Curriculum and assessment centre of the curriculum model and these will This study will particularly focus on student underpin learning across all ‘areas of learning’ self-assessment in Core Competencies, reform in British Columbia, (subjects). With these Core Competencies acknowledging that this is only one Canada as a foundation, the curriculum in each area aspect of a far wider range of reforms. of learning is structured around a Know- Students self-assess against the three Core Do-Understand model, encompassing its: 3.7.1 Introducing the Competencies – rather than knowledge. innovation • Content (‘Know’) The competencies do not have to be • Curricula Competencies (‘Do’, denoting taught explicitly; they underpin teaching Reforms to assessment and the curriculum skills students will develop during and learning across the entire curriculum. are underway in the West-Canadian and across different grades) The assessment approach is intended to: Province of British Columbia (BC). • Big Ideas (‘Understand’, outlining Instigated by the BC Ministry of Education, • help students to self- and peer-assess principles and key concepts covered the reforms are now being enacted in and to draw students into the reporting during a grade in an area of learning). school districts and individual schools. process so that they take greater ownership over their learning These reforms involve considerable In 2011, the Ministry of Education in BC changes to assessment, including: • increase students’ engagement in their work began work on a sweeping set of curriculum • prioritise formative assessment and assessment reforms, culminating in • students self-assessing (and reporting) over summative tests 2015/16 with the launch of a new curriculum against the Core Competencies, at least • promote positive language and ‘strength- in Grades Kindergarten to 9, followed once a year throughout their education based’ assessment, which describes what by a new curriculum for Grades 10 to 12 • the piloting of a proficiency scale students can do, rather than grades in 2017/18, and ongoing revisions to the (‘emerging’, ‘developing’, ‘proficient’ • ensure more students are able to graduate assessment regime. This is the first time in and ‘extending’), for reporting students’ • use a wider range of assessment BC’s recent history that a comprehensive achievements in each area of learning set of reforms to the curriculum have methods (for example, shifting away • teacher assessment of the been designed and implemented. from a reliance on quizzes and tests Curriculum Competencies towards using observation, interviews, BC’s new curriculum and assessment is • Numeracy and Literacy Assessments dialogue, e-portfolios and so on). intended to better prepare students for life replacing exams in Grades 10 and 12
Making Waves A better future for assessment 81 Province-level reforms are now being ability to communicate effectively. Addressing the revised curriculum’s role in helping pupils implemented in districts across BC, although this perceived shortfall and ensuring pupils develop social responsibility. individual teachers’, schools’ and districts’ have support to develop holistically has ii) Keeping pace responses to the reforms vary considerably. been a key focus for the reforms, which reposition the curriculum (and consequently The Ministry has stressed the need for BC 3.7.2 Agenda setting assessment) as a means to an end. to maintain pace with ‘the 21st-century world’, for example in its 2011 Education What were/are the ‘pain points’ that this Plan, which set out the perceived need for “This is not about curriculum and innovation sought to respond to? a “more nimble and flexible [system] that disciplines being the end destination. can adapt more quickly to better meet the The end destination is the process of The reforms respond to three needs of 21st Century learners”. Its revised ‘becoming’ as a human being. It’s about main perceived pain points: curriculum focuses on ‘education for the human development, that’s the end 21st century’, making personalised learning, 1. The need for pupils to develop destination. The confidence around flexibility and choice key tenets. Many competencies and skills, over and communication, around critical thinking, practitioners, administrators, academics and above developing lots of knowledge creative thinking, around social and policy makers in BC are supportive of this emotional wellbeing, around personal 2. The need for BC’s education rationale. For example, one District Assistant identity – that’s our destination. 5 system to keep pace with other, Superintendent argued that “the world is Curriculum? Of course, you can’t leading education jurisdictions changing and education needs to change”. critically think in a vacuum. That’s the 3. The need to include First vehicle, not the end destination.” For the British Columbia Teachers’ Federation Nations communities. (BCTF), the reforms could also help unify Maureen Dockendorf, the curriculum, ensuring subject areas i) Holistic development Superintendent of Literacy and pull in the same direction. There was a Numeracy, Ministry of Education sense that subject learning – especially BC’s scores in international assessments, at secondary level – had become too including the Programme for International siloed, and that the reforms offered an Student Assessment (PISA), were world- Kim Schonert-Reichl, Professor in Human opportunity to open up the curriculum so class but policy makers and educators were Development, Learning and Culture at the that it had greater overarching coherence. concerned that students’ excellent subject University of British Columbia’s Faculty of knowledge was not matched in terms of their Education, explained that another important personal and social development, nor their dimension has been extending and formalising 5 A superintendent and their team are responsible for standards and budgets in a school district.
82 A better future for assessment Making Waves However, some are sceptical of these to reform the curriculum and assessment assessment regime, some argued that arguments. Glen Hansman, Immediate came from concern that lower graduation teachers would not be incentivised to Past President of the BCTF said rates among First Nations6 students were a develop their pedagogy. Specifically, they discourses pushing for modernisation result of a lack of engagement and support would not be incentivised to move away had been around for decades: for them within the school system: from more ‘traditional’ teaching methods.
Students’ self-assessment against the Core “One should be sceptical of rationales “What did they need for support? Competencies reflects these various rationales, like a rapidly changing world. The world When we started to look into the data, placing greater emphasis on individual students’ is always changing. I could produce it was around engagement, culture needs and experiences. for you newspaper articles from this and climate, feeling welcome in school, Who and what influenced this agenda? province from the 1960s that utilised seeing their culture reflected.” similar sort of constructivist language Jennifer McCrea, Assistant Deputy i) The philosophical foundations around getting rid of siloed subject Minister, Ministry of Education The reforms’ foundations were laid in the areas, students take charge of their 1988 ‘Sullivan Report’,7 written because of learning and teachers being facilitators, the ‘unprecedented challenges’ posed by rather than somebody at the front In addition to the three reasons for reform shifting economic and technological trends. of the room sharing information.” outlined above, while most educators see teachers’ practice as a strength of The Ministry of Education responded with Glen Hansman, Immediate the BC system, there is a feeling among Year 2000: A framework for learning, a Past President, BCTF educators that pedagogy needs to evolve. plan for reforming education by enabling In particular, teachers’ assessment practice ‘learners to develop their individual potential and… contribute to a healthy society and iii) Inclusion of First Nations communities has been deemed by policy makers and 8 many educational thought-leaders to be too a prosperous and sustainable economy’. Educators, policy makers and the wider dependent on a narrow range of practices that It emphasised the need for personalised public in BC are keen to improve the quality could not support new system-level priorities and inquiry-based learning, and for pupils of education for, and about, indigenous such as inclusion, the co-construction of 7 Royal Commission on Education (1988) A Legacy communities. This has long been perceived learning and holistic development. Without for Learners: Summary of findings, Victoria: Royal as substandard across the province, and wholesale reform of the curriculum and Commission on Education, available at: https://crofsblogs. there is a fear that it will remain so unless typepad.com/files/legacyforlearnerssummary.pdf curriculum and assessment reform places 6 ‘First Nations’ is a term encompassing many different 8 Ministry of Education (1990) Year 2000: A framework indigenous groups. For more information, see: for learning, Victoria: British Columbia Department this issue front and centre. The motivation https://www.welcomebc.ca/Choose-B-C/Explore-British- of Education, available at: Columbia/B-C-First-Nations-Indigenous-People https://files.eric.ed.gov/fulltext/ED357999.pdf
Making Waves A better future for assessment 83 to develop ‘the thoughtfulness, creativity, “[The Year 2000 report] was just way “The kind of Ken Robinson approach flexibility, motivation, and responsibility that ahead of itself and it was a desire to to revitalising education I think had characterize educated citizens’. However, make public education meaningful to all significant currency in British Columbia many parents and teachers expressed concern learners. It was a desire to ensure that for probably a couple of decades.” about moves towards ‘anecdotal reporting’, they were engaged, that they had a sense George Abbott, former and universities were unhappy with the extent of agency, that they had voice. … So the Minister of Education to which students were prepared for their whole focus around social and emotional studies. 9 Consequently, the New Democratic wellbeing, social responsibility, personal Party (NDP) abandoned many of the Year 2000 awareness began in the nineties.” Notably, these ideas influenced Rod Allen, a pledges when it took office in the early 1990s. Maureen Dockendorf, teacher, Principal and Superintendent turned Nonetheless, influential educators and policy Superintendent of Literacy and Associate Deputy Minister in the BC Education makers who developed the current curriculum Numeracy, Ministry of Education Ministry. Rod worked closely with Deputy and assessment reforms cite both the Sullivan Minister James Gorman (who was, at the and Year 2000 documents as influential in time, concerned about his Grade 4 son’s lack shaping their thinking. For example, Jordan ii) Progressive educational philosophy of reading ability) and with Minister George Tinney, Superintendent in Surrey, the Abbott to design and pass the curriculum and Politicians, educationalists, academics largest school district in BC, explained that assessment reforms. Gorman identifies Allen as and senior policy makers attribute the the documents’ recommendations were “the intellectual leader” behind the reforms. reforms to the growing popularity of “foundational” in shaping his and others’ progressive educational philosophy. views about education, and that many of Organisations such as the Global Education the ideas were “dusted off” in 2011. In the late 2000s, the ideas of John Abbott Leaders’ Programme (GELP) and the Canadian 10 21st Century Skills Consortium were also Maureen Dockendorf was seconded to (author of Overschooled but Undereducated ) and other progressive educationalists were influential in shaping thinking underpinning the Ministry of Education to help engage the reforms (including at the Ministry), school districts and teachers in the reforms, taking hold among BC politicians, policy makers and teachers. For example, former Minister of adding to a desire for radical transformation and explained that the Year 2000 text of the BC system. For example, Allen and “set the tone” for the latest reforms: Education, George Abbott, recalls “considerable affection” for Ken Robinson’s arguments Gorman asked Valerie Hannon, the founder 9 See Sanford, K. and Hopper, T. (2019) Educational of the UK’s Innovation Unit and co-founder Transformation: The BC story, Victoria: about creativity and individualised learning: Houlihan Publishers; Kilian, C. (2011) ‘The last of GELP, to speak to Cabinet in order to time we had an education commission’, The provide an ‘external voice’ and gravitas. Tyee, 9 March, available at: https://thetyee.ca/ 10 Abbott, J. (2009) Overschooled but Undereducated, Opinion/2011/03/09/LastEducationCommission London: Continuum International Publishing Group.
84 A better future for assessment Making Waves iii) Political timing and buy-in – to education. Clark was subsequently ‘hands off’ with education and, because the rationale While Gorman attributes what he sees as the for reform had already been communicated reform’s successes to “good manufacture and effectively to Cabinet, Abbott and his team good design”, he also believes it was “good were left to proceed unimpeded: luck and good timing, too”. When Premier Gordon Campbell11 asked Gorman to run the Ministry of Education, he told Gorman “[Clark] was quite happy to largely let “your job is just to keep it quiet and keep it me go in terms of what we were trying to out of the news”. After the 2006 election, do. And again, what we were trying to do Campbell reprimanded Gorman, saying – in was not unfamiliar to caucus and Cabinet Gorman’s words – “why the hell are you not because of some of the speakers that we making substantive change to education?” George Abbott entered the Ministry for the had at different caucus sessions and so Gorman replied saying the Ministry was ready first time in 2010, and suggests that “when on. … There wasn’t any resistance from to do so, and sought Campbell’s support in I arrived in 2010, 2011, the time was really the Premier’s office under Clark. … I don’t securing the political buy-in from Cabinet: right for this to move forward”. While not an think she had any interest in battling… education expert, Abbott described feeling She might battle on other political issues, “I said, ‘Look, I’ve got people busting at an immediate affinity with the work already but she would not I think expend a lot the seams to make substantive change underway within the Ministry. He and Gorman of political capital on trying to influence here’, and I said, ‘If you’re interested recall how Premier Campbell gave them the educational reforms that clearly were well in doing it, then you help me bring space to develop ideas, and political support defined already and [well] perceived.” to achieve buy-in from Cabinet. They achieved Cabinet along and we’ll get there.’ former this by asking influential thinkers to present to George Abbott, So we started this process and we Minister of Education must have made 15 trips to Cabinet, Cabinet, and Gorman explained how speakers pitching how we were going to do it.” such as Valerie Hannon “did a beautiful job” of setting out the need for change. James Gorman, former Deputy Perhaps ironically for a party associated by Minister of Education Abbott lost the provincial government some with austerity and a reputation for leadership battle to Christie Clark in 2011, but squeezing public services, reforms in education was given his pick of Ministries afterwards and took on a wide-reaching, ambitious and 11 The Premier is the head of the provincial progressive bent. One senior policy maker government. In Canada, provincial governments an assurance that Clark would support his wield a lot of power, with responsibility for agenda for reform. Abbott returned – by choice within the Ministry of Education said ministers education, health and other public services.
Making Waves A better future for assessment 85 seized the opportunity to reform education. Abbott saw the reforms as an opportunity Abbott, Gorman and Allen (among This high-level political support was vital in the to build bridges with the BCTF, and he and others) all felt that Abbott’s investment in reforms gaining traction. others (including Rod Allen and James communicating and building bridges with Gorman) believe that the curriculum and these key groups had been hugely important iv) Involving the British Columbia Teachers’ assessment reforms helped achieve this even in building support for the reforms. Federation and other stakeholders during ongoing and acrimonious negotiations v) Sociocultural imperatives Policy makers and politicians acknowledged about other areas of education. Even though that, in theory, the reforms could have been Lambert always remained “cool” towards The social, cultural and political debates designed without BCTF support. However, the Ministry, dialogue about the reforms taking place in BC about the rights of, and the Ministry’s relationship with the BCTF was helped build trust with her deputies, both reparations owed to, indigenous peoples fundamentally important in both symbolic and of whom later became BCTF Presidents were significant influences on assessment and practical terms. Relations between the Ministry and who speak favourably of the Ministry’s curriculum reform. The First Nations Education and the BCTF were woefully poor when Abbott work to involve the BCTF in the reforms. Steering Committee (FNESC) was guaranteed arrived. Abbott said that his first call when he seats on each of the Ministry’s curriculum According to George Abbott, other arrived at the Ministry was to Susan Lambert, committees. Furthermore, from the early stakeholder groups also played a vital then President of the BCTF: 2000s, the grassroots Networks of Inquiry role in building support for the reforms, and Indigenous Education (as it is now called) including the BC School Superintendents were exerting considerable influence over “The first call I made as an education Association and the British Columbia School the teaching practice of a small (but growing minister - and I think it was in retrospect Trustees Association. Abbott spoke in and prominent) group of teachers who held a really wise thing to do - was to make my person to these groups, attending meetings sway over the education debate. The sorts of first call as a minister to Susan Lambert, to update them on the Ministry’s work: practices these networks supported aligned who was the President of the BCTF, ideologically with the Ministry’s emerging just to say, ‘Hey, this may surprise you, “One of the things that I like[d] to ideas. As Markus Baer, Director of Assessment but I wanted to make my first phone do, particularly when attending their and Reporting at the Ministry, put it, “we call as a minister to you. It’s no secret conferences, [was] give them lots of don’t do anything without the involvement that our government has had a poor time. I would spend the whole day there. of our indigenous educator colleagues”. relationship with you for a long time. When they ask me to speak, I’d say, ‘I’ll I’d like to see if we can change that.’” take questions until you’re tired of [me].’” While the commitment to students self- reporting against Core Competencies emerged George Abbott, former George Abbott, former from the curriculum and assessment drafting Minister of Education Minister of Education and revisions process undertaken by various
86 A better future for assessment Making Waves committees from 2011 onwards, the idea’s being involved in the reforms, and these shifts in pedagogy. Importantly, it is foundations were laid during the 1990s and highlighted a range of ways the reforms hoped that all teachers – rather than an early 2000s. The research and writing of have been important. However, there enthusiastic minority – will embed these Caren Cameron, Anne Davies and Kathleen was lots of overlap in what they said. approaches in their teaching. Sharon Jeroski, Gregory had been influential among teachers a former advisor at the Ministry and key For example, some – including academics and academics, who themselves were later architect of the Core Competencies curriculum involved in teacher training, district heavily involved in designing the revised approach, explained that self-assessment could administrators and teachers themselves curriculum. For example, Dr Leyton Schnellert ensure teachers were not overburdened while – talked about how curriculum and sat on the initial Curriculum and Assessment also gathering more valid information: assessment reforms had the potential Framework Advisory Group, and explained: to reshape teaching practice by: “We kept coming up against the issue of • raising students’ awareness of how “We have a long history of student how secondary teachers, who already they learn, as well as what they learn self-assessment in this province. … we were feeling overburdened, were did that work extensively in the late 90s • giving students a greater role going to be able to somehow develop and early 2000s and all through the in their own learning a reporting system for Core Comps 2000s, and so formative assessment and • encouraging students to develop the for students who had as many as eight student self-assessment was something skills and capabilities associated with the different teachers. We kept coming back we’d been working on extensively. Core Competencies (in turn enabling to the fact that the only person in the So the idea of self-assessment for them to flourish at school and in life) process who had enough knowledge of the Core Competencies built on a • repositioning teachers as facilitators all the various pieces to summarise and practice that we were familiar with.” of learning, rather than ‘lecturers’ report on them was the student, and if somehow, we could find a way to make Dr Leyton Schnellert, Associate • encouraging collaboration between self-assessment the guiding process, Professor, Faculty of Education, teachers, within and across there was a hope it would all work.” University of British Columbia departments and year groups • increasing the inclusion of vulnerable Sharon Jeroski, former advisor and marginalised groups and in to the Ministry of Education Why was it considered an important particular First Nations students. agenda to pursue? Assessment reform, particularly the increased The reforms also continue to be seen as Different individuals and organisations role of self-assessment against the Core important in reducing teachers’, parents’ expressed different motivations for Competencies, is seen as a way of embedding and students’ reliance on summative grades
Making Waves A better future for assessment 87 and percentages. The hope is that formative Students’ self-assessment against the Core the reforms and initial approach. Equally, feedback and self-reflection will gain greater Competencies also reflects the wider education Allen, Gorman and Abbott (among others) all prominence, replacing the current preference reforms’ overall goals of promoting: recognised the role that policy experts such as for summative judgements. It is hoped this Walt played in moving the ideas into policy. will have a knock-on impact for students’ • skills and competencies, wellbeing, and, particularly at secondary alongside knowledge ii) Committees level, reduce the ‘obsession’ with summative • social responsibility Rod Allen emphasised the need for the judgements. In turn, policy makers, teachers, • inclusion Ministry to ‘co-construct’ the curriculum union officials and academics believe the shift • the drawing in of a range of perspectives, and subsequent assessment systems towards formative feedback and greater self- including indigenous perspectives with a wide range of stakeholders. This agency among students has the potential to • flexibility and autonomy for involved establishing various committees to make assessment more inclusive. A number teachers and students shape the curriculum principles and, later, of contributors, including policy makers, • student self-awareness and agency, subject-specific content and approaches practitioners and academics, felt the grade- and with students taking ownership over to assessment. These committees involved percentage-reliant system placed vulnerable and co-constructing their learning. individuals and organisations including: and marginalised students – often including members of indigenous communities – at a • University-based academics disadvantage. This is because these groups 3.7.3 Formulation • The BCTF often perform worse under such assessments, How was a solution arrived at? • Indigenous communities potentially driving disengagement. • Representatives from independent schools i) Key individuals within the Ministry Young people highlighted the stress that • Superintendents. summative scores can cause: As described, above, a small team at the • School administrators including Principals. Ministry was instrumental in setting the • Organisations including GELP and “It’s really competitive. Someone will vision for the reforms and ensuring they the Organisation for Economic Co- be like, ‘I got an 81.’ And other kids gained traction. George Abbott – Education operation and Development (OECD). are like, ‘I got an 82 – hah!’ It’s really Minister between 2010 and 2012 – was crucial • Employers and industry representatives. competitive. I also know some kids who politically, persuading his government’s Cabinet [are] so stressed out about their marks to support the reforms. Within the Ministry, Committees set ground rules, emphasising that they can’t see anything past it.” Nancy Walt – a senior policy maker – said First Nations principles. Rod Allen explained Secondary school student that Deputy Minister James Gorman and Rod that acknowledgement of (and adherence to) Allen worked closely with Abbott to shape these principles was the “starting point for
88 A better future for assessment Making Waves everything”, but also helped build coalitions the curriculum and assessment arrangements A senior policy maker with oversight of the with indigenous communities. The first out for consultation. Early drafts had over eight different committees’ work said there had committee explored underlying principles million online hits. been two key disputes during the committees’ that would go on to shape the curriculum drafting phase. The first was in relation to the content and ‘areas of learning’ while, later, There was generally said to be broad, environment, and specifically the prominence committees had more specific remits. philosophical alignment among committee (or perceived lack thereof) of content relating members. For example, the co-chair of an to climate change, and a clash with mining The process was intended to be very open; assessment review group said: companies who felt their industry was not for example, members of committees given due prominence. The second was in were not required to sign non-disclosure relation to maths, and concern that a cross- “The Ministry had an agenda that it agreements, as can sometimes be the case curricular approach would threaten standards. wanted to move forward, and that’s with this sort of work. Instead, the Ministry The former issue was resolved through raising why [we] were asked to chair the encouraged committee members to talk to the profile of climate change and discussions committee in particular, because colleagues about what they were discussing, about industries’ roles in mitigating it in the they thought we probably shared a and gather feedback for the next session. subsequent iterations of the curriculum. political outlook on moving this agenda This was to build awareness and buy-in for The latter ‘died down’ in the face of broad forward, around assessment.” the reforms, and also so that committee consensus among other committee members. members could gather feedback from their Blye Frank, Dean of the Faculty of peers and colleagues about the ideas. Education, University of British Columbia The Advisory Group on Provincial Assessment submitted its final report in 2015, and One contributor noted the importance of recommended that self-evaluation be ‘grassroot’ contributions to the process, saying This shared ‘philosophical outlook’ meant ‘embedded into the learning and “we would not have this curriculum” without that committee members often broadly assessment process’.12 such “deep partnership”. agreed with one another. A member of the Ministry’s Curriculum and Assessment The idea of student self-reporting against The Ministry, and domestic and international Core Competencies,emerged from this cycle experts also fed into the process including Framework Advisory Group, responsible for the initial design of the revised curriculum of creating and revising curriculum and influential educationalists within the province assessment framework drafts. including were brought into the Ministry to and assessment frameworks, explained the provide guidance on the reforms and shape the group did not start out from a ‘blank slate’. The 12 Magnusson, K. and Frank, B. (2015) Advisory Group on committees’ work. As the committees’ work Ministry selected people who could help build perceived best practice into the heart of the Provincial Learning: The graduation learning years: Final progressed, the Ministry put draft versions of report, available at: https://curriculum.gov.bc.ca/sites/ revised curriculum. curriculum.gov.bc.ca/files/pdf/agpa-final-report.pdf
Making Waves A better future for assessment 89 iii) Research, Consultation, communication
Academics and policy makers working at or with the Ministry to design the reforms emphasise how important research was in shaping decisions about the curriculum and assessment. In particular, research published by the OECD, GELP and other international comparisons was influential.
However, some practitioners expressed scepticism about the presence and quality of research used to justify the reforms: Universities were sometimes more sceptical, parents and other local stakeholders and had questions about how the revised including employers. This communication “I’ve searched and I’ve searched, assessment arrangements would affect looked different in different districts, [and] my opinion is that this has admissions procedures and enable them to but included combinations of: not been based on research; it’s distinguish between applicants. However, not on peer-reviewed evidence. It’s some universities were more supportive than • meetings based on philosophy or ideology.” others and Jan Unwin, the Superintendent of • online communication, including emails Secondary school teacher Graduation and Transitions at the Ministry, and newsletters, and signposting draft was able to leverage support from certain content on the Ministry’s website universities to persuade those who were • paper communication Alongside the committees, districts (normally less supportive. Unwin also visited teacher (particularly with schools). in collaboration with local branches of training faculties at universities to speak the BCTF) held information and feedback to cohorts of new teachers, explaining: 3.7.4 Decision making sessions with teachers, district administrators “Here’s the transformation, here’s why, (including school leaders) and parents. here’s what we’re hearing, here’s what Who was involved in deciding on this Feedback from these sessions was fed up we’re doing, here’s what it looks like.” approach to the pain points? to the Ministry. However, this process of sharing feedback with the Ministry has been Individual districts communicated about Key decision makers tended to be stronger in some districts than in others. the reforms with teachers and schools, ministers and their deputies, including:
90 A better future for assessment Making Waves • Rod Allen, the Assistant Deputy With the political will and permissions in Education Director put it) around the Core Minister of Education place from Cabinet, ministers could therefore Competencies model, and a more skills • James Gorman, the Deputy Minister unilaterally adapt and create the legislation focused curriculum and assessment regime). required to implement the reforms. This • George Abbott, the Minister. A drawback of the high degree of consensus was done for the majority of the reforms is that committees may have been vulnerable Part of the reason ministers played such a key before the change in government in 2017, to ‘groupthink’. One contributor expressed role is that, despite the far-reaching scope by which point sufficient legislative progress concern that the ‘Overton Window’ – that of the reforms, only limited legislation was had been made to safeguard the bulk of the is, the parameters for what it is and is not required. A senior policy maker explained reforms. This is not the case for reforms to acceptable to say as an educator – have that the key legislation relating to schools in the Provincial Assessments at Grades 10 and shifted towards more progressive views of BC (the School Act) is written at such a high 12. The legislation for this was not passed education. This impacted on the reforms’ level, it “doesn’t seem to need to be touched”. before the change in government, and this design, because more traditional views Consequently, the necessary legislation for has proven a political sticking point since (regarding the importance of knowledge, the curriculum and assessment reforms could then, with the new provincial government memory and practice) were seen as outmoded. be passed as Ministers’ Orders, which “don’t scaling back on the initial promises to remove Some contributors argued that while those have to go anywhere else”. She commented graduation assessments altogether. designing and advocating for the reforms that legislation has been a sticking point in Why were other solutions rejected talked about ‘consultation’ and ‘dialogue’, other jurisdictions attempting reform: and this approach preferred? they in practice “just heard what they wanted to hear”. However, one senior policy maker Some contributors during the committee argued that the curriculum still contains lots “They really get bogged down in this, phase advocated more ‘conservative’ of content, but laid out in a different way. with governments changing, or just [in] suggestions prioritising knowledge and more debate. We’re seeing this even in our Sometimes there were disputes in subject- didactic forms of teaching, and expressed neighbouring province, Alberta. This is specific committees about the content and concern about the emphasis in the new playing out a little bit in Ontario. Lots of structure of the ‘areas of learning’. Some felt curriculum and assessment arrangements views [on] what curriculum should be, the reforms were not going far enough, and on skills and competencies, and student but we didn’t have that legislative angle essentially wanted to scrap subject labels agency. In particular, they felt the reforms at all. I think that surprises people.” altogether. Others expressed concern that the would pull the rug from beneath teachers’ reforms were not applicable to their subject. Nancy Walt, Executive Director, feet, undermining existing good practice, For example, there was disagreement about the Ministry of Education, British Columbia and threatening standards. However, there extent to which mathematics could be taught was “critical mass” (as one Ministry of and assessed in a cross-curricular fashion.
Making Waves A better future for assessment 91 Furthermore, one union representative following the change in government. New different districts – this pilot has been suggested that “a lot of the stuff early on was Provincial Assessments (cross-curricular extended by a year, to allow districts sort of predetermined in terms of structure”, literacy and numeracy assessments at Grade more time to embed changes and because this pre-dated the BCTF’s 10, and a cross-curricula literacy exam at • district-level communication with schools, involvement, many teachers were excluded Grade 12) are now being phased in. The teachers and parents about the reforms from initial discussions. This mattered, government decided to slow down the • individual teacher and school- because decisions were taken during these reform of Provincial Assessments in part level experimentation. discussions regarding the nomenclature for because of concerns among parents (that the Core Competencies and the structure is, voters) and universities. A senior policy Implementation of differs between districts. that would be used to present the revised maker within the Ministry explained this was Key differences include: curriculum. These provided the subsequent “not the hill to die on” when far-reaching foundation for future committees’ work. reforms to curriculum and assessment are • Timeframes. Different districts underway across Kindergarten to Grade 9. have become involved in the pilot in This generated some suspicion about how open different years, some as early as 2016. the drafting process was. However, another 3.7.5 Implementation • Numbers of participating teachers and commentator – a university academic involved schools. In some districts a high proportion in several of the committees – observed What mechanisms have been used of schools are involved. In others, the pilot that some teachers who were resistant to to implement the approach? only involves individual teachers and schools. the reforms had also complained during • Approach. Depending on the ‘pathway’ previous sets of reforms in the 2000s. She The Ministry is overseeing the assessment the district has selected, in some districts hypothesised that these teachers’ reservations and reporting pilot, although day-to-day implementation is far more extensive than stemmed more from a conservative outlook responsibility lies with individual teachers, in others. For example, some districts on reform and resistance to change, than schools and school districts. have moved wholesale towards the new an objection to the new reforms per se. Implementation is ongoing and involves: assessment and reporting frameworks There has also been some disagreement across Kindergarten to Grade 9, while with regards to Provincial Assessments. • documentation, including curriculum others are only adopting certain elements. The government had previously announced documents and assessment guidance • Communication. Some districts that it would scrap these end-of-secondary available on the Ministry’s website are communicating more widely assessments on the grounds that they ’stifle • a Ministry-coordinated pilot of than others about the pilot and creativity’ and result in teaching to the test. assessment and reporting arrangements associated reforms. Some districts However, this decision was reversed in 2017 for Kindergarten to Grade 9 in 13 are focusing their communications
92 A better future for assessment Making Waves on teachers and administrators, while Core Competencies is appropriate as there links and other infrastructure challenges others are involving parents, students is a lot of flexibility available to teachers and including broadband access can slow uptake and employers too. Districts use so the reform does not feel heavy-handed. in more sparsely populated and rural areas. different modes of communication, Furthermore, teachers are still required to including in-person meetings, online assess pupils in line with the Curriculum ii) Conceptions of assessment communications including emails and Competencies, providing pupils with feedback The move away from allocating grades and videos, and paper communications on their academic progress. However, even percentages challenges many teachers’ including posters and letters. the teachers, schools and districts who whole- conceptions of assessment and not all heartedly support the reforms, are finding their teachers feel comfortable with the shift. Consequently, in some classrooms, students way, and implementation has been challenging. are self-assessing their learning against the iii) Reliance on summative judgements competencies in most lessons; in others, this is What obstacles had to be overcome as a one-off activity at the end of term. part of implementation? Many teachers, parents and students still prioritise summative judgements, including To what extent is the reified solution Interviewees talked about a range of challenges percentages and grades, over and above consistent with the planned/ in relation to implementing recent education other forms of feedback encouraged by the intended solution? reforms including: reforms (including self-assessment). Even As yet it is unclear how the requirement for teachers supportive of the shift towards i) Geography students to self-report each year will affect different forms of feedback acknowledged teachers’ practice, and the extent to which BC is a vast province, covering nearly 1,000,000 that parents’ and students’ views will take time teachers, schools and districts have been km2 stretching up Canada’s western coast. It to shift. Furthermore, parents’ scepticism responding to the reforms and embedding the has a population of around five million, and has been compounded in some districts changes varies hugely. approximately half the population lives in the by inconsistent reporting; some teachers Vancouver metropolitan area. This means provide narrative feedback, and others stick Many are enthusiastic about the reforms and to grades. One student said her parents how they are playing out. Academics, teachers large areas are sparsely populated. This presents challenges for implementing and “hate” the new proficiency-scale-based and administrators who engaged with the system because they do not understand it. committees and consultation process are monitoring new policies such as the revised curriculum and assessment system. Some generally enthusiastic about the changes, saying iv) Teachers’ confidence and knowledge the reforms represent a logical extension of contributors observed that while implementing work they were doing already. They believe reforms is comparatively straightforward Many teachers are subject experts and the process for self-assessment against the in urban and metropolitan areas, transport experienced at creating and deploying
Making Waves A better future for assessment 93 assessments in their subject areas. However Others questioned whether offering “We had this deputy minister [visit], not all teachers feel comfortable supporting support was worthwhile in the first place: [and] one of our colleagues asked, other forms of assessment (including well, ‘Is there a robust, peer-reviewed helping students self-assess against the research base that supports all these Core Competencies). This challenge is “I’m not going to set up educational changes?’ [The minister…] clearly wasn’t compounded by difficulties in specific systems for dinosaurs when ready for that question. He said, ‘Yes, phases and subject areas as well as by time, mammals are thriving. They’re going there is, we’ll get back to you on that.’” with many teachers feeling they do not to be extinct soon enough.” Secondary school teacher have time to give the reforms the attention needed to embed them effectively. Dr Kris Magnusson, Dean of Education, Simon Fraser University A number of contributors less in favour of Many teachers are unsure how best to help (or in some cases opposed to) the reforms students self-assess and report against explained a sense that alternative viewpoints the Core Competencies. The Ministry has v) Engaging teachers were being shut out of the debate. For example, published examples and guidance, and Some teachers have been resistant to the views that emphasise the centrality of subject- technology companies are offering solutions reforms. One school Principal said that specific knowledge, memory and practice to (for example, FreshGrade, an online reporting while some of her staff had whole-heartedly learning, as opposed to cross-curricular and platform). However, this technological adopted the changes, other colleagues were generic skills, not only felt their views are support has also proven problematic. “dragging their heels”. She believes that this under-represented in the reforms, but that it is taboo to raise such concerns at all. This One teacher said he felt more resources stems from a combination of scepticism has alienated some teachers, administrators, were needed to help teachers (especially new about the reforms themselves, and feeling parents and pupils with different views about teachers) to implement the changes. the reforms are “top down”, and teachers not liking “being told what to do”. what effective teaching looks like. Others While interviewees from virtually all schools counter that teachers’ autonomy remains an This also stems from a sense among some and districts talked about struggling to important principle underpinning education teachers that the reforms are ideologically update their practice, there is disagreement in BC, and that knowledge and content driven, rather than evidence-based. Some about how to respond to the challenge remain important in the revised curriculum. teachers requested more information from of a deficit in teachers’ knowledge and the Ministry about the reforms and felt their vi) Transferability across subjects confidence. Some interviewees said districts concerns had not been adequately addressed: and schools do not have sufficient funds to Most of the teachers and pupils we spoke to support all teachers to shift their practice. observed that the reforms impact on different
94 A better future for assessment Making Waves A lot of content is “buried in hyperlinks”, again meaning that while the curriculum itself is more succinct, teachers need to do more work to find out the content they are expected to cover. Indeed, despite a rhetorical emphasis on teacher autonomy and learner freedom through scaled-back curriculum content, some areas of the curriculum feel cluttered. For example, Neil Stephenson said that in Kindergarten: subject areas differently. For example, teachers not the curriculum specifies what this is. He explained that while English and social studies said responsibility has shifted onto teachers “There are 28 big ideas (although these have been changed quite dramatically, fewer to work out what this component content is: are not mandated), plus curricular and substantive changes have been noted in maths Core Competencies, plus you’re trying to and science (particularly at secondary level). “For example, in social studies, 12-year- develop literacy and self-regulation, right? How [can] teachers... develop and assess vii) Curriculum design olds explore ‘the anthropological origins of humans, human responses and design learning for 28 big ideas for a While the volume of content in the written to particular geographical challenge room full of six-year-olds? The curriculum curriculum has in many areas been scaled including climate, landforms, features requires a lot of interpretation and back, it is still necessary to teach more than is and characteristics of civilizations that filtering to be useful by teachers. The specified in the key curriculum documentation. lead to their rise and fall’. That whole end goals of the new curriculum are For example, one district administrator said learning outcome is stated in just one admirable; I do wish it was a little more that a teacher of Grade 6 social studies is content bullet – but behind it is a great streamlined to help teachers focus in now instructed to cover “huge ideas around deal of specific ideas, vocabulary and on fewer essential learnings connected global conflict and patterns of interconnection skills. So the curriculum looks reduced, between grades and subject areas.” between developed countries”, “so, sure, it fits but the foundational knowledge needed Neil Stephenson, Director of on one page now, but the ideas are massive”. to develop understanding doesn’t Learning Services in a school district change, it’s just not as readily available.” Giving another example, he explained that to Neil Stephenson, Director of understand electricity, pupils still need to know Furthermore, there are some concerns Learning Services in a school district the same component content, whether or about curriculum coherence since objectives
Making Waves A better future for assessment 95 at different grade levels do not necessarily BCTF representatives raised concerns about cohere with objectives lower and higher up the students’ self-reporting at the secondary level, system. For example, a district administrator given the lack of exemplars from the Ministry. explained that Kindergarten outcomes in Perhaps partly as a consequence (as some subject areas do not align with those well as because any new system takes in Grade 1, which in turn do not link fully time to get used to), there are concerns with those set out for Grade 2. In his words, about certification and progression, the curriculum “does not hang together”. particularly when it comes to university. viii) Inconsistency in assessment Despite self-reporting being used only up to While teachers and students said there had Grade 9, some universities are concerned about always been discrepancies in how individual Lots of commentators argued that the revised interpreting and using the revised assessment teachers approached assessment, marking approaches to assessment (including the outputs when offering prospective students and reporting, students expressed concern revamped Provincial Assessments) make places. We heard about some universities that the new system would exaggerate them more accessible and, by extension, changing their entry criteria to make more these differences. This could mean teachers’ something a greater number of pupils can of competency-based judgements about perceptions and biases exert greater engage with. However, others (including applicants’ suitability. However, students influence over students’ outcomes: teachers, parents and some pupils) suggested and teachers alike expressed concern that even though the Provincial Assessments that they do not know how universities “Each teacher teaches differently; they appear on students’ transcripts, they no longer in general will respond to the reforms: have different styles and different need to be passed in order to graduate, which ways to share their knowledge with some pupils find demotivating. One parent their students. Some classes might be felt the process of sitting an assessment that “The thing that I was really concerned harder than other classes, so for two does not need to be passed “wastes times”. [about is that] they [the government] students taking [the same class with] two took off the Provincial Assessments Another issue is the modes of assessments different teachers, one might have 90% and they just put the numeracy and themselves. For example, while gaming was in because that teacher is really easy or the English assessments aside. We theory possible under the previous system, something, and there might be another don’t know if the universities are this risk is heightened especially with students teacher who is really hard… and the other looking at that. Now we’re just solely self-reporting, and these assessments form student gets like a 60 or something.” based on the teachers’ marks on us.” part of their applications to universities or jobs. Secondary school student Glen Hansman noted that on several occasions Secondary school student
96 A better future for assessment Making Waves However, a policy maker at the Ministry of cited an absence of communication about ix) Technology Education said that this will become clearer aspects of the pilot and felt frustrated that this over time and welcomed universities’ shift to a is leading to duplication of effort. Administrators, teachers and students talked more ‘broad-based’ admissions process. about adopting electronic and online systems for assessment (such as sharing comments and Some teachers and students suggested the “I clearly wasn’t paying attention to some photos of students’ work), and reporting scale reforms might ‘move the goalposts’ without meeting somewhere obviously and I results and grades. There are teething issues, tackling the underlying causes of stress when it didn’t realise that all of our proficiency including a lack of alignment in how grades are comes to university applications: scales would have to be converted to reported on these platforms. One teacher said: a letter grade at the end of the year. I didn’t know that that would have to “They look at your extra-curriculars happen. So now at the end of the year we “The computer programme does not on top of your academics, so that’s now have to take all those proficiencies really fit the new assessment styles why there’s a lot of stress on us. If and convert them to a letter grade.” that we’re trying to implement. So you want to get into UBC [University there were a lot of staggered starts and of British Columbia] there’s so many Secondary school teacher stops at the beginning of the year.” things you have to qualify for.” Secondary school teacher Secondary school student This is striking given that converting the proficiency scale results is not in fact a requirement. There have also been challenges in getting viii) Lack of clarity teachers and parents to adopt the new systems. Many of the challenges mentioned here are Some teachers involved in the pilot have particularly pronounced at secondary level, x) Preparation for adult life felt confusion or frustration during the pilot where teachers are more accustomed to Students expressed concern about the fact despite in many cases having self-selected working in discrete subjects and reporting using they are ‘guinea pigs’ trying a new system out into the pilot because they approve of the summative grade or percentage judgements. without being clear whether the new system changes to the curriculum and assessment Furthermore, teachers spend comparatively is better. Some are optimistic and welcome frameworks. This has resulted from confusing less time with their students at secondary level, the emphasis on the assessment of skills and or contradictory guidelines about what was which raises questions about their ability to competencies because they do not believe happening, including the emphasis on moving accurately judge students’ development under that simply learning content will help them away from letter grades and a simultaneous the competencies. requirement to ‘convert’ scale ratings. Others flourish after school. However, others do not
Making Waves A better future for assessment 97 believe the new assessment and reporting • Communication: Many interviewees 3.7.6 Evaluation frameworks are beneficial, and one student felt that the Ministry’s work was well- even said she found the reforms “patronising”. intentioned, but that sometimes the How is success or failure reforms were poorly communicated. being characterised? What conditions contributed to or • Funding: There is not enough detracted from effective implementation? Success is characterised in different funding available to pay for cover ways by different people, at different Interviewees highlighted several factors that and free up teachers’ time so levels but notions include: have inhibited effective implementation: that they can communicate, plan and review implementation. • Improved communication between schools, • The scale of the task: The reforms • Levers to influence practice:BC does parents, employers and universities involve revising an entire curriculum and not have some of the ‘sticks’ present in • Maintaining (and being seen to assessment system which has been in place other systems for incentivising change such maintain) educational standards for years. This has created a huge volume of as a schools inspectorate or league tables. and league table positions. work and uncertainty for lots of teachers. Where enthusiasm and goodwill are lacking • Reducing the perceived need • The substance of the reforms: Some among teachers, principals therefore do for private tuition interviewees felt the curriculum and not possess the levers to force uptake of • Shifting priorities and pedagogical assessment reforms themselves are flawed. the reforms by individual staff members. approaches in the education system. • Confusion: Because of the scale of • Pull-back by government: The change the reforms, some have experienced The extent to which members of each of government in 2017 has led to some confusion and uncertainty about what group are aligned behind these different political reticence, slowing down the final needs to change and when. This has been outcomes and the extent to which they phases of implementation. This applies particularly prevalent for the reporting believe the reforms are the right way of particularly to the reporting arrangements, pilot and for Provincial Assessments. achieving the outcomes are mixed. For with politicians nervous about the scale With regards to the self-assessments, example, teachers want to secure the best of change and removal of ‘universal’ many are unclear how to embed these education possible for their students and benchmarks. While the new government effectively, or what ‘good’ looks like. some see the reforms as a helping with this is committed to seeing implementation • Prioritisation: Student self-reporting goal, while others are less convinced. through, the revised Provincial Assessments has achieved limited traction, in part may more closely resemble their What approaches are being used to because this is such a small part of predecessors than had initially been planned. evaluate the innovation’s success? a much wider set of changes, with many practitioners describing self- The Ministry is currently conducting a pilot in 13 assessment as an ‘afterthought’. school districts. This has been underway since
98 A better future for assessment Making Waves 2016. The pilot has been extended for another from the pilot was being captured. Some educational equity were below the average for year to give schools and districts more time were not aware of any formal mechanisms for Canadian provinces in the 2018 PISA results. It to embed the changes. The extent to which passing feedback back up to the Ministry. is however worth noting important limitations findings from the pilot are being effectively when it comes to Canada’s PISA data14. There is resistance to more structured fed upwards to the Ministry is unclear, and evaluation because of deep-set ideological Lessons learned some contributors suggested there was a views held by many teachers and lack of formal mechanisms for doing so. Educational and philosophical beliefs and superintendents when it comes to comparing ideologies lie at the heart of the reforms The Ministry is conducting surveys of teachers’, and benchmarking schools and pupils. These currently underway in BC. Some say that pupils’ and parents’ understanding of the concerns are linked to fears about excluding the consultation process enabled a wide assessment changes and attitudes towards vulnerable pupils. Representatives from variety of views to surface, and that the these. The results were not yet available at the Fraser Institute observed that while curriculum landed in the middle ground so the time of writing. Ultimately, senior policy many parents and even the Ministry are in as to acknowledge a range of perspectives makers within the Ministry said the reforms’ favour of monitoring system performance, and beliefs. A critical mass of opinion success would be judged using graduation the BCTF’s influence makes it difficult and pre-existing pedagogic practice also rates and the province’s performance in to introduce monitoring systems. prepared the ground for extensive reform. international rankings such as PISA. Although Is the innovation believed to be ministry staff and politicians talked about Others argue there was little meaningful achieving its intended objectives? their hope that graduation rates would rise, opposition while the reforms were designed, some contributors including administrators At the moment, many are optimistic about the claiming that groupthink during drafting and and teachers expressed cynicism about this, reforms, believing they legitimise and embed revision cycles meant that certain perspectives saying that the reforms make graduation burgeoning good practice. However there are were not sufficiently taken into account. easier and represent a lowering of the bar. also some concerns about implementation. Closely aligned priorities were crucial Success is also being judged on a more ad BC’s performance in the 2018 OECD PISA in securing buy-in at all levels, including hoc basis, including through school and tests placed the province slightly below the from ministers, academics, union district feedback to the Ministry, the results Canadian average in reading, maths and officials, administrators and teachers. of surveys undertaken by organisations science, although the province’s performance Council of Ministers of Education, Canada, available at: like BCTF, and potentially through election places it among the top 20 jurisdictions https://www.cmec.ca/Publications/Lists/Publications/ outcomes. However some commentators within the OECD, in each area.13 BC’s levels of Attachments/396/PISA2018_PublicReport_EN.pdf working at different levels in the system 14 Jerrim, J. 2019. Is Canada really an education superpower? 13 O’Grady, K., Deussing, M.-A., Scerbina, T., Tao, Y., The impact of exclusions and non-response on results expressed concerns about how well feedback Fung, K., Elez, V. and Monk, J. (2019) Measuring Up: from PISA 2015. https://johnjerrim.files.wordpress. Canadian results of the OECD PISA 2018 study, Toronto: com/2019/12/canada_papaer_october_2019_final_wp.docx
Making Waves A better future for assessment 99 The stars therefore aligned in terms 3.8 to-day teaching in a range of subjects, of the macro political context and the through diagnostic, formative assessment. perceived educational imperative. Innovations 8 and 9: In New Zealand, assessment resources are While there is broad agreement on the Online assessment resource available through one website. In contrast, in priorities that underpinned the reforms, it is banks in New Zealand and Victoria a range of different portals exist (such increasingly clear that different people are as the Insights Platform), with different tools. defining success differently depending on their Victoria, Australia Moves are currently underway to bring some of position in the system. For instance, students these together in one ‘Digital Assessment Library’ characterise success as making a successful 3.8.1 Introducing (DAL) and to add an online adaptive platform to application to university, whereas politicians this. The first elements were due to be online in see maintaining the province’s international the innovation mid-2019, followed by a full platform from 2020. league table position as an important goal. However, this was delayed and during the second In recent years, both New Zealand and the visit conducted in October 2019 it was not yet Paradoxically, these far-reaching reforms state of Victoria in Australia have created clear whether the DAL will end up including all have in some ways had only limited impact online assessment resources. Other Australian the different resources and how it will develop. on practice. This is because teachers have states have also introduced similar initiatives had to do a lot of interpretation about what but this study focuses specifically on the New Zealand’s ARB includes 2,800 assessments, the reforms mean, but also because there work going on in Victoria and New Zealand. which can be completed online, 1,200 of which are few formal levers for changing practice. are self-marking. Each tool includes an overview Both jurisdictions have set up one or more of the topic being assessed, guidance on how to websites through which teachers can access a use the resources and links to the curriculum. selection of resources. A limited but growing Benchmarking information is also available, to set of assessments is available in Victoria help teachers understand common responses and the assessments tend to be focused on and misconceptions (for example, stating ‘80% core areas of interest within the curriculum of pupils correctly identified that… 30% had the such as literacy, numeracy or cross-curricula following misconception, you might like to…’). competencies. They are intended to help Pupils can access resources directly when the gauge and map-out progression. In New teacher gives them a URL and teachers can go Zealand, a far larger bank of resources is back to previously used resources to see how available in an Assessment Resource Bank pupils performed or to use previous responses (ARB), which can be used to support day- as a prompt for self-reflection among pupils.
100 A better future for assessment Making Waves Online assessment tools and (to teachers and pupils) is generally seen as • a ‘significant gap between its highest resources in Victoria include: equally important as, if not more important and lowest performing students; far than, the assessment itself. greater than in many OECD countries’ • adaptive online tests known as • a link between low levels of achievement ‘assessment on demand’ 3.8.2 Agenda setting and students from low socioeconomic • assessment rubrics and indigenous backgrounds. • intercultural capability What were/are the ‘pain points’ that this • critical and creative thinking assessment innovation sought to respond to? While most of the recommendations focused • ABLE (ability-based learning), on increasing funding, high-quality formative which helps teachers and teaching i) Improving educational outcomes assessment was seen as a key way of improving assistants make judgements about Victoria has currently got a high-level priority teaching and learning in the context of large communication, literacy, interpersonal around becoming ‘The Education State’ attainment gaps within each classroom. and intrapersonal skills and cognition and this involves improving achievement in The so-called ‘Gonski 2.0’ report was among students with additional needs. reading and writing (in the national ‘NAPLAN’ published in 201816 and had a greater • a Transition Learning Development Status assessment – National Assessment Program focus on assessment, though the Victorian tool, which all government-funded early – Literacy and Numeracy), as well as in arts Curriculum and Assessment Authority (VCAA) years provision must complete and which and creative thinking. Some argue that reports that this “confirmed they were is delivered through the Insights Platform this agenda is about securing ‘excellence’ on the right path” rather than the report • a guide to formative assessment designed by rather than tackling specific problems. Melbourne University’s Assessment Research driving the online assessment bank project. Centre (ARC), which includes examples In 2012, a review of Australian education One recommendation of the Gonski 2.0 designed in collaboration with teachers. by David Gonski – ‘The Gonski review’15 – report specifically suggests the need for the highlighted a number of negative trends government to ‘develop a new online and on While many of the assessment resources in the educational achievement of demand student learning assessment tool that are available in Victoria may appear to Australian students, namely: based on the Australian Curriculum learning be summative from an English perspective, progressions’ (p xiii). It recommended that • a decline in performance in comparison in Australia they are viewed as formative the tool should focus on measuring ‘growth’ to international benchmarks because they provide immediate and actionable (progress) and that it should support teachers • a ‘concerning’ proportion of students not information for teachers. In both New Zealand meeting minimum standards of achievement 16 Department of Education and Training (2018) Through and Victoria, the formative dimension of the Growth to Achievement: Report of the Review assessment resources is underlined by the fact 15 Gonski, D. (2012) Review of Funding for Schooling: to Achieve Educational Excellence in Australian that the reporting of assessment information Final report, Canberra: Department of Education, Schools, Canberra: Australian Government, available Employment and Workplace Relations (Australia). at: https://docs.education.gov.au/node/50516
Making Waves A better future for assessment 101 not just with assessment, but also with tailored of progress’ (p 63). This was highlighted by to identify and learn from good practice next steps: the teachers and leaders we interviewed who and it is this, rather than a need for more reported having to dedicate many hours to: accountability, that is driving the agenda ‘The tool would assist them to readily identify (since accountability is being addressed the stage of learning a student has reached and • designing assessment resources, through a separate reform). On the other to provide a choice of possible appropriate duplicating work that is done in hand, if – thanks to better assessment and interventions from which the teacher could all schools but rarely shared monitoring – teaching and learning improve, select to provide the next challenging but • attempting to analyse the results of NAPLAN results might also improve. achievable learning task.’ (p 61) summative tests such as Progressive Achievement Tests (PAT tests) and iii) Implementing the curriculum The focus on signposting towards next steps is NAPLAN in a way that provides diagnostic The link between a relatively non-prescriptive also an increasingly important goal behind the information for teachers, despite the fact ARBs in New Zealand. curriculum and a perceived need for clarity that they are not designed for this purpose. about progression is common to both Victoria The focus on assessment resources around On the other hand, according to some and New Zealand. Schools in Victoria have a intercultural and creative skills in Victoria ties educationalists in Victoria, the DAL and particularly high degree of autonomy compared into a widespread (and international) interest Insights Platform is also intended to address to those in some other Australian states, in cross-curricula skills and capabilities that shortcomings in monitoring pupil progress, making it fertile ground for the innovation. has also been noted in our study of British as well as teachers’ difficulties planning and Teachers in Victoria were particularly keen for Columbia’s reforms. Teachers’ appetite for delivering the curriculum. This issue is linked to: support in making sense of the link between these agendas is demonstrated by the fact that ‘learning areas’ and ‘capabilities’, which were trials for new assessments in these areas have • a top-level and relatively non- intended to mesh together subjects and been over-subscribed. prescriptive curriculum cross-curricula strands. Formative assessment • lack of clarity about progression resources are seen as something which could ii) Improving the quality of assessment practice among teachers fill this gap without making the curriculum • poor-quality assessment resulting in more prescriptive or increasing accountability. The Gonski 2.0 review suggested that the wasted time and inconsistent scoring and lack of an ‘online learning assessment tool’ reporting, which means a lot of time is Issues due to loose curricula are particularly is a key issue in Australia, highlighting that wasted ‘weighing the pig’ and producing pronounced lower down the age range, there is ‘uneven access to validated formative data that is not reliable, valid or comparable. resulting in many schools just mapping back assessment tools’ and that ‘teachers… rely on from secondary level, or having to invent a mix of locally developed… idiosyncratic tools According to the VCAA, lack of monitoring their own frameworks, even if this is very that do not provide the most reliable measure is problematic because it makes it hard time-consuming and inefficient. Saving
102 A better future for assessment Making Waves teachers’ and principals’ time is therefore an Although teachers we spoke to were “We don’t have a good centrally provided important driver behind the innovation. not aware of plans to develop the DAL, curriculum. We have the Australian they recognised the potential benefit curriculum and the Victorian curriculum There are concerns in some policy circles that such a resource could have in supporting but it will say for example ‘you need to autonomy has worked well for some schools them to implement the curriculum. teach magnetic fields’, then a full stop but not all of them. Both the VCAA and the and no additional resources, so every Ministry of Education recognise a groundswell It was clear that teachers felt that the lack of school teaches magnetic fields and they in calls for support in understanding curriculum prescription in the curriculum made it difficult might get other resources but there is implementation. Assessment tools are seen to make accurate assessments and judgements: huge variety in what that looks like.” as a key response to this. There also appears to be a growing interest in curriculum design, which is closely linked to the need For some, this led to new teachers struggling for carefully planned assessment tools – “You just don’t know where your with the challenge of interpreting the particularly in order to map out progression. students stand against others. Compared to countries where you have that detail curriculum and the workload of creating These views were echoed by teachers we fleshed out in the curriculum where assessments based on this interpretation, interviewed in Melbourne. Teachers viewed a you can say for sure ‘I know that you and more experienced teachers disregarding non-prescriptive curriculum as having benefits have to do fractions in Year 8, I know changes to the curriculum: and drawbacks. Benefits mainly centred it’s this kind of thing, I know what that around teacher autonomy and the freedom to looks like’. You haven’t got that objective adapt teaching to pupils’ current levels. This description to compare against.” “It’s difficult if you’re a new teacher is seen as particularly important in a system trying to find your way through. If you’ve that does not set or stream pupils. However, been around for a little bit longer they despite this, we found that teachers still Additionally, teachers highlighted further have more idea of what level to teach wanted their practice to be more informed by downsides to having a non-prescriptive that content… or they pin it to the accurate assessment data and judgements: curriculum, mainly in relation to high workload old curriculum that they just roll out and inconsistency across and within schools, every year because there is nothing “We want to work towards data informed leading to a situation in which no one felt sure else… If they’ve been through a few teaching to pupils at their point of needs, they were delivering the correct content at curriculum changes [they] don’t so rather than teaching the curriculum the right pitch. much see the need to change things to the middle of the bell curve.” in response to curriculum changes.”
Making Waves A better future for assessment 103 Overall, teachers felt that assessment resources address different pain points over time, with The early stages of the ARBs in New – primarily rubrics, tasks and model responses different priorities dominating for different Zealand can therefore be characterised as a to tasks – would be useful when interpreting stakeholders. For the New Zealand Council compromise between two competing agendas: the curriculum levels and would not have a for Educational Research (NZCER) (which one focused on pedagogy, the other on detrimental impact on autonomy. They also did manages the ARBs), the primary purpose has accountability, with some seeing ARBs as a way not appear concerned that extra assessment always been pedagogical, in terms of providing of holding off the ‘threat’ of national testing. resources would be interpreted as an attack on support to teachers in a non-prescriptive autonomy or a push to greater accountability. form. However, different governments have This shifted as the Labour party and teaching had different goals. In particular, during the unions drove forward an alternative agenda late 1990s, the government saw the ARBs as around strengthening teacher professionalism “Because we don’t have a lot of a tool in a move towards closer monitoring and ‘authentic learning’, which they argued accountability tied to testing, any of national standards, particularly at key were under threat. This alternative pain point resources out there are going to help ‘transition points’. However, Cedric Croft, one has since come to dominate, with the new me curriculum planning. The problem of the architects of the ARBs, explains that Labour government abolishing the national now is that that’s not there. I don’t this goal was contested (in his tellingly titled standards. This has allowed the ARBs to move think it would harm the autonomy and article “The Assessment Resource Banks: From further towards NZCER’s preferred pedagogical curriculum design; it would complement National Testing to a School-Based Resource”): agenda, for example helping teachers it. It would be good for teachers to know understand the ‘nature of science’ strand of the if… they interpreted the little dot point curriculum that had previously been neglected, of the Victorian curriculum correctly. ‘There was no deep conviction within perhaps due to a lack of guidance. This has In some cases they would see that NZCER that Intact tests [a set of tests close parallels with interviewees in Victoria, they weren’t interpreting it right and within the ARBs that could be used who also noted a tension between pedagogical that would be a useful conversation.” as part of national testing] would and accountability-based agendas for the DAL. necessarily be of high educational value… but they were a pragmatic Other groups continue to hold differing They felt that comparative data from other response to the continuing notion agendas and to point to different pain points. schools would be particularly useful in order to of national testing, which they Some politicians and influential groups (such provide information in advance of NAPLAN or regarded as somewhat flawed.’17 as the think-tank ‘The New Zealand Initiative’) Victorian Certificate of Education (VCE – GCSE see closer monitoring and better use of more equivalent) tests. standardised and reliable data as routes to iv) New Zealand’s shifting ‘pain points’ 17 Croft, C. (2002) ‘The assessment Resource Banks: From tackling underachievement – particularly National Testing to a School-Based Resource’, New among Maori pupils. Meanwhile, others in The ARBs in New Zealand have sought to Zealand Annual Review of Education, 11, pp 229–43.
104 A better future for assessment Making Waves government recognise that New Zealand has an that might be seen as cultural imperialism or unusually well-developed data infrastructure. domination. This approach is embodied in the They argue that this needs to be used better. country’s curriculum, which is described by some as ‘beloved’ and ‘world leading’ – despite Some in government also believe that the others seeing it as too high level. An aversion current curriculum results in an ‘anything to hard accountability also means that the goes’ culture and that more (quality-assured) ARBs give teachers (rather than schools or guidance is needed. Over the course of this managers) control of the data that is produced. study, the ongoing education consultation also led to further emphasis on the notion of As noted above, teachers have called for progression and the need to map out pupil more guidance, particularly in terms of how trajectories and monitor progress rather they should enact the curriculum, as well as than what thresholds they are reaching. in order to reduce the workload associated This shift in agenda will have implications with developing assessment resources. despite the fact that, in other contexts, for the ARBs and it is not yet clear how The fact that teachers and schools have commissioning an external organisation to this will play out, though it appeared that, been willing to pay for tools provided by develop assessment resources might be seen as in future, online assessment resources the private sector has helped demonstrate removing teachers’ independence. Meanwhile, would be centred around new progression their keenly felt need for a solution and the both jurisdictions are keenly aware of the frameworks rather than the current platform. importance of this agenda. Independent close link between curriculum, progression Schools Victoria and Catholic schools have also and assessment, and online assessments Who and what influenced the agenda? helped to articulate and mobilise demand. are seen as a way of tying these together. The overall culture around education policy in Meanwhile, there is a live debate in Victoria, In New Zealand, prioritising school autonomy Victoria and New Zealand has played a critical including in the media, about how schools ties into the country’s historic, geographical role in shaping the agenda around ARBs. Both report to parents. Schools are free to do this and cultural context. Since the country is jurisdictions prioritise school and teacher how they wish, yet parents are often dissatisfied largely rural and schools are closely tied to their autonomy and online assessment resources with the way they receive information. Non- communities, they are expected to tailor their are seen as a way of guiding and supporting prescriptive nudges towards standardisation provision to their communities, particularly without prescribing. In New Zealand, investing can therefore be seen as a response to parents’ their Maori communities given the country’s in experts, including organisations like NZCER, demands, but this is held in tension with many history. High degrees of autonomy can is seen as a way of equipping teachers to educationalists’ aversion to standardised therefore be linked to an aversion to anything act in a professionally autonomous manner; grading. Assessment tools such as rubrics
Making Waves A better future for assessment 105 to map progression are therefore seen by In New Zealand, schools and teachers are teachers understand where pupils are so that some as a way of negotiating this tension. particularly powerful players in education they can raise standards, David Howes (VCAA policy. Approaches that do not sufficiently CEO) helped ensure the DAL became a “pet Teachers and parents have therefore both respect their autonomy are therefore project” for the minister, who is said to be played a role in building the case for an actively resisted – for example, over 200 well-respected. Effective lobbying seems to online assessment tool in Victoria, though schools (a sizeable number in the New have been a driver in moving along the ‘long some argue they have been important Zealand context) refused to implement a and torturous road’ towards funding. It was supporters rather than a driving force. previous set of education standards. suggested that this may have been helped by the fact that the minister’s wife is a teacher, 3.8.3 Formulation Research institutes have played a particularly giving the minister first-hand experience of How was a solution arrived at? important role in formulating a solution in issues such as workload that result from a lack both jurisdictions, whether the ARC in Victoria In both jurisdictions, current work on of assessment resources. or the NZCER in New Zealand. The ARC has assessment resources is part of a long-running, had a long-standing interest (and considerable The detail of which assessments feature iterative process. As we have seen, in New academic expertise) in formative assessment in Victoria’s bank of resources has been Zealand, the focus of the ARBs evolved over and developing tools to assess cross- influenced by policy makers’ interest in time and in Victoria, current developments curricula skills and capabilities. The VCAA’s particular areas – for example, critical and are partly a response to dissatisfaction strong reputation strengthens the initiative. creative dimensions of learning. with, or a need to evolve, the current Meanwhile, the NZCER has been working on platforms and products (with some seeing the ARBs for nearly three decades, since 1993, In New Zealand, although ministers develop the existing platforms as a ‘sunk cost’). though its involvement with assessment in work programmes and come with their own priorities, there is a culture of consultation, In Victoria, various platforms and tools have New Zealand stretches back to the 1970s. which means that if the profession and the evolved in a piecemeal and somewhat ad hoc public do not back reforms and initiatives, they fashion, with a range of assessment tools 3.8.4 Decision making may not go ahead. Sector and professional emerging, some from the government and Who was involved in deciding on this support was therefore crucial. some from private providers. Schools therefore approach to addressing the pain points? have a range of solutions to choose from – Why were other solutions rejected and The case for the DAL and a $17 million for example, many use a product from the this approach preferred? Australian Council for Educational Research investment in the platform was primarily made (ACER). The DAL therefore evolved to draw by the Head of the VCAA to the minister and Teachers themselves make decisions together and upgrade existing ‘part-solutions’ minister’s office. By arguing that assessment about which resources to use in both and provide a state-sector, state-level solution. was “the missing piece” in terms of helping jurisdictions. Use of ARBs therefore
106 A better future for assessment Making Waves depends on teachers seeing them as the ministry only covers state schools, the team instead of the curriculum team. Later more useful than alternative products. VCAA was considered a better home. Questions on, there were questions around whether remain around how the data from assessments the new “Progress and Consistency Tool” In Victoria, at one point there was a grander should be collected and used. It is currently (PACT) should be part of the ARBs; however, plan, to pull together a range of tests from stored and this feeds fears that it could be used this idea was rejected because of PACT’s different providers, including PAT tests from to monitor teachers. This potential challenge links to the National Standards towards ACER, but this grand dream was moved around has been tackled in New Zealand by linking all which there was considerable resistance. from one person to another and abandoned data to teachers’ individual accounts (rather for practical and software reasons, since it than schools), allowing them to delete the data 3.8.5 Implementation was “like bringing together a set of parts from if they wish to do so. Meanwhile, although the different car manufacturers”. Another reason platform and the data are owned by the New What mechanisms have been used to for this was said to be that ACER, which owns Zealand Ministry of Education, the NZCER is implement the approach? PAT tests, which are used by around half of the data custodian and would anonymise data schools in Victoria, was reluctant to hand over before sharing it with the Ministry. However, i) Developing the initiatives the management of its tool. It was concerned the NZCER notes that the Ministry has never Implementation in New Zealand has proceeded that the platform would be managed by tech requested access to ARBs’ data. experts, rather than assessment experts, and in several phases: thus PAT tests would not be updated and In New Zealand, questions around funding also 1. 1993/94–1998: The curriculum framework/ maintained to a high-enough quality. ACER also affected the formulation of the solution. It vision was produced as well as individual believed that insufficient consideration was was a battle to secure funding to create online subject curricula. Some argue this was given to the reporting side of assessment as ARBs, since making an online resource is not very challenging for teachers at primary the plan focused on the analysis that would be the same as putting paper resources online; level because each was very thick. It possible with large sets of data rather than how at one time the Ministry simply wanted online, gradually became more coherent as teachers would use the platform on a day-to- markable responses. However, the NZCER curricula were more tied to the overall day basis. It decided it had no motivation to join wanted to continue a formative focus and framework. The first feasibility studies for the platform and, as time went on, the idea of felt that this required a more sophisticated the ARBs were conducted in 1993–94 and the creating a bank of new resources (the DAL) (and expensive) approach. Chris Joyce at the first ARBs were made available in 1995. emerged instead. the NZCER therefore had to put in multiple During this time, the ARB remained mainly requests to maintain funding and compromise summative and multiple choice based. There was also a question over whether the was needed, for example when it came to 2. 1998–2003: Assessments gradually ministry or the VCAA should run the DAL but ownership of the platform. The ARBs also became more formative. because the VCAA covers all schools whereas shifted into the remit of the Ministry’s digital
Making Waves A better future for assessment 107 3. 2003: ARBs became explicitly before moving on to other state priorities. “Teacher involvement genuinely formative in intended purpose. Like in the early days of ARBs in New Zealand, improves the assessment, adding to 4. 2006–07: The new curriculum led simpler assessments (such as multiple- their authenticity and credibility.” to a remapping of the resources. The choice format) were also prioritised. Ministry wanted explicit links between On the other hand, he says it is important resources and National Standards. This Teachers have been involved in developing to accept that not all the material teachers led to constant remapping. When the assessment resources and tests in both submit will be up to scratch. Teachers therefore curriculum was released, people said “Is jurisdictions, through both piloting and item have to know they will get feedback and it detailed enough?” The response was creation. In both jurisdictions, resources that this is part of professional learning. that a second tier of supporting materials are piloted in a mix of schools or with a range of teachers (low-, mid- and high- was promised, but never came in any fully In October 2019, the DAL was still in decile schools in New Zealand), while in developed form. ARBs filled this gap. development, slightly behind schedule. Victoria, panels of teachers are selected to 5. 2009–10: National Standards led Although the VCAA reported that the include teachers from both rural and urban to another shift in course. workshops had been useful, more items still schools. This has considerable implications needed to be developed, suggesting that 6. 2018: National Standards were abandoned in terms of needing to pay for travel but it this method alone had not been sufficient and the aim shifted to ‘student-centred is considered important and worthwhile. formative assessment’. The focus of to fully populate the online bank. resources therefore began to move In order to develop a replacement for the old Efforts to involve teachers in item development from ‘What do students know?’ to ‘What ‘on-demand’ tests, the VCAA invited specific in New Zealand have had mixed success. can they do?’. It is this phase of the teachers (government school leads with a Originally, teacher panels came up with innovation that we have focused on. particular interest in the areas) to a two-day ideas for the assessments and items, but workshop. This began with a session in which this did not work at the time because they Over time, things shifted away from a subject manager from the VCAA introduced tended to come up with learning activities downloading resources and towards the project. There was then some expert input rather than assessments. This appeared to online tools. Resources also became more on item development, which prepared teachers be because they lacked the curriculum and sophisticated, for example allowing students to to review and develop items. The teachers assessment expertise to create valid and draw things online or introducing auto-marking. then created items for the bank, which were reliable items. Part of the problem was that, ii) Developing resources subsequently reviewed and quality assured. as Jonathan Fisher of the NZCER explained, Joe Pellegrino at the VCAA argues that: teachers had to think “not just how did In Victoria, the provision of online assessment pupil respond, but why did they respond resources began with literacy and numeracy that way”. Assessment resource developers
108 A better future for assessment Making Waves therefore had to become “researcher writers”. To what extent is the reified NZCER also pointed out that that resource solution consistent with the development needs to be conceived of as a planned/intended solution? form of curriculum development because it involves mapping capabilities and concepts. The current innovation in Victoria looks This is particularly important given New different to the originally intended plan to Zealand’s broad, non-prescriptive curriculum. bring all online assessments together under one platform. This original solution would All assessment resources were piloted and have involved more focus on grouping trialled in schools and this process included different assessments and filling gaps gaining an understanding of the reasons whereas the current plan primarily involves behind certain misconceptions and mistakes. creating new items to form the DAL.
The ARBs in New Zealand are maintained by In New Zealand where the innovation is the NZCER team, with less input from teachers much further along, decades from its initial now, as new resources are not being developed. inception, the ARBs have changed and evolved This is partly driven by funding constraints but multiple times. The main gap between the it also highlights the challenges of depending intended plan for the ARBs and the current too much on input from serving teachers. state of play lies in the extent to which the In time, the VCAA in Victoria may also have ARBs are updated and refreshed, according to respond to this challenge and move task to the NZCER. Current funding levels allow creation away from teachers to expert teams. maintenance of the ARBs rather than Who are the different actors involved? ongoing innovation and new resources.
Bodies that are close to (and/or funded What obstacles had to be overcome by) government play a critical role in both as part of implementation? jurisdiction’s initiatives. These include non- governmental/semi-non-governmental actors The need for constant refining and such as the NZCER, Melbourne University improvement is common to both jurisdictions. and the VCAA. Meanwhile, alternative (and Interviewees in New Zealand pointed out sometimes competing) solutions often the importance of good-quality digital come from the commercial sector. infrastructure to organise the resources. This infrastructure now needs updating
Making Waves A better future for assessment 109 and improving and in spring 2019 it was in the budget) is considered potentially frameworks and mapping tools that will replace highlighted that what is possible will be problematic as difficulties securing interest and the recently removed National Standards. limited by upcoming spending decisions. By exposure could result in low prioritisation. However, new resources would draw on the autumn 2019, it was clear that funding would learning from the current ARBs and maintain remain limited and therefore maintenance Although crucial, the relationship between a focus on formative feedback for teachers. was all that was happening, despite the need the Ministry and the VCAA has not always for new items to be created to keep up with been easy and at one point became strained. ii) Long-term funding changes in the progression frameworks and This was partly because of a lack of clarity the world at large – as well as to maintain over respective remits (for example, whose As noted above, funding constraints (often teacher engagement in the ARBs, since it is role it was to say how the curriculum should driven by changing priorities) can have a huge new content that tends to drive engagement. be interpreted), partly because of specific impact on implementation. Items need near- individuals and partly because of a perception constant updating and maintenance to ensure In Victoria, plans to develop the DAL have that the assessments were not being sufficiently they evolve alongside the curriculum. This been in existence for a long time, but it has prioritised by the Ministry. Changes to staffing is particularly pertinent in a context where been hard to secure funding. Only now has this have since resolved this problem. Changes the curriculum is relatively non-prescriptive. been secured - though it remains much lower to individuals within the Ministry have also Shifting priorities and global issues can than what was hoped for, and possibly what is caused challenges in New Zealand. Churn, in considerably impact the interpretation of a needed. New Zealand provides a comparative terms of individuals and in terms of curriculum/ loosely defined curriculum and what teachers or perhaps predictive lens for this issue: it is policy change and prioritisation, is therefore a teach - and thus, what assessments should clear that funding requirements for online considerable challenge in both jurisdictions. cover. Resource banks can help teachers ARBs remain high even once the platform keep up with change, but only if they are is up and running and if this funding is not In New Zealand, the development of a sufficiently funded to do so. Without new available in the long term then it is hard to new, online digital curriculum is potentially content NZCER is concerned that engagement keep the resources at the desired standard. a competing priority. In our first visit, we with the ARB will plateau or even decline. heard that for the last year or two, new What conditions contribute to or detract assessments have not been developed and iii) Assessment development expertise from effective implementation? this is considered a potential threat to success. During our second visit, some in the Ministry The right combination of technical and i) Prioritisation suggested that if new assessments are to educational expertise is considered important be developed, they would take a different in Victoria, with comparisons drawn to In Victoria, the fact that the DAL is not more form to the current ARBs and will likely be other assessment initiatives that may have high profile (and is thought of as a ‘footnote’ linked more closely to the new progression worked technically but did not work in a
110 A better future for assessment Making Waves classroom context. Recruiting teachers to every six months, are intended to boost Thus, the apparent absence of other forms of work on creating the questions and tasks consistency across teacher judgements assessment that are able to identify knowledge is therefore important and is currently a and check whether teacher judgements and skills gaps that would impact NAPLAN barrier to developing the new resources on map onto Victorian curriculum levels. results means that NAPLAN data ends up being time. The development of assessment on used formatively. Though PAT data fills this demand has also been delayed considerably. However, although schools sometimes try gap to some extent, there is still considerable to use NAPLAN as a diagnostic tool, it is not work to do to map results from one to Current technology and resourcing means intended (or designed) to be a formative another and to the curriculum (and there are that only the more closed assessments in New assessment; rather it was designed to identify questions as to whether this is pedagogically Zealand’s ARB can be auto-marked online. ‘cold spots’ at a state level. Analysis takes desirable). Therefore, should the ARBs allow However, the resources have moved away a huge amount of time (often weeks) and schools to understand pupils’ attainment and from ‘correct/not correct’ questions, making relies on having staff members that can take progress against the Victorian curriculum in auto-marking functions more difficult. time out to perform such analysis. Teachers a way that is predictive of NAPLAN results, explained that due to NAPLAN being the engagement with the resources would be iv) School buy-in: compatibility main form of school accountability, they high and teacher workload considerably with other assessments were highly motivated to spend considerable reduced. However, there is an obvious tension time breaking down NAPLAN results at a here between the original goal for the DAL When considering what a new DAL would pupil level, performing gap analysis and of being formative and low stakes and it have to offer to secure engagement, looking for trends in pupil performance: becoming a tool in ‘teaching to the test’. teachers in Victoria felt strongly that new assessments needed to ‘marry up’ with other assessments such as NAPLAN. 3.8.6 Evaluation “We would be silly if we didn’t use For example, one Melbourne school explained the data in this way [as formative How is success or failure it currently used teacher assessments, based and diagnostic assessment] because being characterised? on teacher-created learning tasks each lesson, we know it’s used to judge us. It’s the Teachers’ use or misuse of the resources as well as doing considerable analysis with performance evaluation the department is seen as an important determinant and NAPLAN data. It also recently began using makes of our school and our Principal metric of how effective they are in both PAT tests to help check teacher judgements and then that our Principal makes jurisdictions. Good uptake in New Zealand and map vastly different types of assessments of us as teachers and leaders.” is seen as an indicator of the resources’ (namely NAPLAN and in-class assessments) success. Meanwhile in Victoria, usage figures onto one another. PAT tests, which are used
Making Waves A better future for assessment 111 are believed to provide some information on helpline calls was given as a potential metric What approaches are being used to what is most useful to teachers and where for the satisfaction element of this. evaluate the innovation’s success? need is greatest. For example, higher uptake of existing maths assessment tools compared to There were no plans to measure effectiveness The teachers and leaders we spoke to English resources is believed to demonstrate by tracking pupil performance or using were not aware of plans to develop the low teacher confidence in the former. comparison groups and some emphasise DAL, though they could see a need for that success needs to be characterised as something that helped them perform The NZCER highlighted that it is interested not gaining pedagogically useful information, accurate formative assessments linked to just in how much resources are being used, but rather than pupils achieving high marks. the curriculum and NAPLAN. This is perhaps how they are used (for example, the extent to Meanwhile, the VCAA recognises that to be expected as the development and which teachers are using supporting resources, frequency of use should not be compared rollout of the DAL has been delayed. guidance on misconceptions, the tagging across tools too bluntly since usage will vary system and so on). Training is therefore depending on the assessment (for example, The primacy of uptake and popularity among prioritised as a means of ensuring success. ‘on-demand’ assessments are likely to be teaching professionals as a valid measure of used at the start and end of the year, whereas success in New Zealand was demonstrated Impact on pupils is also an important goal, others would be in a specific context). by the fact that one interviewee said formal but given that there is less emphasis on evaluation was not needed because it was clear supporting materials in Victoria compared In Victoria, much like in British Columbia, there the resources were popular. The need to see to New Zealand, there is some concern is some hope that assessment innovation the ARBs as deeply embedded within a national that this could diminish impact. will drive education in a particular direction culture that elevates teacher professionalism towards critical and creative skills – in this to an extremely high degree is therefore key. In Victoria, the VCAA’s dream is that way, assessment resources might ‘educate there will one day be an accurately tagged the educators’ by shifting teacher practice. bank of high-quality items that could be combined into bespoke tests covering a On the other hand, the intention is not wide range of subjects. However, this is a that online assessment tools should long-held dream and currently a long way become the sole means of assessing pupils. off. Success is also described as “successfully Interviewees therefore emphasised that deployed assessment resources used in all the tools should not compromise teachers’ schools and all sectors, with high teacher freedom to develop their own assessments. satisfaction”. A high ratio of usage to
4 Conclusion
114 A better future for assessment Making Waves In the last few months, countless remarkable makers’ did was adapt and change tack as unhelpful workload linked to assessment. responses to covid-19, from every quarter they found their way through an uncertain However, assessment innovation tends to be of the education sector have been a landscape. This is not surprising. In her book pragmatic and iterative. Thus, making waves testament to the sector’s endless ingenuity. Cleverlands, researcher Lucy Crehan notes that: in assessment is not just about identifying effective approaches to assessment, it is This report was written prior to the pandemic, also about meeting teachers where they but the innovations it catalogues show that “Despite all this uncertainty, politicians are, tapping into their motivations or the same ingenuity is being deployed to tackle and system leaders have to keep on pain points, and equipping them to make long-standing and intractable challenges in making educational decisions. And changes that work in their context. the field of assessment. Importantly, almost these decisions are influenced by the all the innovations we looked at focused beliefs and desires of the society that 4.1 primarily on enhancing pupil learning, rather they represent, and mediated by the than the top-down accountability that many beliefs and desires of the teachers that Recommendations discussions about assessment focus on. work for them; so to a degree we all have to work with this uncertainty ”18 Based on what we have observed and In this conclusion we summarise what we analysed over the last two years we make the have learned about how teachers, schools, following four recommendations to anyone groups of schools and states can channel The same holds for anyone making decisions seeking to make waves in assessment their professional expertise into change. In about how to conduct assessment in doing so, the main challenge is that the Making schools. Professionals make highly contextual 4.1.1 Prepare for innovation Waves study was not designed to measure the decisions based on imperfect information, As we learned in Testing the Water, teachers impact of a set of innovations - they were at refining their approach in an iterative and school leaders are not always confident too early a stage for that. Instead we followed manner. Thus, reflecting back on the last about their assessment expertise. In this people as they embarked on the process of two years conducting this study we find study too, we saw that innovators generally innovation and our conclusions are therefore that our conclusions are less about ‘what drew from a limited range of sources impressionistic - highlighting what we have to do’, and more about ‘how to do it’. and options when determining their learned from our observations and analysis. We have found that given the right approach. Yet limited expertise constrains conditions and support, alongside a keenly We began with a tidy framework structured people’s repertoire of options and their felt need, professionals are able to develop around ‘the policy cycle’ but soon found that ability to make informed decisions. their assessment expertise and reduce the reality was far messier. Indeed, one of the In contrast, at Heathfield Community College most important things some of our ‘wave 18 Crehan, L. (2018). Cleverlands: the secrets behind the success of the worlds education superpowers. UNBOUND. we saw that members of the assessment
Making Waves A better future for assessment 115 innovation team received training prior to “At the school-level, many assessments functions can compromise the effectiveness selecting their approach. With also saw with are expected to provide both formative of an innovation. In contrast, clarity of other innovations that bringing together and summative information. That is, one purpose can reduce the risk of overlooking different partners can ensure that innovation assessment may be used to generate a solutions suited to hyper-specific issues. is based on mutually reinforcing collaboration grade that shows how a pupil is doing We therefore recommend that assessment that does not begin with a blank page. relative to their peer group, but it innovators, particularly at a classroom or may also then be expected to provide We therefore recommend that those department level, focus sharply on a specific diagnostic information about what a challenge that they are seeking to address. looking to make waves in assessment grow pupil has to do next to improve. But their repertoire of potential solutions by They can then tailor their response to this, the kind of assessment that is ideal for rather than seeking one approach which they learning more about assessment, building these two purposes is very different 19” up their technical knowledge and forming hope will tackle all their pain points in one go. new partnerships and collaborations. Without a crystal ball we cannot know which of 4.1.3 Consider the best 4.1.2 Be explicit about what the innovations we studied will succeed in the level to innovate at long run. However those we have the greatest We studied innovations originating at purpose assessment is serving concerns about are the ones that have tried to various levels; some were led by individual kill too many birds with one stone. In contrast, in this instance teachers, some by leadership teams, others some wave makers designed their innovations by trusts and some by whole states. One of the reasons the innovations we to serve a hyper-specific purpose: not just “how studied looked so different from each other shall we go about formative assessment” but It is important to consider how these is that they focused on different functions “how can we ensure teachers are aware of key different levels relate to each other. For of assessment. Rosie Osborne’s approach misconceptions at the start of a lesson,” “what example in academy trusts, how do initiatives at Eltham Hill was a great way of providing impact are our (non-academic) interventions and insights move between teachers, credible feedback to pupils in an informal having,” or, “how can we use homework to departments, schools and the trust? It is environment, but it would not have worked as identify what has and has not been learned”. easy for knowledge to get stuck at different a means of identifying pupils’ support needs points and we have explored the benefits across a multi-academy trust, or identifying As Christodoulou points out, trying to find and challenges of various approaches misconceptions to inform lesson planning. an approach that serves multiple assessment to ‘cascading’ throughout this study. 19 Millard, W., Small, L., & Menzies, L. (2017). Testing the We found that in secondary schools there As Daisy Christodoulou points Water: How assessment can underpin, not undermine, appeared to be considerable benefits to out in Testing the Water: great teaching. Retrieved from https://cfey.org/wp-content/ uploads/2017/11/Testing-the-Water-Final-Report-WEB.pdf Heads of Department leading an innovation.
116 A better future for assessment Making Waves Firstly, assessment looks different in different 4.1.4 How tight or loose? We therefore recommend that innovators disciplines and innovating at subject level maximise opportunities to gather and act recognises this, as was clear at Heathfield Few if any of the innovations we studied on feedback fast, rather than waiting until where each department’s approach reflected were particularly rigid. Even where externally and end-point evaluation. This involves an academic field’s particular characteristics. provided platforms were deployed, the planning opportunities for reflection opportunity to make modifications was highly and incremental steps that facilitate Secondly, Heads of Department tend to have valued. On the other hand, this needs to adaptation. However the appropriate strong relationships with their team combined be balanced with the risk of compromising level of flexibility will depend on the with a degree of authority. This allows them impact when using a tried and tested model. approach that is being implemented since to take a context-informed approach when well-evidenced, established approaches planning implementation whilst ensuring Furthermore, innovators noted that need to prioritise fidelity more than compliance with ‘non negotiables’ (see below). maintaining flexibility would be difficult early-stage experimental initiatives. or undesirable as an innovation scaled up. Thirdly, where other departments could Throughout our research we heard about benefit or learn from an innovation, Heads of innovators grappling with the right balance Department are well situated within school between non-negotiables and flexibility. Explicit hierarchies to share insights or secure senior decision making about the parameters of what leadership buy-in. Annual or termly cross- can be adapted is therefore an important department meetings could potentially consideration when planning implementation. help this process if middle leaders worked together to identify shared priorities and When implementing an external solution like decide which approaches could be scaled Isaac Physics, Eedi or ImpactEd, innovators up. This could provide an alternative to the had to make critical decisions about how to proliferation of top-down, untested initiatives. implement their chosen approach, and they tended to adjust their approach along the We therefore recommend that, subject- way. Thus, even where a platform or tool does specific assessment expertise should be not afford for flexibility, schools still have to recognised as a critical element of professional show flexibility in how they communicate or development and competence for middle deploy their chosen approach. Identifying leaders. Schools could then explicitly recognise emerging difficulties with implementation and that research-informed, assessment innovation responding to these fast is therefore crucial. is one of middle leaders’ responsibilities.
Making Waves A better future for assessment 117 Appendix: Research framework
118 A better future for assessment Making Waves Phase Research question Question Prompt
Introduction / (As much as possible to Check consent form etc. What are the demographics? context be covered in pre-visit Tell me about this school/MAT/jurisdiction – what is it like? How effective do you think it is? communication) What is your role in the school/jurisdiction? Has it always been like this/how has it changed? How long have you been here? Formative pedagogical information, predictions, validated summative grading What has your role/involvement been with the innovation? for labour market/higher education Tell me about your/the innovation / what you know about it? progression, accountability? • What does this type of assessment aim to achieve? Does it feed in to shape teachers’ lesson- • How does information from this type of to-lesson planning? Feed into reflection assessment feed into teachers’ practice about effectiveness of pedagogy? • How does pedagogy/curriculum shape this type of assessment? Shape curriculum planning? • How much of the content that is covered in Is the assessment based on the schemes teaching is included in the assessment? of work? End of Key Stage expectations? • What is the balance between assessing skill, knowledge, technique or application, in the assessments you are conducting? How do you decide what to include and not to include?
Agenda setting What are ‘pain points’ What was/is the situation (with regard to x) What was the impact of x? this innovation sought like before this innovation began? Was it universally recognised or were to respond to? What was/is problematic about that? some people more aware/concerned about it than others? Why? Who recognised that problem?
How did these How long had/has this been an issue for? become a priority? How did you first become aware of the issue? Has anything else been tried to address the issue? Why was action taken at this point rather than another? What other priorities are there for your classroom/school/MAT/state? Is there any competition between these priorities and your innovation?
Making Waves A better future for assessment 119 Phase Research question Question Prompt
Formulation How was this Who was involved in shaping your/this innovation? What role did they play? solution developed? Did you consider any other possible solutions? Please tell me about them.
In what ways, if any, has your idea/has this innovation changed or evolved?
What sources of information did you use/are you using to shape your idea?
Decision making How was this When was the decision made to go with this particular approach? solution chosen? Who was involved in pushing forward this particular solution?
Who was involved in signing off or giving permission for this solution?
Did anyone push against this approach or argue for different solutions? If so, what led to this one being the preferred option? Why were other potential solutions rejected?
Implementation How is the innovation What was the first step in implementing this E.g. policies, instructions from leaders, new being implemented? innovation? What were the subsequent steps? resources, training, communications materials.
Who has been involved in implementation? What has their role been?
What mechanisms have you used?
What has Has the solution changed in any way over the What have been the main challenges implementation course of its implementation? during implementation? How were looked like? these challenges overcome? What has made it easier to implement the approach? Have there been any key supporters? What has made it harder? Have there been any expected or unexpected barriers?
120 A better future for assessment Making Waves Phase Research question Question Prompt
Evaluation How is it When considering how successful the approach Which factors do you look at to consider being evaluated? has been, what do you consider? the success of the approach?
Has there been/will there be any formal evaluation? If so what?
How is success What would it look like for this innovation to be successful? being characterised? What is needed for it to be successful?
What would failure look like? What might lead that to happen?
Has it/is it having any unexpected effects?
What are the best things that have happened/ are happening as a result of this innovation?
What are the worst things that have happened/ are happening as a result of this innovation?
Overall, how successful do you think this innovation has been and why?
What will the medium/ What do you think will happen with this innovation Will the innovation continue? Will it change? If long-term consequences in the next term? Next year? Longer term? so, how? How will it affect pupils? How will it of the innovation be? affect teachers? How will it affect the school? What do you think will happen as a result of this innovation in the next term? Next year? Longer term?
Making Waves A better future for assessment 121 cfey.org @TheCFEY [email protected] +44 203 488 2765
©2020