Aspire to STEM Independent Evaluation October 2017 – March 2020 Written by Skyblue Research Ltd Contents Executive summary ............................................................................................................... 4 1 Introduction ........................................................................................................... 11 2 Methodology .......................................................................................................... 15 3 Context: schools engaging in the AtS Programme ................................................. 19 4 Teacher feedback .................................................................................................. 25 5 Whole school case studies .................................................................................... 31 6 Key findings by audience ....................................................................................... 40 6.1 Outcomes for teachers .......................................................................................... 40 6.2 Outcomes for leaders ............................................................................................ 44 6.3 Outcomes for students .......................................................................................... 48 6.4 Outcomes for colleagues, the school and wider community .................................. 54 7 Key findings by Programme aim ............................................................................ 58 7.1 Improving leadership to support the teaching of STEM subjects ............................ 58 7.2 Developing teacher pedagogy within STEM subjects to support disadvantaged students ................................................................................................................ 63 7.3 Increasing science capital within disadvantaged communities (including enrichment to raise awareness of STEM) ................................................................................ 67 7.4 Strengthening careers information and guidance (particularly technical pathways) 70 8 Attribution and Added Value .................................................................................. 72 9 Conclusions ........................................................................................................... 80 Appendix 1: Aspire to STEM intervention logic model ......................................................... 82 Appendix 2: School sampling information............................................................................ 83 Appendix 3: Detailed methodology for Ofsted report review ................................................ 85 Appendix 4: Aspire to STEM Monitoring Data ..................................................................... 87 2 List of Tables Table 1: Changes in Ofsted inspection ratings for AtS targeted schools ............................. 19 Table 2: Baseline STEM subject performance ratings by a sample of 15 AtS schools ......... 21 Table 3: Comparative performance of STEM and non-STEM subjects in AtS schools ........ 21 Table 4: STEM needs and challenges in schools at the start of Aspire to STEM ................. 22 Table 5: The main role of Aspire to STEM CPD recipients .................................................. 25 Table 6: How satisfied have you been with the CPD provided by STEM Learning? ............ 25 Table 7: How would you rate the overall quality of CPD you have received? ...................... 25 Table 8: Aspire to STEM CPD Teacher Feedback .............................................................. 26 Table 9: 25 reasons that participants found Aspire to STEM CPD useful ............................ 27 Table 10: Usefulness of Aspire to STEM CPD for different teacher roles ............................ 28 Table 11: Outcomes by audience for 29 schools engaging in the Programme .................... 39 Table 12: Example school contexts and changes in STEM during the intervention period .. 73 Table 13: Changes in STEM maturity within 7 case study schools ...................................... 74 Table 14: Haydock School pre- and post-AtS STEM subject performance criteria .............. 75 Table 15: Counterfactual speculation .................................................................................. 76 Table 16: List of 29 schools that have been the focus for this evaluation ............................ 83 Acknowledgements Skyblue Research would like to thank Ben Dunn, Erin Gray and Wayne Jarvis of STEM Learning for their support and co-operation with this evaluation. We would also like to thank the individuals at seven participating Aspire to STEM (AtS) schools and 6 AtS Educational Leads for completing depth interviews with the research team at a time when the Government was just about to close all schools. Without their insights this evaluation would have been more limited, so we are extremely grateful for their contributions. 3 Executive summary 1. Rationale for Aspire to STEM (AtS) Aspire to STEM (AtS) is a complex whole-school intervention aimed at improving primary and secondary school pupil outcomes in subjects related to Science, Technology, Engineering and Mathematics (STEM). It has been implemented with a £1 million investment as a part of the Teaching and Leadership Innovation Fund (TLIF), a 3-year initiative in 1 England to support high-quality professional development for teachers and school leaders . The Programme’s aims have been to: 1. Improve leadership to support the teaching of STEM subjects. 2. Develop teacher pedagogy within STEM subjects to support disadvantaged students. 3. Strengthen careers information and guidance (particularly technical pathways). 4. Highlight the vast resources available to help support teaching and promote careers in STEM. 5. Increase Science capital within disadvantaged communities. 6. Encourage enrichment activities in schools for students, their families and the community to raise awareness of STEM careers. 2. Programme scope The Programme started in October 2017 and finished in March 2020. It has supported 206 schools across 40 Delivery Partnerships located either in 12 areas designated as Opportunity Areas (OAs) or in Local Authority Districts (LADs) which are rated lowest in England (identified as LADs 5 or 6). 3. The Programme’s offer to schools AtS used a model where clusters of between 3-10 schools in the same geographical area shared resources and support to form a Delivery Partnership, in order to build a sustainable ‘community of practice’ in STEM, adapted to their local needs and context. Each Partnership received funding(£25,000) for bespoke Continuing Professional Development (CPD) for teachers and leaders; career guidance for students; access to STEM inspiration and enrichment activities delivered by a network of STEM Ambassador volunteers; and access to online resources via the STEM Learning ‘STEM Club infrastructure and eLibrary’. Each school and Partnership was able to access the support and expertise of a dedicated Aspire to STEM Educational Lead to support needs analysis, action planning and help to access appropriate solutions and resources. 4. Programme outputs2 • 206 schools (92 primary and 113 secondary or middle deemed secondary schools, 1 all-through school) were recruited to the Programme across 40 Delivery Partnerships. • 2,8023 teachers were recruited, significantly exceeding its target of 880. • A total of 2,852 CPD events and community events have been delivered • 3,563 unique participants have attended one or more of the 2,852 CPD activities run: cumulatively accounting for 10,923 participants (‘incidences of activity’). • A minimum of 9,000 CPD hours was delivered across all Partnerships. 2 Please see main report and Appendix 4 for more detailed information about these outputs. 3 These are the total number of individuals for which AtS holds all the teacher data required by DfE therefore count towards the Participant Recruitment KPI. 4 5. Programme intermediary outcomes Using an outcomes framework applied during the Programme’s school action planning process4, and based on a review of agreed data for 29 schools that have participated in the Programme, the researchers found that: • 184 outcomes could be observed across the sample, an average of 6 outcomes per school. For primary schools, the average was 8 outcomes per school, while for secondary schools the average number of outcomes was 5. • Schools recruited earlier in the Programme (‘cohort 1’) had more observed outcomes than those recruited later in the Programme (‘cohort 2’). • Outcomes for teachers and pupils were most prevalent (in primary and secondary schools) accounting for 28% each of all 184 outcomes that were classified; followed by outcomes for colleagues / school (20%), leaders ( 17%) and 7% for ‘wider communities’ (including other feeder / local schools). • Primary schools were more likely to report outcomes for each audience. • Schools each engaged with the Programme differently and this is measured by CPD Units. 1 unit equates to 5 hours of CPD or learning. Analysis found that schools gaining less than 10 CPD units gained on average 6 outcomes, schools receiving between 10-20 units averaged 4 outcomes, while those that received more than 20 CPD units gained an average of 8 outcomes per school. 6. Outcomes for teachers Feedback gathered from over 1,300 teachers5 at 122 AtS CPD events finds extremely high levels of satisfaction with the CPD and its quality provided by STEM Learning: • Overall satisfaction (31% excellent, 38%