Edition 9: December 2015 Research Brief Evaluating Access

Pressure is mounting on universities to increase the The best outreach strategies What makes effective outreach? number of non-privileged students they enrol, and the • Mentoring • Combining several strategies Government is calling on the • Multi-year combined into one longitudinal sector to double the proportion of interventions programme young people from disadvantaged • Personalised application • Improving academic information and assistance attainment backgrounds it admits by 2020.1 • Residential programmes • Intervening early • Tutoring • Involving teachers Entry rates for 18 year-olds • Working closely with parents from the most disadvantaged group to universities with the highest entry requirements have initiatives in 2015/16, growing - body of international increased considerably2 and Outreach iniatives come in many evidence. The Trust has collated higher education participation shapes and sizes, and include the most promising research from the most disadvantaged summer schools, mentoring into a literature review entitled 40% of young people has schemes, tutoring, teacher Evaluating Access which gives an also increased over the last engagement, parental support, initial indication of which outreach ten years. Yet disadvantaged information campaigns and more. strategies may work best. The students remain chronically Although some universities review also highlights some underrepresented in the most seek feedback from their common features of the most selective institutions. A student participants and track their later successful outreach programmes. from a privileged background academic success rates, very Taken all together, there is strong is still 8.5 times more likely to few institutions are rigorously evidence that outreach activities, attend a selective university than evaluating the impact of their in general, do succeed in their least privileged counterpart.3 outreach programmes. attracting and admitting students Ultimately, there are still far too from non-privileged backgrounds, many high-attaining low-income There is a lack of robust evidence but there is not enough evidence students failing to gain access to from the UK indicating which indicating which particular higher education. outreach strategies work initiatives work. best, yet there is a small - and Since tuition fees increased to £9000 a year, the Office for Fair Universities’ spending on Widening Participation over time5 Access (OFFA) has developed spending guidelines outlining the proportion of their budgets universities should spend on Widening Participation. As a result, universities now spend £750 million a year on activities to improve outreach, retention and progression.

OFFA has encouraged a growing proportion to be spent specifically on outreach, with universities set to spend £124 million on such 1 include early intervention, a particular area should not be Evaluating Access: Research attainment-raising, interaction equated with a lack of impact. Review with parents and teachers, and combined outreach programmes. Evaluating Access: Joint Existing research in the UK Research Project too often lacks a control group The review includes both against which to benchmark quantitative and qualitative The overall conclusion of the participants’ outcomes. Failing research undertaken within the Evaluating Access Research to provide a counterfactual last 15 years focusing solely on Review is that, while outreach makes it very difficult to quantify outreach strategies, rather than initiatives are having a positive the impact of an initiative and strategies related to student impact, there is a severe therefore to ascertain whether it attainment, retention or post- lack of evidence identifying actually works. The gold standard university outcomes. The review which iniatives work - despite for such quantitative research looks specifically at the impact on universities spending £120 is the randomised control trial low-income students, and does million annually on this activity. (RCT) and a report authored by not consider evidence related to The Trust is working to address Professor Carole Torgerson of mature or part-time students. this by partnering with OFFA published These caveats should be carefully and universities from across the alongside this review, finds that considered before drawing sector to undertake a series of there is very little research of this conclusions from the review’s research trials measuring the standard in the UK. Across all findings. Bursaries, for example, effectiveness of their outreach outreach strategies, the majority have not been shown to have programmes. The project is set to of robust research has been a positive impact on university begin in spring 2016. undertaken in an international access and retention, however context. Nevertheless, initial this doesn’t mean they don’t have By working collaboratively findings from the Evaluating an impact on either. Similarly, with other institutions, and Access review shed light on there is a lack of evidence guided by a lead academic, the several promising interventions indicating the impact of combined project will enable universities that have had measurable positive interventions that result in a across the sector to develop impact within their own national transferrable offer of a university a research methodology that context. Some common elements place. Yet this appears to be tackles common problems: of effective outreach to emerge because these interventions are such as isolating the impact from the research literature relatively new and remain largely of each intervention, achieving untested. A lack of evidence in

Embedding Evaluation Targeting and Longitudinal Comparison Group Trials Tracking In 2011, academics from the University The has an of Bristol undertook a quasi- overarching evaluation plan for pre- An evaluation framework for outreach experimental study of the Sutton Trust’s entry activity which targets students for activity has been developed at the UK summer school programme. They particular activities; collects detailed University of Chester using POLAR3 found that summer schools generate data; and tracks students on particular neighbourhood-based participation data proportionately more UCAS applications programmes. The university employs to target resources. The university is and registrations, and attending a dedicated officer with specific undertaking detailed assessments of the summer schools reduces these responsibility for evaluation within the application data from targeted schools differences to the relative advantage of outreach team to coordinate this work. and colleges to provide initial indicators of influence. The institution intends to the more underprivileged students. make use of the new Higher Education The university is also involved in a PhD Access Tracker (HEAT) to support its They used two sets of control groups linked research project with the Centre longer term evaluation activity. - ‘inner’ controls of students who for Higher Education Equity and Access. applied for a summer school place The study’s control and experimental unsuccessfully, and ‘outer’ controls, groups span the 14-19 phase and by recruiting students with similar include young people who meet all, characteristics to the Trust’s eligibility some or no widening participation criteria, but who never applied. targeting criteria.

2 good response rates, and national evidence base requires developing robust data collection coordination among universities. References practices. The project will Evaluations of financial support emphasise the importance of are plentiful, yet open days 1https://www.timeshighereducation.com/ using a comparison group in are very rarely evaluated for news/cameron-access-targets-major- all evaluations. The ultimate their impact on non-privileged factor-higher-education-green-paper aim is to foster collaboration applicants. These disparities 2https://www.offa.org.uk/uncategorized/ between universities in order to can only be addressed by celebrating-ten-years-of-access- build expertise among widening communication and collaboration agreements/#sthash.5YvujBde.dpuf participation practitioners so across the sector. 3http://www.suttontrust.com/wp-content/ that robust evaluation can be uploads/2015/07/ICOF-REPORT-2015.pdf sustainably embedded in all Universities must ensure that university outreach work. their outreach budgets are spent 4http://www.hefce.ac.uk/news/ Universities, by their very nature, in a way that maximises their newsarchive/2015/Name,104404,en.html are home to world-class analysts impact on under-represented 5 and researchers, yet many groups. This report calls on https://www.offa.org.uk/wp-content/ uploads/2015/07/Access-agreements- widening participation teams do universities to take responsibility for-2016-17-key-statistics-and-analysis. not harness the expertise within for improving the evidence pdf their own institutions. The base and puts forward several project will act as a call to action recommendations to universities. for universities, challenging them The Trust also calls on the to use their institutional expertise Government to support the scale- to conduct rigorous evaluations up of those outreach projects that and to make the results publically have robust evidence of impact. available. Improving the

Recommendations

1. Universities should increase their spending on evaluation with the ultimate goal of undertaking robust research trials. Universities should spend at least 10% of their outreach budget on evaluation of this kind. 2. Universities should utilise the research expertise within their own institutions to undertake evaluations using comparison and control groups. 3. Universities should publish the outcomes of their evaluations fully and independently, in order to contribute to the wider evidence base. 4. Universities should make use of joined-up data collection methods like the Common Evaluation Framework (CEM) and the Higher Education Access Tracker (HEAT). 5. Universities should collaborate with each other to build research designs that can be replicated across the sector. 6. Widening participation teams should further educate themselves about the existing evidence base and how they could contribute to it.

We would also encourage universities to register an interest in joining the Sutton Trust/OFFA Research project, aimed at building a common research methodology and undertaking robust evaluations, which is set to begin in Spring 2016.

Credits Acknowledgments

Sophie Maddocks, The Trust is grateful for the of Michigan; The Sutton Trust Dr Lee Elliot Major, assistance of Prof A Hoare, Common Evaluation Advisory Prof Carole Torgerson ; Kirsty Group; the Education Endowment Younger and Dr Andy Wiggins, Foundation; and The Boston Durham University; Prof Anna Consulting Group Vignoles, ; The Widening Participation Team; Prof Susan Dynarski and delegates from the University 3