Ymchwil gymdeithasol Social research
Number: 79/2013
Evaluation of Further Mathematics
Support Programme Pilot:
Interim Report
1 Evaluation of Further Mathematics Support Programme Pilot: Interim Report
Claire Kerby, Kerry Lewis, Nick Miller, Miller Research (UK) Ltd
Views expressed in this report are those of the researcher and not necessarily those of the Welsh Government
For further information please contact: Sara James Knowledge and Analytical Services Welsh Government Cathays Park Cardiff CF10 3NQ Tel: 0300 025 6812 sara.james@wales.gsi.gov.uk
Welsh Government Social Research, 2013 ISBN: 978-1-4734-0788-6 © Crown Copyright, 2013 Table of Contents
1 Background to the Study 4
2 Key Findings and Recommendations from the Study 18
3 Approach to the Evaluation 29
4 Detailed Findings - Structure of the FMSP Pilot 43
5 Detailed Findings. 48
6 Outcomes and Impacts 67
Appendix 1: Questioning Framework and Data Sources 95
Appendix 2: Topic Guides 108
Appendix 3: List of Schools and Colleges Visited 149
Appendix 4: Results of Online Survey of HE Students 150
Appendix 5: Results of Online Survey of School and College Staff 161
Appendix 6: Suggested Format for Management Reports 176
2
Glossary of acronyms A2 An examination which, in combination with an AS, forms a full A Level qualification AES Advanced Extension Award AS Advanced Subsidiary (A standalone qualification that also forms half of a full A Level qualification) FEI Further Education Institution FMSP Further Mathematics Support Programme HEI Higher Education Institution HESA Higher Education Statistics Agency JCQ Joint Council for Qualifications LLWR Lifelong Learning Wales Record MEI Mathematics in Education and Industry STEM Science, Technology, Engineering and Mathematics STEP Sixth Term Examination Paper WED Welsh Education Database WIMCS Wales Institute of Mathematics and Computational Sciences
3 1.1 1 Background to the Study
Introduction
1.1 This is the first of two reports from an evaluation of the Further Mathematics Support Programme (FMSP) pilot. This first report presents findings from a process evaluation of the pilot and early indications of its impact. The second report is due for publication in the autumn of 2014.
1.2 The pilot has been funded to a total of £581,485 between 2010 and 2014 from Welsh Government and has been managed by the Wales Institute of Mathematical and Computational Sciences (WIMCS).
1.3 The FMSP initially covered Carmarthenshire, Neath Port Talbot, Pembrokeshire and Swansea and was originally intended to run between July 2010 and October 2013; however, in early 2013, the decision was taken to expand and prolong the pilot. From April 2013, FMSP support has also been provided in Anglesey, Conwy, Gwynedd and Rhondda Cynon Taf and the pilot programme has been extended for an additional year.
1.4 The aims of the FMSP Wales are to: Raise awareness among students and their parents of the value of studying mathematics at higher levels; Increase the number of students in Wales studying AS/A2 Level Mathematics and Further Mathematics; Increase the number of schools and colleges in Wales offering further mathematics; Increase the number of students from Wales applying to study higher education courses in mathematics and mathematically-rich subjects such as Engineering, Physics, IT, Finance and Economics; Improve the transition of students from further to higher education courses in mathematics or courses which have a significant element of mathematics thus benefiting the wider economy.
4 1.5 Support offered to schools and colleges through the FMSP Wales includes: Student tuition for AS/A Level Further Mathematics; Free single user teacher access to the Mathematics in Education and Industry (MEI) Resources Website (supporting all AS/A2 Level Further Mathematics specifications plus Additional Mathematics and GCSE extension resources); Mathematics enrichment courses for students in Key Stage 4 and post- 16, such as mathematics master classes, mathematics career talks and revision events; AS and A2 Mathematics and Further Mathematics revision days.
The evaluation
1.6 The evaluation set out to provide an assessment of both the process and impacts of the pilot to date, through:
examining the effectiveness of the monitoring processes and governance arrangements, and the specific targeted events, activities and resources that the programme provides
Setting out the extent to which the pilot is meeting its objectives, as set out in paragraph 1.4, looking particularly at the following:
(i) The contribution of the programme in increasing the number of students studying GCE further mathematics; attainment levels in the subject and applications to higher education courses in mathematics and mathematically-rich subjects
(ii) The contribution of the programme in increasing the number of institutions offering further mathematics and the number of mathematics teachers in the pilot area
(iii) Making comparisons, in relation to achievement of the programme’s objectives, between the pilot area and areas where similar programmes do not exist
(iv) The wider economic benefits of the FMSP and the vale for money it offers.
5 1.7 Most of the evaluation work took place in the 12 month period from September 2012 and included both qualitative and quantitative approaches to fieldwork, underpinned by extensive desk research and interrogation of existing datasets. Approaches to the counterfactual (i.e. what would have happened in the absence of the pilot) were trialled, using spatial comparisons, (pilot area vs Wales, England and Northern Ireland) and time series (pre-pilot and during pilot).
1.8 Originally, the evaluation also set out to explore the wider economic benefits of studying further mathematics. As the evaluation progressed it became clear that these were not the key issues to be resolved as the wider economic benefits of studying further mathematics were already recognised both here and in England. The elements of the evaluation that focused on the process and impact of the programme were therefore given prominence.
1.9 This interim report sets out the findings from the fieldwork, desk research and data analysis during 2012-13. The final impact evaluation report in Autumn 2014 will set out results of analysis of a further year’s FMSP participation data, full 2013 examination entry and attainment data, 2013- 14 information from HESA regarding uptake of STEM subjects in HE and further analysis of gender issues and links back to GCSE performance of further mathematics students. The final report will draw together findings and recommendations from the whole evaluation, and will be published in the autumn of 2014.
Rationale for the pilot
UK Context
1.10 Between the late 1980s and the late 1990s the number of students in the UK taking further mathematics at Advanced Level fell by approximately two thirds1. The growing popularity of modern and often more vocationally-
1 The Economist, September 21st 2006: http://www.economist.com/node/7950110?story_id=7950110
6 oriented A Levels2 contributed to falling demand from students for this traditionally difficult subject. In many cases smaller class sizes made it difficult for schools to justify the viability of delivering A Level Further Mathematics. These factors led to many schools dropping the subject altogether.
1.11 The trend of decline has been reversed, however, with growing interest across the UK in STEM (science, technology, engineering and mathematics) subjects over the last decade. The numbers of pupils in the UK studying A Level Further Mathematics started to increase dramatically, so that by 2006 further mathematics was the fastest-growing subject at A Level3 in the UK and numbers have continued to rise to date. This is likely to reflect increasing emphasis on the potential career benefits to students of studying STEM4 subjects compared with some arts and humanities subjects and also changing attitudes of HEIs towards the subject. In addition, research such as that carried out by the Advisory Committee on Mathematics Education in 2011 into the mathematical needs of both universities and the workplace noted a range of ways in which employers increasingly needed people with competent mathematics skills5. Particular skills included: The ability to communicate mathematical concepts, calculations and results to colleagues The ability to understand and interpret information produced by computer-generated analyses, and The ability to apply theoretical mathematical concepts in a range of familiar and unfamiliar contexts.
1.12 State-funded schools (particularly comprehensives) in the UK have historically been far less likely to offer A Level Further Mathematics than private sector competitors, primarily due to timetabling and staffing issues.6
2 For example Media Studies and ICT 3 The Economist, September 21st 2006. (op cit) 4 Science, Technology, Engineering and Mathematics 5 Mathematical Needs Mathematics in the workplace and in Higher Education, ACME, June 2011 6 Evaluation of the MEI Further Mathematics Network, Initial Report, Durham University, 2006
7 However, this too is changing, with particular growth in state sector support for the subject.
Development of the Further Mathematics Support Programme in England
1.13 In 2004, the Further Mathematics Network (FMN) was established in England, in recognition of the risk that students in state-funded schools were potentially disadvantaged in terms of higher education access and career prospects, because of a lack of opportunity to study A Level Further Mathematics.
1.14 At that time fewer than 40 per cent of state schools and colleges had students taking A Level Further Mathematics and the Further Mathematics Network offered pupils at schools that could not provide A Level Further Mathematics the opportunity to study the subject at one of the forty-six centres set up as part of the initiative. In 2009 the Further Mathematics Network was superseded by the Further Mathematics Support Programme (FMSP). In England the FMSP is funded by the Department for Education and managed by Mathematics in Education and Industry (MEI), an independent educational charity.
1.15 Since its inception, the take-up of A Level Further Mathematics in England has continued to increase dramatically and by 2013 delivery of A Level Further Mathematics in the state-maintained sector had grown to include 65 per cent of English schools. Between the establishment of the Further Mathematics Network and 2013, the number of students in England taking AS Further Mathematics had increased five-fold, whilst those taking the full A Level had more than doubled.
Table 1 Students taking up Further Mathematics in England 2004-2013
AS Level Further Mathematics A2 Level Further Mathematics Male Female Total Male Female Total 2004 2526 1235 3761 3894 1549 5443 2013 15366 6620 21986 9452 3780 13232 Percentage 508% 436% 485% 143% 144% 143% Increase Source: Joint Council for Qualifications (JCQ), 2013.
8 1.16 It can be seen that the proportionate increase in participation by female students was broadly in line with that for male students, although at a much lower level in numeric terms.
1.17 Another way of looking at this is to look at further mathematics entries as a proportion of mathematics entries by year, and by entries in all subjects. As can be seen, there has been a steady increase in England in the proportion of further mathematics entries compared with mathematics entries, at AS Level in particular. Similarly, the representation of further mathematics entries as a proportion of all AS and A2 Level entries has increased steadily over the period.
Table 2 Further Mathematics Entries as a Proportion of Mathematics / All Subject Entries, England 2004-2103
Per Cent 2004 2010 2011 2012 2013 Further Mathematics / AS Level 6.6 13.8 13.6 14.6 15.5 Mathematics A Level 11.3 16.0 15.4 16.1 16.3
Further Mathematics / AS Level 0.4 1.3 1.4 1.6 1.8 All Subjects A Level 0.8 1.4 1.5 1.6 1.7 Source: JCQ, 2013.
1.18 A recent evaluation update report on the FMSP in England7 presented very positive findings in relation to the FMSP there. It reported on numbers of students for both mathematics and further mathematics growing strongly year on year and these being among the four fastest growing A Level subjects throughout the period of the FMSP. Its achievements are supported by its ‘priority schools’ initiative, ‘Access to further mathematics’ events, continuous professional development (CPD) for teachers, revision programmes (online and face-to-face), and an extensive programme of enrichment events.
1.19 For students, FMSP support under the programme in England included tuition in further mathematics, revision in preparation for exams and also
7 Evaluation of the FMSP 2009-2012. Durham University Centre for Evaluation and Monitoring, August 2012.
9 enrichment activities; for teachers, support included advice in course management, opportunities for professional development (via extended professional development courses such as Teaching Further Mathematics (TFM) and Teaching Advanced Mathematics (TAM) courses), and the use of the FMSP’s online resources. The evaluation of the programme in England concluded that the FMSP’s work was ‘very effective’, and succeeding in its key aims. It also stated that ‘the FMSP’s work was essential if current levels of participation in Further Mathematics were to be sustained and further expanded’.
1.20 In terms of the sustainability and development of Further Mathematics, the evaluation identified the need for viable numbers in order to form teaching groups, at least one teacher who is confident and capable in delivering further mathematics tuition, and support of the senior management team at the school.
1.21 In comparison with the FMSP in Wales, less emphasis was placed in the evaluation report of the programme in England on problems of accessibility to further mathematics; teacher confidence and competence; time-tabling, curriculum choices and resources; problems posed by distance with its inherent cost implications; or the value of parental support. It did draw attention however to a marked gender imbalance in favour of male students (twice as many male students taking further mathematics as female students).
Engagement in AS/A Level Further Mathematics in Wales, Northern Ireland and England
1.22 To put the above increases in further mathematics entries into perspective, the graphs below look at the changes in proportions of further mathematics / mathematics entries and further mathematics / all subject entries between 2004 and 2010 across England, Wales and Northern Ireland; prior to the start of the pilot in SW Wales.
1.23 The data shows a stark increase in further mathematics entries as a proportion of mathematics entries at A2 Level and especially AS Level in England between 2004 and 2010. This rise was not mirrored in Wales
10 (although there was a slight rise in both AS and A2 Level proportions over the period), or in Northern Ireland, where the proportion of further mathematics to mathematics entries fell between 2004 and 2010.
Figure 1 Further Mathematics Entries as a Proportion of Mathematics Entries, 2004 and 2010
Source: JCQ, 2013.
1.24 The picture was similar in terms of further mathematics as a proportion of entries for all subjects. In England in 2004, further mathematics accounted for 0.4 per cent of all AS Level entries and 0.8 per cent of all A2 Level entries. By 2010, these proportions had risen to 1.3 per cent and 1.4 per cent respectively. In Wales, the AS Level proportion rose from 0.2 per cent to 0.5 per cent and A2 Level proportion from 0.4 per cent to 0.6 per cent over the period.
11 Figure 2 Further Mathematics Entries as a Proportion of Entries for all Subjects, 2004 and 2010
Source: JCQ 2013
1.25 In real terms, then, there was a rise in the proportion of entries in further mathematics at both AS and A2 Levels between 2004 and 2010 in Wales. This is also true in straightforward numerical terms, as the number of A2 Further Mathematics candidates increased from 138 in 2004 to 240 in 2010, with a corresponding rise from 2,487 to 3,362 at AS Level. In spite of this increase however, the rate of A2 Further Mathematics entries per head of population was still much lower in Wales in 2012, at 7.2 per 1,000 eighteen year olds8 than in England in that year, where the corresponding figure was 18.9. Hence Wales needed to increase representation of candidates sitting further mathematics at a very much faster rate than England if there was to be a chance of parity being established. This in itself provides a powerful rationale for establishing a pilot programme in Wales.
1.26 The above comparison data was also used by the team conducting the 2011 Evaluation of FMSP Phase 2 in England9,10 to arrive at the
8 ONS Data for Mid-2102 population estimates: England=670,895, Wales=40,243. 9 Evaluation of the Further Mathematics Support Programme Report on Phase 2 Searle, J. March 2011 10 It should be noted that there is a calculation error in the report and in fact the number of students in Wales studying A Level Further Mathematics actually increased by 74 per cent
12 conclusion that the FMSP (and its predecessor the Further Mathematics Network) had been effective in terms of influencing greater take-up of AS and A Level Further Mathematics in England.
Further Mathematics Support Programme in Wales
1.27 Despite consistent indications of the FMSP in England supporting an increase in the take up of further mathematics, no support was offered in either Wales or Northern Ireland11 during the period 2004-2010 (although those with a keen interest could access some support online via the FMSP England Website12). However, academics at Wales Institute of Mathematical and Computational Sciences (WIMCS) became increasingly concerned over the disparity in support and, following the first evaluation report on the English project which demonstrated success; moves were made to introduce a similar level of support in Wales. In July 2010 Welsh Government introduced a pilot Further Mathematics Support Programme (FMSP Wales) to trial a similar approach to that taken in England, initially on a regional basis in South West Wales. WIMCS, a collaborative partnership of the universities of Aberystwyth, Bangor, Cardiff, Glamorgan and Swansea, was contracted to deliver the pilot through a series of annual grants within a three year outline commitment. WIMCS in this case was represented by Swansea University, who were effectively sub- contracted by Welsh Government, on behalf of both WIMCS and Mathematics in Education and Industry (MEI). MEI is the organisation contracted to manage the FMSP in England on behalf of The Department for Education (DfE).
Principles of the FMSP Wales
1.28 As outlined above there was a layered, sequential rationale underpinning the FMSP pilot in Wales, in that it aimed:
between 2004 and 2010, which is still a lower rate of change than England (which was 108 per cent) but an increase, rather than a decease, as posited in the report. 11 Scotland delivers Highers rather than GCE A/AS levels and so is not included for comparison. 12 http://www.furthermaths.org.uk/
13
1.29 These principles were to be supported in two key ways: Building provision through direct support for schools and colleges in the context of the Regional Learning Partnership, 14-19 Learning Networks and local consortia arrangements; Providing tuition to those who could not access it directly in their local schools or colleges, through: o Predominantly distance learning, with occasional face-to-face tuition by the Further Mathematics Support Programme (FMSP) Wales; o A combination of distance learning and more frequent face-to-face tuition, e.g. once or twice a month, by the FMSP Wales; o Entirely face-to-face tuition by the FMSP Wales, with online resources as a support; o Entirely face-to-face tuition by a school/college being supported by FMSP Wales to act as a hub, with online resources to support; o A combination of FMSP Wales and school/college delivery, with online resources to support.
1.30 The original scope of the pilot was the four local authority areas of Pembrokeshire, Carmarthenshire, Swansea and Neath Port Talbot, which accounted for 48 secondary schools, of which 25 had sixth forms and 14 were Welsh medium (all with sixth forms). Each local authority area also
14 housed a Further Education (FE) College13. The total number of institutions with further mathematics students increased from 21 in 2010, to 26 in 2013. In 2013, the pilot was extended geographically into NW Wales and RCT, in partnership with the Universities of Bangor and South Wales. Although the defined pilot area covers only eight – and originally only four - counties in Wales, all secondary schools and colleges in Wales were eligible to register with the FMSP Wales free of charge, and to access the on-line resources. However, in terms of the logistics of programme coordination and delivering the face-to-face support (such as student tuition, careers talks, revision days etc.), this has been limited to the pilot area.
Additional / alternative sources of further mathematics support in Wales
1.31 The WIMCS and other bodies have been active in promoting Mathematics amongst school children through various programmes. For example WIMCS held Mathematics Applications Roadshows (in conjunction with Science made Simple) aimed at Key Stages 3 and 4, and the University of Glamorgan’s Faculty of Advanced Technology14 held mathematics master classes (through the WIMCS) for years 9 and 10, revision conferences, and pop quizzes. Also through the WIMCS/FMSP, the Royal Institute of Great Britain (Ri)15 supports mathematics master classes across the UK: “Ri mathematics and engineering master classes are hands-on and interactive extracurricular sessions led by top experts from academia and industry for keen and talented young people all around the UK”.
1.32 Similarly, but outside of WIMCS/FMSP, Aberystwyth University’s Institute of Mathematics and Physics hosts the Aberystwyth Mathematics Club16, and Bangor University School of Computer Science hosts the Royal Institute mathematics master classes for young people in northwest
13 At the time of contracting, there were two colleges in Swansea, but Gorseinon and Swansea merged to form Gower College in Summer 2010. 14 www.wimcs.ac.uk/schools.html 15 www.rigb.org 10 www.aber.ac.uk › Home › IMAPS › Aberystwyth Mathematics Club
15 Wales.17 WIMCS also holds the LEADERS Awards18, aimed to engage students with STEM subjects, and the National Science Leaning Centre19 holds ‘Cluster Awards’ available for clusters of schools to provide professional development for teachers with the support of the National Science Learning Centre. The aim of these is to raise standards in science and increase pupils’ engagement with and interest in science in the classroom and beyond. Also for teaching staff, was the National Science Academy / WIMCS STEM CPD Programme20, which ended in 2012.
1.33 Finally, the UK Mathematics Trust (UMT) offers an annual Team Challenge (for years 8 and 9), and Senior Team Challenge (years, 10, 11 12, 13). This was designed to “promote mathematical dexterity, team working and communication skills”21.
STEP and AEA qualifications
1.34 The STEP (Sixth Term Examination Paper) Mathematics is ‘a well- established mathematics examination designed to test candidates on questions that are similar in style to undergraduate mathematics’.22 The test consists of up to three 3-hour paper-based examinations: STEP I, STEP II and STEP III. Candidates are usually required to sit either one or two of the examinations, depending on the requirements of the universities they hope to attend. The Admissions Testing Service claims that STEP ‘acts as a hurdle’, (a good indicator of potential to do well on a difficult course), that it is useful preparation for an undergraduate mathematics course, and that it tests motivation by demanding dedicated preparation. Cambridge and Warwick Universities typically require undergraduates on their G100 mathematics degree courses to have taken STEP mathematics, and Bristol University, Bath University, Oxford University, and Imperial College London all encourage applicants to take the papers.
17 www.bangor.ac.uk/cs/ 18 www.wimcs.ac.uk/schoo 19 www.leadersaward.com 20 http://www.wimcs.ac.uk/schools.html 21 www.ukmt.org.uk 22 http://www.admissionstestingservice.org/our-services/subject-specific/step/about-step/
16 1.35 The FMSP/MEI offers CPD for teachers for both STEP and Advanced Extension Award (AEA) examinations. The website includes a series of problem-solving questions for student use, and a list of events are provided offering day-long courses to support teachers delivering STEP or AEA at a number of locations in England. For Wales (and the rest of the UK), an online course is available with the aim of advising, supporting, and helping teachers to prepare students for STEP, via six one hour live online sessions running through October and November, at a cost of £85. Other online CPD resources include Meikleriggs’ Mathematics, NRICH, and Master Mathematics (Durham University).
1.36 The MEI also offers support to students preparing for STEP and AEA via 10 fortnightly tutorials between February and June each year. NRICH23 provides support for teachers and students via numerous exercises to help prepare students for these qualifications and/or University in general.
23 http://nrich.maths.org/10047
17 2 Key Findings and Recommendations from the Study
2.1 This section of the report presents an overview of the main findings of the evaluation to date: process issues, the progress against outcome targets and recommendations for the future. The detail underpinning this content is then expanded upon in subsequent chapters of the report.
Process issues
2.2 Stakeholders feel that the programme has been managed effectively and prudently, with a very hands-on, committed team. An active management committee reports feeling engaged with the project and other stakeholders are highly complementary of the approach taken to date.
2.3 Schools and colleges engaged with the FMSP. The pilot’s stakeholders and management committee members were generally very positive about the pilot approach of providing a number of different ‘routes’ for pupils (and their teachers) to support them achieving a further mathematics qualification. The evidence showed that pupils were able to access the most effective support to match their individual circumstances.
2.4 Awareness raising has been underpinned by a thorough programme of publicity, backed by attendance at events and extensive personal contacts from the project team. Mathematics teaching staff in target schools generally had a good awareness of the pilot. Some stakeholders were not sure of the success of the project in engaging parents.
2.5 Student tuition has been very well received, despite the project having to charge a non-recoverable fee of more than £200 per student to schools. Face-to-face tuition was highly rated, especially when easily accessible to students, and online support was also appreciated despite some issues of scheduling and student access. There were some concerns about the quality of support through the medium of Welsh and this is an area that should be investigated further.
2.6 Face-to-face tuition was clearly the preferred method of teaching and learning amongst those interviewed. Teachers spoke warmly of the added value offered by online materials, however; especially past papers and
18 revision exercises, although students were not always aware that resources they were using had originated on the FMSP site.
2.7 The general FMSP website was felt to be in need of further refreshment, in order to make it more engaging and broaden its appeal beyond those already committed to further mathematics.
2.8 Enrichment events were a very popular element of the pilot, comprising events targeted at KS4 Pupils and post-16 students, mathematics master classes held in university buildings, careers talks in schools and colleges and revisions days in Swansea and Pembrokeshire. The inclusion of careers talks for pupils in years 10 and 11 and masterclasses for year 9 pupils were especially well received for building pupils’ interest in studying mathematics and STEM subjects at a higher level. The revisions sessions were also used as refresher courses by staff who had not been involved in teaching further mathematics for some time.
2.9 There is no one element of the pilot that has brought individual success, but it is the integrated approach to support that has been valued by teachers and students alike.
Monitoring and reporting
2.10 Currently the quarterly reports compiled by the FMSP team present an overview of school and college registrations to date and the provision of further mathematics by type of delivery setting, summary of recent and planned events, promotional activities and future priorities. However, the information lacks benchmarking or reference to desired outcomes of the pilot.
Value for money
2.11 Assessing value for money of a pilot programme can be challenging, given the extent of capacity building and initial programme development entailed, in addition to the delivery of support for further mathematics itself. As discussed above, there is no single clear measure for the number of additional students engaged in further mathematics as a result of the pilot. However, if examination entry data from the LLWR / WED is used, as the
19 most conservative estimate of outcomes, a total of 165 additional examination entries have occurred against what might have been expected, using 2010 data as a baseline. Against an overall programme cost of £581,485, this represents a cost per examination entry of £3564. However, the pilot has delivered extensive benefits beyond examination entries, in terms of awareness raising and encouragement of students to undertake mathematics and STEM subjects at a higher level, capacity building amongst teachers and setting in place a process which should develop further and bring additional benefits in the coming years.
Progress against outcomes
2.12 The FMSP was set six key outcomes at the start of the pilot. Although there is not yet sufficient time-series data to draw robust, long-term conclusions, the majority of outcomes have been achieved, at least in part. The evidence from the evaluation, as to how far these have been achieved is set out below.
Outcome 1: Increased numbers of students in the pilot area studying Further Mathematics at GCE A/AS levels, over the life of the pilot.
2.13 There has been a clear increase in the number of students undertaking A Level Further Mathematics in the pilot area, although the data is less conclusive in terms of AS level at present. In terms of a counterfactual comparison with Northern Ireland, where figures have not changed, examination entries have clearly increased in the pilot area and in Wales to a lesser extent. There has also been an increase in the numbers of students studying mathematics, although the indicative rate of increase has been lower than that for further mathematics.
2.14 Data from FMSP registration data shows a year on year increase in the number of students studying further mathematics at both AS and A Level in the pilot area. Lifelong Learning Wales Record (LLWR) data and Welsh Examinations Database (WED) data on examination entries shows a sharp rise in the level of A2 Level entries since the beginning of the pilot, increasing four-fold from 21 in 2010 to 105 in 2011 and 87 in 2012. There was also an increase across Wales during that period, where the number
20 of A2 entries more than doubled from 142 in 210 to 290 in 2012. However, this has not been reflected in AS Level entries in the pilot area to date; there were 50 entries in 2010, falling to 31 in 2011 and 34 in 2012. The reasons for the lack of consistency between the FMSP data and the examinations data sources are not clear and will need to be monitored once further data becomes available. However, it is likely that a contributing factor is that some students do not “cash in” modules taken as AS Levels, but treat them as part of an A Level qualification, which is then recorded in entry and attainment data returns.
2.15 If the proportion of examination entries in further mathematics is looked at as a proportion of entries in mathematics, there has been a relative increase in the pilot area, against the rest of Wales.
2.16 Importantly, there are also emerging signs of an increase in further mathematics attainment levels, especially at A Level; both in terms of passes at grades A*-E and at the highest levels (A*A) in both the pilot area and the rest of Wales. In terms of passes at the highest levels (A*A), between 2010 and 2012, attainments in the pilot area increased four-fold (from 11 to 44) and in the rest of Wales by two thirds (from 69 to 115).
2.17 However, whilst further mathematics entries from both genders have increased in the pilot area, the proportion from females has reduced between 2010 and 2012. By contrast, figures for England showed that the proportionate increase in participation by females has been broadly in line with that for males. Data is available at a Wales level (but not for the pilot area) for 2013 and shows a partial reversal of the sharp fall in the proportion of female entries in 2012. When 2013 data for separate genders are available for the pilot area in 2014, it will be possible to see whether the gender gap is still as wide.
Outcome 2: More schools and colleges in Wales offering Further Mathematics, either individually, or via consortia.
2.18 On this measure, good progress has been made in the pilot area: Coverage of further mathematics delivery in the pilot area has increased
21 from 16 11-18 schools in 2010 to 22 of these schools. FE Colleges were delivering further mathematics throughout this period.
2.19 In 2010, 21 out of 32 school sixth forms and FE colleges in the pilot area offered further mathematics. By February 2013, this number had increased to 26 out of 29 centres in the area24. The number of schools delivering further mathematics in a classroom setting (either timetabled or at lunchtime / after school) rose from 11 in 2010 to 20 by 2013. Over the same period, the number of schools without classes, but with a single supervised student or up to two unsupervised students fell from five to two.
2.20 The FMSP secured registrations from all sixth form centres in the pilot area by 2013, along with the majority of 11-16 schools in the region25.
Outcome 3: Increased numbers of mathematics teachers in Wales who are trained to teach further mathematics.
2.21 There is no clear means of measuring the “stock” of teachers qualified to teach further mathematics in Wales and so this outcome could not be assessed effectively in the course of the evaluation. Delivery of CPD for teachers was not part of the original pilot programme and online support was introduced in 2013. Discussions with teaching staff during the evaluation suggest that this element should be a focus for continued effort, in order to ensure that Wales has the capacity to meet demand for further mathematics in future. The evaluation found that many mathematics teachers are not confident to teach further mathematics either because they have never received training in the relevant modules, or because they are out of practice in delivering them.
2.22 Unlike the programme in England, the pilot FMSP in Wales did not initially include specific resources to address CPD issues; relying instead on the online resources available on the MEI website. The call for teacher CPD within the pilot area led to the start of Live Online Professional
24 Mergers of two colleges into one and four sixth forms into two brought about the reduction in total centres from 32 to 29. 25 All of the 11-16 schools and a small number of 11-18 schools that were registered with FMSP did not deliver GCE level further mathematics tuition.
22 Development from October 2013, which has already proved a very popular resource. As a measure of the demand for CPD, approximately 130 teachers had taken part in various events for students and 17 teachers have taken part in various workshops organised to date.
2.23 In terms of wider levels of qualification to teach mathematics (although not necessarily further mathematics), the number of all secondary teachers trained in mathematics and registered with GTCW increased from 1,204 in 2009 to 1,412 in March 2013, accounting for 9.9 per cent of all teachers by that date26 and second only to the number of English teachers (10.2 per cent of the workforce). The data shows that 73.8 per cent of those teaching mathematics at secondary level were known to be trained in the subject; the highest for any subject area. For comparison, 30.1 per cent of science teachers were qualified in their subject. Further, the number of newly qualified mathematics teachers registered with GTCW grew from 70 in 2009 to 84 in 2013, accounting for 11.8 per cent of the total at that time; the largest proportion of any subject specialism.
Outcome 4: Overall raised awareness among students and their parents of the importance of studying mathematics at higher levels.
2.24 Awareness of the opportunities offered by studying further mathematics is difficult to measure specifically. However, discussions with existing students and teachers suggest that there is a general understanding of the potential that of the subject to support higher level study. The increased take up of further mathematics AS/A Level provision and increased applications for undergraduate further mathematics and STEM courses corroborates this.
2.25 The evaluation included interviews with and surveys of students and other stakeholders, to test their attitudes towards mathematics and STEM subjects at HE Level. In the online survey of undergraduates, for example, improving career options was one of the main reasons for choosing further
26 http://www.gtcw.org.uk/gtcw/images/stories/downloads/Annual%20Statistics%20Digest/Annual_ Stats_13_E.pdf. No. of teachers registered by GTCW by ITET Subject trained (secondary only).
23 mathematics at A Level for almost 40 per cent of respondents. It should be noted, however, that the main motivation for undertaking further mathematics was a personal interest in the subject, rather than an expectation of financial gain, or a university entrance requirement.27
Outcome 5: Increased numbers of students from Wales applying to study higher education courses in mathematics and related subjects, such as engineering and physics.
2.26 Data from HESA shows a clear increase in the number of applicants to mathematics courses from the pilot area over the last three years; from 155 in 2007/8 to 240 in 2011-12. This 55 per cent increase in mathematics course take-up from the pilot area outstripped the growth from Wales as a whole, which was 28 per cent. Increases in applications for STEM subjects were less pronounced and appeared more volatile. When 2012-13 HESA data are available, it will be possible to assess this outcome more fully.
Figure 3 First Degree Students Domiciled in SW Wales / Wales on Mathematical Sciences / STEM Courses in UK HEIs, 2007-2012
Source: HESA 2013
2.27 However, engagement levels still lie below those of English students and more work needs to be done to narrow the gap. There is a backdrop of low
27 For example only 40 per cent of those who said they were motivated by a personal interest in the subject also listed university entry requirements as a motivating factor.
24 growth in undergraduate numbers from Wales and it was the only UK country to experience a decline in university applications in 2013, with numbers falling by 0.2 percentage points across all subjects.
Outcome 6: Improved transition of students from further to higher education courses in mathematics, or from courses which have a significant element of mathematics, thus benefiting the wider economy.
2.28 Undergraduate students are clear that transition from A level to degree courses in mathematics, and to a lesser extent STEM subjects is made considerably easier by taking further mathematics at A/ AS Level. Generally, however, this advantage is eroded after the first year of undergraduate study.
2.29 There was general consensus amongst STEM students and lecturers interviewed during the evaluation that further mathematics at A Level was a clear advantage in the first year at university. In the online survey of undergraduates, the joint most important advantage of studying further mathematics at AS/A2 Level was that “It helped with the jump from AS/A level to undergraduate level”; cited by 27 per cent of respondents, alongside “It gave me a better understanding of the application of mathematics in general”.
Recommendations
2.30 The following recommendations, based on the lessons learnt through this stage of the evaluation, provide suggestions for developing and sustaining the progress made to date through the pilot.
(i) The pilot has proved successful in building engagement in further mathematics in the pilot area and the extension to Rhondda Cynon Taf and North West Wales is to be welcomed. This approach should be allowed to consolidate by maintaining support for the extended pilot, whilst considering a change of emphasis within the initial pilot area away from direct support to building sustainability through capacity building amongst teaching staff and sixth form centres. The full breadth of
25 support should be continued where possible, in recognition of the value of an integrated approach to developing awareness, engagement and support for pupils, without prioritising or discontinuing any individual elements.
(ii) The timescale of the pilot to date is such that there is currently insufficient data to robustly evaluate progress, or to recommend a changes to or extension of the programme in the future. It is therefore recommended that any decision to develop the programme is delayed until findings from the final impact assessment and counterfactual are known.
(iii) More needs to be done to continue to promote further mathematics to female students and encourage them to take further mathematics at AS/A2 level, as the gender gap in examination entry levels increased during the period up to 2012, despite indications at an all-Wales level of improvements in 2013. Actions could include enrichment events targeted at female students, presenting case studies and using gender- specific materials.
(iv) Attention needs to be paid to the quality of provision and support in the medium of Welsh, to ensure equality of access and standards to all students in Wales.
(v) The evaluation has shown the importance at an institutional level of school principals and senior management team members in promoting further mathematics provision – not least because of the financial implications of support. Brokerage work with this group could help to build commitment and embed further mathematics in a sustainable manner.
(vi) It is imperative that CPD is incorporated more fully into the FMSP in Wales at the earliest possible stage. The clear demand for online resources provides compelling evidence of need. Research with teaching staff has revealed a widespread lack of confidence at best and in many cases staff have not
26 received training in delivering further mathematics at AS/A Level. Future CPD should incorporate a module on use of online resources, to enable more effective use to be made of these. The accreditation of CPD would allow for tracking of take-up and this should be considered.
(vii) Collaboration and networking across schools should be encouraged to share resources beyond the formal collaboration resulting from the Regional Learning Partnership and 14-19 Partnership arrangements. In particular, any actions to increase levels of face-to-face tuition at convenient times and locations would be welcomed by practitioners and students alike.
(viii) The project management and delivery of the pilot have been acknowledged as generally very effective. However, the quality and clarity of progress reporting should be reviewed, to ensure that all partners have a clear understanding of the achievements of the pilot and of where barriers to success occur. The main report sets out suggestions for a reporting framework which clearly presents activities delivered by quarter, progress against outcomes and future plans and priorities. Securing the suggested data for this report will mean some additional monitoring, most notably monitoring outcomes in schools and colleges supported by the pilot.
(ix) The FMSP Website should be further updated and enhanced, to provide a more effective marketing and engagement tool for the pilot and a stronger identity for support in Wales. More work could be done to alert teachers to the breadth of materials available on the site, possibly including a quick guide to what is available.
(x) Where online activities are provided, their availability requires greater promotion and publicity, and they must ‘work’ in terms of easy, straightforward connectivity. Timing of sessions also requires more consideration.
27 (xi) There is currently no textbook for further mathematics modules available to students, which is reported to be a barrier to engagement for some individuals. The development of offline materials of this type would be a useful development in future.
(xii) Finally, there is a vulnerability in the pilot, in that it has substantially relied on the high level of commitment and support from the programme leader. Plans for any future investment will need to be mindful of the critical nature of this role.
28 3 Approach to the Evaluation
Overview
3.1 In recognition of both the rationale for the FMSP Wales, and the various levels of support available to teachers and students alike, it was recognised that this evaluation needed to gather evidence across a range of activity areas.
3.2 At a basic level, the evaluation firstly needed to determine the extent to which the FMSP Wales has influenced an expansion in the delivery of further mathematics, in terms of the number of schools and colleges starting to offer AS and A2 Further Mathematics for the first time28 and subsequently the number of learners engaged.
3.3 The other main area of enquiry concerned the degree to which FMSP Wales has helped to improve delivery and increase levels of achievement in AS and A2 Further Mathematics in South West Wales. This was a far more challenging undertaking, given both the range of support mechanisms that make up the FMSP Wales and the difficulty of attributing impact to individual activities, as well as the range of other factors that could influence take-up and attainment in mathematics in Key Stage 4 and 5 and in Higher Education, either before or during the course of the FMSP Wales. These factors included: Work undertaken by the South West Wales Regional Learning Partnership29 and the South West and Mid Wales Consortium; Effects of the school reorganisation policy and the Learning and Skills Measure (both in terms of increasing choice);
28 Please note, we are aware that delivery of A Level Further Mathematics was more widespread in the UK during the 1980s, and that schools in South West Wales may have previously offered the course, but had not done so in recent years. Therefore we had to agree during the inception stage what would constitute an appropriate period of time that schools and colleges were not offering A Level Further Mathematics prior to the start of the FMSP, in order to calculate the realistic impact that the FMSP Wales has had in the pilot area. It was agreed that we should look back to 2007 to do this. 29 The Regional Learning Partnership South West Wales was formed in 2007 and is made up of education and regeneration partners with the purpose of improving opportunities for learners and potential learners across the South West Wales region.
29 The newly introduced National Literacy and Numeracy Framework and associated annual tests.
3.4 In terms of the latter, although these are unlikely to have had a direct effect on take-up of AS and A2 Further Mathematics, given the timescales for their introduction and this project, the increased pressure30 on institutions to prioritise literacy and numeracy in the aftermath of Wales’ disappointing PISA31 results is likely to have already had an influence on mathematics teaching throughout Key Stages 3, 4 and 5.
Establishing an evaluation methodology
3.5 The evaluation was designed to provide a sound approach to reviewing the effectiveness of the pilot to date, based on the HM Treasury Magenta Book methodology. Following that pattern, the evaluation was centred on a simple logic model (Figure 4 overleaf), which allowed the team to consider the rationale, inputs, process, outputs, outcomes and emerging impacts for the project and to develop an initial evaluation framework and associated research instruments. This logic model forms the structural basis for the evaluation and guides the structure of the subsequent chapters of this report.
30 For example, the introduction of the secondary school banding system, where schools are placed in bands according to a range of performance indicators which includes Key Stage 4 results. 31 PISA (Programme for International Student Assessment) is an international study launched by the OECD in 1997. It aims to evaluate education systems worldwide every three years by assessing 15-year-olds' competencies in the key subjects: reading, mathematics and science. To date over 70 countries and economies have participated in PISA.
30
Figure 4 Logic Model for Further Mathematics Support Pilot Evaluation
3.6 The full evaluation questioning framework, mapped by planned data sources, is appended at Appendix 1 of this report.
31 Programme logic: Pre-evaluation scoping
3.7 As a precursor to the main evaluation, a short scoping exercise was conducted, holding a review workshop with the project team and interviewing a total of thirteen stakeholders to understand the rationale for the programme, the approach taken and the design process. These interviews included representatives of WJEC, Estyn, GTCW, Welsh Government School Standards and statistics students on placement with Welsh Government; this last group providing early insights into undergraduate experiences of the effects of further mathematics on performance at HE Level. A copy of the topic guide used for these interviews is included at Appendix 2.
3.8 An initial review of programme documentation and existing monitoring data was also undertaken at this point, comprising the contract agreement with Welsh Government, performance indicators, budget allocations and timescales and reports of delivery progress to date. Finally an overall evaluation framework and associated research instruments were devised to guide the project. These are attached at Appendix 1.
Process evaluation
3.9 The process evaluation provided discrete reviews of the inputs, process, outputs and outcomes elements of the programme, although common information sources and related fieldwork approaches were used to inform these elements to a large extent. The evaluation was structured as follows:
Preparatory Work
Management committee interviews
3.10 Qualitative telephone interviews were carried out with eight32 of the 24 active members of the FMSP Management committee, to inform the main fieldwork programme. Members were asked about their opinions on AS/A2 Further Mathematics generally, their awareness of and any views
32 This included representatives from Careers Wales West, MEI, WIMCS and universities, schools and colleges in the pilot area.
32 on the FMSP specifically, and their views on the potential implications of increasing take-up (and completion) of AS/A2 Further Mathematics in Wales, for students, schools and colleges, HEIs, employers and the wider Welsh economy. A copy of the topic guide used for these interviews is included at Appendix 2.
Initial review of school and FE performance data
3.11 The preparatory phase also included an analysis of examination entries and attainments in AS/A2 Mathematics and Further Mathematics in schools (using Welsh Examination Database (WED) data) and colleges (using LLWR data) over the period 2008-201233 to identify trends in attainment and to ascertain levels of drop-out in the pilot area and across Wales as a whole. The review was additionally used to inform the sampling frame for the qualitative research with schools and colleges in the pilot area, by identifying institutions where AS/A2 Further Mathematics was, and was not delivered.
Schools and colleges research
Qualitative interviews with staff and students at 14 case study schools and colleges
3.12 Of the 52 secondary schools and FE colleges in the pilot area, a total of 14 were selected as case studies, representing a sample of approximately 25 per cent. Schools were selected using a complex sampling frame, to include: Schools and colleges; Welsh-medium, English-medium and bilingual institutions; Schools with and without sixth-forms; Broad representation across each local authority; Institutions currently delivering AS/A2 Further Mathematics; Institutions not currently delivering AS/A2 Further Mathematics; Institutions that have engaged with the FMSP, and;
33 2008, 2009, 2010, 2011 and 2012. Provisional schools data was subsequently secured for 2013.
33 Institutions that have not engaged with the FMSP (for the purpose of assessing the counterfactual),
3.13 The sub-sample of schools and colleges that had engaged with the FMSP included institutions that had engaged with each type of support offered through the FMSP including pre- and post-16 activities. In each case, visits included interviews with relevant staff and focus groups with students. A list of the schools and colleges included in the qualitative research is provided in Appendix 3.
Online survey of mathematics teachers and tutors in the pilot area
3.14 A bilingual on-line survey of mathematics teachers and tutors across the four local authorities was run in parallel to the qualitative research. The survey was conducted as a census of the 48 state secondary schools and four colleges in the region; links were emailed to the head of mathematics in each institution via the relevant local authority mathematics advisor. A total of 17 responses was received, 12 from 11-18 schools (a 48 per cent response rate) and 5 from 11-16 schools (a 22 per cent response rate). No response was achieved from FE colleges in the area, which is a limitation of the dataset.
University fieldwork
Qualitative interviews with 12 university STEM admissions tutors or lecturers
3.15 Telephone interviews were carried out with 12 admissions tutors or lecturers in STEM subjects from a selection of universities in the UK. Interviewees were selected at random, but included representation of academics from: Universities in Wales and in England; Russell Group and non-Russell Group institutions; Academics from mathematics, physics, engineering (various fields) and medicine departments;
34 Departments where A’ Level34 Further Mathematics is, and is not a pre- requisite to study.
3.16 The purpose of the interviews was to understand how and to what extent AS/A2 Further Mathematics advantages students applying to and studying STEM subjects at university.
Online survey of first year undergraduates on STEM related courses
3.17 The university fieldwork included an on-line survey of students part-way through their first year in university who were studying mathematics, physics, engineering (various fields) or medicine. Survey links were emailed to students via the admissions tutors and lecturers included in the telephone interviews described above. The link was also disseminated more widely to students using social media and therefore the sample was self-selecting. The purpose of the survey was to identify differences between students who had, and had not attained AS35/A2 Further Mathematics, in terms of access to university and the ease of transition to their respective degree course as well as other factors in relation to undergraduate study. The survey generated 120 responses; all from mathematics, physics or joint honours students. No responses were achieved from other STEM subject undergraduates, such as engineering or medicine. Due to the nature of the sampling structure, it is not possible to establish the response rate. Results from the survey are reported in Appendix 4
Qualitative interviews with 20 undergraduate students on STEM related courses
3.18 Survey respondents were asked whether they would be willing to participate in a more in-depth telephone interview and were offered an incentive of a £25 voucher for their time. A total of 20 students in their first year of a STEM subject degree course were interviewed over the telephone. The sample included:
34 Please note, none of the universities included in the sample made degree offers based on the attainment of AS qualifications. 35 Some students will attain AS Further Mathematics despite it not being a condition of their university offer.
35 Students who had and had not taken AS/A2 Further Mathematics; Students who had and had not engaged with the FMSP (in either England36 or Wales); Students on mathematics, physics and economics courses. As above, there were no opportunities to interview students on wider STEM-related courses with a further mathematics background
3.19 The interviews sought to explore the issues included in the on-line survey questions in greater detail, in order to add depth to the quantitative results. The topic guide used for the interviews is included in Appendix 2.
3.20 The process evaluation aimed to answer some key questions about both the effects of further mathematics and the influence of the FMSP on students and teachers.
Impact assessment and the counterfactual
3.21 A range of possible approaches was considered for assessing the emerging impacts and the counterfactual, in relation to the evaluation of the Further Mathematics Support Programme pilot. Exploring the emerging impacts of any programme can be a significant challenge, not least because of the issue of attributing causality of any observed changes to the effects of the programme and its constituent activities. In this case there were several dimensions to consider in estimating impacts: Shorter-term impacts, which could be measured through either official data or through primary research; Longer-term impacts, which will require time to accrue and which will require a more econometric approach to assessment. For example, do FMSP pilot beneficiaries experience longer term differences in behaviour or circumstance (such as career path, salaries etc.) against those who did not receive support? If so, is the difference attributable to the support received under the FMSP pilot?
36 Given the small number of first-year university students who took their A’ Levels in the pilot area and the similar nature of the FMSP in England to the FMSP Pilot in Wales, the decision was taken to use students who had engaged with the FMSP in England as a proxy for those who had engaged with the FMSP in Wales.
36 3.22 Some of these are summarised in Figure 6 below.
Figure 5 Potential Short and Long-Term Impacts of FMSP
3.23 In terms of short-term impacts, data is available from both DfES in Welsh Government and JCQ37 for the numbers of entries and attainments relating to further mathematics and mathematics AS/A levels, although there are differences in both the data recorded and the intended scope of the information. For example, it is not possible to obtain JCQ data to a level lower than a Wales level and so is not available for the pilot area. Also, it includes all ages and delivery settings, whereas LLWR and WED data reflects year 12 and 13 students, excluding independent schools. However, JCQ provides useful information for comparisons with England and Northern Ireland.
3.24 It was felt that early impacts of the FMSP pilot would potentially be observable in terms of any increased entry to mathematics and related courses in HEIs from the pilot area. The FMSP began only in 2010 and so the first beneficiaries of a full, supported A Level course would only have entered HE in 2012. Statistics are available from HESA for numbers of first year undergraduate students in mathematics and STEM Subjects, which will cover this initial (2011-12) cohort. Additional data for 2013-14 will be available for the final impact evaluation, in 2014.
3.25 However it was also clear that the longer term questions of economic impact would be more difficult to estimate at this relatively early point in support. For example, a longer term measure would be whether those
37 Joint Council for Qualifications; a membership organisation comprising the seven largest providers of qualifications in the UK.
37 engaging with the pilot went on to enjoy enhanced career prospects or increased salaries as a result. At this stage, however, it was recognised as being too early to establish the achievement of these longer term advantage in terms of salaries and career paths. However, it was hoped that it might be possible to model the likely advantage accruing to beneficiaries, as a result of acquiring increased skills.38
The counterfactual
Spatial approaches
3.26 In terms of the counterfactual – that is the assessment of impact and causality by asking the question “what would have happened in the absence of the intervention?” the evaluation set out to consider comparisons of activity in the pilot region against non-pilot areas in a number of ways, including: High level data for participation in AS and A Level Further Mathematics at a SW Wales Regional level against Wales, England and Northern Ireland data. Limitations were acknowledged to be the timeframe for the support pilot to have had an impact (data were available up to 2013 from JCQ and WED for schools but only to 2012 from the LLWR for FE). There was also the issue that not all schools/colleges in the pilot area received support from FMSP, as some were already delivering further mathematics prior to the intervention. As there is a long standing FMSP in England, it was felt that this should be included as the benchmark for performance that Wales should aspire to at least emulate, and preferably exceed. Northern Ireland provides a better benchmark for the ‘no intervention’ state, as it has not developed a support programme for further mathematics. Local authority level data – comparing either the SW Wales regional average or individual local authority areas within SW Wales against performance in a selected local authority area or areas within Wales, but
38 For example, there is a standard methodology from BIS in England for linking skills increases to salary advantage among work-based learners, which could potentially be extended to school or university students.
38 outside the pilot area. It was felt that the evaluation could look at SW Wales data against each of the three other Welsh regions, for example, for entries and attainments in both mathematics and further mathematics over the pilot period. In the outturn, however, the very small numbers of students in some authorities suggested that data disaggregation below the Wales level would not provide sufficiently robust data for the evaluation. School/pupil level data – using a case study approach to match a small number of comparative schools/colleges (for example using school banding and location: urban/rural etc.) within and without the pilot area for a more intensive comparison. Given the relatively low numbers of further mathematics students at any given institution, this seemed to be an unlikely route to take, although worthy of consideration for highlighting best practice or individual examples. Were data to be available, it would be possible to investigate matching pupils across schools using a propensity score matching approach based on LLWR/WED-based indicators such as: o Gender o GCSE Points o AS/A Level portfolio o Residence deprivation score
3.27 There are other potential tools available to reduce the effects of externalities (such as varying cohort numbers, economic circumstances, economic downturn etc.) when looking at participation and attainment data at AS/A2 Level. These could include looking at the proportion of further mathematics pupils achieving A*/A Grades at A Level in Mathematics. There is a caveat to this, in that as further mathematics is promoted and mainstreamed, it will possibly attract less able pupils and so grade dilution can be expected to occur. Hence, it would be necessary to control for this by referencing GCSE Mathematics grades of candidates and comparing students of similar abilities in terms of their A Level experiences in both mathematics and further mathematics.
Time Series Approaches
39 3.28 The obvious alternative to a spatial approach was to conduct a time series review of data, again within and without the pilot area, to identify differences in changes taking place with respect to performance data pre and post intervention. The limiting factor here was the issue that the first full cohort of supported pupils only took A2 Level examinations in 2012, implying a need to draw conclusions from one, or at best two, years’ data. This approach and the spatial one outlined above were not mutually exclusive, however, and it was expected that the evaluation should be able to undertake a combined approach, which would allow for some triangulation of data.
3.29 Hence it was decided that, by taking time series and spatial data for all available indicators, it would be possible to map the relative progress of students in the pilot area against a wider Wales / UK average. By doing this, a ‘difference in difference’ model could be employed to extract the effects of external influences, such as economic growth, cultural factors, changes in education policy etc.
3.30 This might be summarised as follows:
Figure 6 Approach to the Counterfactual
Pilot (Treatment) Area
Comparator (Control) Area
Source: Miller Research
Where:
40 c is the difference in performance in the treatment area over time; d is the difference in performance in the comparator area over time; b is the difference in difference; a proxy for change attributable to the intervention.
3.31 The approach taken assumes similar characteristics of externalities in pilot and non-pilot areas, to cancel out noise through the use of comparative measures of progress. In real terms, this approach provides a more practical measure of success, in that an approach which controlled for all external factors in the pilot area in isolation may have registered an impact arising from the intervention. However, if this impact was insufficient to keep up with change in comparator areas, it would be a hollow success. The spatial comparison over time therefore provides a more useful measure of progress.
3.32 The results of the calculations can be found in Chapter 6, on outcomes and impacts of the FMSP.
Final evaluation report
3.33 The analysis aspects of the evaluation will be revisited in 2014 to provide a final report, which includes more robust data arising from a further year’s examination results. The final report, to be compiled in late spring 2014, will: Update participation data from the FMSP pilot (including the extended pilot); Prepare a review of all 2013 attainment data (incorporating FE data from the LLWR which was unavailable for this report); Add in 2012-13 HESA data relating to Welsh domiciled first year students on STEM subjects in UK HEIs; Revisit the gender analysis, to investigate whether the gender gap in further mathematics participation is narrowing; Analyse KS4 participation rates in mathematics and attempt tracking of A Level Further Mathematics students to investigate the relative success rates of those with different grades in GCSE Mathematics. Feed results
41 into the gender analysis and assess any differences between pilot and non-pilot areas in respect of this information;
3.34 The final report will present an extended version of the analysis contained in this interim report, reporting by outcome. Whilst the numbers involved will still require an element of caution in interpreting results, the additional data will serve to identify emerging trends more effectively and provide a more extensive evidence base for future shaping of policy.
Summary
3.35 This evaluation is based on the HM Treasury Magenta Book methodology. Following that pattern, the evaluation was centred on a simple logic model which allowed the team to consider the logic, inputs, process, outputs, outcomes and emerging impacts for the project and to develop an initial evaluation framework and associated research instruments. A spatial comparison over time was then used to provide a measure of the programme’s progress. It was recognised, however, that the limitations of a short time series prevented robust conclusions being drawn, highlighting the need for a further review of the pilot. The final impact evaluation in 2014 will utilise an update of the available data to provide a more robust analysis of the outcomes and emerging impacts of the FMSP pilot.
42
4 Detailed Findings - Structure of the FMSP Pilot
4.1 This section of the report looks at the “inputs” section of the logic model, covering an overview of the pilot, management and governance, monitoring and finance, communication and awareness.
Overview of the FMSP
4.2 The FMSP set out to support a range of activities aimed at widening access to further mathematics in the pilot area, under two main headings: Strengthening existing provision, through promoting (further mathematics) and supporting schools and colleges as part of local consortia, 14-19 learning networks and the Regional Learning Partnership for SW Wales, and; Providing tuition in further mathematics GCE AS/A2 level units for all students who could not access them through their local schools or colleges. This included a range and mix of approaches as follows; Predominately distance learning, with occasional face-to-face tuition by the FMSP; A combination of distance learning and more frequent face-to-face tuition by the FMSP; Entirely face-to-face tuition by the FMSP, supported by online resources; Entirely face-to-face tuition by a school / college, supported by the FMSP and supported by online resources, and; A combination of FMSP and school/college delivery, supported by online resources.
Management and governance arrangements
Project management
4.3 WIMCS has been responsible for the overall management of the FMSP Wales pilot from the outset. Three posts were initially funded under the programme, namely a full-time programme leader, a half-time administrator and a half-time project support assistant, supported by a team of six tutors,
43 some of whom were already working full-time in schools and FE colleges in the region.
4.4 The programme leader has been responsible for day-to-day management of the project, liaising with schools and colleges and reporting to the management committee; as a qualified mathematician she also delivers a substantial amount of the revision and enrichment activities. The programme leader deals directly with individual schools, but also works with mathematics leads in local authorities. WIMCS acknowledges that participation depends very strongly on individual teacher commitment and the project leader communicates with each school every three to four weeks to maintain commitment. Since the extension of the pilot into RCT and North Wales, two additional coordinators with relevant skills (a STEM Background in RCT; a Welsh speaker in NW Wales) have been appointed to manage the delivery of the programme in these areas.
Findings
4.5 Considering the geographic breadth of the pilot area and the limited capacity of the FMSP Wales team, the evidence gathered showed that the pilot appears to have been managed very efficiently and in a hands-on manner. At an administrative level, the general consensus amongst relevant stakeholders has been positive and members of the management committee generally felt sufficiently informed about project progress to contribute to discussions. The energy, enthusiasm and charisma of the programme leader was commended by teachers, students and other stakeholders as being an important factor contributing to the success of the pilot:
4.6 “[Programme leader] was very keen [at a Head of Mathematics meeting], which was an attraction ... she was also very engaging at the revision events.” (Head of Mathematics, school currently delivering further mathematics). “The management works well, partly because it sits within WIMCS and partly because [programme leader] is highly capable.” (Management committee member)
44 4.7 Contracting with Welsh Government has been organised on an annual basis, with an extension each year based on performance during that year. Whilst the members of the management committee recognise that this has introduced a sense of prudence and performance-related support into the pilot, it has created difficulties for the project team in terms of planning and ensuring continuity of staff. Aside from this, project staff reported having a very good relationship with Welsh Government, despite changes in project management representatives.
Governance
4.8 A management committee was convened, made up of representatives from WIMCS, MEI, Careers Wales and Welsh universities in addition to a number of local authority mathematics advisors, and mathematics teachers and tutors. The committee generally meets three times a year, in November (to report on the start of tuition for the year), March (to reflect changes in provision of further mathematics in schools and colleges) and July (to discuss results and plans for the following year). Hence there is an important role for the committee in shaping delivery and acting as critical friend/sounding board for the pilot. Committee members generally felt that the group had sufficient coverage and had brought together professionals from very different perspectives, albeit all with a mathematical interest: “It’s also helped to bring a bit of reality to the pilot ... you get mathematicians in academic departments talking directly to schools and colleges.” (Management committee member). Membership can be rather fluid, however, as different representatives of the same organisation may attend different meetings.
4.9 Over the course of the pilot programme, a good relationship has been developed between the management committee and those responsible for delivering the pilot; similarly, committee members have noted unanticipated outcomes that have arisen from the regular meetings of this disparate group: “There have been benefits beyond the project itself ... it (the management committee) has enabled an exchange of communication that wouldn’t necessarily have happened.” (Management committee member.)
45 4.10 Members of the management committee were also aware of some of the challenges facing the FMSP in Wales, such as ensuring that stakeholders, schools and college staff remained open to the activities of the FMSP and did not become defensive or disengaged. There was a need for the FMSP team to stay diplomatic and tactful, given that many colleges and schools would “ideally like to service their students themselves”. (Management committee member).
4.11 Consultation with members of the committee suggests that most feel they are part of something worthwhile and are valued as members: “Each person has the chance to make a contribution and you’re not made to feel stupid if you don’t understand something.” (Management committee member).
4.12 There were those who questioned this, however, reflecting that the extent to which the committee actively influences the pilot is limited. Several members described their role as being insufficiently influential: “I don’t think we’ve got the clout we should have ... I don’t think the Committee was designed to steer the programme considerably but play more of a sounding board for [programme leader].” (Management committee member).
Project monitoring and finance
Project finance
4.13 Details provided to the evaluation team by FMSP showed that a total of £581,485 was allocated for the project from its inception in 2010 through to the end of the extension period in March 2014. The great majority of costs related directly to staff salaries for the programme leader, administrator and project support assistant, for example accounting for slightly less than £99,000 in 2011-12, out of a total operating budget of £126,500.
Table 3 Financial Support to FMSP from Welsh Government
Pounds Sterling Jul 10 – Jul 11 – Jul 12 – Apr 13 – Total Jun 11 Jun 12 Jun 13 Mar 14
Budget 147,497 126,519 126,560 180,909 581,485
Source: FMSP. Note the final year includes funding for the extended pilot.
46 4.14 Hence general operating expenses were kept to a minimum; covering such items as payments to MEI for consultancy and resources, general administration and travel.
4.15 One of the financial challenges to the pilot was that when the programme was originally formulated, schools providers were funded per qualification delivered, at a rate of £210 per module, or approximately £630 per head. It was therefore proposed that FMSP would provide support at this rate, effectively offering a revenue neutral solution for schools to provide delivery in further mathematics. Later, however, there was a change to support so that schools received a fixed amount per pupil on a capitation basis. Hence if a school does not provide a given subject, additional tuition (from projects such as FMSP) does not attract additional funds. This was felt to have had an adverse impact on demand for support and presented the FMSP Team with a more difficult task in securing engagement from providers; in that mathematics leads in individual schools had to make the case internally for financing tuition.
Progress reports and monitoring
4.16 Progress reports are compiled each quarter for submission to the management committee. These contain some useful information, although some committee members were critical that the reports go into too much statistical detail rather than providing a strategic overview of progress and the issues facing the pilot, with straightforward narrative to explain the figures included in the reports.
4.17 Currently the quarterly reports present an overview of school and college registrations to date and the provision of further mathematics by type of delivery setting, summary of recent and planned events, promotional activities and future priorities. However, it was pointed out that the information lacks benchmarking or reference to desired outcomes of the pilot. A suggested structure for content of future management reports is laid out in Appendix 6 of this report.
47 5 Detailed Findings.
Delivery of FMSP
5.1 This chapter looks at the process and output elements of the evaluation, dealing with delivery of the pilot and the extent to which outputs were achieved.
Delivery aims
5.2 The FMSP aimed to engage pupils and students at state schools and FE colleges, heads of mathematics and mathematics teachers. The delivery programme was designed to provide student tuition for AS/A2 Level Further Mathematics and support for teachers via the following activities: Promotion and publicity Student tuition for AS/A2 Level Further Mathematics; Access to the MEI Resources Website; Mathematics enrichment courses for students in Key Stage 4 and post-16; AS and A2 Mathematics and Further Mathematics revision days39; CPD support for teachers.
5.3 Mostly, the schools and colleges engaged with the FMSP, the programme’s stakeholders and management committee members were highly positive towards the activities on offer, deeming them to be both appropriate and valuable. Critically, what the activities succeeded in doing was to provide a number of different ‘routes’ to pupils (and their teachers) to help them towards achieving a further mathematics qualification. The evidence from the primary fieldwork in schools and colleges revealed that not all routes could be used by all students (often for reasons of logistics), nor were all always needed, but having the different options available increased the choices available to any individual student at any time during their course of study. 40
39 Revision Days are incorporated within Enrichment Courses in the following pages 40 Expanding on this point, some students were unable to attend sessions held on Saturdays because of part-time jobs or difficulties with transport , or sessions being too far away. Others did not need to access FMSP activities or resources because they received fully time-tabled teaching and support through their school.
48 Promotion and publicity
5.4 The first aim of the FMSP pilot is to raise awareness among students and their parents of the value of studying mathematics at higher levels. A structured communications plan was put into place and in the three years that the pilot has been running, the team has been active in raising awareness of the existence of the programme and what it sets out to deliver across all stakeholder groups. The FMSP Team were themselves responsible for promotion and publicity of the programme and this included: Contacting each secondary school in the pilot area, initially by letter Making contact with local authority mathematics advisors, 14-19 learning network co-ordinators and Careers Wales contacts Press releases to raise initial awareness Adaptation and translation of publicity materials from FMSP England Visits to individual schools and launch events to ensure face-to-face contact between a representative of each school / college and the Programme Leader for the FMSP. These contact sessions had a dual aim; of raising awareness of the benefits of studying further mathematics and introducing students, teachers and parents to the FMSP Convening enrichment sessions for year 11 pupils to promote take-up by raising awareness of careers opportunities and options for studying mathematics and STEM subjects at a higher level Promotion to year 12 students of the benefits of taking AS Further Mathematics in year 13, through attendance at university open days and other first year sixth form events.
5.5 In total, promotional events were attended by 2,850 students and 40 teachers across the pilot area.
Management committee awareness and understanding
5.6 Most members of the management committee claimed to have a good understanding of the FMSP, its aims and objectives for individuals involved in the short term, and its broader, more wide-ranging and longer term
49 benefits. In conversation, the majority were knowledgeable about the pilot and had a good sense of the issues affecting progress. As one committee member described: “(The aim of the FMSP) is to enable more students to take further mathematics and to support schools and colleges to provide further mathematics on their own or through a consortia arrangement … both approaches (direct delivery and capacity building) are needed ... even in England, the delivery of further mathematics is never going to be self- sufficient.”
5.7 Management committee members expressed strong support for the programme leader, and the role she plays in engaging with schools, colleges, teachers, and their pupils. From their perspective, the pilot was engaging with FE students, school pupils and their parents at every opportunity, whether at open evenings, heads of mathematics meetings, or revision events. The challenge of connecting with secondary school parents appeared to be recognised by pilot staff, however, and one commented that: “I am not sure if the project reaches parents enough ... although they have gone to some college open days … We need to think of ways to interact better with parents at open evenings.”
5.8 This was a point echoed by several members with links to teaching, given their appreciation of parental influence on children’s subject choices, and the value of their support for their child in making the decision to study FM. (In more practical terms, it was also mentioned amongst teachers interviewed, how parents could be helpful in transporting their child to and from events, assisting in accessing online activities as and when necessary and in some cases, paying for support).
Wider stakeholder awareness and understanding
5.9 Generally, stakeholders were clear in their understanding of the value of further mathematics and the role of the FMSP, for example: “It is to make further mathematics available to the whole of Wales. In areas where it isn’t delivered as part of the mainstream curriculum, it should directly benefit students.” (FMSP tutor).
50 “It’s aiming to address the position that Wales has insufficient pupils at A Level taking further mathematics”. (Management committee member). “Its purpose is to raise awareness of the benefits of taking Further Mathematics and to provide support to schools and colleges to deliver the course” (Head of Mathematics, school not currently delivering further mathematics.) “The purpose is to encourage more people to take Further Mathematics and to help mathematics teachers by reminding them of the course context in order that they may teach the modules themselves.” (Mathematics teacher, school currently delivering further mathematics.) “It aims to develop an enjoyment of mathematics and take the fear out of mathematics. It’s also about increasing demand for mathematics at A Level and beyond.” (Mathematics tutor in FE College currently delivering further mathematics.)
5.10 Within the schools and FE context, delivery of further mathematics was seen to be an important mechanism for raising the academic status of a school, enhancing its competitive status and prestige in their local area, potentially encouraging more students to attend. The view was also expressed that schools benefited from delivering a larger range of qualifications, in helping to retain pupils in sixth form. Teaching further mathematics was also seen to benefit teaching staff as illustrated by teachers’ comments below: “It’s good for teacher morale” “It presents new challenges for us all!” “To reach a critical point in mathematics, the more you do, the more cross benefits you get” “It offers keen teachers the opportunity to inspire a love of mathematics in pupils” “If working as an FMSP tutor you have greater access to resources”
5.11 Other Stakeholders talked of the potential for greater numbers of pupils in Welsh schools and FE colleges to take further mathematics, and the impact this could have in the long term by encouraging easier access to STEM in HE: “... [further mathematics] thereby brings about better outcomes for
51 graduates in terms of progression into higher tech jobs, for example...” (Local authority mathematics advisor). “...the creation of a virtuous circle whereby more skilled mathematicians in the workforce … benefit students … who should have more job prospects at a higher level.”(Management committee member).
5.12 In the shorter term, the primary research has identified potential, amongst other benefits, for: Helping schools in the achievement of higher results Stretching higher ability students, providing opportunities to fast track, doing mathematics in the lower sixth and FM in the upper sixth Helping develop talented students and offers provision for gifted children Providing a “good grounding” for those who know they want to pursue mathematics at HE level, and “getting youngsters as high as possible” Enhancing standards of teaching thanks to smaller class sizes, more dedicated teachers, and more able and motivated pupils, and Providing a foundation for a future degree in mathematics or other STEM subjects.
5.13 Overall, therefore, teachers and other stakeholders have demonstrated widespread awareness of FMSP and its aims. Mathematics students interviewed during the evaluation generally had a good grasp of the issues around further mathematics and the potential benefits of studying the course for those considering mathematics as a future career or HE option.
Student tuition for AS/A Level Further Mathematics
5.14 Further mathematics tuition was provided in-house by some schools and FE colleges, who had the experience and resource to do so. (Some had been offering FM before the advent of the FMSP). For all schools and colleges, both within and outside the pilot area, the FMSP offered tuition for the modules they were unable to deliver themselves, however many that was.
52 This tuition was available face-to-face or online at a cost of £202 per module per student.41 Further details are provided below.
Face-to-face tuition
5.15 For face-to-face tuition, expert tutors employed on a sessional basis by the FMSP delivered a structured learning programme; online resources were used alongside as additional support. The tutor liaises with the school/college mathematics lead (typically the head of department), and tuition is provided to students at their school/college or a neighbouring institution.
5.16 Face-to-face teaching approaches were the preferred delivery method for further mathematics students, especially if held at the individual’s own school. Where it necessitated travel from one school or centre to another, it was compromised because of the time taken and costs involved, but it was still seen as “better than nothing” by students and generally preferable to online teaching and learning42. The provision of particular modules as required by the school was seen to be invaluable, especially being able to access this delivery (when and where required) through the medium of Welsh. One FMSP tutor described doing face-to-face tuition for several modules. In her experience: “Very low numbers had attended the on-line teaching sessions, which were delivered more like the lectures. The face-to- face sessions were far more interactive and therefore a better way of teaching.” (Head of Mathematics, FE College).
Live Online Tuition (LOT)
5.17 This was designed to offer additional support, especially to meet gaps in face-to-face tuition occurring as a result of distance/other impracticalities affecting the ability to tutor face-to-face. For online tuition, which supported key modules of Numerical Methods and Differential Equations, students
41 Cost of tuition was calculated as 1/3 of the base rate schools receive towards an AS or A2 qualification, £606 for 2011-12 academic year. It provides each student with 18 - 20 contact hours per module, with tutor fees comprising most, if not all of this fee. Access to integral resources was free. 42 It may be that acceptance of online teaching will increase over time as it becomes more familiar and wide-spread in schools.
53 logged in to the online classroom at set times and were able to interact with the tutor (and other logged in students). The majority of participants contacted during the evaluation recognised the role and value of online tuition, especially through real-time online tutorials. Live Online Tuition, via Blackboard Collaborate (previously known as Elluminate), potentially offers the benefits of flexible times and locations for both students and staff. The support is based around students and tutors located remotely, but communicating via keyboard, microphone and headset. The system also allows for written communication in real time using a graphics tablet, so that tutors can check students’ work whilst on line. Homework can be set by the tutor, with follow-up support via the web and students were also made aware of other online Integral Resources, comprising extensive exercises, multiple choice assessments and interactive material such as animated worked examples to support learning. The platform also allows for messaging between students, for discussion forums and customised calendars. “We went to some on-line sessions too which were really good. … They were quite slow, but this was good because it meant that they went through everything with a detailed explanation. They (pupils) listened to them recorded, generally because it was more convenient that way. They listened in to some on-line revision sessions too, which went through the whole course and were useful. They were live at 6.30 however, which they felt was perhaps a bit late.” (FE Lecturer). “These kids are often more independent anyway, so distance or online learning is very appropriate.” (Stakeholder).
54
5.18 The FMSP has successfully provided increasing numbers of tuition modules year on year, as follows:
Table 4 FMSP Student Tuition Modules Delivered43
Academic Year Modules Delivered 2010 – 2011 1 2011 – 2012 17 2012 – 2013 21 2013 – 2014 (to date) 37 Source: FMSP
5.19 For many schools in the pilot area it was unlikely that it would be possible to deliver further mathematics tuition in-house until a critical mass of approximately ten students wanting to study further mathematics had been achieved. Typical numbers at the schools visited in the pilot area were in the region of two or three students (albeit with significantly larger numbers in FE Colleges), and timetabling was not a viable option. Sharing tuition with other schools, as discussed elsewhere, was also problematic because of the distances involved and timetable difficulties.
5.20 Online tuition, while extremely beneficial for some students and schools, presented a number of barriers to take-up. First, was that online tuition had cost implications. In some schools the parents paid, whilst in others it was met from the school’s budget. However, the outcome in one school contacted was that one FMSP module was purchased from school funds for one pupil and a second had to “share” it informally, as the school would not pay for the additional materials. Second, some teachers did not fully value online tuition: “We don’t make use of online revision. It is a bit of a problem in that different teachers teach things in slightly different ways which can lead to considerable confusion and uncertainty, which then makes kids anxious”. (FE Mathematics lecturer.)
43 No targets were set for numbers of tuition models to be provided. Figures shown comprise both online and face-to-face tuition
55 “I’m not a huge fan of on-line tuition ... it’s not as good as tutoring face-to- face ... especially in such small groups.” (FE mathematics lecturer).
5.21 A third barrier to online learning was a reported lack of awareness of the availability of online tuition amongst teachers and students interviewed, and/or of what it comprised. Within individual establishments, different staff members displayed very different levels of awareness of what was available, largely citing a lack of time to devote to research for this. (It is important to note in this regard that communication from teachers was essential in encouraging student take-up, as few students would have found their way to the resources unprompted).
5.22 Fourthly, difficulties were occasionally described with access to revision sessions on-line. At two schools and one FE College included in the study, for example, teachers and lecturers described how access was only sometimes successful, whilst a student described how their username had not worked for more than one week. One teacher from a Welsh medium school described that he had thought it was possible to download the tuition (and revision) sessions as podcasts, but had not manage to do this himself and nor had any of his students when they had tried. Sometimes the IT support in schools and colleges created barriers: One Year 13 student complained that his college had obtained ten accounts for student access but that they had not been distributed to the students.
5.23 Linked to the above, another barrier for the online revision sessions was the IT confidence and competence of some teaching staff. Although many were very happy working online and making use of internet resources, a small number expressed their reluctance to work in this way, with one teacher favouring ‘old fashioned’ text books and believing that face-to-face tuition and revision was the better teaching method.
5.24 Scheduling of the online sessions was not always at the most appropriate times (although it was recognised that not all people’s needs could be met all of the time). Many felt that 4.30pm starts, for example, were too early, especially for FE students, although later start-times could then clash with meal times. Distances involved in the pilot area were often an issue, with
56 journey times between school/college and home sometimes being quite lengthy. Hence there was no clear or consistent indication as to what time was most convenient: “Some like it, some don’t ... there is only one session per module so it’s easy to miss them ... also it’s at 4.30 so a bit too early ... they aren’t all home by then.” (Mathematics teacher)
5.25 The gap between sessions, sending off work and receiving comment /feedback was sometimes felt to be too long. Students admitted to being impatient; saying that they wanted feedback (virtually) immediately so they could feel confident to move on.
Access to the FMSP (MEI) Integral Resources website
5.26 Single use teacher access to the MEI Resources Website was delivered free on registration to the FMSP both within, and outside, of the pilot area44. The resources, through the MEI, covered further mathematics specifications from a number of examining boards, including WJEC, the board most commonly used within the pilot area. It included all AS and A Level Further Pure (FP) modules (FP1-3), and most of the Applied Mathematics modules (S1-3, M1-3 of the WJEC specification).
5.27 At a cost of £240 per 10 student accounts, schools and colleges could also subscribe to online student access (to GCSE, Additional Mathematics, A Level Mathematics, and Further Mathematics resources)45. Additional student accounts could be requested (at a cost). Information on the FMSP website details the benefits of this student access: the provision of students’ record systems for teachers, students’ online tasks and progress tracking systems, as well as core A Level Mathematics resources for teachers. The website also describes the role of the FMSP Wales Programme Leader, and
44 Within the qualitative interviews, references to websites focused on the FMSP website, rather than MEI, and there was some confusion over sources/brands 45 Some teachers expressed concern about this cost, feeling it was too high and so encouraged ‘sharing’ of resources between students
57 how she will help arrange further mathematics tuition for students if their schools and colleges are unable to provide it themselves46.
5.28 Overall, reports of the use of integral resources from FMSP were quite limited but this could, in part, be because of under-reporting (or no reporting) by students of their individual use, and because of some confusion and misunderstanding over ‘source’ of the resources: MEI, or FMSP.
5.29 To those using them, the FMSP website and integral resources were felt to be of considerable value, including for some teachers who were already delivering further mathematics without the help of the FMSP. An example of the value to the latter was given by one teacher who described how he “cherry-picked” or “dipped into” certain aspects of the FMSP for further support as and when he felt the need. Another, experienced at delivering further mathematics, described how: “We’ve always timetabled it (further mathematics). What FMSP did was to add something.” (Head of Mathematics, FE College).
5.30 The key role of the resources was to provide access to past papers and exercises: a function of considerable benefit to further mathematics teachers and their students. Students, however, did not generally know the source of material given to them in the classroom, and so were possibly using material from the FMSP or MEI, without necessarily realising. Further, take-up of online resources by students at home was not necessarily reported to their teachers/lecturers, and so staff were not aware of the full extent of resource utilisation.
5.31 Continued, strengthened marketing of the FMSP website was called for since a significant number of teachers admitted to knowing very little of what was available on the website, especially in certain schools. The key barrier, in this respect, was reported to be a lack of time, but it is possible that a few were using time as an excuse for not feeling very comfortable or competent with online resources. “I don’t really know anything about online FMSP activities. I don’t really have the time to find out, or explore the options” (a school with otherwise quite a high level of knowledge of the FMSP, and of
46 One school expressed gratitude to the Area Coordinator for having received some ‘free’ help
58 teaching further mathematics, but who went on to suggest that a further meeting with the Programme Leader may be beneficial to fully explain what resources were available). “I’d like a textbook, and maybe someone I can call if I’m really stuck” (Mathematics teacher – Pembrokeshire).
5.32 Although the resources available could be of considerable value, some teachers and management committee members felt the FMSP website under-performed, and that it was in need of refreshment. Views from the management committee included the following: “It’s is pretty boring and unattractive – it needs to be jazzed up … Also there is a lack of clarity in the language used on the site which makes it not easy to understand what the project does: schools might be confused over what needs to be paid for and what is free … The quality and clarity of the website is awful ... it needs to be concise.” (Management committee member). “It needs a refreshed website with an emphasis on ‘selling’ Further Mathematics to parents … I think it looks very amateurish.” (Management committee member). “(The FMSP needs) leaflets and FAQ sections targeting parents on the FMSP website.” (Management committee member).
Mathematics Enrichment Events
5.33 The FMSP’s enrichment events were seen to be core to the FMSP, and elicited positive responses from teachers, students, and pupils. They comprised the following: Mathematics enrichment events (aimed at students in Key Stage 4 and post-16) Mathematics master classes, held at Swansea, Swansea Metropolitan, Bangor and Glamorgan Universities Mathematics careers talks at local schools and colleges, aimed at demonstrating the scope, value and potential of (further) mathematics qualifications
59 Revision days/events, mostly held at Swansea University, but also at Pembrokeshire College. These were designed to concentrate on specific modules or units, for example, FP1, M1, across various specifications.
5.34 The FMSP succeeded in delivering the enrichment courses shown below, attracting the following numbers:
Table 5 Enrichment courses delivered by FMSP Events Students Attending Teachers Attending Revision Events 810 27 Enrichment (Mathematics Conferences for 549 46 Y10, Y11,Y12) Sub Total 1359 73 Further Mathematics Enhancement Events: Study Days 45 Problem Solving Class 22 Live Online Lectures (FP1) 79 Masterclasses for Yr12 188 Total 1693 73 Source: FMSP
5.35 Approximately 130 teachers were reported to have taken part in events for students47 and workshops organised specifically for teachers. Responses to these enrichment events, in whatever form they took, were generally very positive. The following section describes these responses, from the perspective of a variety of evaluation participants.
The Mathematics Inspiration Days, Conferences, Master classes, and Mathematics Careers Talks
5.36 These enrichment events were the most talked-about element of the FMSP by both pupils and staff. The programme included careers talks for pupil in years 10 and 11, as well as GCSE revision sessions in algebra. There were also master classes for pupils in year 9, plus revision sessions for those taking Level 2 additional mathematics alongside their GCSE. Two year 9 pupils had been to a Saturday morning event, their teacher reporting that:
47 Made up of 40 teachers taking part in promotional events, 27 attending revision events, 46 in enrichment events and 17 in teacher workshops,
60 “They were very bright girls anyway but the classes had given them more confidence … they are doing more difficult work in school since going to the classes ...they have been better at problem solving ... no longer expecting us to show them the way to the answer.” (Mathematics teacher, Neath Port Talbot).
5.37 Based on pupil and teacher feedback from three institutions that accessed them, all the enrichment activities were particularly well received. Sessions in cryptology and code breaking in the inspiration days were said to capture pupils’ imaginations, as did the mathematics conferences for younger year groups. They were fun, interactive, engaging, and memorable. They also served to create a feeling of community, alongside an element of (healthy) competition between the schools attending. Further too, actually attending an event at a University was something many pupils enjoyed: it provided a ‘taster’ of what it was like to go to university as a student. “The Swansea enrichment lecture was really valuable: it helps open their eyes.” (Mathematics teacher). “Fifteen students in Year 10 went to the conference in Haverfordwest which was very good. The last speaker in particular was excellent. He was a mathematics busker and performed tricks. This session was interactive and involved getting the students up on stage. The speakers were all very different; although not all of them were relevant to Mathematics A Level, many of them relates to mechanics, for example one talked about building a car to break the 1,000 km/hour speed record. They felt this helped to bring mathematics alive and to demonstrate the relevance to future careers.” (Mathematics teacher, Pembrokeshire). “It helped us think about solving a problem in mathematics ... they took apart the problem and then showed us how to solve it.”(Pupil, Pembrokeshire).
5.38 There were some drawbacks with these events, primarily based around the logistics of transport and timing, given the geographical spread of schools within the pilot area. Holding events in Swansea deterred a significant number from attending, although some were helped by the Pembrokeshire location of certain events. Events held on Saturdays could be a barrier too,
61 with some Carmarthenshire A Level students interviewed reporting having jobs, and / or parents being unable to help with transport.
5.39 Finally, it is important that the sessions delivered through the medium of Welsh are of the same standard as those delivered through English: there was occasional comment that this was not the case, and also that students normally taught through the medium of Welsh sometimes struggled with terminology used when taught in English. One head of mathematics at a Welsh Medium school reported that he was not aware that revision sessions were available through the medium of Welsh.
5.40 Careers talks in schools were very well received because they overcame some of the difficulties of timing, costs, and logistics for pupils and staff. Generally, these were reported to be inspirational to interested students, demonstrating the wide-ranging benefits of studying further mathematics, and its application in various fields. Parental involvement was also valuable given its potential role should the pupil decide to study further mathematics.
5.41 Revision days and master classes in Pembrokeshire and Swansea were very well received by those who attended. It appeared that attendance was helped because they were free to state schools in the pilot area. Revision sessions were found helpful to staff as well as students: “None of us had taught further mathematics for years so without the revision sessions we probably wouldn’t have been confident in our own ability to deliver further mathematics.” (Mathematics teacher, Pembrokeshire).
5.42 An important barrier to attendance at the revision sessions, however, revolved around their scheduling. Several members of staff, especially in two of the FE Colleges contacted, commented that they were too close to actual exam dates; a few days in some instances, and this was felt to be a problem for a several reasons: It did not allow much time for the revision to become fully embedded There was limited time left to clarify understanding (if required). Different teachers have different styles, and it can take time to adjust to a ‘new’ teacher and the way they present information
62 The timing overlooks the fact that pupils at this stage of their education have many exams to take (including Welsh Baccalaureate, as well as GCSEs/AS/A2 Levels), and so spending one day going to and from a FMSP mathematics revision session may be detrimental to revision on other subjects: “Could you schedule revision sessions one month before exams, rather than two days please?” (FE Lecturer wanting to take students to Swansea sessions, but who felt the date was too close to the mathematics modular exams).
5.43 Evaluation participants appreciated that the timing of events was never going to please everyone or be convenient for everyone. Certainly some felt that having revision sessions towards the end of the course/module was better than nearer the beginning because it entailed running briefly over everything learnt on the course, and attendees had to have learned it first.
5.44 One further negative mentioned of a revision session related to content and quality. One FE College head of mathematics talked of “several mistakes in the delivery of the event … and the feedback was very poor”. A stakeholder too described hearing some criticism from teaching staff of the sessions delivered by the project in the early days, in terms of the quality of delivery, and even the relevance of some of concepts discussed; for example, “one session looked at several aspects of mathematics that weren’t on the syllabus”. (Stakeholder).
5.45 One stakeholder again raised the issue of barriers for pupils studying mathematics through the medium of Welsh: “Language is an issue, as one of the Welsh medium schools in the county is sending pupils to Swansea, who consequently receive further mathematics tuition in English, rather than Welsh”.
Support for CPD
5.46 Initially, the provision of CPD was not a component of the FMSP in Wales and could only be accessed in England. One argument in favour of CPD was given by a Stakeholder who expressed the view that schools “need to become more self-reliant if more and more pupils want to take further mathematics”.
63 “We shouldn’t have to go all the way to London to access CPD” (Head of Mathematics, Swansea).
5.47 However, issues in relation to teacher confidence and competence were raised in particular by teachers themselves (secondary and FE), especially heads of mathematics, and also stakeholders and management committee members. “The complexity of the subject would be a deterrent to some schools ... you don’t always get mathematics specialists teaching in mathematics departments.” (Welsh Government stakeholder).
5.48 Qualification levels of teachers and associated issues had certain ramifications; for example one management committee member described how only “63 per cent of mathematics teachers have a mathematics degree”, making the point that competencies were thinly spread and there may only be one or two teachers capable of delivering mathematics to further mathematics level in a school. Further, as one teacher reported, some teachers had received their degrees “yonks ago” (sic), and were concerned that their abilities were not sufficiently up to date. A stakeholder echoed this point: “I imagine that there are teachers who are used to teaching mathematics, but less confident about further mathematics.” (Welsh Government stakeholder).
5.49 The demand for CPD was particularly based on a lack of confidence in teaching certain modules. Although many teachers had impressive credentials, some were relatively new to further mathematics teaching, and they were keen to adopt the terminology and skills required in order to be effective. Some had received help via the FMSP, especially those who became tutors for the programme48, but not all. Further, one school stated that no further mathematics textbook was available from the WJEC: “I’d like to have someone I could talk to on the phone, who I could call if I was stuck with something, like an on-line mentor” (Mathematics teacher, Pembrokeshire). “I’d like ‘teach the teacher’ days” (Mathematics teacher).
48 One teacher in particular spoke of this support and training which he deemed excellent. As a result, he had felt highly confident in delivering further mathematics to a number pupils within his school and in tutoring on-line
64 “You have to hunt around in books to find what you need” (Mathematics teacher, Pembrokeshire).
5.50 Further, turnover in teaching staff could pose problems, if, for example, a teacher of further mathematics left a school to be replaced by one with less, or no further mathematics experience. For other schools, the ability to ‘brush up’/refresh their skills was something they sought by way of reassurance. One head of mathematics explained how two teachers from his school went to three half-day (9.30-1.30) FP1 revision sessions in Swansea last year (2012) as they hoped to familiarise themselves with the module in order to be able to deliver it in-house the following year. Nonetheless, they would still only feel competent and confident to teach FP1 (which, with additional statistics and/or mechanics modules, would enable students to get an AS in further mathematics). They were currently delivering the four “core” modules (C1-4), mechanics 1 and 2 (M1 and M2) and statistics 1 and 2 (S1 and S2): “If we had a little bit of help from FMSP to brush up our skills we could do the full A Level in further mathematics here.” (Head of Mathematics, Carmarthenshire). “None of us had taught further mathematics for years so without the revision sessions we probably wouldn’t have been confident in our own ability to deliver.”
5.51 The call for teacher CPD within the pilot area led to the start of Live Online Professional Development which, from October 2013, led to a busy programme comprising online classroom environments for a range of aspects of further mathematics. Courses on offer for October and November 2013 are shown below, with more than one course of each of those listed generally being offered49: STEP support Further Pure 1 Statistics 1 and 2 Decision 1 and 2
49 At the time of writing, most of the October courses had been fully booked, demonstrating teacher demand
65 A2 Core: Topics from C3, C4 Mechanics 1 and 3 Topics from FP2, 3, 4 Use of GeoGebra in A Level Mathematics
5.52 One barrier to accessing support was that with such small numbers of further mathematics qualified teachers it was also difficult to release teachers from school for training, because of the cost/logistics of providing cover. Hence some staff members might have to make their own arrangements for CPD training out of school hours; especially those working in small schools.
5.53 A cautionary issue raised by one stakeholder was that teachers who are competent to teach further mathematics may be more expensive, and so there may be more reason to make them redundant at a time of budget pressures. In some schools an external tutor (for example a retired teacher) was employed to deliver further mathematics, despite the cost implications for this to the school.
5.54 There was a view expressed by one interviewee that schools in Wales do well with regard to retaining qualified teachers, because of a lack of competition or choice: “Many new teachers struggle with mathematics – never mind further mathematics although this is less of an issue in Wales than in England, given the lack of other jobs in Wales and the fact that national pay scales make these posts more attractive” (University lecturer).
5.55 An important consideration here, and in the provision of further mathematics in general, was that the successful delivery of further mathematics within a school depended, at least in part, on having “good, enthusiastic Heads of Mathematics and Local Authority numeracy advisors” in order to gain maximum support for CPD activities.
66 6 Outcomes and Impacts
6.1 The specification for the FMSP identified six outcomes to be achieved by the end of the pilot. These were all comparative targets, relating to increases in provision and participation, rather than hard numeric targets against a baseline position.
6.2 The outcomes were: Outcome 1: Increased numbers of students in the pilot area studying further mathematics at GCE A/AS levels, over the life of the pilot Outcome 2: More schools and colleges in Wales offering further mathematics, either individually, or via consortia Outcome 3: Increased numbers of mathematics teachers in Wales who are trained to teach further mathematics Outcome 4: Overall raised awareness among students and their parents of the importance of studying mathematics at higher levels Outcome 5: Increased numbers of students from Wales applying to study higher education courses in mathematics and related subjects, such as engineering and physics Outcome 6: Improved transition of students from further to higher education courses in mathematics, or from courses which have a significant element of mathematics, thus benefiting the wider economy.
6.3 In this section of the report, we examine the progress made against each outcome, within the constraints of available data and draw some conclusions about the overall efficacy of the pilot. This analysis will be extended in the final report, due for publication in summer 2014. The final report will utilise: additional FMSP data for participation in further mathematics, all 2013 attainment data for AS/A2 examinations, HESA data for Welsh-domiciled, first year undergraduate students on STEM related courses in 2013-14 The report will also aim to track A Level students in respect of their GCSE history to develop an understanding of the impact of the FMSP pilot on widening access to students other than the most able, and to look at impacts of the pilot in terms of female students.
67 Outcome 1: Increased numbers of students in the pilot area studying further mathematics at GCE AS/A2 levels, over the life of the pilot.
6.4 There is a stark disparity in data between the numbers of students recorded by FMSP as studying for AS / A2 examinations in further mathematics and the data returns from the LLWR and WED for the number of examination entries and attainments. Whilst no clear explanation is available, it is likely that some students’ AS Levels are not counted in their own right, but they are combined with A2 results as part of their A Level qualifications, which are then recorded in entry and attainment data returns. At the same time, some students are likely to embark on an AS or A2 further mathematics course in order to boost their grades in mathematics (where there are common modules across the two subjects, such as M2 or S2) and others will not complete the course due to workload.
6.5 Data from FMSP shows that there were 60 students studying A2 Further Mathematics and 129 students studying AS in Years 12 or 13 in state- funded sixth forms or FE in the pilot area in 2010-11. By 2011-12, this had increased to 66 A2 Level and 164 AS Level students studying further mathematics in the region.
Table 6 Students Studying Further Mathematics: AS/A2 Levels, 2010-12
Year AS A2 Total 2010-11 129 60 189 2011-12 164 66 230 Source: FMSP Registration Data
6.6 Whilst there is no hard data from DfES on the total number of students studying further mathematics, a worthwhile proxy is the number of examination entries at AS/A2 Levels, taken from the LLWR for FE students and WED for school sixth form pupils. Table 7 below shows the change in entries over time in the pilot area, demonstrating an increase in A2 entries from 17 in 2007-8 to 87 in 2011-12, change in AS Level entries from 17 in 2007-8 to 34 in 2011-12, and changes in combined AS/A2 entries from 34 in 2007-8 to 121 in 2011-12, albeit with large fluctuations in numbers between these dates. However, these 121 entries were in the context of 230 students said to be studying within the pilot area.
68 Table 7 Further Mathematics Entries at AS/A2 Levels: Pilot Area FE and Schools 2008-2013
Examination Years 2008 2009 2010 2011 2012 2013* A2 Level FE 10 30 7 87 61 - Schools50 7 24 14 18 26 26 Total 17 54 21 105 87 - AS FE 10 31 47 28 27 - Level Schools 0 1 3 3 7 7 Total 17 32 50 31 34 - Combined Total 34 86 71 136 121 - AS/A2 Source: LLWR / WED. *Denotes partial data, as 2013 information for FE will not become available until April 2014.
6.7 Even allowing for AS registrations converting to A2 Levels, total FMSP registrations for 2010-11 were 189, whilst examination entries taken from LLWR/WED came to 136 in that year. This could represent an attrition rate of some 28 per cent over the year across all students, but there is no firm data to support this. In the evaluation fieldwork, stakeholders were asked for their views on drop out, but there was no consistent view expressed. It was pointed out that some students would drop out after completing the AS in order to take another subject to AS level, to “add another string to their bow”. Stakeholders were generally of the view that drop-out rates arising from the difficulty of the subject should not be especially high, since only the more able students would undertake further mathematics anyway. However, one interviewee explained that numbers for both entry and completion of further mathematics are increasing and that although he had never noted a comparatively high drop-out rate in further mathematics, there was a chance that the broadening of entry might bring in those less able to keep up with the course and so attrition might increase as a result.
6.8 One approach to validating LLWR/WED data is to compare all-Wales data from these sources with returns from JCQ. These datasets are not directly comparable, as JCQ data includes returns for learners of all ages (whereas
50 Excludes independent schools
69 LLWR/WED data is shown for 16-17 year olds) and all institutions (LLWR/WED data shown excludes independent schools). However, there are some stark differences in the two sets of results, most notably the increased numbers of AS Level qualifications achieved. There is no obvious explanation for the varying degree of difference between the two sets of data on a year to year basis. The comparisons are shown in Table 8 below.
Table 8 Further Mathematics Entries at AS/A2 Level, Wales: LLWR/WED and JCQ Data
Numbers and percentages Examination Years 2008 2009 2010 2011 2012 A2 Level LLWR/WED 96 204 142 278 290 JCQ 195 250 240 309 380 Percentage 103 23 69 11 31 difference AS LLWR/WED 41 100 116 89 107 Level JCQ 192 246 275 289 339 Percentage 368 146 137 224 216 difference Source: LLWR / WED / JCQ
6.9 The data for A Level entries would appear to show a step change in numbers in 2011, when the first students receiving support for a one year A Level qualification will have featured in the data. However, the change in numbers of AS level entries was less clear, with the number of entries falling between 2010 and 2011.
6.10 Another approach to looking at this is to take further mathematics examination entries as a proportion of mathematics entries and to explore changes over time.
70 Figure 7 Further Mathematics Entries as a Proportion of Mathematics Entries: A2 and AS Levels: Pilot Area vs. Rest of Wales 2008-2012 (Examination Years)
Source: LLWR / WED/ Miller Research
6.11 As before, this shows a step change in the proportion of A Level Further Mathematics entries in 2011 in the pilot area and a consistently higher proportion of further mathematics entries at AS Level than in the rest of Wales from 2009 onwards (i.e. before the pilot started).
6.12 Given that the FMSP has engaged with a substantial number of schools outside the pilot area and that this may have impacted on engagement levels, it may be more illustrative to compare changes with examination entries for Northern Ireland, where no support for further mathematics is in place and for England, where long term support has been in place.
71 Figure 8 Further Mathematics: A2 Entries: Pilot Area vs. Wales, Northern Ireland and England 2008-2012 (Examination Years. Indexed to 2008)
Source: JCQ / Miller Research. Figures in brackets are entries for 2008.
6.13 This shows that whilst levels of engagement in further mathematics remained relatively constant in Northern Ireland over the period and England experienced a gradual rise in numbers, there was a substantial increase in engagement in the pilot area in 2011 and 2012, and a smaller rise in engagement in the rest of Wales. The rate of A2 Further Mathematics entries as a proportion of mathematics entries in Wales was 3 per cent in 2008 and 12 per cent in 2012, whereas in England it was 15 per cent in 2008 and 16 per cent in 2012. In Northern Ireland the proportion fell slightly over the same period from 6 per cent to 5 per cent. However, the increase in Wales should be seen in the context of starting from a low baseline, in that by 2012, there were approximately 7.2 entries per 1,000 population (of 18 year olds) in Wales, against a corresponding figure of 18.9 in England.
6.14 It should be hoped that the FMSP will help to increase the quality of tuition and support, as well as the numbers of students engaged. Hence the extent to which students achieve a qualification on completion of their course is also important. Data for achievements of further mathematics A*-E grades as a proportion of A*-E grades in mathematics are shown below for the pilot area and rest of Wales. There are indicative signs of an increase in
72 achievements at A Level in 2011 and 2012 in both the pilot area and, to a lesser extent rest of Wales. There is no clear message as yet in terms of AS Level data.
Table 9 Further Mathematics Attainments at A*-E as a Proportion of Mathematics Achievements; 2008-2012
Per Cent 2008 2009 2010 2011 2012 A Level Pilot Area 3 8 3 12 12 Rest of Wales 4 7 5 8 9 Total 7 15 8 20 21 AS Level Pilot Area 1 3 4 2 4 Rest of Wales 1 3 2 2 3 Total 2 6 6 4 7 Source: LLWR / WED / Miller Research
6.15 Another variable to consider is the proportion of A* and A grades achieved in Further Mathematics. Table 10 below shows the number of A Level achievements at A*-A grades51 in the pilot area between 2008 and 2012, along with this number as a proportion of A*-A results in the rest of Wales. In both cases, there appears to be a step change in 2011, which is broadly sustained in 2012. Students in the pilot area accounted for 23 per cent of the A*-A grades in Wales in Further Mathematics in 2008, rising to 31 per cent in 2011, before falling back slightly in 2012. A more robust measure will be to see whether this level persists over the medium term.
Table 10 Further Mathematics Achievements at A*A: Pilot Area vs. Rest of Wales, 2008-2012
Numbers and percentages 2008 2009 2010 2011 2012 Pilot Area Achievements 12 26 11 40 44 Rest of Wales A Level Achievements 41 79 69 89 115 Pilot Area / Rest of Wales, % 29 33 16 45 38 Pilot Area / Wales Total, % 23 25 14 31 28 Source: LLWR / WED / Miller Research
51 A* was only introduced as a grade option in 2010.
73 6.16 The data can also be analysed by the proportion of A* and A grades achieved in further mathematics as a percentage of A* and A grades achieved in mathematics. This can potentially control for the presence of a more able cohort of students in a given year and validate data which demonstrates an increase in the proportion of A* and A grades. As is shown in Table 11 below, the rate of A-A* in further mathematics has grown more strongly in the pilot area than in the rest of Wales, in relation to A-A* mathematics grades as well.
Table 11 Further Mathematics Achievements at A*-A as a Proportion of Mathematics Achievements; 2008-2012
Per Cent 2008 2009 2010 2011 2012 A Level Pilot Area 5 9 4 16 15 Rest of Wales 5 9 8 10 13 Source: LLWR / WED / Miller Research
6.17 The data post-2011 appears to validate a proportional increase in achievement of top grades at A Level. However, there may be signs of convergence with the rest of Wales on this measure. This raises a possibility for further exploration, which is the “democratisation” of further mathematics mentioned by stakeholders, that may bring in less able pupils along with the core audience that have traditionally been interested in further mathematics. Once more data is available, it will be possible to track individual further mathematics students by their GCSE mathematics grades to identify whether the above is the case and to what extent.
6.18 Finally in this section, there is the question of equality, especially by gender. The recent evaluation update for the FMSP in England made the point that twice as many male as female students were engaged in further mathematics. In Wales, the gap is larger, with female students accounting for only 27 per cent of A Level Further Mathematics entries in 2012. In the pilot area, the relative proportion of female further mathematics A2 entries has fallen as overall numbers have risen in association with interventions from the pilot, accounting for 23 per cent of all entries in 2012 (although this was a significant fall on the previous year and may not represent a trend). This is also the case when the proportion of female students is examined in
74 terms of the percentage of further mathematics entries as a proportion of mathematics entries, as female participation in Mathematics A2 examinations has remained a relatively constant proportion of the total in both the pilot area and Wales as a whole52. The data shows that the gender imbalance in further mathematics has grown to a larger extent in the pilot area than in the rest of Wales, between 2008 and 2012, suggesting that the pilot has been relatively successful in engaging male students, but may need to focus some attention on how to attract more female students into the subject. Across Wales as a whole, there was a sharp increase in the proportion of female A2 Further Mathematics entries in 2013, although there is no data for the pilot area at the time of writing. This increase follows a fall in the proportion of female entries in 2012, illustrating the volatility of the data over a short period.
Figure 9 Further Mathematics and Mathematics A2 Examinations: Female Entries as a Proportion of Total for Pilot Area and Wales
Source: LLWR / WED / JCQ / Miller Research. Note datasets are not strictly comparable. 2013 data not yet available for pilot area.
52 In the pilot area, female entries accounted for between 40 and 43 per cent of all mathematics A Level entries between 2009 and 2012, whilst in Wales as a whole, the range was between 40 and 42 per cent.
75
Outcome 2: More schools and colleges in Wales offering further mathematics, either individually, or via consortia.
Schools and colleges engaged
6.19 At the outset, it was identified that there were 27 secondary schools with sixth forms in the pilot area, of which 13 were English medium maintained schools. 11 were Welsh medium maintained schools and three were independent. There were five further education colleges to begin with, but the merger of Gorseinon and Swansea Colleges in 2010 to form Gower College reduced this to four.
6.20 In 2012, the number of school sixth forms was reduced to 25, following the merger of four sixth forms into two.
6.21 The FMSP secured registrations from all maintained sixth form centres (in schools and colleges) in the pilot area by 2013. Registrations were also secured from the majority of 11-16 schools, and half of the independent schools in the region.
Table 12 Coverage of FMSP in the Pilot Area, July 2013
School Categories Schools/colleges in the Pilot Area Carmarthen Neath Port Pembroke- Swansea Total shire Talbot shire 11-16 schools 5 7 0 8 20 11-18 schools 8 2 8 7 25 FE Colleges 1 1 1 1 4
area Independent 2 0 0 1 3 schools
Total schools in the the in schools Total Total 16 10 9 17 52
14 11-16 schools 2 5 0 7
25 11-18 school 8 2 8 7
FE Colleges 1 1 1 1 4 Independent 2 0 0 0 2 Schools
FMSP Wales Registered Wales FMSP Total 13 8 9 12 45 Source: FMSP
76 6.22 In addition to schools and colleges in the original pilot area, FMSP secured a further 24 registrations from schools in the extended pilot area, incorporating Anglesey, Gwynedd, Conwy and RCT. There were also 65 registrations from sixth form centres in other parts of Wales.
Table 13 FMSP Registrations Outside the Pilot Area, July 2013
Local Authority School FMSP Wales Registrations Blaenau Gwent 0 Bridgend 0 Caerphilly 6 Cardiff 11 Ceredigion 3 Denbighshire 5 Flintshire 5 Merthyr Tydfil 1 Monmouthshire 4 Newport 3 Powys 9 The Vale of Glamorgan 5 Torfaen 1 Wrexham 1 Total 65 Source: FMSP
6.23 Table 14 shows the change in both the number of sixth form centres in schools and colleges in the pilot area between 2010 and 2103, and the growth in the number and proportion of these delivering further mathematics. In 2010, there was provision in 21 out of 32 centres, whilst in 2013 the proportion had changed to 26 out of 29 centres.
Table 14 Number of Schools with Further Mathematics Students in the Pilot Area in 2010 - 2013
2010 2013 No. of school sixth forms in the pilot area 27 25 No. of FE colleges in the pilot area 5 4 No. of schools with FM students 16 22 No. of colleges with FM students 5 4 Proportion of centres with FM students % 66 90 Source: FMSP
77 6.24 Table 15 shows the change in the proportion of schools and colleges delivering further mathematics in a classroom setting (either timetabled or non-timetabled), as opposed to individual or small numbers of students taking supervised or unsupervised modules, It can be seen that the share of centres offering further mathematics classes increased from 16 out of 21 in 2010 to 24 out of 26 in 2013; that is both the number of centres offering provision increased, but the quality of setting also increased over the pilot period.
Table 15 Further Mathematics Provision in the Pilot Area in 2010 - 2013
Oct Feb Oct Feb Nov Feb 2010 2011 2011 2012 2012 2013 Schools / Colleges with groups of students in a 16 15 19 21 23 24 class (timetabled/ un-timetabled)
Schools / Colleges with 1 supervised student 5 6 3 4 3 2 per module or up to 2 unsupervised students Total 21 21 22 25 26 26
Source: FMSP
Outcome 3: Increased Numbers of Mathematics Teachers in Wales who are Trained to Teach Further Mathematics.
6.25 Data are not available to show numbers of teachers in Wales who are qualified to teach further mathematics and so this outcome could not be assessed effectively in the course of the evaluation.
6.26 The only CPD for teachers that was originally included in the FMSP were online resources provided through the English website, as other organisations, such as Careers Wales, were understood to have been delivering CPD at the time and there was a desire to avoid double funding or duplication of activities. However, an increase in capacity for teaching further mathematics is one of the targets for the FMSP. Through the FMSP website, teachers are able to access Live Online Professional Development (LOPD) courses, covering a range of topics relating to all modules of further mathematics A / AS Level courses. There is no clear data to measure the extent of engagement with these materials. However, as a measure of the demand for CPD, it should be noted that according to the FMSP,
78 approximately 130 teachers took part in various events aimed at students and 17 teachers also took advantage of tailored workshops aimed at capacity building, including content such as Introduction to integral resources; Past papers questions in further mathematics modules; Enrichment in Key Stage 4, and: Post-16 to encourage students into study further mathematics at AS/A Level. In interviews with teachers, several commented that they found these sessions very useful, as refresher courses or in order to pick up tips on how to support students more effectively. In addition, seventeen teachers were reported to have taken part in various workshops organised for specifically for them (this included Introduction into Integral Resources, Past Papers Questions in Misconceptions in Further Mathematics modules, and Enrichment in KS4 and post-16 to encourage students to study Further Mathematics).
6.27 In terms of wider levels of qualification to teach mathematics (although not necessarily further mathematics), the proportion of all secondary teachers trained in mathematics and registered with GTCW increased from 1,204 in 2009 to 1412 in March 2013, accounting for 9.9 per cent of all teachers by that date53 and second only to the number of English teachers (10.2 per cent of the workforce). The data shows that 73.8 per cent of those teaching mathematics at secondary level were known to be trained in the subject; the highest for any subject area. For comparison, 30.1 per cent of science teachers were in this category. Further, the number of newly qualified mathematics teachers registered with GTCW grew from 70 in 2009 to 84 in 2013, accounting for 11.8 per cent of the total at that time; the largest proportion of any subject specialism.
53 http://www.gtcw.org.uk/gtcw/images/stories/downloads/Annual%20Statistics%20Digest/Annual_St ats_13_E.pdf. No. of teachers registered by GTCW by ITET Subject trained (secondary only).
79
Outcome 4: Overall raised awareness among students and their parents of the importance of studying mathematics at higher levels.
Attitudes towards Further Mathematics
6.28 Across all those participating in this research, there was agreement that the study of further mathematics was beneficial, albeit at different levels.
Figure 10 Students and other Stakeholders: Understanding of the Benefits of Further Mathematics
6.29 Students and other influencers demonstrated strong appreciation of the wider benefits of the study of further mathematics, both from an individual perspective and from a wider viewpoint.
The Individual Perspective
6.30 Individual students were asked about their motivations for taking further mathematics at AS/A Level in both the qualitative, focus group research, and the online survey of first year undergraduate students. In each case there was an awareness of the benefits of studying mathematics in HE, although the main rationale for students engaging in further mathematics was primarily out of a personal interest in the subject, complemented by an appreciation of the benefit in terms of university entry requirements and building future career options.
80 6.31 “Further mathematics has that extra edge to it for pupils who don’t feel sufficiently stretched by straight mathematics ... it’s not process driven mathematics ….. it is more interesting.” (Former mathematics student).
6.32 Almost 30 per cent of surveyed students recognised the potential for further mathematics to assist in the transition from AS/A level to undergraduate study.
Figure 11 Main Reasons for Choosing to Study AS / A level Further Mathematics (Unprompted)
Source: Online Survey of Students: Base=93. Multiple responses were permitted.
6.33 Almost half of those responding said that they had been recommended to take further mathematics by a member of school/college staff or a careers advisor, highlighting the importance of staff brokerage for the subject. Family and friends were also important influencers, mentioned by 15 per cent of respondents.
6.34 School students identified the advantages of further mathematics as: Offering an increased chance of getting admitted into (a ‘good’) University (or, perhaps, lowered grade boundaries for acceptance) Providing personal satisfaction through enjoyment of mathematics, gaining a deeper knowledge of mathematics and its application Sense of achievement from having succeeded in passing further mathematics at AS/A Level, (especially if not time-tabled at school)
81 Helping to boost their grades in Mathematics, and To help them stand out ‘from the crowd’, and give them a head-start on reaching University.
6.35 This view was reinforced by a mathematics teacher interviewed: “With pupils getting such brilliant results at A Level it [further mathematics] is an extra something over the other pupils ... most do it as an additional subject on top of the other three”
6.36 One interviewee was of the view that because further mathematics is not a subject that is widely taken up in Wales, it could give students an edge in terms of entry into higher education and employment: “It looks good to employers”.
6.37 Younger pupils who had attended master classes, for example, were more likely to simply display a youthful enthusiasm for the subject being brought alive through demonstrations of mathematics in action: “It’s cool!” (Year 10 pupil).
6.38 From a teacher’s perspective, one of the incidental benefits of further mathematics is developing more mature and confident pupils, because of the determination and self-motivation required. It can also foster lateral thinking skills, creativity and imagination; “(Further mathematics) encourages children to do their best, to stretch themselves.” (Teacher).
6.39 Another stakeholder talked of the potential financial rewards for learners, and, linked to this, the Welsh economy as a whole: “Students taking mathematics (and further mathematics) are more likely to make more money in business or finance, but also in engineering or technological sectors: further mathematics opens doors for learners, and, it is also beneficial to the economy.” (HE mathematics lecturer).
The Wider Perspective
6.40 In terms of the longer-term, macro benefits, it was recognised that greater participation in further mathematics would help create a high quality workforce in Wales, potentially making the country more attractive to high value industry and becoming more competitive internationally;
82 “Getting more highly skilled mathematicians into the workforce will be invaluable in terms of keeping Wales competitive: we need to keep raising the bar.” (HE mathematics lecturer). “Current pupils are the future mathematicians … assuming they stay in Wales … who will help to build up local businesses and contribute to the Welsh economy.” (Head of Mathematics, Pembrokeshire secondary school). The general view was that further mathematics should benefit employers in the long term by contributing to, and building on a skilled workforce. One added the view that further mathematics is “about problem solving and developing the analytical skills that employers are looking for … further mathematics has insight into parts of the brain that PISA tests, for example.” “It will help Wales to build a technically skilled workforce.” (School head of mathematics). “It will help prevent Wales from lagging behind in the global stakes.” (FE lecturer).
Outcome 5: Increased numbers of students from Wales applying to study higher education courses in mathematics and related subjects, such as engineering and physics.
6.41 Data is available from the Higher Education Statistics Agency (HESA) to show the number of first degree students from the pilot region and from Wales as a whole engaged on mathematical sciences or STEM courses54 at UK Higher Education Institutions (HEIs). The number of students on mathematical science courses from the pilot area grew from 155 in 2007/8 to 240 in 2011-12, with some slight growth in STEM course students. The growth in pilot area mathematics students, of 55 per cent, outstripped that of Wales as a whole over the period, for which the figure was 28 per cent, although it can be seen that the increased rate of growth began before the pilot commenced. Future data will be necessary to show that the trend of increased growth was sustained, or indeed sustainable.
54 This data takes a narrow definition of STEM, including: Physical sciences, mathematical sciences, computer science, engineering & technology.
83
Figure 12 First Degree Students Domiciled in SW Wales / Wales on Mathematical Sciences / STEM Courses in UK HEIs, 2007-2012
Source: HESA
6.42 The growth in the number of first degree mathematics students from Wales as a whole will reflect the contribution of the SW Region, as well as any increase in student numbers from the rest of Wales.
6.43 According to UCAS Data55 there is significant variation by country in the proportion of 18 year olds applying to university across all subjects. In 2013, 48.3 per cent of 18 year olds from Northern Ireland applied (a 1.6 percentage point increase on 2012), whilst the figures for England, Scotland and Wales were 35.1 per cent, 31.6 per cent and 29.9 per cent respectively. Wales was the only country to experience a general decline in applications in 2013, with applications falling by 0.2 percentage points. This is in spite of the relatively advantageous position of Welsh students in relation to tuition fees, compared with their counterparts from England. Wales and Scotland have jointly experienced the slowest growth in application rates of any parts
55 Demand for full-time undergraduate higher education (2013 cycle, March deadline): UCAS Analysis and Research, July 2013.
84 of the UK, with applications at age 18 increasing by 13 per cent over the period 2004-2013, compared with an increase of 38 per cent in NW England and 32 per cent in London.
6.44 Looking at subject groups, however, mathematical and computational sciences were one of the subject areas to experience an increase in undergraduate applications from 18 year olds across the UK in the period 2010-2013. Of all the groups experiencing an increase in applications over the period, all but two could be considered STEM subjects. Subjects experiencing a decline in applications over the same period were led by non-European languages (-37 per cent), architecture, building and planning (-22 per cent) and combined arts (-17 per cent).
Table 16 Proportional Changes in UK 18 year olds Application Rates by Subject Group (2010 to 2013): Subjects becoming More Popular
Subject Group Change 2010-2013 % Group F: Physical sciences 14.9 Group D: Vet. science, agriculture and related 12.8 Group H; Engineering 8.6 Group B: Subjects allied to medicine 7.6 Group L: Social studies 6.1 Group Y: Combined sciences 5.7 Group G: Mathematical and computational sciences 5.4 Group C: Biological sciences 1.7 Group M: Law 0.8 Group A Medicine and dentistry 0.6 Source: UCAS, 2013.
6.45 Linked to this outcome is the issue of how universities themselves perceive a further mathematics qualification in terms of entry requirements. Across those interviewed, there was considerable variation, from those who saw it as essential, to others who wanted to ensure widening participation and who felt that it was somewhat elitist. Background desk research, in conjunction with telephone interviews, indicates that few UK universities explicitly require further mathematics for studying a mathematics degree, but it is quite clear that it is recommended, if not expected, especially amongst the ‘top ten’ universities for mathematics undergraduate study.
85 6.46 For example, University College, London (UCL) is a ‘top tier’ university where further mathematics is a requirement for Mathematics BSc, and has been for approximately five years (starting just before the A* grades were introduced). Initially they made exceptions for applicants from Wales where no support programme was available, but this is no longer the case.
6.47 Desk research into ‘Top Ten Universities for Mathematics Degrees’56 revealed the following requirements and/or offers for 2013.
Table 17 Entry Requirements of “Top Ten” Universities for Mathematics Degrees
University Course Requirements for Mathematics Degrees
Cambridge A*(Mathematics), A, A, STEP; AS Further Mathematics essential, plus CU test Oxford A* (Mathematics), A* (Further Mathematics, if taken), A*, plus OU test Imperial College London A*(Mathematics), A*(Further Mathematics), A, plus ICL test Warwick A*(Mathematics), A*(Further Mathematics), A, STEP Level 2 Durham A* (Mathematics), A* (Further Mathematics), A (adjusted if no Further Mathematics available) London School of Economics A*(Mathematics), A, A (Mathematics and Economics BSc) Bath A*, A, A (if no AS in Further Mathematics, then STEP/AEM Level 2) St Andrews A, A, A (No subjects specified) Bristol A (Mathematics), A (Further Mathematics), A, plus Level 2 STEP, OR A (Mathematics,), A (Further Mathematics), B, plus Level 2 STEP Lancaster A, A, A (No subjects specified) 6.48 At some Universities, members of staff (and students) described the practice of reducing grade boundaries if a student applied with further mathematics. At one university, for example, further mathematics was ‘strongly encouraged’, but not a condition of acceptance. ‘The standard grade requirement is A*AA, but if a student has further mathematics the
56 www.thecompleteuniversityguide.co.uk
86 offer is AAA’. The reason for not insisting on further mathematics was given as “access … we want to make it open to all”.
6.49 At another university, the view was expressed that “Although someone with further mathematics would probably be selected over someone else of exactly the same calibre (but without further mathematics), it is one of the last things to be taken into consideration. This is because we recognise that it would not be fair to penalise students on the grounds that not all students have access to further mathematics and there can be an element of risk in taking further mathematics (particularly for those who are taking it out-of- timetable), which can put students off. For example, students would not want to take further mathematics as a fourth subject if it jeopardised their grades in other subjects by proving to be too much work”.
6.50 Outside pure mathematics degrees, and/or outside of Russell Group universities, further mathematics was mentioned with less frequency, but several university websites suggest that it could be of significant value given the competitiveness of the market place.
6.51 In discussion with HE lecturers, it was occasionally mentioned that further mathematics could be a disadvantage in some circumstances, especially for degrees other than mathematical sciences. This was felt to be the case if it meant, for example, two of a student’s three A Level choices comprised mathematics.
6.52 Examples included: An admissions tutor for medicine, who said that they would rather have students with a broader range of subjects on their CV, possibly containing an arts subject, or drama, than having science/mathematics subjects exclusively. A lecturer in physics and astronomy (at a Russell Group university) responded that they would “prefer students to take a third A Level (in addition to mathematics and physics) that they enjoy, rather than doing further mathematics because they felt that that was what the university wanted.“ At one of the non-Russell Group universities, the university’s own foundation course in mathematics for BEng and BSc Engineering
87 applicants was felt to be more important and relevant than having further mathematics. In relation to the study of economics, a lecturer from a Welsh Russell Group university described preferring another A Level to further mathematics (assuming they had ‘standard’ A Level Mathematics), and/or evidence that the student had gained experience in economics in one way or another. This lecturer’s concerns were less to do with ability or inability to do mathematics, “because they could fill in the gaps anyway”, but more to do with a lack of intuitive understanding of economics, and a lack of curiosity and real interest in the subject. From his point of view, “There has been a gradual deification of mathematics, when it should be seen as a tool.”
6.53 Overall, therefore, discussions about admissions policies suggested that a further mathematics qualification can help identify the more able mathematicians amongst student applicants, (particularly valuable if competition amongst students intensifies), but that outside of a stand-alone mathematics degree, it is important that students develop other skills too, such as the ability to write well, which may be an important component of third year projects. This, in turn, could be taken to suggest that to develop as ‘well-rounded’ mathematicians, students ideally need to be taking either four A Levels, or, as practised by some students, only taking further mathematics to AS Level. Some university staff corroborated this view: “Further mathematics should be taken as a fourth and not a third subject. It’s only disadvantageous if it’s taken with only mathematics and another mathematical course.”
Outcome 6: Improved transition of students from further to higher education courses in mathematics, or from courses which have a significant element of mathematics, thus benefiting the wider economy.
6.54 There was a general consensus that taking further mathematics at AS/A Level eased the path of transition for students going into mathematics and STEM courses. This was especially true in the case of those universities which specified further mathematics as an entry requirement, or as recommended for the course: “We now have 200 students with an A* in
88 further mathematics ... the quality of the students is significantly better (than it was before further mathematics became mandatory) and we are now looking to make our first year harder ... the improvement correlated with when we insisted on further mathematics … we have a higher level of assumption now as to our students’ ability.” (University lecturer).
6.55 (Note that the Lecturer (from an English ‘top tier’ university) quoted above also commented on noticing an increase in the number of students from Wales, particularly Swansea and Cardiff): “The old A Levels were not preparing students for a mathematics degree. Now we have further mathematics and STEP.” (University lecturer). “I wished I’d had the full Further Mathematics A Level rather than just AS. Those with further mathematics had an easier time of it.” (Mathematics undergraduate at an English university). However, there were very few students or HE lecturers who felt that the difference between those with or without further mathematics persisted beyond the first year of undergraduate study: “On my course, for the first 6 to 12 months you could tell the difference between those who had and those who hadn’t got further mathematics” (Mathematics undergraduate at a Welsh university). “Further mathematics helps the progression from A Levels and from school to uni. It certainly helps for the first year.” (Physics undergraduate at a Welsh university).
6.56 University staff talked of other benefits, such as problem solving and approach to learning: “A Level Mathematics only tests ‘can you do this’, not ‘can you solve this problem’.” “It gives students a greater depth of knowledge”; “It broadens their minds”, and; “Studying for further mathematics positively stretches more able students, and helps them develop more applied, problem solving skills compared to standard mathematics A Level (and it can be stimulating and fun).” (University lecturer).
6.57 Some lecturers expressed the view that studying further mathematics reaped benefits beyond the application of mathematics; it also contributed to
89 student’s personal development in terms of building confidence, their ability to self-learn, and a broader appreciation of mathematics’ application in everyday life.
6.58 One stakeholder continued by saying how some HEIs claim that it is a big advantage to have further mathematics if applying for a mathematics degree or, indeed, engineering, “because it spans pure, mechanics and statistics, it strengthens all aspects of their mathematical understanding.” In particular, he highlighted the value of algebraic skills, reasoning and proof skills, setting out working, and problem solving.
6.59 Without exception, undergraduates claimed that the possession of Further Mathematics A Level was beneficial to the transition to HE. It was seen as especially valuable for their first year, if studying mathematics, and at any time on their course if studying a mathematics-rich subject such as physics, engineering, or economics depending on when particular topics arose. Students without further mathematics were often seen to be at a disadvantage, (those with further mathematics would help others out), but universities had systems in place to help any struggling. Several reported using their notes from studying further mathematics to ‘help them out’ with university work.
6.60 In the online survey of students, help with the transition to undergraduate study was identified as the joint most important advantage of studying further mathematics (27 per cent of responses), along with gaining a better understanding of mathematics in general.
90
Figure 13 Undergraduate Students: Main Advantage of Studying Further Mathematics A/AS
Source: Online Survey of Students. Base=93. Only single response allowed.
Longer Term Impacts
6.61 In terms of longer-term impacts, there is generic data on graduate destinations, which provides some insights into the relative position of mathematics and STEM graduates. The data shows that mathematical sciences graduates are less likely than average to go into employment and more likely than average to continue on to further study, or a combination of work and further study.
Table 18 Destinations of Full-Time First Degree Leavers by Subject Area Six Months After Graduating 2011-12 (Known Destinations) Percentages Mathematics STEM All Subjects UK work 52 65 63 Overseas work 2 2 3 Work and further study 8 5 6 Further study 25 15 15 Unemployed 9 8 9 Other 5 4 5 Total percentage 100 100 100
Source: HESA 2013
91 6.62 In terms of those who entered employment, mathematics graduates were more likely than average to secure careers in professional /associate professional roles (although less likely to become professionals than STEM graduates as a whole).
Table 19 Occupation of Full-Time First Degree Leavers Entering Employment in the UK by Subject Area of Degree 2011-12
Percentages Mathematics STEM All Managers, directors and senior officials 3 3 4 Professional occupations 43 53 34 Associate professional and technical occupations 28 18 26 Total professional 74 74 64 Administrative and secretarial occupations 10 5 9 Skilled trades occupations 1 1 1 Caring, leisure and other service occupations 3 6 7 Sales and customer service occupations 8 9 13 Process, plant and machine operatives 0 0 0 Elementary occupations 4 5 6 Total non-professional 25 26 36 Source: HESA 2013
6.63 Perhaps the starkest difference in terms of employment profile was in the sector of employment: mathematics graduates were very much more likely than average to be employed in the financial sector, or in the property development, business and research sector than other graduates and very much less likely to enter health and social work, for example.
92
Table 20 Industry of Full-Time First Degree Leavers Entering Employment in the UK by Subject Area of Degree 2011-12
Mathematics STEM All Standard Industrial Classification % % % Agriculture, forestry and fishing 0 0 0 Mining and quarrying 1 1 1 Manufacturing 5 5 4 Electricity, gas and water supply 1 1 1 Construction 1 2 1 Wholesale and retail trade/repair(2) 12 13 17 Hotels and restaurants 4 4 6 Transport, storage and communication 11 7 8 Financial activities 21 4 5 Property development, renting, business and research activities 23 14 17 Public administration and defence/social security 3 3 4 Education 11 8 12 Health and social work 3 33 18 Other community, social and personal service activities 4 4 6 Private households with employed persons 0 0 0 International organisations and bodies 0 0 0 Source: HESA 2013
6.64 With regard to salaries, HESA Statistics show that the average salary for a mathematics graduate after six months of leaving university is £24,29657, against an average for all graduates of £21,762. Dentistry was the highest earning degree, at £30,775 and all of the twelve highest earning subjects were STEM-related. It should be noted, however, that the setting in which graduates were employed had a significant effect on their salaries.
Value for Money
6.65 Assessing value for money of a pilot programme can be challenging, given the extent of capacity building and initial programme development entailed, in addition to the delivery of support for further mathematics itself. As
57 Those in “graduate jobs”.
93 discussed above, there is no single clear measure for the number of additional students engaged in further mathematics as a result of the pilot. However, if examination entry data from the LLWR/WED is used, as the most conservative estimate of outcomes, using 2010 data as a baseline it can be seen that the total number of examination entries at AS and A Level for further mathematics was 165 more than might have been expected58. Against an overall programme cost of £581,485, this represents a cost per examination entry of £3,564. This figure ignores the additional outcomes achieved in terms of increased entry levels in Wales outside the pilot area and any impacts on increased recruitment onto mathematics courses, or engagement of pre-16 pupils. The pilot has delivered extensive benefits beyond examination entries, in terms of increased numbers of students studying further mathematics in the pilot area, awareness raising and encouragement of students to undertake mathematics and STEM subjects at a higher level and capacity building amongst teachers.
58 Using estimated data for 2013 based on initial schools results and subtracting 2010 entry levels form each year.
94 Appendix 1: Questioning Framework and Data Sources
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
Understand the impact the FMSP is having in the pilot area
compared to the rest of Wales Determine if the FMSP is offering value for money Begin to understand what the wider economic benefits of
studying further mathematics might be Process Evaluation
Evaluate existing monitoring data, Management Information X X Systems, processes and governance arrangements
Do the internal project steering group and Management X X X Committee provide strong project steer? Is the membership
59 Acknowledging the need to differentiate between those that deliver further mathematics in school and those who use collaborative arrangements.
95
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
appropriate?
Do the Management Committee and internal project group ensure the pilot is meeting its targets and stated outcomes X X X X and that processes and systems are working appropriately? Are staffing levels and other administrative systems adequate X X X and effective? Are existing MIS systems robust and fit for purpose? How is X X X performance measured against targets? Are there any gaps in data collection? X X X Is the ongoing monitoring of various activities fully utilised for X X X the improvement of the pilot? Evaluate contractual arrangements between the Welsh Government and WIMCS (Wales Institute of Mathematical and Computational Sciences), specifically progress against X X X X the stated outcomes as set out in Schedule IV of the grant agreement(s)
96
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
Are there any barriers or constraints for stakeholders (as X X X named in the contract) in meeting the contract? Do contractual arrangements facilitate and encourage X X participation? Could the contract be operated or delivered in a more X X efficient or effective way? Is the budget/grant allocation adequate enough to deliver to X X all of those it intends to and in the way that was envisaged? Evaluate the effectiveness of tuition in further mathematics GCE AS/A2 level units for all relevant English and Welsh awarding bodies to students who cannot access them through their local schools or colleges (in the pilot area). Evaluate specific targeted events, activities and resources as set out in the Programme of Work Schedule III. These X X include: Tuition Enrichment and Enhancement Activities Promotion and Publicity
97
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
Website and Online Resources
Assess the contribution the FMSP has had in ‘increasing the X X X X number of students studying GCE AS/A2 Level Further Mathematics in the pilot area (outcome 1) How are students recruited or encouraged to take up further X X X X mathematics? Who is involved in recruiting and awareness-raising? How successful are these people and organisations in increasing X X X X levels of participation? What criteria, if any, are used to select potential participants? X Are there any barriers to participation? X X X What do students see as the benefits to the further X X X mathematics qualification? How are disadvantaged and minority groups encouraged to participate? What is the variation in participation by gender, X X X X ethnicity, type of school, language (Welsh/English),
98
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
geographical area etc.? Do participation rates need to be improved? If so how? X X X
Assess the contribution the FMSP has had on ‘increasing the number of schools and colleges, or consortia, who offer X X X X X further mathematics’ (outcome 2) To what extent do the local consortia, 14-19 learning networks and the Regional Learning Partnership approaches help to strengthen existing provision? Are local authorities X X X X supportive of the initiative? What do they perceive the benefits of the FMSP to be? Are all schools in the pilot area are able to join the programme? Explore any barriers or constraints, also benefits. Are there any differences in the level of take up and interest between types of schools? Include secondary, Welsh medium and bilingual schools, sixth forms and colleges. How X X X X X are schools and colleges encouraged to participate? How can participation rates be increased?
99
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
What is the general overall awareness of the FMSP outside X X X X the pilot area in the rest of Wales? How effective is the support for schools that do not offer further mathematics, to enable teachers to supervise a single X student who is studying further mathematics on his/her own? Explore the effectiveness of: Promoting interactive online resources in schools where teaching of further mathematics is delivered in reduced hours. Providing extra classes for students to help to close the gaps X X which may occur when students learn further mathematics as part of local consortia. Providing tuition of single topics only. Assess the contribution of the FMSP in ‘increasing the number of mathematics teachers in Wales (confined to the X X X X pilot area) who are trained to teach further mathematics’ (outcome 3)
To what extent does the FMSP contribute to teacher CPD X X X X and progression and meets the needs of all teachers (full
100
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
time, part time, temporary and Welsh medium and bilingual) Is the FMSP integrated with teacher CPD programmes led by X X X teaching professions and the Welsh Government? Should CPD be included in the WIMCS contract in the same X X X X way it is in England? What incentives or benefits are there for teachers? X X Are teachers in the pilot area able to attend revision sessions and enrichment events and access online resources and X X student activities free of charge, to support their professional development? How are teachers recruited to become teachers of further mathematics? How are they identified and approached? What X X qualifications do they need? Are there any resourcing issues for schools/colleges and X X X teachers? How are these managed? Are there any differences between schools and sixth form colleges/FE colleges in terms of having sufficient numbers of X X X X mathematics teachers?
101
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
Explore the extent to which the FMSP contributes towards ‘raising student and parent awareness of the value of studying mathematics at a higher level’ (outcome 4) Explore how effective promotional events and materials have been in raising awareness for all potential students and their X X X X X X parents? Does awareness raising positively change opinions and X X X X increase levels of understanding? What is the role and impact of careers advisors in raising X X X awareness of students and parents? Are there any differences in awareness which may be the X X result of FMSP awareness raising activities? What do students perceive are the medium and long term X benefits of further mathematics? Impact Evaluation
Evaluate the extent to which the FMSP pilot is meeting its
102
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
stated aims and objectives (expressed as anticipated outcomes) and assess the impact the FMSP is having in the pilot area. Assess the impact of the FMSP by exploring the counterfactual position. Make comparisons between the pilot area and one Welsh local authority area outside the pilot or between Wales and England (or other area(s) as appropriate and agreed). Also make comparisons between further mathematics students and those following A level mathematics only as appropriate. How do participation and achievement rates compare? X X X Can differences between be attributed to the FMSP? X X Does the FMSP facilitate a greater uptake of further X X mathematics? Assess the impact the FMSP has had in relation to ‘increasing the number of students from the pilot area applying to study higher education courses in mathematics and mathematically rich subjects such as engineering, physics, IT, finance and economics’ (outcome 5). Notably this
103
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
will involve just two cohorts60. Explore further mathematics and mathematics pass rates and X grades achieved What proportion of further mathematics students apply for HE X courses? What proportions are successful? What subject areas are these in? X What is the variation by gender, ethnicity, type of school, X language (Welsh/English), geographical area etc? Assess the extent to which the FMSP ‘improves the transition of students from Further to Higher Education courses in
Mathematics, or courses which have a significant element of Mathematics’ (outcome 6)
Explore the issues experienced by students transferring from X X X FE to HE for: students who have been successful at further
60 The first cohort Year 2010 would reach full A2 certification on August 2012, Year 2011 candidates would reach full certification in August 2013. Year’s starting in 2012 would not enter HE courses until 2014 after the evaluation comes to an end
104
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
mathematics and those who studied mathematics only, focusing on study elements Does studying further mathematics make the transition X X easier? What is the dropout rate of students studying further X X X mathematics? Does this vary between universities or courses? X What is the variation by gender, ethnicity, language X (Welsh/English), etc? In addition to mathematics subjects, what other undergraduate subject areas does having the further X X mathematics qualification benefit How many UK universities require applicants to have the further mathematics qualification for mathematics X undergraduate courses? Explore the wider economic benefits of the FMSP and the value for money it offers (outcome 6).
105
Source
SchoolsOutside the Pilot
SchoolsCollege / Pupils
ExternalStakeholders
HEFCW/HEFCEData ProjectManagement
Non CareersAdvisors
Schools/College Schools/College Undergraduates
UniversityStaff
ProjectTeam
Participating
UCASData
Teachers
C
LitReview
University
MIS Data MIS Teachers
-
ommi Participating
Area
ttee
59
Activity
Identify and measure the relevant costs and benefits of the pilot programme. What is the added value of the FMSP to X X Wales? Does the FMSP increase levels of participation significantly to justify the costs? Compare the actual outputs and/or outcomes with target outturns, investigating any deviation(s) from the expected outputs and/or outcomes, including in relation to previously X identified risks. Are some individuals or groups likely to benefit more than others in the long term? Compare the identified impacts with relevant counterfactual(s), including the “do nothing” option Benchmark the pilot programme outputs and outcomes, and the cost effectiveness and value for money, against other X similar programmes elsewhere
106
107 Appendix 2: Topic Guides
Evaluation of FMSP
Stakeholder Topic Guide
November 2012
Please note, this topic guide has been designed for use with a broad cross- section of stakeholders, and therefore not all questions will be relevant to every individual; moreover, there may be additional issues to explore, that emerge during the course of individual interviews, which are not covered in this topic guide.
Introduction
Awareness of/attitudes towards AS/A2 Further Mathematics
What would you perceive to be the main advantages/disadvantages of taking an AS/A Level qualification in further mathematics for students currently in Wales?
In your experience who tends to take AS/A Level qualification in further mathematics?
(If applicable) Has this changed over time?
What would you perceive to be the main advantages/disadvantages of delivering AS/A Level qualifications in further mathematics for schools and colleges currently in Wales?
(If applicable) How do completion rates for AS/A2 Further Mathematics compare with other AS/A2 courses? Please explain your answer. Probe for any substantive evidence to support responses.
(If applicable) How is AS/A2 Further Mathematics perceived amongst school teaching staff, compared to other AS/A2 courses? Please explain your answer.
108 On a scale of one-to-five (five being high and one being low) how highly do you think the value of AS/A Level qualifications in further mathematics are rated amongst the following key groups:
Schools/teachers; FE colleges/lecturers; Students; Parents.
On a scale of one-to-five (five being high and one being low) how highly do you think universities in the UK rate the value of AS/A Level qualifications in further mathematics?
Do you think attitudes towards AS/A Level qualifications in further mathematics amongst HEIs in the UK have changed in recent years?
Can you point to any differences in attitudes towards AS/A Level qualifications in further mathematics amongst different groups of universities, for example Russell Group/non-Russell Group, universities in England/Wales, pre- and post- 92 universities?
Please explain your answers to the above questions.
Are you aware of any barriers for schools/colleges to the delivery of AS/A Level qualifications in further mathematics? If yes, please describe these barriers
(If applicable), do you think these barriers have become more/less significant in recent years?
What factors do you think might have brought about any chances in significance?
Please explain your answers.
Similarly are you aware of any barriers for students to the take-up of AS/A Level qualifications in further mathematics? If yes, please describe these barriers
(If applicable), do you think these barriers have become more/less significant in recent years?
109 What factors do you think might have brought about any chances in significance?
Please explain your answers.
Awareness of the FMSP
Are you aware of the Wales Institute of Mathematical and Computational Science (WIMCS)?
If yes, have you had any direct dealings with the WIMCS?
If yes, please explain the nature of any interaction, and your perceptions of the organisation.
What do you understand to be the purpose of the Further Mathematics Support Programme in Wales (FMSP)?
(If applicable) Are you aware of the specific objectives for the programme?
If yes, how realistic are these objectives, in your opinion?
Have you had any direct dealings with the FMSP?
If yes, please explain the nature of any interaction, and your perceptions of the programme.
How aware do you think schools and colleges in south west Wales are of the FMSP?
How aware do you think schools and colleges in the rest of Wales are of the FMSP?
How aware do you think universities in Wales are of the FMSP?
How aware do you think universities in the rest of the UK are of the FMSP?
Progress to date of the FMSP
(If applicable) Please describe the nature of any activities carried out by FMSP that you are aware of, and the success of the programme to date in delivering these activities.
110 Why do you think that is? Probe for specific awareness of, and attitudes towards the value of individual strands of the Project, for example:
The student tuition for AS/A level further mathematics;
On-line teaching materials, specifications and resources;
Mathematics master classes, Mathematics career talks and revision events for GCSE and post-16 students;
AS and A2 Mathematics and Further Mathematics revision days.
What do you perceive to be the main challenges for the FMSP?
To what extent do you think the programme is overcoming these challenges? Please explain your answer.
The Future
Are there any modifications you would make to the objectives/delivery approaches for the programme? Please explain your answer.
What would you see as the main benefits of increasing delivery/take-up of AS/A Level qualifications in further mathematics, from the perspective of:
Students in Wales;
Schools/colleges in Wales;
HEIs
Employers;
The UK/Welsh economy?
Are you aware of any problems associated with increasing delivery/take-up of AS/A Level qualifications in further mathematics that have not already been discussed?
Are there any particular issues that you feel should be explored in relation to the FMSP or AS/A Level qualifications in further mathematics generally, as a part of this evaluation?
Thank you for your time
111 Evaluation of FMSP
Management Committee Members Topic Guide
January 2013
Introduction
Individual role and experience
When did you first become a member of the FMSP management committee and for what reason did you join? For example; were you nominated?
What knowledge and experience do you feel you bring to the group?
Do you feel that the current management committee has sufficient coverage of relevant stakeholders? Please explain your answer.
What contribution do you feel you personally make to the management committee?
How would you rate the effectiveness of the management committee as a whole in guiding and supporting the FMSP project? I.e.: would you rate the committee as being very effective, somewhat effective or not effective at all?
Please explain your answer.
Attitudes towards the FMSP
What do you understand to be the purpose of the Further Mathematics Support Programme in Wales (FMSP)?
Are you aware of the specific aims of the programme?
NB: Read out (if necessary): The aims of the FMSP Wales are to:
Raise awareness among students and their parents of the value of studying Mathematics at higher levels;
Increase the number of students in Wales studying AS/A Level Mathematics and further mathematics;
112 Increase the number of schools and colleges in Wales offering further mathematics;
Increase the number of students from Wales applying to study higher education courses in Mathematics and Mathematically-rich subjects such as Engineering, Physics, IT, Finance and Economics;
Improve the transition of students from further to higher education courses in Mathematics or courses which have a significant element of Mathematics thus benefiting the wider economy.
If yes, how realistic are the specific aims of the programme, in your opinion?
Please explain your answer.
How aware do you think schools and colleges in south west Wales are of the FMSP?
Management and Governance
Regarding the management and governance arrangements for the FMSP project, can you point to which aspects work well and which might need some improvement?
Please explain your answer.
What are your views on the level of communication between the FMSP project team and the management committee?
Please explain your answer.
Do you feel that you are sufficiently aware of progress on and issues facing the FMSP?
If not, why not? Can you suggest any measures that could be taken to improve your levels of awareness?
Please explain your answer.
Do you feel able to contribute to decisions on issues facing the FMSP?
If not, why not?
Please explain your answer.
113 Awareness of/attitudes towards AS/A2 Further Mathematics
What would you perceive to be the main advantages/disadvantages of taking an AS/A Level qualification in further mathematics for students currently in Wales?
In your experience who tends to take AS/A Level qualification in further mathematics?
(If applicable) Has this changed over time?
What would you perceive to be the main advantages/disadvantages of delivering AS/A Level qualifications in further mathematics for schools and colleges currently in Wales?
(If applicable) How do completion rates for AS/A2 Further Mathematics compare with other AS/A2 courses? Please explain your answer. Probe for any substantive evidence to support responses.
(If applicable) How is AS/A2 Further Mathematics perceived amongst school teaching staff, compared to other AS/A2 courses? Please explain your answer.
How highly do you think the value of AS/A Level qualifications in further mathematics are rated amongst the following key groups?
Schools/teachers;
FE colleges/lecturers;
Students;
Parents.
How highly do you think universities in the UK rate the value of AS/A Level qualifications in further mathematics?
Do you think attitudes towards AS/A Level qualifications in further mathematics amongst HEIs in the UK have changed in recent years?
Please explain your answers to the above questions.
Are you aware of any barriers for schools/colleges to the delivery of AS/A Level qualifications in further mathematics? If yes, please describe these barriers
(If applicable), do you think these barriers have become more/less significant in recent years?
114 What factors do you think might have brought about any chances in significance?
Please explain your answers.
Similarly are you aware of any barriers for students to the take-up of AS/A Level qualifications in further mathematics? If yes, please describe these barriers
(If applicable), do you think these barriers have become more/less significant in recent years?
What factors do you think might have brought about any chances in significance?
Please explain your answers.
Progress to date of the FMSP
How successful has the programme been to date in delivering against the following specific activities, providing an explanation for your answer and (if applicable) examples of what has worked well and what might need some improvement in each case.
The student tuition for AS/A level further mathematics;
On-line teaching materials, specifications and resources;
Mathematics master classes, Mathematics career talks and revision events for GCSE and post-16 students;
AS and A2 Mathematics and further mathematics revision days.
What do you perceive to be the main challenges for the FMSP?
To what extent do you think the programme is overcoming these challenges? Please explain your answer.
The Future
Are there any modifications you would make to the objectives/delivery approaches for the programme? Please explain your answer.
What would you see as the main benefits of increasing delivery/take-up of AS/A Level qualifications in further mathematics, from the perspective of:
115 Students in Wales;
Schools/colleges in Wales;
HEIs
Employers;
The UK/Welsh economy?
Are you aware of any problems associated with increasing delivery/take-up of AS/A Level qualifications in further mathematics that have not already been discussed?
What would you see as the main benefits of expanding the programme across Wales?
Are there any particular issues that you feel should be explored in relation to the FMSP or AS/A Level qualifications in further mathematics generally, as a part of this evaluation?
Thank you for your time
116
Evaluation of FMSP
Discussion guide for focus groups with AS/A2 Level Further Mathematics students in schools and colleges in South-West Wales who have not engaged with FMSP
April – May 2013
Please note that these questions need to be asked in conjunction with the information acquired through Stage 1, and may require adaptation in some cases.
Clarify at the outset:
The age profile of the students; and
The numbers of students in the group studying AS and A2/A2 Level Further Mathematics.
General Introduction
What do you think are the advantages of studying AS/A2 Level Further Mathematics (from now on I will use the term further mathematics to describe both AS and A2 Level Further Mathematics) (?
What do you think are the disadvantages of studying further mathematics?
What were the main reasons that you chose to study further mathematics? If university entry requirements are mentioned, probe for which universities they hope to attend.
Awareness of the FMSP
Were you aware of the FMSP or of any of the support available from the FMSP before we mentioned it to you?
(If yes, ask questions 2.2 – 2.5)
117 Which activities specifically were you aware of before we contacted you?
Were your first impressions of the purpose of the FMSP?
How did you first become aware of the FMSP/ [specify FMSP activity/activities]?
Why do you think you chose not to participate in the [specify FMSP activity/activities]?
(Ask all)
Have you received any other support in A2 Level mathematics and/or further mathematics that is additional to your regular class teaching (for example revision sessions, on-line materials and resources)
(If no)
Why not?
(If yes)
How did you hear about the [specify other activity/activities]?
What motivated you to participate in the [specify other activity/activities]?
How useful was the [specify other activity/activities] in terms of helping you with your further mathematics?
(For those aware of some FMSP activities)
What changes would you make to the support available from the FMSP?
(Ask all)
Can you suggest any type of support activity or resources that could have helped you with your further mathematics course?
Thank you for your time
118
Evaluation of FMSP
Topic guide for mathematics teachers in schools and colleges in South-West Wales who have not engaged with the FMSP
April – May 2013
Please note that these questions need to be asked in conjunction with the information acquired through Stage 1, and may require adaptation in some cases.
Background context to delivery of A2 Level Further Mathematics
Do you have capacity in your school/college to deliver some or all modules of AS/A2 Level Further Mathematics (hereafter referred to as further mathematics, in terms of teaching expertise?
Do you have capacity in your school/college to deliver some or all modules of further mathematics, in terms of timetable space?
Does your school collaborate with other schools in the delivery of further mathematics?
Is funding for delivering further mathematics or enabling students who are enrolled on courses at your school/college to access further mathematics elsewhere an issue?
Are there logistical problems associated with delivering further mathematics or enabling students to access further mathematics elsewhere?
What other factors affect the capacity of your school/college to deliver further mathematics or enable students to access further mathematics elsewhere?
How strong is demand for further mathematics amongst students in your school/college? Has this changed at all over the last five years?
What factors or influences (either internal or external) affect demand for further mathematics in your view?
Awareness of the FMSP
119
Were you aware of the FMSP or of any of the support available from the FMSP before we contacted you?
(If yes, ask questions 2.2 – 2.5)
What were your first impressions of the purpose of the FMSP?
Which activity/activities specifically were you aware of before we contacted you?
How did you first become aware of the FMSP or [specify FMSP activity/activities]?
Why do you think you chose not to participate in the [specify FMSP activity/activities]?
(Ask all)
Have you received any other CPD support specifically in teaching A2 Level Further Mathematics/Mathematics that is additional to general teacher training/INSET provision?
(If yes)
How did you hear about the [specify other activity/activities]?
What motivated you to participate in the [specify other activity/activities]?
How useful was the [specify other activity/activities] in terms of improving your abilities to teach A2 Level Further Mathematics/Mathematics?
(For those aware of some FMSP activities)
What changes would you make to the support available from the FMSP?
(Ask all)
Can you suggest any type of support activity or resources that could help improve your abilities to teach A Level Further Mathematics?
Are you aware of any other sources of support for practitioners who are teaching further mathematics/Mathematics at A Level?
Are you aware of any other programmes of activities designed to increase student demand for A Level Further Mathematics?
120 If yes, has your school/college and/or your students engaged with these activities? If so, probe for if they are available in the local area, what they encompass (content, mode of delivery, quality etc.) and who delivers the programme.
If not, why has your school/college and/or your students not engaged with these activities
Have you any final comments to make about the FMSP or about A Level Further Mathematics?
Thank you for your time
121
Evaluation of FMSP
Discussion guide for focus groups with pre and post-16 students in schools and colleges in South-West Wales
April – May 2013
Please note this discussion guide has been designed for use with students who will experience different delivery arrangements for A Level Further Mathematics in their school/college (including no provision at all) and will have had varying levels of engagement with the FMSP. Therefore not all questions will be relevant to every individual; moreover, there may be additional issues to explore, that emerge during the course of individual interviews, which are not covered in this topic guide.
Please also note that these questions need to be asked in conjunction with the information acquired through Stage 1, and may require adaptation in some cases.
Clarify at the outset:
The age profile of the students;
The numbers of students in the group studying AS and A2 Level Further Mathematics;
The different types of FMSP activity they have engaged with in the last 3 years.
General Introduction NB: for pre-16 students, ask questions 1.1 and 1.2 hypothetically.
What do you think are the advantages of studying AS/A2 Further Mathematics? (From now on I will use the term further mathematics to describe both AS and A2 Level Further Mathematics)
What do you think are the disadvantages of studying further mathematics?
122 (For students studying further mathematics)
What were the main reasons that you chose to study further mathematics? If university entry requirements are mentioned, probe for which universities they hope to attend.
(Ask all)
How did you first find out about the FMSP?
What did you think was the purpose of the FMSP?
AS/A2 Further Mathematics Tuition
(For those who have received further mathematics tuition (on-line or face-to- face) via the FMSP)
NB: Please note that students may not necessarily be aware that the tuition is via FMSP. Clarify whether or not this is the case at the outset and articulate the questions accordingly.
How did you first hear about the tuition?
Why did you choose to receive tuition in further mathematics from the FMSP?
What was the alternative if you had not received tuition from the FMSP?
What were you expecting the FMSP tuition to be like before you started?
Did you have any concerns about receiving tuition in further mathematics from the FMSP before you started?
What did you think of the quality of tuition provided via the FMSP?
What did you think of the teaching approach provided via the FMSP (i.e.: on-line or face-to-face)?
How did this compare to tuition you receive for other subjects in your school/college?
Do you think the tuition was as good as you expected beforehand? If not, why not?
Were there any unexpected benefits of the tuition in further mathematics that you received from the FMSP?
123 How do you think the tuition could have been improved?
FMSP Revision Session
(For those who have participated in any face-to-face/on-line revision sessions - including GCSE mathematics and A Level Mathematics and further mathematics)
NB: Please note that students may not necessarily be aware that the revision was via FMSP. Clarify whether or not this is the case at the outset and express the questions accordingly.
How did you first hear about the revision sessions?
Where and when was the revision taking place? Was this convenient for you?
What motivated you to participate in the revision sessions?
Where there any barriers to participating the revision sessions?
What were your expectations of the revision sessions?
What were your views on the quality of the revision sessions?
What were your views on the teaching approach (i.e.: face-to-face or on-line) used for the revision sessions?
How did this compare to other types of revision you have carried out yourself or participated in with others?
To what extent were your expectations of the revision met by your experience of participating in the sessions?
How could the revision sessions have been improved in your view?
FMSP Enrichment Activities
(For those who have participated in any FMSP enrichment activities - including Mathematics conferences, Mathematics careers talks, Mathematics master classes and On-line mathematics seminars / lectures)
124 NB: Please note that students may not necessarily be aware that the activities were provided by the FMSP. Clarify whether or not this is the case at the outset and express the questions accordingly.
How did you first hear about the [specify FMSP activity]?
Where and when was the [specify FMSP activity] taking place? Was this convenient for you?
What motivated you to participate in the [specify FMSP activity]?
Where there any barriers to participating the [specify FMSP activity]?
What were your expectations of the [specify FMSP activity]?
What were your views on the quality of the [specify FMSP activity]?
Can you compare the [specify FMSP activity] to any other similar activity?
To what extent were your expectations of the [specify FMSP activity] met by your experience of participating in the [specify FMSP activity]?
Were there any unexpected benefits of attending the [specify FMSP activity]
How could the [specify FMSP activity] have been improved in your view?
On-line further mathematics Resources
(For those who have accessed any on-line resources for Mathematics)
NB: Please note that students may not necessarily be aware that the resources are provided by the FMSP. Clarify whether or not this is the case at the outset and express the questions accordingly.
How did you first hear about the on-line resources?
What motivated you to access the resources?
What were your expectations of the resources?
Did you face any problems accessing or using the resources?
What were your views on the quality of the resources?
125 Can you compare the resources to any other similar materials intended to support your mathematics education?
To what extent were your expectations of the resources met by your experience of using them?
Were there any unexpected benefits of using the resources?
How could the resources be improved in your view?
Outcomes and Impacts
Have the FMSP activities you have participated in helped you in your further mathematics studies? How much and in what ways? (For those studying A Level Further Mathematics)
Do you think the [specify FMSP support activity received] has improved your understanding of mathematics generally and further mathematics specifically?
Do you think the [specify FMSP support activity received] has had a positive effect on your results in further mathematics and/or Mathematics?
If so, how?
If not, why not?
(For those in years 9, 10 and 11)
How interested are you in taking further mathematics at AS or A Level?
To what extent do you think your views have been influenced by [specify FMSP support activity received]?
(Ask all)
Are you interested in studying mathematics or a mathematically-rich subject such as Engineering, Physics, IT, Finance and Economics at university?
Do you think your views have been influenced by [specify FMSP support activity received]?
Future improvements
How could the FMSP be improved?
126 What other kind of further mathematics would be useful?
If the support available via the FMSP not been available would this have affected you in any way?
Are you aware of any other programmes of activities designed to increase demand for further mathematics?
Have you any final comments to make about the FMSP
Thank you for your time
127
Evaluation of FMSP
Topic guide for mathematics teachers in schools and colleges in South-West Wales who have engaged with the FMSP
April – May 2013
This topic guide has been designed for use with teaching practitioners who will have different delivery arrangements for A Level Further Mathematics in their school/college (including no provision at all) and will have had varying levels of engagement with the FMSP (but will have engaged at least to some extent). Therefore not all questions will be relevant to every individual; moreover, there may be additional issues to explore, that emerge during the course of individual interviews, which are not covered in this topic guide.
Please note that these questions need to be asked in conjunction with the information acquired through Stage 1, and may require adaptation in some cases.
Please also note, that this topic guide should be used for all staff who have had direct engagement with the FMSP as well as for the full interview with the key contact, which is to be carried out first (either via telephone or face-to-face), to ensure the necessary contextual information is acquired prior to carrying out research with students and other members of staff.
Background context to delivery of A Level Further Mathematics
Do you have capacity in your school/college to deliver some or all modules of AS/A2 Further Mathematics (hereafter referred to as further mathematics), in terms of teaching expertise?
Do you have capacity in your school/college to deliver some or all modules of further mathematics, in terms of timetable space?
Does your school collaborate with other schools in the delivery of further mathematics?
128 Is funding for delivering further mathematics or enabling students who are enrolled on courses at your school/college to access further mathematics elsewhere an issue?
Are there logistical problems associated with delivering further mathematics or enabling students to access further mathematics elsewhere?
What other factors affect the capacity of your school/college to deliver further mathematics or enable students to access further mathematics elsewhere?
How strong is demand for further mathematics amongst students in your school/college? Has this changed at all over the last five years?
What factors or influences (either internal or external) affect demand for further mathematics in your view?
Process of engagement with FMSP
How did you first become aware of the FMSP?
Were your first impressions of the purpose of the FMSP?
For each FMSP activity that the practitioner has engaged with personally, please ask the following questions:
What were your motivations for engaging with this activity?
What were your expectations of the activity?
To what extent were these expectations met by your experience of engaging with this activity?
How would you rate the quality of the tuition/support/activity that you received?
Were there any unexpected benefits of engaging with this activity?
How could your experience of the activity have been improved?
(For those accessing CPD modules)
Who paid for the CPD?
What were the time implications of accessing the CPD?
Ask all
129 What was your experience of dealing with the people delivering the FMSP support?
Are you aware of which organisation(s) the people delivering the support came from? Probe for whether it was internal FMSP staff, from WIMSC, or contracted teachers/tutors.
How could your experience of dealing with them have been improved?
Outcomes and Impacts of Engagement with the FMSP
What contribution has the project made to your personal abilities as a mathematics teacher?
Has teaching competence within the school/college generally improved following engagement with the FMSP programme? To what extent do you think this can be attributed to the FMSP?
Has your school/college begun delivering/extended delivery, of further mathematics following engagement with the FMSP? To what extent do you think this can be attributed to the FMSP?
Has the FMSP helped improve success in further mathematics in your school/college and if so in what ways? 3.5 Has demand for further mathematics in your school/college increased following engagement with the FMSP? To what extent do you think this can be attributed to the FMSP? (If applicable) can you quantify this claim?
Are more students from your school/college applying to study mathematics related courses at university following engagement with the FMSP? To what extent do you think this can be attributed to the FMSP? (If applicable) can you quantify this claim?
Are more students from your school/college getting accepted to university to study mathematics related subjects following engagement with the FMSP? To what extent do you think this can be attributed to the FMSP? (If applicable) can you quantify this claim?
What are your personal views on the benefits of taking further mathematics?
130 To what extent do you think your views have been affected by engagement with FMSP?
Future Improvements
How could the FMSP be improved?
What would have been the likely effects and outcomes if the support available via the FMSP not been available?
Are you aware of any other sources of support for practitioners who are teaching mathematics at A Level?
Are you aware of any other programmes of activities designed to increase demand for A Level Further Mathematics?
If yes, has your school/college and/or your students engaged with these activities? If so, probe for if they are available in the local area, what they encompass and who delivers the programme
If not, why has your school/college and/or your students not engaged with these activities;
Have you any final comments to make about the FMSP?
Thank you for your time
131
Quantitative Survey of Teaching Practitioners: Online Questionnaire.
English The Welsh Government has commissioned Miller Research to carry out an evaluation of the Further Mathematics Support Programme (FMSP).
The FMSP has run as a pilot in Carmarthenshire, Neath Port Talbot, Pembrokeshire and Swansea since 2010. The FMSP provides further mathematics GCE AS and A Level tuition to students who cannot access further mathematics through their local schools or colleges within the pilot area. Additional support provided by the programme includes Mathematics and further mathematics revision days, mathematics careers talks and master classes as well as online mathematics resources for teachers, tutors and students and mathematics enrichment activities for students in Key Stages 3 and 4.
Part of the research for the evaluation includes a survey of all mathematics teachers/tutors, heads of mathematics departments and head teachers/principals in secondary schools and colleges in the pilot area.
The survey asks about current availability of and delivery arrangements for AS and A2 Mathematics and further mathematics in your school or college, awareness of the FMSP and your experience of any support received via the FMSP.
We would be grateful if you would complete the survey even if you teach in an 11 16 school. We thank you in advance for taking the trouble to participate in this survey. It should take between five and ten minutes to complete the survey. By taking part, you will be entered into a prize draw with all other respondents, to win a £100 Amazon voucher for your school.
Please note that all responses are anonymous, and will be reported collectively with those received from other teaching practitioners. Individual responses will not be shared with either Welsh Government or any other organisation (including your local authority, school or college).
General Information In which of the following institutions is your main teaching role? Please select all that apply. NB: please do not include ad hoc teaching/lecturing delivered, for example, as part of the Further Mathematics Support Programme. School (11-16 years) School (11-18 years) Further Education College Other (please specify) In which of the following local authorities do you work? Carmarthenshire Neath Port Talbot Pembrokeshire Swansea
132 Other (please specify) Which of the following best describes your school/college? English medium Welsh medium Bilingual (English and Welsh) Other (please specify) Which of the following best describes your role in your school/college? Mathematics teacher/lecturer Head of mathematics department Headteacher/ principal Other (please specify) About you Is mathematics/STEM (science, technology, engineering and/or mathematics) your main subject teaching area? Yes No A Level Mathematics provision in your school/college Which one of the following combinations of AS /A2 Mathematics is delivered in your school/college in the current academic year (201213)? Neither AS nor A2 Mathematics AS Mathematics only A2 Mathematics only Both AS and A2 Mathematics Other (please specify) Reasons for limited/no delivery of A Level Mathematics Why is AS and/or A2 Mathematics not delivered in your school/college in the current academic year? Please select all that apply. Not enough students wanting/academically able to take the course Not enough space in the curriculum Lack of teaching expertise/experience in the school/college Decision taken by the consortium that it should be delivered in another school/college in the authority Other (please specify) Delivery of A Level Further Mathematics (1) We are interested in whether any modules of AS and/or A2 further mathematics are currently delivered in your school/college. Which of the following statements best describes the arrangements in your school/college in the current academic year (201213)? Yes, all modules of AS and/or A2 further mathematics are formally timetabled and delivered in my school/college Yes, all modules of AS and/or A2 further mathematics are delivered out of timetable in my
133 school/college Yes, some modules of AS and/or A2 further mathematics are formally timetabled and some are delivered out of timetable in my school/college Yes, some modules of AS and/or A2 further mathematics are formally timetabled and delivered in my school/college and some are delivered in a neighbouring school/college Yes, some modules of AS and/or A2 further mathematics are delivered out of timetable in my school/college and some are delivered in a neighbouring school/college No, but all modules of AS and/or A2 further mathematics are delivered in a neighbouring school/college No, but some modules of AS and/or A2 further mathematics are delivered in a neighbouring school/college No, neither AS nor A2 further mathematics is available to pupils in my school/college Don’t know Other (please specify) Reasons for non-delivery of A Level Further Mathematics Why does your school/college not currently deliver any modules of AS and A2 further mathematics? Please select all that apply. Not enough students wanting/academically able to take the course Not enough space in the curriculum Lack of teaching expertise/experience in the school/college Decision taken by the consortium that it should be delivered in another school/college in the authority Don’t know Other (please specify) Delivery of A Level Further Mathematics (2) Which of the following combinations of AS /A2 further mathematics is delivered in your school/college in the current academic year (201213)? NB: please do not include any tuition that is provided in-house via the Further Mathematics Support Programme. Some modules only of AS further mathematics only All modules of AS further mathematics only Some modules only of A2 further mathematics only All modules of A2 further mathematics only Some modules only of both AS and A2 further mathematics All modules of both AS and A2 further mathematics Don't know Other (please specify) Delivery of A Level Further Mathematics (3) Why does your school/college not currently deliver all modules of AS and A2 further mathematics?
134 Please select all that apply. Not enough students wanting/academically able to take the course Not enough space in the curriculum Lack of teaching expertise/experience in the school/college Decision taken by the consortium that it should be delivered in another school/college in the authority Don’t know Other (please specify) Awareness of the Further Mathematics Support Programme (FMSP) Have you heard of the Further Mathematics Support Programme Yes No Engagement with the Further Mathematics Support Programme (FMSP) How did you first become aware of the Further Mathematics Support Programme? Through a visit from one of the programme staff members From a flier/poster or other marketing material From the further mathematics Support website From my local authority Directly from Swansea University I can’t remember Other (please specify) Have you personally obtained any of the following support from the further mathematics Support Programme (FMSP)? Where relevant, please rate the usefulness of each form of support on a scale of one to five, where five is very useful and one is not very useful at all. Leave blank any type of support you have not received. 1 not very useful at all 5 very useful Free single-user teacher/lecturer access to online integral resources Workshop for teachers/lecturers on Interactive Resources in further mathematics (free) General guidance or advice Information about student tuition, revision sessions, master classes and careers talks available through the FMSP Information about other support available to mathematics teachers/lecturers outside of the FMSP Live Online Professional Development courses (at a charge) Biannual electronic newsletter General marketing information and promotional material Other (please specify) Have any post-16 students in your school/college received any of the following forms of support from
135 the Further Mathematics Support Programme (FMSP)? Please select all that apply. Student tuition for AS/A2 Level Further Mathematics modules (at a charge per module) Online student access to A Level mathematics and further mathematics resources (at a charge per 10 students) Mathematics master classes Mathematics career talks Online mathematics lectures A Level mathematics and further mathematics revision events (face-to-face) A Level mathematics and further mathematics revision events (online) None of the above Not applicable – I teach in an institution catering for students aged 11-16 years I don’t know Other (please specify) Reasons for limited or non-engagement with the Further Mathematics Support Programme (1) For what reasons do you think students post-16 students in your school or college have not received any/more support from the Further Mathematics Support Programme? Please select all that apply. Lack of awareness amongst staff of the available support Lack of interest amongst students in accessing the available support Lack of encouragement or support within the school/college for students wishing to access available support Financial cost to the school/college of accessing support (for example tuition fees, student access to online integral resources) Logistical barriers for students of accessing support (for example travel costs, internet connection and costs) The available support is not relevant to the students Not applicable – they have accessed all the support they need Not applicable completion and achievement rates in A Level Further Mathematics in this school/college are excellent and support is not needed Other (please specify) Key Stages 3 and 4 engagement the Further Mathematics Support Programme The FMSP also provides support to KS3 and KS4 learners to try to encourage them to follow a mathematics programme post 16. Have any students in Key Stages 3 and 4 in your school/college received any of the following types of support from the Further Mathematics Support Programme? Please select all that apply. Mathematics conference Online student access to GCSE and Additional Mathematics resources (at a charge per 10 students) Mathematics master classes Mathematics career talks I don’t know
136 None of the above Not applicable – I teach in an institution catering for post16 students only Other (please specify) Reasons for limited or non-engagement with the Further Mathematics Support Programme (2)
For what reasons do you think students in Key Stages 3 and 4 in your school might not have received any/more support from the Further Mathematics Support Programme? Please select all that apply. Lack of awareness amongst staff of the available support Lack of interest amongst students in accessing the available support Lack of encouragement or support within the school for students wishing to access available support Financial cost to the school of enabling access support Financial cost to students of accessing support (for example travel costs, ICT costs) The available support is not relevant to the students Not applicable – our students have accessed all the support they want from the FMSP Other (please specify) Quality and effectiveness of the Further Mathematics Support Programme On a scale of 1 to 5, where 1 is very poor and 5 is very good, how would you rate the quality of support FOR STUDENTS provided via the further mathematics Support Programme? 1 very poor 5 very good Don't know If you have any comments to make about the quality of support FOR STUDENTS provided via the Further Mathematics Support Programme, please make them in the box below On a scale of one to five, where one is very poor and five is very good, how would you rate the quality of support FOR TEACHERS AND LECTURERS provided via the Further Mathematics Support Programme? If you have any comments to make about the quality of support FOR TEACHERS AND LECTURERS provided via the further mathematics Support Programme, please make them in the box below Based on your professional experience and feedback from students, on a scale of one to five (whereby one is very ineffective and five is very effective), how effective has the Further Mathematics Support Programme been in raising awareness among students and their parents in your school/college of the value of studying Mathematics at higher levels? 1 very ineffective 5 very effective Don't know If you have any comments to make about the effectiveness of the Further Mathematics Support Programme in raising awareness of the value of studying Mathematics at higher levels, please make them in the box below
137 Based on your professional experience, on a scale of one to five (whereby one is very ineffective and five is very effective), how effective has the further mathematics Support Programme been in increasing the number of students in your school/college studying AS/A level Mathematics and further mathematics, either in your school/college or another institution within your consortia area? If you have any comments to make about the effectiveness of the Further Mathematics Support Programme in increasing the number of students studying AS/A level Mathematics and further mathematics, please make them in the box below Based on your professional experience, on a scale of one to five (whereby one is very ineffective and five is very effective), how effective has the further mathematics Support Programme been in increasing the number of students from your school/college applying to study higher education courses in Mathematics and Mathematically-rich subjects such as Engineering, Physics, IT, Finance and Economics? If you have any comments to make about the effectiveness of the Further Mathematics Support Programme in increasing the number of students applying to study higher education courses in Mathematics and Mathematically-rich subjects please make them in the box below Based on your professional experience, on a scale of one to five (whereby one is very ineffective and five is very effective), how effective has the further mathematics Support Programme been in increasing the number of students from your school/college getting accepted onto higher education courses in Mathematics and Mathematically-rich subjects such as Engineering, Physics, IT, Finance and Economics? If you have any comments to make about the effectiveness of the Further Mathematics Support Programme in increasing the number of students getting accepted onto higher education courses in Mathematics and Mathematically-rich subjects please make them in the box below Are there any alterations you think could be made to the further mathematics Support Programme in Wales that could make it more effective in supporting students and mathematics practitioners in Wales? No Yes Please specify Delivery of A Level Further Mathematics (i) Is AS and/or A2 further mathematics delivered in your school/college in the current academic year (201213)?
Yes No Reasons for non-delivery of A Level Further Mathematics Why does your school/college not currently deliver any modules of AS and A2 further mathematics? Please select all that apply. Not enough students wanting/academically able to take the course Not enough space in the curriculum Lack of teaching expertise/experience in the school/college Don’t see the value of offering further mathematics
138 The cost of delivery Decision taken by the consortium that it should be delivered in another school/college in the authority Not applicable – my institution caters for students aged 11-16 years Delivery of A Level Further Mathematics (ii) What, in your view, are the benefits of delivering AS and/or A2 further mathematics in your school/college? Please select all that apply To attract/retain higher ability students; To gives the school/college a competitive edge; To help achieve higher results; To stretch higher ability students; Is AS and/or A2further mathematics formally timetabled? Please select one of the following Yes, all modules are formally timetabled No, all modules are delivered out of timetable Some modules are formally timetabled and some are delivered out of timetable Don’t know Which of the following combinations of AS /A2 further mathematics is delivered in your school/college currently? Some modules only of AS further mathematics only All modules of AS further mathematics only Some modules only of A2 further mathematics only All modules of A2 further mathematics only Some modules only of both AS and A2 further mathematics All modules of both AS and A2 further mathematics Reasons for limited delivery of A Level Further Mathematics Why does your school/college not currently deliver all modules of AS and A2 Further Mathematics? Please select all that apply. Not enough students wanting/academically able to take the course Not enough space in the curriculum Lack of teaching expertise/experience in the school/college Don’t see the value of offering further mathematics The cost of delivery Decision taken by the consortium that it should be delivered in another school/college in the authority Awareness of the Further Mathematics Support Programme (i) Have you heard of the Further Mathematics Support Programme? Yes No Engagement with the Further Mathematics Support Programme
139 How did you first become aware of the Further Mathematics Support Programme? Through a visit from one of the programme staff members From a flier/poster or other marketing material From the further mathematics Support website From my local authority Directly from Swansea University From one of my colleagues I can’t remember Please select from the following table all forms of support that you or any of your staff members have received from the Further Mathematics Support Programme? Please select all that apply. Free single-user teacher/lecturer access to online integral resources Workshop for teachers/lecturers on Interactive Resources in further mathematics (free) General guidance or advice Information about student tuition, revision sessions, master classes and careers talks available through the FMSP Information about other support available to mathematics teachers/lecturers outside of the FMSP Live Online Professional Development courses (at a charge) Bi-annual electronic newsletter General marketing information and promotional material Please select from the following table all forms of support that students at your school/college have received from the Further Mathematics Support Programme? Please select all that apply. Student tuition for AS/A Level Further Mathematics modules (at a charge per module) Online student access to A Level Mathematics and further mathematics resources (at a charge per 10 students) Online student access to GCSE and Additional Mathematics resources (at a charge per 10 students) Mathematics master classes Mathematics career talks Mathematics conference Online mathematics lectures A Level Mathematics and further mathematics revision events (face-to-face) A Level Mathematics and further mathematics revision events (online) None I don’t know Other (please specify) Implications of engagement with the Further Mathematics Support Programme Have there been any direct financial and/or HR implications for your school/college of engaging with and receiving support from the Further Mathematics Support Programme? Yes No
140 Please specify What are the advantages of engaging with and receiving support from the Further Mathematics Support Programme? What (if any) are the disadvantages of engaging with and receiving support from the Further Mathematics Support Programme? How (if at all) could the Further Mathematics Support Programme be improved in order to make it more suited to the needs of your school/college? Thank you Many thanks for completing this survey. Please can I remind you that all responses are anonymous, and individual responses will not be shared with either Welsh Government or any other organisation (including your local authority, school or college). For more information about this evaluation of the Further Mathematics Support Programme in Wales, please feel free to contact Kerry Lewis at Miller Research on Kerry@miller-research. co.uk or Julie Owens at Welsh Government on [email protected]. Please provide your email address to be entered into a prize draw with all other respondents, to win a £100 Amazon voucher for your school.
141
Evaluation of FMSP
Topic guide for STEM-subject lecturers in Universities in the UK
May – June 2013
This topic guide has been designed for use with lecturers in subjects such as mathematics, engineering, physics, astrophysics etc. in both Russell Group and non-Russell Group universities.
The purpose of these interviews is to explore lecturers’ views on the value of A Level Further Mathematics generally (in terms of preparation for, and transition to university to study STEM subjects) as well as their knowledge of the FMSP and the impact of the programme’s support on students.
Background context to university admissions policy and attitudes towards AS/ A Level Further Mathematics
What is your admissions policy in relation to AS/ A Level Further Mathematics for [specify course]? Are students required to have AS/ A Level Further Mathematics in order to be accepted to study [specify course] at your university?
(If not a requirement) would students be given preference (in terms of being accepted to study [specify course] at your university) if they had studied AS/ A Level Further Mathematics? (In terms of lower offer grades, for example).
What would you perceive to be the main advantages/disadvantages for students of taking AS/ A Level Further Mathematics prior to studying [specify course] at your university? Prompt for: wider syllabus knowledge / syllabus knowledge more aligned to HE requirements, greater depth of knowledge, increased reasoning / problem solving capabilities, ease of transition to university.
In your experience have you identified any differences in the characteristics of student who take AS/ A Level Further Mathematics and those who don’t? Prompt for differences in ability, effort, etc.
(If applicable) Has this changed over time?
142
On a scale of one-to-five (five being high and one being low) how highly do you rate the value of AS/ A Level Further Mathematics?
Has your attitude towards AS/ A Level Further Mathematics changed at all in recent years? Please explain your answer.
Should AS/ A Level Further Mathematics be made compulsory for [specify course(s) taught by interviewee]?
Could anything else be done to prepare further mathematics/M students for study for [specify course(s) taught by interviewee]?
Awareness of FMSP
Are you aware of the Wales Institute of Mathematical and Computational Science (WIMCS)?
If yes, have you had any direct dealings with the WIMCS? If yes, please explain the nature of any interaction, and your perceptions of the organisation.
Are you aware of the Further Mathematics Support Programme in Wales (FMSP)?
NB: interviewees may be aware of the Further Mathematics Support Programme in England.
(If yes) What do you understand to be the purpose of the FMSP in Wales? NB: is lecturers are not aware of the FMSP in Wales, but are aware of the England equivalent, then ask the following questions in relation to the programme in England.
(If applicable) Are you aware of any of your students on [specify course] who have received support from the FMSP in Wales [or England]?
(If applicable) in what ways and to what extent do you think the support they received from the FMSP in Wales [or England] benefited these students?
Probe for impact on students’ interest and ability in mathematics, their decision to take [specify course], transition to [specify course] in HE and their competency in studying [specify course].
143 (If applicable) In your view, have there been any disadvantages for students who have received support from the FMSP in Wales [or England]?
The Future
(If aware of the FMSP in Wales [or England]) Are there any modifications you would make to the objectives/delivery approaches for the FMSP in Wales [or England] in terms of support for schools and/or students to help students to prepare for studying [specify course]? Please explain your answer.
(If unaware of the FMSP in Wales [or England]) based on your experience of teaching students who have studied AS/ A Level Further Mathematics, is there any type of support that schools and/or students could receive to help students to prepare for studying [specify course]? Please explain your answer.
What would you see as the main benefits of increasing delivery/take-up of AS/A Level qualifications in further mathematics, from the perspective of:
Your university/HEIs in general?
Employers?
The UK/Welsh economy?
Are you aware of any problems associated with increasing delivery/take-up of AS/A Level qualifications in further mathematics that have not already been discussed?
Are there any particular issues or concerns in relation to the FMSP or AS/A Level qualifications in further mathematics generally, that you wish to discuss?
Thank you for your time
144
Evaluation of FMSP
Discussion guide for students studying STEM subjects in university who previously engaged with the FMSP
May - June 2013
General Introduction
What subject are you currently studying in university?
Did you require AS/A2 Further Mathematics to get accepted onto your current university course?
Was AS/A2 Further Mathematics course timetabled in your school/college? Was it delivered by a teacher/lecturer from your school/college? (NB: if not delivered by internal staff, ascertain how the course was delivered).
What do you think are the advantages of studying AS/A2 Further Mathematics?
Are there any disadvantages of studying AS/A2 Further Mathematics?
What were the main reasons that you chose to study AS/A2 Further Mathematics?
(For those who took AS/A2 Further Mathematics but are studying a course for which they DID NOT require AS/A2 Further Mathematics) Do you think you were better prepared for your current university course than others on your course who did not take AS/A2 Further Mathematics?
How did you first find out about the FMSP?
What did you think was the purpose of the FMSP?
Engagement with FMSP
What type(s) of support did you receive from the FMSP? NB: it may be necessary to list the different types of support offered by the FMSP. If student is not aware of any support they received, as specifically whether they attended any revision sessions at Swansea University, logged onto to any on-line revision
145 sessions, attended any mathematics conferences, careers conventions or Master-classes at Swansea University, received a ‘Careers in mathematics’ talk in school from someone from Swansea University or used any revision materials accessed via the MEI website (the integral resources). Refer back to their answer to question 1.3; if AS/A2 Further Mathematics was NOT delivered by a teacher/lecturer within their school/college; ask whether they received tuition from an external tutor or via on-line tuition.
For each type of support received, ask the following questions:
Why did you choose to receive [specify support] from the FMSP?
What was the alternative if you had not received [specify support] from the FMSP?
What were you expecting the [specify support] to be like before you started?
Did you have any concerns about receiving [specify support] from the FMSP before you started?
What was particularly good about the [specify support] from the FMSP?
What (if anything) was particularly bad about the [specify support] from the FMSP
What did you think of the quality of the [specify support] provided via the FMSP?
What did you think of the teaching approach adopted in the [specify support] provided via the FMSP (i.e.: on-line or face-to-face/interactive or non-interactive etc.)? What type of support did you prefer?
How did this compare any support you received for other A Level subjects?
Do you think the [specify support] was as good as you expected beforehand? If not, why not?
Were there any unexpected benefits of the [specify support] that you received from the FMSP?
How do you think the [specify support] could have been improved? Was there anything missing?
Outcomes and Impacts
146 In what ways and to what extent do you think the support you received from FMSP helped you with your AS/A2 Further Mathematics course?
In what ways and to what extent do you think the support you received from FMSP has helped/is helping you with [specify course taken in university] Prompt for: wider syllabus knowledge / syllabus knowledge more aligned to HE requirements, greater depth of knowledge, increased reasoning / problem solving capabilities, ease of transition to university.
In your view, have there been any disadvantages of receiving specify FMSP support activity/activities received] from the FMSP, in terms of your university education?
Do you think the [specify FMSP support activity/activities received] improved your understanding of mathematics generally and further mathematics specifically?
Do you think the [specify FMSP support activity received] has had a positive effect on your results in further mathematics and/or Mathematics?
If so, how?
If not, why not?
To what extent do you think your decision to study [specify course taken in university] was influenced by [specify FMSP support activity received]?
Future improvements
How could the FMSP be improved?
Looking back on your AS/A2 Further Mathematics course with your experience of studying [specify course taken in university], is there anything more that the FMSP could offer to AS/A2 students, in order to assist them with AS/A2 Further Mathematics and prepare them for [specify course taken in university]?
In what year do you think support from FMSP is best introduced to students in Wales? For example, pre GCSE choices? Pre-A Level choices? During A Level courses only?
What other kind of further mathematics would be useful?
147 If the support available via the FMSP not been available would this have affected you in any way?
Have you any final comments to make about the FMSP
Thank you for your time
148 Appendix 3: List of Schools and Colleges Visited
Local Authority School / College
Carmarthenshire Coleg Sir Gar
Ysgol Gyfun Dyffryn Taf
Ysgol Gyfun Maes yr Yrfa
Neath Port Talbot Dwr y Felin Comprehensive
Neath Port Talbot College
Ysgol Gyfun Ystalyfera
Pembrokeshire Thomas Picton School
Ysgol Bro Gwaun
Ysgol y Preseli
Swansea Bishop Gore School
Gower College
Ysgol Gyfun Gymraeg Bro Tawe
149 Appendix 4: Results of Online Survey of HE Students
The online survey of STEM students in HE was distributed via a link sent out to tutors involved in the qualitative fieldwork. The survey attracted 120 responses from six different HEIs.
Survey respondents were asked about which undergraduate course they were currently studying at University. Ninety-two per cent of respondents were reading mathematics, seven per cent physics and one per cent economics. No respondents involved in the survey were studying engineering or medicine, although the survey had been distributed to students on a range of STEM courses.
Main Subject Studied
Figure 14 Subjects Studied by Respondents
Base:116
Source: Miller Research Student Survey (Single selection)
Eighty-nine per cent of surveys users sat their A/AS Levels in England, with the remaining eleven per cent taking them in Wales.
150 Figure 15 Country Sat A/AS exams
Base:110
Source: Miller Research Student Survey (Single selection)
Eighty-two per cent of the survey respondents had studied FM A/AS Level with remaining eighteen per cent stating they had not taken the subject.
Figure 16 Number of Participants who had taken Further Mathematics AS/A Level?
Base:120
Source: Miller Research Student Survey (Single selection)
151 Figure 17 Main Reasons for Taking Further Mathematics
Base: 93
Source: Miller Research Student Survey. (Multiple Selection)
Participants were asked their main motivation for taking FM. The most popular responses among survey respondents were that they had a personal interest in higher level mathematics (75 per cent), it was a university requirement (43 per cent), it would improve their future career options (39 per cent) and that it was recommended by a member of staff or careers advisors (37 per cent). The least prominent reasons included becoming interested in the subject after attending a FM open day (15 per cent) and that it was recommended by friends or family (4 per cent).
152 Figure 18 Completed Course/ Result?
Base: 93
Source: Miller Research Student Survey. (Single selection)
Eighty-two per cent of survey respondents completed A Level qualification in Further Mathematics. Fifteen per cent received an AS level qualification whilst three per cent completed the course but did not achieve a qualification.
153 Figure 19 Advantages of Further Mathematics
Base:92
Source: Miller Research Student Survey. (Single selection)
When asked about the advantages of taking FM the most popular responses among survey participants was that it had provided them with a better understanding of the application of mathematics in general (27 per cent) and had eased the progression to undergraduate study at university (27 per cent). A further twenty-two per cent stated it improved their chances of getting into their first choice university, whilst eighteen per cent stated it was a compulsory entry requirement for their chosen course. Only five per cent of the respondents outlined that FM provided them with a better understanding of AS/A level mathematics.
Disadvantages
Survey respondents were also asked to highlight the main disadvantage of taking FM. The most prominent reason discussed by participants was that FM has to be taken as an additional AS/A level
154 which many felt was just too much work on top of the other AS/A level subjects they were currently studying.
Timetabled
Eighty-eight per cent of the respondents indicated that FM was offered at their school or college, with only twelve per cent unable to access the subject.
Figure 20 Was Further Mathematics Timetabled?
Base:92
Source: Miller Research Student Survey. (Single selection)
155 Figure 21 Additional Support
Base: 52
Source: Miller Research Student Survey (Multiple selection)
Participants were asked whether they had received any additional support when undertaking their A/AS in Further Mathematics. The largest group of the respondents (48 per cent) had received additional face-to-face tuition from their school or college. A further seventeen per cent had visited a mathematics enrichment event, thirteen per cent had received private tuition and twelve per cent viewed an online revision session or had been to an additional face-to-face revision event. The least common response among participants was that they had attended an additional face-to-face tuition at another school or college, with only two per cent of the respondents selecting this response.
156 Figure 22 Awareness of FMSP
Base: 93
Source: Miller Research Student Survey (Single selection)
Only a fifth of participants had heard about FMSP with just over eighty per cent of the respondents stating there were unaware of the programme’s existence.
Figure 23 How did you Hear of FMSP?
Base:18
Source: Miller Research Student Survey (Multiple selection)
157 Of those who selected that they were aware of the programme, the majority of participants had heard about FMSP through member of staff (12) or online (8). A further four participants had heard about the programme through parents and friends or had attended a mathematics career day, whilst one respondents had heard about support via social media such as Twitter and Facebook.
Participants were also questioned about whether there were any benefits of taking FM. There were only four respondents to this question and so the data should be taken as indicative, but the most prominent response from survey respondents was that FM had not only increased their confidence in mathematics at school or college but also their confidence in the subject more generally. Participants also stated that FM has strengthened their understanding of both AS/ A Level Mathematics and FM as well as helped with the progression to undergraduate studies. The least common responses among survey respondents was that taking FM had gave them a better application of mathematics and had encouraged them to study a mathematics-related subject at University.
When asked if taking FM had helped their progression into undergraduate studies at University participant ratings amongst the small number of respondents who voiced an opinion were extremely positive. Two respondents highlighted that FM had a strong positive effect upon the progression, whilst the remainder indicated that it had a slight effect.
158 Figure 24 Why did you not receive tuition?
Base: 83
Source: Miller Research Student Survey (Multiple selection).
The majority of participants (70 per cent) did not receive tuition from FMSP as they were unaware of its existence. A further fifth stated it was not available in their school, whilst twenty of participants stated they did not require support. The final sixteen of the respondents indicated that their school or college did not have the necessary resources at their disposal.
Figure 25 Main reasons didn’t take A/AS level FM
Base: 17
Source: Miller Research Student Survey (Multiple selection)
159 The main reasons put forward by survey respondents for not taking Further Mathematics at AS/A level was due to their school or college not offering the subject as well as the number of subjects they were already taking. Less prominent reasons included that FM was not part of university requirement, timetabling issues and lack of staff and resources.
Figure 26 Would taking FM have been beneficial in hindsight?
Base: 20
Source: Miller Research Student Survey (Single selection)
Nineteen participants admitted that in hindsight taking FM A/AS level would have been beneficial to their personal development.
160
Appendix 5: Results of Online Survey of School and College Staff
Figure 27 Teaching Role
Base: 17
Source: Miller Research Teacher Survey (Single selection)
Twelve participants taught 11-18 years whilst the remaining five respondents were teachers who taught 11-16 years. No FE staff took part in the survey.
Figure 28 Local authority school / college
Base:15
161 Source: Miller Research Teacher Survey (Single selection)
The research sample consisted of six teachers from Carmarthenshire, four from Neath Port Talbot, three from Pembrokeshire and two from Swansea
Figure 29 Type of school
Base: 16
Source: Miller Research Teacher Survey (Single selection)
Ten participants were teachers from English medium schools whilst the remaining were from Welsh medium (3) and bilingual establishments (3)
Figure 30 Teaching role
Base: 15
162 Source: Miller Research Teacher Survey (Single selection)
Nine participants were heads of mathematics departments whilst the remaining six were mathematics teachers or lecturers.
Figure 31 Courses offered
Base:16
Source: Miller Research Teacher Survey (Single selection)
Ten survey respondents indicated that their school offered both AS and A2 mathematics. The remaining five highlighted that their establishment did not offer either of the subjects.
Figure 32 Reason for not being offered
Base:4
163 Source: Miller Research Teacher Survey (Single selection)
When asked about why their establishment did not offer those subjects, four of the five participants to whom this question was applicable responded. Three of these participants indicated that it was a decision taken by a large consortium due to lack of teaching expertise and experience in the school or college. The other reason selected by the remaining participant was that there were not enough students wanting to take the course.
Figure 33 Level of FM offered
164 Base:15
Source: Miller Research Teacher Survey (Single selection)
Survey respondents were also questioned about the extent to which their associated teaching establishment delivered FM. Four participants stated that some modules in FM were formally delivered by their school or college. A further four respondents stated that FM was not available to pupils attending their school or college. A fifth of respondents stated that modules of both AS and A2 were formally timetabled and delivered within their school. Two survey respondents highlighted FM was delivered out of schools timetable. The remaining two survey respondents stated that even though they did not offer the subject FM modules could be accessed at a neighbouring school or college.
Figure 34 Modules offered
Base:10
Source: Miller Research Teacher Survey (Single selection)
Five participants highlighted that their teaching establishment offered all modules both AS and A2 in FM. Two offered some modules both AS and A2. The remaining respondents indicated that their teaching establishment only offered some AS and A2 modules.
165
Figure 35 Reasons for not delivering all modules
Base:5
Source: Miller Research Teacher Survey (Multiple selection)
The most common answers to explain why schools and colleges were unable to deliver all the modules was that there was not enough room in the curriculum for delivery and that there was not enough interest from students wanting to take up FM.
166 Figure 36 Awareness of FMSP
Base: 11
Source: Miller Research Teacher Survey (Single selection)
Survey respondents were also asked how they first became aware of FMSP. Five participants couldn’t remember how they first heard of the programme, however the remaining had heard about it from a flier or poster (3), through a visit from programme staff (1), from the FMSP website (1) and directly from Swansea University (1).
167 Figure 37 FMSP support received
Base:12
Source: Miller Research Teacher Survey (Multiple selection)
According to survey respondents, the most popular forms of support received by schools were FM master classes, face-to-face revision events, and online revision events. Student tuition at a charge per module and mathematics career talks were also popular with three participants selecting this option. The least common form of support received by participants was online student access to FM resources at a charge per 10 students and online lectures
168 Figure 38 Reasons for not receiving support
Base: 9
Source: Miller Research Teacher Survey (Multiple selection)
Participants were then asked for reasons to explain why they were not receiving support from FMSP. The most popular reason among participants was that there was a lack of interest among students. Other reasons included a lack of awareness among staff of available resources and logistical barriers to students accessing support.
169 Figure 39 Key Stage 3 and 4 support received
Base:13
Source: Miller Research Teacher Survey (Multiple selection)
Schools and colleges were also provided with KS3 and KS4 support from FMSP to encourage post 16 take up of mathematics based subjects. Eight survey respondents outlined that had taken advantage of mathematics master classes whilst a further five stated they had attended mathematics career talks. Two respondents had attended a mathematics conference and accessed online GCSE resources
170 Figure 40 Reason for lack of Key Stage 3 and 4 support
Base: 13
Source: Miller Research Teacher Survey (Multiple selection)
Those participants who did not access support from FMSP for KS3 and KS4 were then questioned to provide reasons for not taking up the available resource. The most prominent response from teaching staff who engaged with the survey included a general lack of awareness of resources available and also the financial cost to students of accessing support for instance travel and ICT costs. A further fifth of participants stated there was a general lack of interest among students in accessing the support. One participant stated there was a lack of support within the school or college for student wishing to access available support.
171 Figure 41 Rating support for students
Base:12
Source: Miller Research Teacher Survey (Single selection)
Respondents were then asked to rate on a scale of one to five the quality of support available to students from FMSP. The majority of the respondents selected don’t know (5) whilst the remaining seven participants selected very good (4) or neutral (3)
Figure 42 Rating support for teachers
Base:12
Source: Miller Research Teacher Survey (Single selection)
172 Support to teachers from FMSP was also rated by the survey respondents. Five participants rated support as good or very good. Five participants felt they were unable to rate support, whilst the remaining two survey respondents gave support a neutral rating.
Figure 43 Feedback from Students
Base:12
Source: Miller Research Teacher Survey (Single selection)
Based on their professional experience and feedback from students participants were then asked to rate how effective FMSP has been in raising awareness of taking higher level mathematics. Five participants rated activities as effective and very effective. Three respondents rated activities as neutral.
173 Figure 44 Increase uptake in higher level mathematics
Base:12
Source: Miller Research Teacher Survey (Single selection)
Participants were then asked to rate how effective FMSP has been in increasing the uptake of higher level mathematics. Half of the respondents felt they were unable to answer this question, whilst the remaining six participants provided above average ratings.
Figure 45 Increase number of people applying to higher education
Base: 12
Source: Miller Research Teacher Survey (Single selection)
174 Survey respondents were also asked to rate how effective FMSP activities have been in increasing the number of people applying to mathematic based subjects at higher education. Half of the respondents felt they were unable to comment on this question. Two participants outlined that FMSP has been very effective in increasing the number of people applying to higher education. Three respondents rated FMSP as neutral whilst the remaining participants felt FMSP had been ineffective in increasing the number of applicants.
175 Appendix 6: Suggested Format for Management Reports
The evaluation found that current management reporting provides useful information, but that there are elements missing, or not fully clear. It might therefore be a positive development to structure future reports around three sections: Activities delivered to date (i.e. from beginning of the pilot period, with quarterly breakdowns) Progress against outcomes Future plans and priorities The progress against outcomes section could be generated from the indicators listed in Table 4 below.
Table 21 Suggested indicators to inform progress against outcomes
Outcome Suggested Measure Increased numbers of students in the pilot No. of pupils participating in further area studying Further Mathematics at GCE mathematics by centre, level and gender A/AS levels No. of examination entries by centre, level and gender More schools and colleges in Wales offering No. of centres registered with FMSP from Further Mathematics, either individually, or via within and without the pilot areas consortia No. of centres delivering further mathematics, by setting Increased numbers of mathematics teachers in Engagement with LOPD sessions Wales who are trained to teach further No. of teachers attending support events mathematics Overall raised awareness among students and Feedback from awareness raising events their parents of the importance of studying mathematics at higher levels Increased numbers of students from Wales UCAS Data for applications from Wales applying to study higher education courses in domiciled students for STEM courses at UK mathematics and related subjects, such as universities. engineering and physics HESA Data for Wales domiciled first year undergraduates in UK universities studying STEM subjects Improved transition of students from further to Destination tracking surveys of students after 9 higher education courses in mathematics, or months of A Level results. from courses which have a significant element of mathematics, thus benefiting the wider economy.
176