Performance management in the Scottish Qualifications Authority
Prepared for the Auditor General for Scotland November 2006 Auditor General for Scotland The Auditor General for Scotland is the Parliament’s watchdog for ensuring propriety and value for money in the spending of public funds.
He is responsible for investigating whether public spending bodies achieve the best possible value for money and adhere to the highest standards of financial management.
He is independent and not subject to the control of any member of the Scottish Executive or the Parliament.
The Auditor General is responsible for securing the audit of the Scottish Executive and most other public sector bodies except local authorities and fire and police boards.
The following bodies fall within the remit of the Auditor General:
• departments of the Scottish Executive eg, the Health Department • executive agencies eg, the Prison Service, Historic Scotland • NHS boards • further education colleges • Scottish Water • NDPBs and others eg, Scottish Enterprise.
Audit Scotland is a statutory body set up in April 2000 under the Public Finance and Accountability (Scotland) Act 2000. It provides services to the Auditor General for Scotland and the Accounts Commission. Together they ensure that the Scottish Executive and public sector bodies in Scotland are held to account for the proper, efficient and effective use of public funds. 1 Contents
Summary Part 3. The SQA’s performance management framework Introduction How effective are the SQA’s Summary of key findings performance indicators and Page 2 measures?
Summary of key recommendations Are corporate planning processes Page 3 well developed? Page 13 Part 1. Introduction Is the SQA’s performance The role of the Scottish Qualifications management framework fit for Authority purpose? Page 4 Page 16
Background to the report Part 4. The SQA’s use of performance information The study Page 6 The SQA’s development of a balanced scorecard approach Part 2. Developments in the SQA Page 19 since 2000 The SQA’s approach to the analysis Response to the problems of 2000 of costs and activities Page 7 The development of benchmarking Additional resources information Page 8 Page 20
Changes in the wider environment Next steps Page 9 Page 22
Customer and stakeholder Appendix 1. SQA objectives confidence in the SQA Page 23
Role of the board and senior Appendix 2. Advice on the management development of a balanced Page 10 scorecard approach Page 26 The SQA’s approach to good governance and the management Appendix 3. Findings from Audit of risk Scotland’s benchmarking exercise Page 11 Page 28
2 Summary
Introduction to inaccuracy and incompleteness in continuous improvement and it is 2.7 per cent of the results. A total of now in a position to more critically 1. The Scottish Qualifications 17,000 candidates were affected. challenge its operational processes. Authority (SQA) is the national awarding body for Scotland, 4. The overall aim of this study was 6. The SQA has received additional responsible for the development, to assess the extent to which the government funding since 2000 accreditation, assessment and organisation has recovered from its and has also increased the income certification of qualifications position in 2000. We considered generated by charges for its (other than degrees and some whether the SQA’s current risk qualifications. This has helped to professional qualifications). It has and performance management secure the SQA’s recovery and also an annual turnover of around £51 framework is appropriate, whether to support additional costs created million and employs around 650 the organisation displays the by the more complex educational full-time equivalent (FTE) staff. Its principles of good governance environment in which the SQA qualifications are taken by around and whether it demonstrates a now operates. Customers and 500,000 candidates each year and commitment to best value through stakeholders have indicated that around 1,470 centres are approved continuous improvement. they are increasingly confident in the to offer its qualifications. SQA’s performance since 2000. Summary of key findings 2. The SQA is responsible for a large 7. The board and executive team range of academic and vocational 5. Since 2000, the SQA have shown strong leadership in qualifications. This includes the has been successful in re- driving forward improvements. annual set of national examinations establishing customer and This has taken the form of ‘hands- known as the Diet. The results of stakeholder confidence in its on’ management with detailed these examinations are issued in core functions. In response to monitoring and involvement in all August and include Standard Grades the problems encountered, the stages of activity. There is, however, and Highers. SQA has developed a much better a risk that concentrating on detailed understanding of its key business monitoring can detract from 3. In 2000, the SQA encountered activities. It has gone beyond simply strategic-level thinking. The board significant problems with processing recovering from 2000, demonstrating has recognised this by agreeing to the summer exam results, leading an ongoing commitment to adopt a more strategic approach from Summary 3
June 2006. The executive team also Summary of key needs to continue to develop a more recommendations strategic approach, particularly in relation to the management of risk. 11. The key recommendations from our study are: 8. The SQA’s corporate planning process has established clear • The SQA should use objectives. There is still work to do its reviews of staffing, in linking performance management qualifications portfolio and information to these objectives. The business areas to provide a SQA’s key objectives are backed by more critical challenge of its corporate and business objectives products and processes. and they are expressed in the corporate plan. Progress is, however, • The board and senior monitored by management mainly management of SQA should at the level of detailed business continue to develop a more objectives, with an emphasis on strategic focus based on the process. key risks facing the business.
9. The SQA’s annual report includes • The SQA should develop its key 20 performance indicators. As performance indicators, linked they stand, these indicators do to key objectives, as part of not present a clear picture of its development of a balanced performance. The SQA recognises scorecard. that the current performance management framework needs • The SQA should continue to reviewing and is introducing a explore the scope for efficiency balanced scorecard approach to savings including opportunities performance management in order presented by the SQA’s to provide a clear and coherent modernisation agenda. picture of its performance. The development of key performance • The SQA should continue to indicators (KPIs) and the balanced refine its work linking costs scorecard provides an opportunity to with activities, to further give a clearer picture of performance improve its understanding of for both internal and external users. its cost base and to inform the setting of charges for 10. The SQA is developing a deeper qualifications. understanding of the link between costs and activities and is beginning • The SQA should explore the to benchmark its functions against opportunities for greater use other organisations. Continued of benchmarking to inform progress in these areas will allow performance management the SQA to explore its scope for activity. efficiency savings and to further improve its understanding of its cost base. 4 Part 1. Introduction
The role of the Scottish are offered at approximately 1,470 Qualifications Qualifications Authority centres, including schools, colleges, 16. The SQA is responsible for employers and private training a large range of academic and 12. The SQA is the national awarding providers in Scotland and elsewhere. vocational qualifications split into body for Scotland, responsible for In 2005, the SQA received more three main types: units, courses and the development, accreditation, than 2.5 million entries from around group awards (Exhibit 1). assessment and certification of 500,000 candidates. qualifications other than degrees and 17. Most Scottish qualifications some professional qualifications. It 15. The SQA’s functions are to: are covered by a single framework was established under the Education known as the Scottish Credit and Act (Scotland) 1996 which was • devise, develop and validate Qualifications Framework (SCQF) subsequently amended by the qualifications, and keep them which was developed by the SQA Scottish Qualifications Authority Act under review along with Universities Scotland, the 2002. This Act outlines the functions Quality Assurance Agency Scotland and the governance of SQA along • accredit qualifications and the Scottish Executive with the composition and structure (Exhibit 2, page 6). This framework of its board. • approve education and training brings together all the qualifications establishments as being suitable awarded and accredited by SQA 13. The SQA was formed following for entering people for these and by degree-awarding institutions. the merger of two existing bodies, qualifications The SCQF provides a means for the Scottish Examination Board and comparing different qualifications by the Scottish Vocational Education • arrange for, assist in, and carry their level and credit points. The level Council. It is an executive non- out, the assessment of people of qualification shows how difficult departmental public body (NDPB) taking SQA qualifications it is and the credit points show how sponsored by the Scottish Executive much learning has to be done to Education Department. • quality assure education and achieve it. training establishments which 14. The SQA has an annual turnover offer SQA qualifications of around £51 million and a staff of around 650 FTEs. Its qualifications • issue certificates to candidates. Part 1. Introduction 5
Exhibit 1 SQA Qualifications
Units National Units Take around 40 hours of teaching time. Can be built into National Courses, National Progression Awards and National Certificates. Higher National Units Mainly taken at college. The building blocks of Higher National Certificates (HNCs) and Higher National Diplomas (HNDs) although they are qualifications in their own right. SVQ Units Defines one aspect of a job or role. Can be built up into SVQs or Modern Apprenticeships. Courses Standard Grades Can be taken at three levels, Foundation, General and Credit. Generally progression from these to National Courses. National Courses Available at six levels including Higher and Advanced Higher. Most are made up of three units plus, and external assessment. These courses include Skills for Work which offer practical experience linked to a particular career for school-aged leavers. Group Awards National Progression Awards Assess a defined set of skills in specialist vocational areas. Linked to SVQs. National Certificates Designed to prepare candidates for progression to HNC/HND level. HNCs and HNDs Made up of Higher National Units. May allow entry to 2nd/3rd year of degree courses. SVQs Based on job competence – primarily delivered to candidates in full-time employment. Professional Development Awards (PDAs) Often taken after degree or vocational qualifications to extend or broaden skills. Customised awards In addition to SVQs and PDAs, the SQA also offers specially designed vocational qualifications to meet an organisation’s specific needs for skills and expertise.
Source: SQA 6
Exhibit 2 Scottish Credit and Qualifications Framework
SCQF SCQF level level
12 Doctorate 12 11 Masters 11 10 Honours Degree 10 Professional 9 Ordinary Degree Development 9 Scottish Awards 8 HND/DIP HE Vocational 8 7 Advanced Higher HNC/Cert HE Awards 7 6 Higher 6 5 Intermediate 2, Credit Standard Grade 5 National National 4 Intermediate 1, General Standard Grade Progression 4 Certificate 3 Access 3, Foundation Standard Grade Awards 3 2 Access 2 2 1 Access 1 1
Source: SQA
Background to the report • Poor reporting of progress to the • data management management board. 18. One of the SQA’s key functions • the Awards Processing System. is to set, deliver and mark the annual • Lack of clarity of the role of the set of national examinations known board in dealing with the concerns 21. The SQA developed an action as the Diet. The results of these which were presented to them. plan to address these issues and examinations are issued in August all subsequent Diets have been and include Standard Grades 20. The problems with the Diet completed successfully. and Highers. in 2000 and subsequent media criticism led to investigations into 22. A set of performance measures 19. In 2000, the SQA encountered the management and performance relating specifically to the Diet has significant problems with the of SQA being conducted by the been agreed annually with the processing of the summer exam Scottish Executive (using Deloitte & Scottish Executive since 2001. These results. As a consequence, 2.7 per cent Touche) and the Scottish Parliament’s detailed measures cover areas such of results and 17,000 candidates were Enterprise and Lifelong Learning as the accuracy and timeliness of affected by missing or incomplete data. and Education, Culture and Sport results. All measures have been met The following problems contributed Committees. The reviews identified or exceeded in each year since 2001. to the errors: weaknesses and recommended action across a number of areas The study • The late delivery of software for including: SQA’s new Awards Processing 23. The overall aim of this study System computer system. • governance was to assess the extent to which the organisation has recovered from • Mismanagement of candidate • planning its position in 2000. We considered data. whether the SQA’s current risk • staffing and performance management • Lack of properly trained staff to framework is appropriate, whether process data. • communication the organisation displays the principles of good governance • Failure to recruit sufficient script • operational processes and whether it demonstrates a markers within the required commitment to best value through time frame. • exam procedures continuous improvement. 7 Part 2. Developments in the SQA since 2000
Key messages • the SQA’s approach to developments and lessons learned. achieving the principles of Weekly meetings of the Certification Since 2000, the SQA has been sound governance, including its Planning Group focus on managing successful in re-establishing approach to risk management. this process and involve senior customer and stakeholder managers. confidence in its core functions. Response to the problems of 2000 27. The SQA has monitored progress The board and executive team 25. Following the problems of 2000, against the recommendations made have shown strong leadership in the SQA undertook an extensive by the Scottish Executive following driving forward improvements. mapping of business processes. Diet 2000. Other developments in This has led to a much better response to issues highlighted post- 24. To set the context for the SQA’s understanding of the processes 2000 include: performance, this part of the report underlying its core functions, looks at: which are now linked to individual • internal restructuring, including responsibilities and accountabilities. a smaller executive and senior • the SQA’s response to the management team and a reduced problems of 2000 26. The SQA’s approach to managing number of board members the annual cycle of activities leading • the additional resources received up to the issue of examination • improved communications with by the SQA since 2000 certificates in August has significantly centres, including the introduction improved. Risks are highlighted of customer account managers • changes in the wider and managed on an ongoing basis, linked to each centre environment within which the and critical events and milestones SQA operates and the associated through the year are clearly • more active stakeholder funding implications understood and regularly monitored. communication and engagement An annual certification plan has been • customer and stakeholder prepared in each year since 2000 • improved management of those confidence in the authority which sets out key tasks, owners appointed to mark examination and timescales. This is updated papers, including a 50 per cent • the role of the board and annually following the August increase in payments to these executive team certification process to reflect new markers in 2001. 8
Exhibit 3 SQA income and expenditure
The chart shows the increase over time of SQA’s expenditure and how it has required additional grant funding from the Scottish Executive to cover its development costs.
60,000 Expenditure
Additional grant funding 50,000 Government funding
40,000 Other income
Entry charges 30,000 £ (000) £
20,000
10,000
0 1999/2000 2000/01 2001/02 2002/03 2003/04 2004/05 2005/06
Year
Source: SQA
28. The SQA has gone beyond core processes will be delivered 31. The SQA’s increased expenditure simple recovery from the problems to schedule and to the standard reflects a range of factors. For it faced in the immediate aftermath required, the SQA is now in a example, the more diverse range of of 2000. It is now demonstrating position where it can use the review qualifications delivered by SQA has a commitment to continuous process to more critically challenge led to additional costs associated improvement, as evidenced in a its operational processes. with assessing these qualifications. wide range of areas of operation, Current levels of activity could including regular reviews of not be sustained without support staffing, a major review of the Recommendation 1 from government. In Exhibit 3, portfolio of qualifications and a The SQA should use its reviews of ‘Additional grant funding’ covers rolling programme of business staffing, qualifications portfolio and non-core expenditure on activities area performance reviews. A business areas to provide a more such as the development of new Best Value review in 2006 by the critical challenge of its products qualifications or e-enablement of SQA’s internal auditors concluded and processes. processes. However, a certain level that the organisation was able to of government support (labelled demonstrate a commitment to ‘Government funding’) is also best value principles. The auditors Additional resources required to support the core activities cited a number of examples of associated with delivering and good practice such as its detailed 30. The SQA is funded by the fees assessing qualifications. corporate planning cycle and its it charges for its qualifications and use of a quality management assessments and by government 32. In 2005/06, the SQA generated framework. An important current grants. Since 2000 it has received £33.8 million in revenue through, area of development for the SQA additional resources to help ensure for example, examination fees involves exploring the potential for the successful delivery of its core and consultancy assignments. The e-enablement of various areas of functions. Exhibit 3 shows the revenue generated largely covers the activity, for example marking and increase in the SQA’s income and SQA’s core functions of assessment processing of exam scripts. This has its expenditure since 1999/2000. and certification. In addition, the SQA the potential to deliver considerable Government funding, including received £10.6 million additional grant efficiency savings. additional grant funding, has funding and £6.9 million of other increased significantly (by 160 per government funding. The additional 29. Having reached a point where cent) during this period (Exhibit 4). grant funding is principally to cover the organisation is confident that ‘development costs’, which include Part 2. Developments in the SQA since 2000 9
Exhibit 4 SQA government funding and staffing
The chart shows the increase over time of the SQA’s government funding and staffing.
30,000 750 Staff
700 Total government 25,000 ) funding 650 f 20,000 600
550 15,000 500 Number of staf of Number 10,000 450
400 otal government funding £(000 funding government otal
T 5,000 350
0 300 1999/2000 2000/01 2001/02 2002/03 2003/04 2004/05 2005/06
Year
Source: SQA
qualifications development. Total much broader range of qualifications, • International benchmarking and government funding reduced from a and a greater number of options is intelligence. peak of £23.8 million in 2003/04 to available to candidates to provide a £17.5 million in 2005/06. system that recognises a wider range • Promoting Scottish education of abilities and skills than in the past. and training. 33. The SQA has increased income generated through charges for its core 35. The Skills for Work initiative, to • Commercial activities to work by 50 per cent since 2000. Much encourage young people to become support education and training of this is accounted for by increases in familiar with the world of work, is development. charges for exams, set in agreement a recent example of expansion to with the Scottish Executive. Between the portfolio of qualifications. The 37. The SQA participates in 2002/03 and 2005/06, charges for SQA is also involved in developing projects and initiatives funded by National Qualifications and Scottish and distributing 5-14 National Tests the UK Government, the European Vocational Qualifications increased by to schools as part of the Scottish Commission and other agencies. The 48 per cent and charges for Higher Executive’s 5-14 Programme, and in SQA’s strategy for these and other National Qualifications increased by the development of e-assessment commercial activities is to ensure 20 per cent. This reflects a move and wider application of new that all projects should at least towards setting charges at a level that technology throughout its processes. break-even and should aim to make fully recovers the costs of core Other potential future developments a reasonable return on investment service delivery. include more than one opportunity where possible. during the year for sitting National Changes in the wider environment Courses, such as Highers, involving a 38. Despite increased income move away from a single annual Diet generation, it is apparent that without 34. The SQA’s need for increased of examinations. additional government funding the resources is not simply a product SQA could not continue to deliver of its need to recover from the 36. The SQA is also committed to its current range of services. On the problems it faced in 2000. The developing an international profile with basis of its current workload, the educational environment in which a vision ‘to be recognised nationally SQA is likely to require even more the SQA operates has become much and internationally as a leader in government funding in future as more complex since then as a result qualifications and assessment’. The Scottish Executive forecasts show of new initiatives and developments SQA’s international strategy is based falling numbers in the school-age in education. The SQA now offers a on three strands of activity: population in Scotland over the next 10
1 few years. The SQA is concerned • the SQA’s expertise • HM Inspectorate of Education that the gap between its expenditure and the income it generates through • contact and openness with • Highlands & Islands Enterprise charges for exams will widen as a customers result of this. • Learning and Teaching Scotland • provision of information. 39. The SQA’s modernisation • UCAS. agenda offers some opportunities 42. The SQA plans to develop its for efficiency savings through, for customer surveys through increased 44. In general, stakeholders were example, increased application frequency and by focusing on key positive about the progress that of new technology to existing issues identified by its customers. the SQA has made since 2000 and processes and activities. This is The aim is to link the findings from about its ability to deliver its core explored further in Part 4. these surveys into the corporate functions. The SQA’s engagement planning process, helping the SQA to and relations with its customers, its Customer and stakeholder become more directly accountable to communications and the robustness confidence in the SQA its customers. of its core business were all cited as strengths by stakeholders. The main 40. Customer and stakeholder 43. The SQA is developing a concerns expressed were about the confidence suffered as a result stakeholder engagement plan for SQA’s levels of funding and about its of the problems of 2000 and broader consultation on the SQA’s ability to sustain its core business in the SQA has worked to address performance and is proposing to future without being distracted by this situation through a range of meet with stakeholders formally commercial activity. initiatives. Successful delivery of once a year. We consulted the the Diet has obviously been a major following bodies as part of our study: Role of the board and senior factor in restoring confidence, but management the SQA has also focused on areas • Scottish Executive such as communication, openness, 45. Stakeholders also spoke responsiveness and accessibility. • Headteachers Association positively of the role of the SQA’s Scotland board and senior management. 41. To monitor the success of these Following the problems of 2000, the efforts and direct future activities, • Sector Skills Alliance Scotland board and executive team developed the SQA has carried out three a very ‘hands-on’ approach to customer surveys since 2000, • Careers Scotland management, with detailed reporting findings in 2004, 2005 and monitoring and involvement in all 2006. In the 2006 survey, overall • Scottish Enterprise stages of activity. The improvements customer confidence was rated at in the SQA’s understanding of 80 per cent, putting the SQA in the • Scottish Parent Teacher Council its key business activities has second quartile when compared to been led and overseen by senior other educational bodies.2 Customer • Scottish Funding Council management. This has allowed the satisfaction levels improved from 78 SQA to drive forward change and per cent in 2004 to 80 per cent in • Scottish Council for Independent instil a commitment to continuous 2006 and are higher for schools than Schools improvement. However, it has also they are for colleges. The areas with meant that the board and executive which customers expressed most • Educational Institute of Scotland team have remained actively satisfaction were: involved at a detailed level of activity. • Scottish Further Education Unit • qualifications 46. There is a risk that concentrating • Association of Scottish Colleges on detailed monitoring can detract • effectiveness of account from strategic level thinking. The management • Association of Directors of executive team meet on a weekly Education Scotland basis and the management team
1 Results of Teacher Workforce Planning for 2005/06 Scottish Executive National Statistics 2 The comparative bodies include other UK examinations authorities, accountancy bodies and other examinations providers such as City & Guilds. Part 2. Developments in the SQA since 2000 11
hold monthly review meetings which and risk register can be used to core principles of good governance are supported by detailed information put more emphasis on challenges (Exhibit 5 overleaf). on all aspects of the business. While and strategy summaries are provided, which use 51. The SQA has made significant a priority system to help users focus • a review of the patterns of progress as an organisation over the on areas of concern, the information meetings and agendas for last six years, partly in response to would be more transparent and 2007 should include a different the problems it faced in 2000, but useable if it was reported on an approach to the board’s also through an ongoing commitment exception basis. In addition, a wide engagement with the to continuous improvement. The range of committees and groups corporate plan. principles of good governance are meet on a regular basis to address evident within the organisation and specific areas of business, such 48. This proposed approach is where there are areas for further as qualifications management and consistent with the principles of development, there is a commitment commercial activity. There are a best value and good governance and willingness to work to make the number of regular meetings that and should promote greater required improvements. For example, senior management routinely attend, board effectiveness. It will require the SQA has worked hard to engage ensuring that core business is tightly streamlining of the information with customers and stakeholders managed. However, as these groups presented to the board and a greater since 2000 and it is committed to and committees typically involve focus on strategic risk assessment. developing even more effective and senior level staff, there is some This will allow the board to develop meaningful consultation. overlap and duplication of issues even further the SQA’s commitment considered. to continuous improvement and 52. However, a key area that the SQA its approach to best value. The could develop further is that of the 47. The board regularly receives executive team should continue to management of risk. The SQA’s risks updates from business group develop a more strategic approach are managed through a risk register areas or committees that focus on now that systems and processes and there are regular meetings particular aspects of the organisation. are well developed and functioning across all areas of activity to ensure A review of board effectiveness was effectively. This will involve a greater comprehensive coverage of issues. reported to the board in June 2006. concentration on key risks and issues, The SQA continues to develop The paper argued that, of necessity, rather than regular review of such an its approach to risk management the board had adopted a hands-on extensive array of information. through the development of a risk approach post 2000 but that it could management policy statement, a now safely devote more time to 49. Although much has been risk management framework and strategic issues and less time to achieved, the latest staff survey risk management guidelines. At scrutinising operational matters. The suggests that there remains scope present, the processes in place paper was supported by the board for further improvement in the role highlight risks, but there needs to and argued that: of management. In the 2005/06 staff be greater focus on prioritising risks survey, only 37 per cent of staff felt and developing responses to address • decision-making powers needed that senior management provided the key risk areas. Development of to be reviewed both for the board staff with a clear sense of direction, a more strategic approach to risk and the executive team reinforcing the need for a more management will be important in strategic approach by the senior supporting the board in its more • agenda items should be management team. strategic approach to management. scrutinised before meetings to identify matters for noting rather The SQA’s approach to good Recommendation 2 than discussion governance and the management The board and senior of risk management of SQA should • committee chairs should consider continue to develop a more issues that would warrant 50. In 2005, the Independent strategic focus based on the key extended discussion by the board Commission on Good Governance in risks facing the business. Public Services published The Good • consideration should be given as Governance Standard for Public to how the balanced scorecard Services. The standard detailed six 12
Exhibit 5
The six principles of good governance
ma
k
d i
n n
a g
s
r a
c
e
c
d
l
o
o u
h
n
e t
k a
a b
t
i
s l
it
y
g
n
i r
e
g
a
a
l
g
n
E
Promoting values
Performing for the whole
effectively in organisation and
clearly defined demonstrating
functions and good
roles governance
Focusing on the
through
organisation’s
behaviour purpose and
on outcomes
for citizens and
service users
Developing
Taking
the capacity
informed,
and capability
transparent
of the governing
decisions and
body to be
managing risk effective
E l n a g re ag y in ilit g s tab tak un eho cco lders and making a
Source: The Good Governance Standard for Public Services, CIPFA/OPM, 2004 13 Part 3. The SQA’s performance management framework
Key messages • whether the SQA’s performance to activities rather than outcomes or management framework is fit impacts. For example, the objective The SQA’s corporate planning for purpose. to inform and support national policy process has established clear development and implementation objectives, but performance How effective are the SQA’s (Objective 2, page 15) is only management information is not performance indicators and supported by indicators that refer clearly linked to these objectives measures? to satisfaction ratings for partners at present. and the development of strategic 54. The SQA’s annual report includes arrangements with partners. These 53. In order to assess whether 20 performance indicators. On the indicators do not provide a sense of the SQA is operating effectively, basis of those indicators, the SQA’s what the SQA is seeking to achieve it is important to have access to performance has generally improved or the extent to which it is meeting appropriate performance information, since 2001/02. Exhibit 6 (overleaf) this objective. which in turn depends on the shows the SQA’s progress with each robustness of its performance of its current performance indicators. Are corporate planning processes management systems. Effective well developed? performance management relies 55. The first indicator summarises a on high-quality management subset of indicators linked specifically 57. The SQA has developed a sound information and the effective use to the delivery of the summer Diet of corporate planning process which of this information within the examinations, which were developed has supported the development of framework of a well developed and and agreed with the Scottish clear objectives for the organisation. coherent corporate plan. This section Executive post-2000. The planning cycle is now an examines: integrated process bringing together 56. Exhibit 7 (overleaf) shows the corporate plan, the business plan • the effectiveness of the SQA’s how the objectives and indicators and resource planning. The corporate performance indicators and have been linked in the 2005/06 plan is a rolling three-year plan, the measures annual report. Although most of latest of which covers the period the indicators can be linked to a 2006-09 and is combined with the • how well developed the SQA’s particular objective, many of them SQA’s business plan for 2006/07. The corporate planning processes are do not present a clear picture of corporate plan sets out the vision, performance in a given area of mission and values of the SQA, and activity. The indicators tend to relate its five key objectives. 14
Exhibit 6 SQA performance indicators for Annual Report 2005/06
Although some performance indicators were established in 2001/02 and 2002/03, the first year in which they were fully applicable was 2003/04, as part of the Corporate Plan 2003-06. Note: blank entries mean that the indicator was not measured.
Indicators 2001/02 2002/03 2003/04 2004/05 2005/06 Ensure the exam Diet and certification arrangements ¸ ¸ ¸ ¸ ¸ continue to meet the published schedule and agreed performance indicators. Ensure high customer satisfaction ratings for the credibility of SQA qualifications: - Standard grade 75% 75% 78% 84% 74% - Intermediate 1&2 70% 69% 72% 78% 73% - Highers 83% 84% 91% 92% 88% - Advanced Highers 81% 80% 87% 89% 88% - HNC/Ds 86% 83% 83% 91% 90% - SVQs 74% 77% 74% 80% 78% Ensure an 80% customer satisfaction rating for n/a n/a 79% 80% 81% SQA’s quality assurance arrangements. Respond to customer complaints within published n/a n/a - 62% 77% timescales (10 days). Achieve 75% customer satisfaction for consultation n/a n/a 72% 74% 74% and responsiveness. Achieve 80% satisfaction rating for SQA’s relations n/a n/a - - - with partners. Develop strategic agreements with all partners. n/a n/a - - - Ensure that staff and other non-appointee costs remain n/a n/a - - - constant in absolute terms for the period 2003-05. Reduce the area of office space by 10%. n/a n/a 3% - - Achieve an 85% customer satisfaction rating for n/a n/a 79% 80% 80% SQA’s operational efficiency. Achieve an 80% satisfaction rating for SQA’s website. n/a n/a 75% 75% 77% Achieve an 80% customer satisfaction rating for the n/a n/a 79% 79% 79% delivery of SQA’s products and services. Achieve more than 95% of data transfer through n/a n/a 96% 97% 97% electronic means. Achieve 75% staff satisfaction with SQA as an n/a 70% 82% 80% 73% employer. Maintain staff turnover between 10-15%. n/a 10% 10% 10% 9% Percentage of staff who have a clear understanding n/a 85% 89% 88% 80% of personal objectives. Percentage of staff who feel sufficiently informed n/a 45% 73% 74% 56% about what is going on at SQA. Reduce the average staff absence rate to below 8.5 n/a 10 10 10 11 days per annum (days). Become self-financing.3 n/a n/a - ¸ ¸ Achieve a 75% credibility rating. 48% 65% 70% 75% 77%
3 The current definition of self-financing used by the SQA refers to ensuring that the costs of its core services are met by charges and other income.
Source: SQA Part 3. The SQA’s performance management framework 15
Exhibit 7 SQA indicators for Annual Report 2005/06
The SQA’s indicators can be linked to its corporate objectives. As they stand they do not present a clear picture of overall performance. The six corporate objectives in the 2005/06 annual report have now been replaced by five key objectives included in the Corporate Plan 2006-09 and business plan for 2006/07.
Objective Indicator 1. Provide qualifications and support to match • Ensure the exam diet and certification arrangements the needs of individuals, society and the continue to meet the published schedule and agreed economy of Scotland. performance indicators. • Ensure high customer satisfaction ratings for the credibility of SQA qualifications. • Ensure an 80% customer satisfaction rating for SQA’s quality assurance arrangements. • Respond to customer complaints within published timescales. • Achieve 75% customer satisfaction for consultation and responsiveness. 2. Inform and support national policy • Achieve 80% satisfaction rating for SQA’s relations development and implementation. with partners. • Develop strategic agreements with all partners. 3. Redesign our services and processes to • Ensure that staff and other non-appointee costs remain make them better to use and more efficient. constant in absolute terms for the period 2003-05. • Reduce the area of office space by 10%. • Achieve an 85% customer satisfaction rating for SQA’s operational efficiency. • Achieve an 80% satisfaction rating for SQA’s website. • Achieve an 80% customer satisfaction rating for the delivery of SQA’s products and services. • Achieve more than 95% of data transfer through electronic means. 4. Develop a highly skilled, motivated and • Achieve 75% staff satisfaction with SQA as an effective workforce. employer. • Maintain staff turnover between 10-15%. • Percentage of staff who have a clear understanding of personal objectives. • Percentage of staff who feel sufficiently informed about what is going on at SQA. • Reduce the average staff absence rate to below 8.5 days per annum. 5. Raise the profile of Scottish education and • No indicators. training internationally to benefit the Scottish economy and users of Scottish qualifications. 6. Develop new markets and income streams. • No indicators. Not related to specific objectives. • Become self-financing. • Achieve a 75% credibility rating.
Source: SQA 16
The five key objectives in achieving Is the SQA’s performance and appropriate for use within its vision to be recognised nationally management framework fit for individual teams and as a basis for and internationally as a leader in purpose? exception reporting to management. qualifications and assessment are to: However, the performance 60. The SQA operationally manages management framework needs • provide qualifications and support its performance by producing to offer a higher-level picture of to match the needs of individuals, monthly reports to the executive performance against key objectives society and the economy of team and high-level quarterly reports to give management and external Scotland to the board and Scottish Executive. stakeholders an understanding of The reports to the executive team overall achievement. • inform and support national policy are very detailed and concentrate on development and implementation the 73 business objectives detailed 63. Performance indicators in the SQA annual business plan. should be specific, measurable, • redesign our services and Some examples of the business achievable, realistic and time bound. processes to make them better objectives used in the performance They should also contribute to to use and more efficient management framework are given the measurement of continuous in Exhibit 9 (overleaf). improvement and should, ideally, be • develop a highly skilled, motivated suited to benchmarking with other and effective workforce 61. This example highlights how organisations. The SQA recognises performance management within the that the current set of indicators • increase the use of SQA SQA relies on detailed monitoring is not fit for purpose and is in the qualifications and services of progress against a large number process of reviewing them as part nationally and internationally. of very specific business activities. of its work in developing a balanced While such monitoring has its place, scorecard approach to performance 58. The annual business plan there is also a need for a higher-level management. provides detailed objectives to picture of progress towards key support the key objectives. The objectives and a means for assessing 64. The SQA is demonstrating key objectives are backed by 19 the impact that the organisation a commitment to continuous corporate objectives and 73 business is achieving. improvement. It is important objectives which are allocated to therefore that its performance named employees with an indicative 62. The emphasis of the business management framework is adequate timescale for their achievement. objectives against which to support that commitment and Appendix 1 (page 23) sets out performance is reported is largely to be able to demonstrate that the these objectives. on the processes of the SQA rather organisation is achieving its aims. than its outputs and the impact The introduction of a balanced 59. Stakeholders are consulted of its work. However, there is a scorecard offers an opportunity for during the development of the risk that providing such a large the SQA to develop its performance corporate plan and 86 per cent of volume of operational information to management framework to provide staff stated they understood how management on a regular basis can information of this nature (see Part 4). their work contributed to SQA’s detract from strategic level thinking. strategies and goals. There is now a In reviewing detailed activity- clear timescale for development of related information, management the plan (Exhibit 8). This integrated and the board spend a significant planning process worked effectively amount of time on operational in 2005, with the corporate plan for management and are therefore 2006-09 presented to the Scottish less able to devote time to strategic Executive for approval on schedule in development and risk management. December 2005. The framework is comprehensive Part 3. The SQA’s performance management framework 17
Exhibit 8 SQA planning cycle 2006
Stage Planning cycle processes Timescale 1 Review statement of priorities and check alignment with broad corporate May-June objectives, through:
• Review environment trends • Review organisational processes and challenges • Review current objectives 2 Create detailed business plan objectives and distinguish between July-August business-as-usual and developmental objectives. 3 Resource planning through: September -November • Submit operational plan • Submit staffing review • Submit ‘rationale’ for developmental objectives
Then prioritise developmental objectives according to ‘rationale’ review and amend business plans, operational plans, staffing plans and budget requests. 4 Get board and Scottish Executive Education Department approval for December- revised business plan and budget. January 5 Begin financial year. April-
• Ongoing review – business plan reporting and quarterly forecasting process
Source: SQA 18
Exhibit 9 Examples of the SQA’s business objectives
The SQA’s 2006/07 business plan includes 73 business objectives. The examples below are the corporate and business objectives linked to the key objective: ‘To redesign our services and processes to make them better to use and more efficient.’
Corporate objective Business objectives Expand services for the management of • Deliver our web-based ‘Candidate Online candidate information. Facility’ according to plan. This facility will include access to results information and text alerting. • Extend online access to SQA validated candidate information (ELECTRIS) to Highlands & Islands Enterprise. Expand provision of e-assessment and • Provide e-assessments and e-learning e-learning resources. support for selected SQA units. • Undertake research to inform the development of e-assessment and e-learning approaches to support SQA qualifications. • Introduce e-assessment of National Courses from Diet 2006. Enhance question paper production processes • Implement a secure system for electronic and security. drafting of question papers. • Roll-out question paper delivery direct to centres in conjunction with education authorities and the Scottish Council for Independent Schools. Introduce new web-based marking and quality • Implement further phases of Appointees assurance processes. Online according to plan. This facility will include provision for appointees to maintain their own details, to mark online and to submit claims. • Implement a phased programme of web- based processes for approval, moderation and quality assurance monitoring. • Pilot and evaluate quality control processes for marking of National Courses.
Source: SQA 19 Part 4. The SQA’s use of performance information
Key messages a valuable tool and can help in out an approach to the development providing assurance to management of the balanced scorecard approach. and stakeholders that the SQA is It described the overall purpose of The SQA is introducing a operating efficiently and effectively. the scorecard approach being ‘to balanced scorecard approach Information to support benchmarking enable managers to develop a robust to performance management needs to be backed up by robust set of performance measures that in order to provide a clear costing of the SQA’s activities. provides a comprehensive view and coherent picture of its of the overall performance of the performance. 66. This section of the report organisation but that is also visibly examines: linked to the key strategies and 4 The SQA is developing a deeper priorities of the organisation.’ understanding of the link • the SQA’s development of a between costs and activities balanced scorecard approach 68. The SQA needs to provide a high- and is beginning to benchmark level picture of performance which its functions against other • the SQA’s approach to the internal and external stakeholders organisations. analysis of its costs and activities can use to assess performance. The SQA plans to introduce a balanced • the SQA’s approach to the scorecard approach to performance 65. Having examined and assessed development of benchmarking management in order to provide the robustness of the SQA’s information. a clear and cohesive performance performance management systems, measurement framework. The aim is this part of the report assesses how The SQA’s development of a for the scorecard to: well the SQA uses performance balanced scorecard approach information. It is important that • support the organisation’s the SQA is able to demonstrate its 67. To present a balanced view objectives performance to internal and external of an organisation’s performance, stakeholders and the development information reflecting a range of • provide intelligence for decisions of a balanced scorecard approach perspectives needs to be considered. will help it to do so. Within this In 1998 the Accounts Commission • effectively communicate SQA’s framework, benchmarking is published a discussion paper setting performance to internal and external stakeholders.
4 The measures of success – Developing a balanced scorecard to measure performance, Accounts Commission, 1998. 20
69. The executive team plans to 72. Further details of the advice present the balanced scorecard provided are reproduced in Appendix 2 Recommendation 4 to the board in November 2006. (page 26). The SQA should continue to The process gives the SQA explore the scope for efficiency the opportunity to focus on key savings including opportunities measures of impact, linked to its five Recommendation 3 presented by the SQA’s key objectives, rather than the more The SQA should develop its key modernisation agenda. process-driven indicators that form performance indicators, linked the focus for performance reporting to key objectives, as part of Recommendation 5 at present. The development of key its development of a balanced The SQA should continue to refine performance indicators (KPIs) and scorecard. its work linking costs with activities, the balanced scorecard provides an to further improve its understanding opportunity to give a clearer picture of its cost base and to inform the of performance for both internal and The SQA’s approach to the setting of charges for qualifications. external users. analysis of costs and activities
70. In formulating its approach to 73. In the past two years the The development of developing a scorecard, the SQA SQA has made progress in linking benchmarking information asked Audit Scotland to comment costs with activities and income on the approach being taken. The generated (for example, through 76. One of the main ways in which key issues for the SQA in developing fees for examinations). This is giving an organisation can demonstrate their approach are a need for: the SQA a clearer understanding of its commitment to continuous the financial impact of its work and improvement is by searching for • clarity between: helping it to align plans and targets examples of better practice in other with available resources. organisations that can be integrated 1. Objectives – which should into its own service delivery. An describe the desired outcome 74. The continued development and Accounts Commission publication5 eg, develop the right refinement of this work is particularly summarises benchmarking as follows: qualifications. important in developing a set of key performance indicators that can be ‘Benchmarking is best thought 2. Measures – which should set used in performance management. of as a structured and focused out a quantifiable means for The SQA is introducing a new approach to comparing with others assessing progress towards financial system which will help its how you provide services and the objective eg, uptake of approach to this work. the performance levels you have qualifications. achieved. The purpose of such 75. The SQA should continue to comparison is to enable you to 3. Targets – which define the adopt a more strategic approach, identify where and how you can do desired level for the measure eg, and explore systematically the scope better. It is about understanding why x per cent of relevant age group for efficiency savings. The SQA’s there are differences in performance entered for SQA qualifications. modernisation agenda will highlight between organisations and this some opportunities for savings and involves looking in detail at the way • development of a concise set of the organisation is already working services are delivered and managed clearly defined and quantifiable towards generating efficiency savings and at the processes and activities measures, each of which has a through e-learning and involved in service delivery that lie clear purpose. e-assessment. However, many of behind the benchmark measures of these will require upfront investment performance.’ 71. The process needs to start from in order to realise future savings. the top down to ensure the linkages are clear. The high-level KPIs should be capable of providing a concise picture of performance to both internal and external audiences.
5 Measuring up to the best – A manager’s guide to benchmarking, Accounts Commission, 1999. Part 3. The SQA’s use of performance information 21
77. The document also gives details benchmarking exercise with a are administered in the different on a number of different approaches number of other organisations that parts of the UK, which limits the to benchmarking and describes a are concerned with the provision of opportunities for benchmarking. four-stage process as follows: qualifications in the United Kingdom. The object of the exercise was to: • The ability to compare information Planning is made more complex by the • Decide what you want to • establish the extent of confidentiality requirements benchmark and why. comparative information available of private sector providers for benchmarking purposes Edexcel, the Assessment and • Decide what type of benchmark Qualifications Alliance (AQA) and you are going to start with. • assess the degree to which OCR, which operate largely other qualifications authorities are in England. • Decide who is going to be engaged in, or would be willing to involved. engage in, benchmarking • There was, however, willingness expressed by the bodies Analysis • where possible, produce we spoke to as part of this • Understand your own performance. benchmarking information. exercise (CCEA and Edexcel) to explore further opportunities • Undertake data benchmarking. 80. Our exercise initially gathered for benchmarking costs and information readily available from processes with the SQA. • Assess key performance gaps. public sources such as organisational websites and published documents. • Some detailed information • Understand your own processes. We met with two bodies; Edexcel was obtained to allow limited and the Council for the Curriculum, comparisons for some areas • Choose your benchmarking Examinations and Assessment of activity, such as unit costing partners. (CCEA), to discuss the availability for types of examination. and comparability of benchmarking Further work would have to be • Complete benchmarking. information. We also consulted with undertaken to refine costing the Qualifications and Curriculum information if a similar approach Action Authority (QCA). was to be extended to other • Decide what you want to improve business activities. Information and how you are going to do it. 81. Our exercise identified that there on specific costs for private are complexities in benchmarking sector providers is commercially • Implement the improvement plan. the SQA with other UK organisations sensitive but Edexcel has as roles and responsibilities vary indicated its willingness to Review considerably. However, the exercise discuss activity costs with SQA • Monitor progress and did highlight potential areas for on a confidential basis. achievements. further investigation, such as unit costing for types of examinations. • Audit Scotland will share the • Decide what comes next. We have derived high-level unit detailed information it has costs as a starting point for further obtained during this exercise 78. To date, the SQA has undertaken work on benchmarking but further with the SQA. Audit Scotland limited benchmarking of its discussions will need to be held will follow up any work done on performance with the Qualifications between SQA and other bodies to benchmarking in due course. and Curriculum Authority in England, arrive at meaningful information. although it proposes to extend this Further results of the exercise are activity. The information obtained presented at Appendix 3 (page 28). Recommendation 6 should be incorporated in the Our key findings were: The SQA should explore the balanced scorecard. opportunities for greater use • There are fundamental of benchmarking to inform 79. As part of this study, Audit differences in the ways in which performance management activity. Scotland carried out a high-level qualifications to degree level 22
Next steps
82. The SQA has made good progress in its recovery from the problems it faced in 2000. It now has a better understanding of its key business activities and it is demonstrating an ongoing commitment to continuous improvement.
83. Much of this success has been driven by the board and executive team, and customers and stakeholders have indicated that they are increasingly confident with the SQA’s performance. The SQA now has clear objectives generated through a robust corporate planning process.
84. There is still room for further improvement. The ‘hands-on’ management of the board and executive team can present a risk by detracting from strategic level thinking, which the board has recognised. The SQA’s performance information needs to be more clearly linked to its key objectives and it is working to introduce a balanced scorecard approach to performance management. This should provide a clearer and more coherent picture of its performance. It is also working on establishing a greater understanding of the links between its activities and its costs. Progress with this will allow the SQA to effectively benchmark its costs and activities with other bodies.
85. Our recommendations reflect this position and support the SQA in its efforts towards further improvement. 23 Appendix 1. SQA objectives
The objectives are included in the SQA’s Corporate Plan 2006-09.
Corporate objectives Business objectives
Objective 1: Provide qualifications and support to match the needs of individuals, society and the economy of Scotland. 1.1 Deliver assessment, quality 1.1.1 Complete preparation for the 2006 Diet of National Qualifications assurance and certification of examinations according to certification plan. SQA’s qualifications. 1.1.2 Deliver the 2006 Diet of National Qualifications examinations according to specified performance measures agreed with the Scottish Executive. 1.1.3 Conduct Quality Assurance, and deliver timely and accurate certification, for all qualifications in SQA’s portfolio. 1.1.4 Introduce an improved Scottish Qualifications Certificate in 2006. 1.1.5 Complete the review of Quality Assurance. 1.1.6 Investigate alternative models of quality assurance and assessment for international centres. 1.2 Develop SQA’s portfolio of National Qualifications qualifications. 1.2.1 Develop National Courses and Access Units. 1.2.2 Develop National Qualification Group Awards (National Certificates and National Progression Awards). 1.2.3 Pilot and evaluate Skills for Work Courses. 1.2.4 Implement the National Courses Development Programme. Higher National Qualifications 1.2.5 Develop Higher National Certificates/Diplomas and Professional Development Awards. Scottish Vocational Qualifications 1.2.6 Develop Scottish Vocational Qualifications. Customised Awards 1.2.7 Extend our Customised Awards service in line with customer needs. Cross-cutting developments 1.2.8 Complete the review of Core Skills and implement the support plan. 1.2.9 Investigate possible models for recognition of informal learning with key providers. 1.2.10 Develop occupational qualification profiles in key areas. 1.2.11 Extend coverage of sector panels to cover all occupational sectors. 1.2.12 Work with and benchmark ourselves against international bodies to support qualifications development. 1.3 Develop and deliver the 1.3.1 Implement audit and monitoring activities. Accreditation function. 1.3.2 Implement common quality assurance principles for N/SVQs awarding bodies amongst all United Kingdom regulators. 1.3.3 Implement the accreditation of qualifications other than SVQs. 1.3.4 Implement credit rating of SVQs. 1.4 Provide information, 1.4.1 Publish learning, teaching and assessment support materials. materials and training to support 1.4.2 Deliver professional development programmes to support understanding understanding and delivery of SQA qualifications, policies, procedures and methodologies. of SQA qualifications and 1.4.3 Deliver marketing and promotional activities to position and increase assessment. awareness of SQA qualifications and services.
1.5 Increase our understanding 1.5.1 Implement revised methods of gathering and analysing customer and of stakeholder and customer market information to inform qualifications development. needs and priorities. 1.5.2 Implement revised communications arrangements for customers and stakeholders. 1.5.3 Implement and monitor the SQA Customer Charter. 24
Objective 2: Inform and support national policy development and implementation 2.1 Maintain existing and 2.1.1 Work in partnership with the Scottish Executive and other UK regulatory establish new strategic bodies to inform developments relating to the Scottish and UK Skills Agendas. relationships with key national 2.1.2 Implement a strategic engagement plan for Sector Skills Councils. and international partners. 2.1.3 Implement an engagement strategy for the Higher Education sector to ensure understanding and recognition of our qualifications. 2.1.4 Implement joint workplans with Her Majesty’s Inspectorate for Education, Learning and Teaching Scotland (LTS), Scotland’s Colleges and the Convention of Scottish Local Authorities. 2.1.5 Maintain and seek membership of key national and international networks to promote understanding of Scottish education and training, and of SQA qualifications. 2.2 Provide information and 2.2.1 Produce the Annual Statistical Report, Statistical Digest and quarterly advice to inform policy and briefings. decision making. 2.2.2 Produce a series of research bulletins. 2.2.3 Respond to consultations within set timescales. 2.3 Implement the outcomes of 2.3.1 Work with partners to review relevant provision against the 3-18 policy developments in relation Curriculum principles and purposes. to qualifications and assessment. 2.3.2 Contribute to the continued development of the Scottish Executive’s ‘Assessment is for Learning’ programme. 2.3.3 Take forward initiatives arising from the Scottish Executive’s Lifelong Learning Strategy. 2.4 Play a leading role in the 2.4.1 Deliver SQA’s ongoing commitment to the SCQF. implementation of the Scottish 2.4.2 Publicise and promote the implementation of the SCQF to SQA centres, Credit and Qualifications customers and stakeholders according to the defined communications strategy. Framework. 2.4.3 Contribute to the development of credit and qualifications frameworks at UK and European levels. Objective 3: Re-design our services and processes to make them better to use and more efficient 3.1 Expand services for the 3.1.1 Deliver our web-based ‘Candidate Online Facility’ according to plan. This management of candidate facility will include access to results information and text alerting. information. 3.1.2 Extend on-line access to SQA validated candidate information (ELECTRIS) to Highlands & Islands Enterprise. 3.2 Expand provision of 3.2.1 Provide e-assessments and e-learning support for selected SQA Units. e-assessment and e-learning 3.2.2 Undertake research to inform the development of e-assessment and e- resources. learning approaches to support SQA’s qualifications. 3.2.3 Introduce e-assessment of National Courses from Diet 2006. 3.3 Enhance question paper 3.3.1 Implement a secure system for electronic drafting of question papers. production processes and 3.3.2 Roll-out question paper delivery direct to centres in conjunction with security. education authorities and the Scottish Council for Independent Schools. 3.4 Introduce new web-based 3.4.1 Implement further phases of Appointees Online according to plan. This marking and quality assurance facility will include provision for appointees to maintain their own details, to processes. mark online and to submit claims. 3.4.2 Implement a phased programme of web-based processes for approval, moderation and quality assurance monitoring. 3.4.3 Pilot and evaluate alternative quality control processes for marking of National Courses. Appendix 1. SQA objectives 25
Objective 4: Develop a highly-skilled, motivated and effective workforce 4.1 Design and introduce a set 4.1.1 Define the leadership competences required to support the vision, of integrated human resource mission and values of SQA. policies which demonstrate best 4.1.2 Introduce HR practices in training and development, performance practice and support the delivery management and succession planning which will have the greatest influence on of SQA’s business plan. creating and sustaining the desired culture and behaviours in SQA. 4.1.3 Introduce improved HR processes for appointees including recruitment and selection, training and development and succession planning. 4.1.4 Implement an effective communications strategy for appointees. 4.1.5 Complete a review of appointee remuneration. 4.1.6 Implement SQA Academy training for staff and appointees. 4.1.7 Develop a staff survey and action plan which incorporates the requirements of Best Value. 4.2 Embed more effective 4.2.1 Define and measure key performance indicators. performance measurement 4.2.2 Implement a benchmarking strategy to encourage continuous and monitoring to encourage improvement. continuous improvement. 4.3 Implement an integrated 4.3.1 Implement Phase 1 of integrated human resources and financial system. human resources and financial 4.3.2 Implement an e-commerce and e-procurement framework. system. 4.3.3 Implement a revised pricing structure for our qualifications. 4.3.4 Scope Phase 2 of the integrated human resources and financial system to include the outcomes of the review of appointee remuneration. 4.4 Implement an estates 4.4.1 Agree and progress the relocation of SQA in Dalkeith. and facilities management 4.4.2 Develop and implement a joint efficiencies programme with LTS for the programme. new shared premises in Glasgow. 4.4.3 Initiate the development and implementation of integrated systems infrastructure in partnership with LTS. 4.4.4 Implement revised approaches to health and safety and environmental management, at all SQA sites. 4.4.5 Complete a review of the extent to which SQA fulfils its corporate social responsibilities. Objective 5: Increase the use of SQA qualifications and services nationally and internationally 5.1 Increase national and 5.1.1 Work with SQA centres and partners. international awarding and 5.1.2 Forge European and international strategic alliances. consultancy. 5.1.3 Promote selected SQA products and services in identified markets. 5.2 Establish a separate trading company to manage and develop commercial activities. 26 Appendix 2. Advice on the development of a balanced scorecard approach
The notes below are summarised Corporate plan/business plan Balanced scorecard development thoughts from the Audit Scotland team involved in the SQA The current practice of combining In introducing the BSC, it is performance management study. these two documents into one has important that there is clear linkage They were intended as an overview merits in providing linkage between with the SQA’s corporate plan, so of key points from our review of objectives and activities, but there is as to avoid creating an extra layer of the material available in relation a danger that objectives become too objectives. The five key objectives to the SQA’s balanced scorecard activity-focused, rather than focusing set out in the latest corporate plan (BSC) development. The notes were on desired outcomes. should be the starting point and the provided to the SQA in July 2006 and BSC model should flow from this. were used to help inform the SQA’s The current document has three ongoing work in this area. It should levels of objectives: The current model uses three be noted that the SQA has continued perspectives – qualifications, working to develop its approach since this • Key objectives (5) – these provide with others, management – rather 6 review was undertaken. a good balance, reflect the than the traditional four. This is fine nature of the organisation, and in principle, but it is not clear how Existing performance address both internal/external this links to the five key objectives management framework perspectives. They provide in the corporate plan. Similarly, the a helpful starting point for objectives set out within these At present, the focus of performance development of the BSC. three categories are different from management is on business those in the corporate plan. To objectives (as set out in the business • Corporate objectives (19) ensure transparency and clarity, plan). Progress towards achievement – this is a reasonable number the objectives should be consistent of these is monitored on a monthly of objectives against which to throughout. basis at monthly management team develop KPIs, but the current meetings and also reported to the set includes several which are The model as described in the board. outputs rather than objectives SQA’s implementation progress (eg, implement an integrated HR summary does not refer to impact The 20 KPIs developed post-2000 and financial system). or continuous improvement – these relate to a range of areas of activity are likely to be captured within and there is an additional set of • Business objectives (73) – in ‘qualifications’ and ‘management’ indicators relating specifically to the the latest corporate plan, these respectively, but it would be Diet results – performance against describe activities and/or helpful to make these important these is published in the annual outputs required eg, implement considerations more explicit in report, but they are not used for benchmarking strategy rather the objectives. regular monitoring purposes aside than desired outcomes. from this. Ownership for each objective In the current model, there is not a is identified and timescales are clear enough distinction between The introduction of a BSC offers set out. The number of business objectives, measures and targets: an opportunity to define a more objectives is too large to be a appropriate set of KPIs that: manageable set for managing • Objectives – should describe the performance at a corporate desired outcome eg, develop the • are clearly linked to key objectives level. These activities/outputs right qualifications. are appropriate for team-level • provide a clear picture of progress management. • Measures – should set out a and impact quantifiable means for assessing In the business plan, it would be progress towards the objective • can present a summary of helpful to have a clearer distinction eg, uptake of qualifications. performance to inform both between objectives, outputs and internal and external audiences. activities – using the term ‘objective’ at all levels is potentially confusing.
6 Since these comments were provided, the SQA has revised the model to provide four perspectives. Appendix 2. Advice on the development of a balanced scorecard approach 27
• Targets – defines the Ideally, the BSC approach should be Summary of key points desired level for the measure, developed from the top down, not eg x per cent of relevant bottom up. It should start from the age group entered for SQA corporate plan and work downwards, • Make sure there is a clear qualifications. but always showing clear linkage linkage through from corporate from one level to the next. There is a objectives to measures and The measures described in the risk that, in starting at individual team targets. latest version of the model are often level, there will be gaps, overlap or objectives rather than measures. inconsistencies at higher levels. • Work from top down to help Measures should describe the ensure this linkage. quantifiable factor that is being used The Corporate Strategy and to assess progress. For example, Performance Team (CSPT) model • Make sure that measures are ‘Improve commercial focus, is very detailed – it may not be clearly defined and quantifiable. capability and profitability’ is not a appropriate for each team to have a measure, this is a desired outcome. performance measure against every • Think about desired impact A measure might be ‘Improve single objective, as different teams and outcomes when defining profitability of commercial contracts’ will make contributions in different measures. (which currently appears as a target). areas. The main requirement is that Meanwhile the target might be there is coverage of all objectives at • Avoid measuring activity, ‘five per cent increase in average organisational level, rather than team instead aim to capture the profitability of commercial contracts’. level (this should be guaranteed if a quality of outputs. top-down approach is used). The measures and associated targets • Focus on a concise set of need to be challenging and focus The focus should be on defining indicators and avoid duplication. on quality. Measures should also be measures that are informative and clearly defined and unambiguous. valuable in managing performance, • Each indicator should have a Where possible, targets should not measuring everything that clear purpose. avoid simple yes/no responses as is done by a particular team or this gives no basis for assessing the the organisation as a whole. The • At top level, the KPIs should scale of achievement. For example: CSPT model is heavily focused on give a balanced picture of measuring activity, but does not give performance and provide • Credibility rating for much indication of the quality of a useful summary for both qualifications – why should outputs. For example, measuring the internal and external audiences. Standard Grades have a lower number of staff reviews conducted target for credibility than Highers? says nothing about the quality of • More detailed sets of KPIs may How is credibility defined (and if these reviews – a better measure be appropriate for individual this is based on survey results, might relate to staff satisfaction with teams, but should still aim to will ‘credibility’ mean the same the review system. be concise and manageable. thing to everyone?). There is a lot of information in • Research output – number of the team level model – for each bulletins produced does not measure, it is important to think capture quality, better to identify a about the value of data, ie how will measure that indicates the value the information be used? Who needs of these bulletins to users. to know? What action might result from the information gathered? • Become self-financing – this is a milestone; a better measure might be non-government income as a percentage of expenditure, with an associated target of 100 per cent. 28 Appendix 3. Findings from Audit Scotland’s benchmarking exercise
We found that there are certain similarities and some significant differences between the various qualifications organisations in the UK. Table 1 gives a broad outline of the functions of the various organisations and how they form frameworks for the provision of qualifications. Table 1 Functions of UK qualifications authorities
The table provides high-level details of the functions provided by different authorities in the UK.
Scotland England The SQA is responsible for: The QCA is responsible for:
• accrediting and regulating awarding bodies • accrediting and regulating awarding bodies • developing, validating and quality assuring • validating and quality assuring qualifications qualifications • approving and quality assuring centres • approving and quality assuring centres • handling appeals (partially) • assessment arrangements • curriculum development. • undertaking assessment • awarding qualifications Awarding bodies (Edexcel, AQA and OCR) are • handling appeals. responsible for:
• developing and quality assuring qualifications Learning and Teaching Scotland has the lead role on curriculum development in schools, with SQA having • approving and quality assuring centres an advisory role in that respect. • assessment arrangements • undertaking assessment • awarding qualifications • handling appeals. Wales Northern Ireland The Welsh Joint Education Committee (WJEC) is The CCEA is responsible for: responsible for: • accrediting and regulating awarding bodies • developing and quality assuring qualifications • developing, validating and quality assuring • approving and quality assuring centres qualifications • assessment arrangements • approving and quality assuring centres • undertaking assessment • assessment arrangements • awarding qualifications • undertaking assessment • handling appeals. • awarding qualifications • handling appeals • curriculum development.
Source: Audit Scotland Appendix 3. Findings from Audit Scotland’s benchmarking exercise 29
Producing comparative unit cost information for these organisations is difficult, primarily because of the differing remits of the various organisations. In addition, for commercial confidentiality reasons, Edexcel were unwilling to share detailed costing information with us. We were therefore only able to generate cost and activity data at a high level for the SQA. These data are not suited to direct comparison and need to be considered in the context of the activities and responsibilities of the various organisations. They can, however, highlight areas that merit further exploration. The areas for potential comparison we explored are shown in Table 2. Table 2 Benchmarking analysis
The table shows the areas of expenditure and activity that we explored as part of our benchmarking exercise.
Type of information Detail Operational • Functions and responsibilities of different organisations • Employment data • Income • Expenditure • Charges for qualifications Activity • Numbers of entries, marks and scripts • Range of qualifications • Certificates issued • Centres offering qualifications Customer-facing • Customer satisfaction levels • Appeals handled • Special arrangements Performance management • Comparison of key factors in relation to performance management, including aspects of corporate plans and objectives, balanced scorecards and performance information.
Source: Audit Scotland
We have not been able to obtain information on detailed costings for specific activities or qualifications that would provide the basis for more useful benchmarking information. The SQA should consider whether it is able to produce such information through its commitment to activity-based costing and should discuss the extent to which other bodies would be able to compare similar information. Edexcel indicated during our work that it would be willing to discuss more detailed benchmarking information with SQA on a confidential basis and this provides an opportunity for a more rigorous comparative exercise.
A starting point for that piece of work could be a comparison of charges for qualifications. Again, this is complicated by the differences between qualifications offered by different bodies but there may be scope for discussion around their comparability. The SQA has in the past compared charges for qualifications with other bodies in the UK in its letters to customers outlining increases in charges. For example, the letter outlining charge increases for 2004/05 compared the cost of a Higher course with the cost of an AS level course. This could open up discussion on how charges are calculated and the costing of the activities involved, allowing efficiency comparisons to be made. Performance management in the Scottish Qualifications Authority
If you require this publication in an alternative format and/or language, please contact us to discuss your needs. It is also available on our website: www.audit-scotland.gov.uk
Audit Scotland 110 George Street Edinburgh EH2 4LH
Telephone 0845 146 1010 Fax 0845 146 1009 www.audit-scotland.gov.uk ISBN 1 905634 36 6 AGS/2006/11
This publication is printed on uncoated paper, made from a minimum of 80% de-inked post-consumer waste.